[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 12:20:24.317339   561 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.140.126:46575
I20260709 12:20:24.319120   561 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 12:20:24.320257   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:24.340258   570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:24.340651   567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:24.340272   568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:24.345743   561 server_base.cc:1061] running on GCE node
I20260709 12:20:25.571638   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:25.571923   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:25.572109   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599625572091 us; error 0 us; skew 500 ppm
I20260709 12:20:25.573004   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:25.579308   561 webserver.cc:533] Webserver started at http://127.0.140.126:35603/ using document root <none> and password file <none>
I20260709 12:20:25.580248   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:25.580472   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:25.580922   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:25.587566   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/master-0-root/instance:
uuid: "b0b5c3d3ac00431bbf5e9232d0839896"
format_stamp: "Formatted at 2026-07-09 12:20:25 on dist-test-slave-7mcn"
I20260709 12:20:25.596609   561 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:20:25.602747   576 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:25.604368   561 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 12:20:25.604826   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/master-0-root
uuid: "b0b5c3d3ac00431bbf5e9232d0839896"
format_stamp: "Formatted at 2026-07-09 12:20:25 on dist-test-slave-7mcn"
I20260709 12:20:25.605302   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:25.683284   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:25.685073   561 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 12:20:25.685613   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:25.771185   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.126:46575
I20260709 12:20:25.771308   627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.126:46575 every 8 connection(s)
I20260709 12:20:25.781136   628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:25.808637   628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896: Bootstrap starting.
I20260709 12:20:25.816246   628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.818658   628 log.cc:826] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896: Log is configured to *not* fsync() on all Append() calls
I20260709 12:20:25.824672   628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896: No bootstrap required, opened a new log
I20260709 12:20:25.848021   628 raft_consensus.cc:359] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b5c3d3ac00431bbf5e9232d0839896" member_type: VOTER }
I20260709 12:20:25.848691   628 raft_consensus.cc:385] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.849033   628 raft_consensus.cc:740] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0b5c3d3ac00431bbf5e9232d0839896, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.850239   628 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b5c3d3ac00431bbf5e9232d0839896" member_type: VOTER }
I20260709 12:20:25.850862   628 raft_consensus.cc:399] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:20:25.851202   628 raft_consensus.cc:493] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:20:25.851646   628 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.856499   628 raft_consensus.cc:515] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b5c3d3ac00431bbf5e9232d0839896" member_type: VOTER }
I20260709 12:20:25.857399   628 leader_election.cc:304] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0b5c3d3ac00431bbf5e9232d0839896; no voters: 
I20260709 12:20:25.859589   628 leader_election.cc:290] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:20:25.860097   631 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:25.863171   631 raft_consensus.cc:697] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 1 LEADER]: Becoming Leader. State: Replica: b0b5c3d3ac00431bbf5e9232d0839896, State: Running, Role: LEADER
I20260709 12:20:25.864339   631 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b5c3d3ac00431bbf5e9232d0839896" member_type: VOTER }
I20260709 12:20:25.865471   628 sys_catalog.cc:565] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:20:25.877024   633 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0b5c3d3ac00431bbf5e9232d0839896. Latest consensus state: current_term: 1 leader_uuid: "b0b5c3d3ac00431bbf5e9232d0839896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b5c3d3ac00431bbf5e9232d0839896" member_type: VOTER } }
I20260709 12:20:25.877827   633 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:25.878129   632 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0b5c3d3ac00431bbf5e9232d0839896" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0b5c3d3ac00431bbf5e9232d0839896" member_type: VOTER } }
I20260709 12:20:25.878950   632 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:25.882426   640 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:20:25.896086   640 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:20:25.899683   561 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:20:25.914310   640 catalog_manager.cc:1383] Generated new cluster ID: a1f7662502224ddd9691bee41b5aee39
I20260709 12:20:25.914645   640 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:20:25.937331   640 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:20:25.939862   640 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:20:25.957959   640 catalog_manager.cc:6070] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896: Generated new TSK 0
I20260709 12:20:25.959334   640 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:20:25.967286   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:25.979849   650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:25.980984   651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:25.987094   653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:25.988250   561 server_base.cc:1061] running on GCE node
I20260709 12:20:25.989249   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:25.989506   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:25.989719   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599625989700 us; error 0 us; skew 500 ppm
I20260709 12:20:25.990543   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:25.996057   561 webserver.cc:533] Webserver started at http://127.0.140.65:38557/ using document root <none> and password file <none>
I20260709 12:20:25.996969   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:25.997260   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:25.997630   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:25.999294   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-0-root/instance:
uuid: "be75e7f361dd41209a39df0af0d6a113"
format_stamp: "Formatted at 2026-07-09 12:20:25 on dist-test-slave-7mcn"
I20260709 12:20:26.006976   561 fs_manager.cc:696] Time spent creating directory manager: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:20:26.013007   658 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.014171   561 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.001s
I20260709 12:20:26.014603   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "be75e7f361dd41209a39df0af0d6a113"
format_stamp: "Formatted at 2026-07-09 12:20:25 on dist-test-slave-7mcn"
I20260709 12:20:26.015059   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:26.034543   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:26.036439   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:26.039705   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:26.045604   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:26.045957   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.046298   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:26.046545   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.146284   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:37195
I20260709 12:20:26.146404   720 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:37195 every 8 connection(s)
I20260709 12:20:26.154937   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:26.170908   725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:26.179538   726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:26.186259   561 server_base.cc:1061] running on GCE node
W20260709 12:20:26.187949   728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:26.190057   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:26.190379   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:26.190606   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599626190587 us; error 0 us; skew 500 ppm
I20260709 12:20:26.191431   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:26.192399   721 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46575
I20260709 12:20:26.193024   721 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:26.194623   721 heartbeater.cc:507] Master 127.0.140.126:46575 requested a full tablet report, sending...
I20260709 12:20:26.197018   561 webserver.cc:533] Webserver started at http://127.0.140.66:35645/ using document root <none> and password file <none>
I20260709 12:20:26.197732   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:26.197932   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:26.198146   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:26.199360   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-1-root/instance:
uuid: "2851d9518eb34a4c94883642f0fd37b6"
format_stamp: "Formatted at 2026-07-09 12:20:26 on dist-test-slave-7mcn"
I20260709 12:20:26.199414   593 ts_manager.cc:194] Registered new tserver with Master: be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195)
I20260709 12:20:26.202986   593 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46614
I20260709 12:20:26.205899   561 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.002s	sys 0.004s
I20260709 12:20:26.211661   733 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.212857   561 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
I20260709 12:20:26.213291   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-1-root
uuid: "2851d9518eb34a4c94883642f0fd37b6"
format_stamp: "Formatted at 2026-07-09 12:20:26 on dist-test-slave-7mcn"
I20260709 12:20:26.213712   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:26.244402   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:26.246367   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:26.249549   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:26.253674   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:26.253888   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.254218   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:26.254374   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.374770   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.66:45033
I20260709 12:20:26.375084   795 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.66:45033 every 8 connection(s)
I20260709 12:20:26.382252   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:26.391111   799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:26.396113   800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:26.398809   796 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46575
I20260709 12:20:26.399274   796 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:26.400563   796 heartbeater.cc:507] Master 127.0.140.126:46575 requested a full tablet report, sending...
W20260709 12:20:26.403201   802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:26.403540   593 ts_manager.cc:194] Registered new tserver with Master: 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:26.404672   561 server_base.cc:1061] running on GCE node
I20260709 12:20:26.405961   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:20:26.405987   593 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46630
W20260709 12:20:26.406333   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:26.406553   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599626406536 us; error 0 us; skew 500 ppm
I20260709 12:20:26.407387   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:26.410542   561 webserver.cc:533] Webserver started at http://127.0.140.67:37233/ using document root <none> and password file <none>
I20260709 12:20:26.411113   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:26.411345   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:26.411643   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:26.412839   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-2-root/instance:
uuid: "f40ae05748a5432199f3d7ca95a3311c"
format_stamp: "Formatted at 2026-07-09 12:20:26 on dist-test-slave-7mcn"
I20260709 12:20:26.418298   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:20:26.423336   807 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.424381   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:20:26.424778   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-2-root
uuid: "f40ae05748a5432199f3d7ca95a3311c"
format_stamp: "Formatted at 2026-07-09 12:20:26 on dist-test-slave-7mcn"
I20260709 12:20:26.425259   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599624140507-561-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:26.451363   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:26.453032   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:26.455991   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:26.459489   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:26.459771   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.460076   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:26.460292   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.552714   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.67:34293
I20260709 12:20:26.552814   869 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.67:34293 every 8 connection(s)
I20260709 12:20:26.574141   870 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46575
I20260709 12:20:26.574643   870 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:26.575793   870 heartbeater.cc:507] Master 127.0.140.126:46575 requested a full tablet report, sending...
I20260709 12:20:26.578208   593 ts_manager.cc:194] Registered new tserver with Master: f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:26.578825   561 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019557342s
I20260709 12:20:26.579879   593 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46644
I20260709 12:20:26.643532   593 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46646:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:20:26.648561   593 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:20:26.768590   834 tablet_service.cc:1511] Processing CreateTablet for tablet 53e3310bec7842c1ad43b21aa1128e00 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:20:26.773367   835 tablet_service.cc:1511] Processing CreateTablet for tablet 9f4e57df4d0d4f8793bd68db29de1067 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:20:26.771747   833 tablet_service.cc:1511] Processing CreateTablet for tablet ddb09687dc34423aadf5b8bebf5f34cb (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:20:26.777145   835 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4e57df4d0d4f8793bd68db29de1067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.775993   832 tablet_service.cc:1511] Processing CreateTablet for tablet 29a1f6ed8add4ddeb88b7832dad139ca (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:20:26.780612   834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e3310bec7842c1ad43b21aa1128e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.785265   830 tablet_service.cc:1511] Processing CreateTablet for tablet 55d9233a4c174bf6ac4f1608d6444455 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:20:26.792826   833 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb09687dc34423aadf5b8bebf5f34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.792017   829 tablet_service.cc:1511] Processing CreateTablet for tablet 00fda6319cef4443a37dc908e5245c54 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:20:26.783457   831 tablet_service.cc:1511] Processing CreateTablet for tablet 47e1e3ade8a3474da6f42ae3abe03d41 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:20:26.797859   830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d9233a4c174bf6ac4f1608d6444455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.797689   828 tablet_service.cc:1511] Processing CreateTablet for tablet 3ef1618bc7c94ff28a5312b7bb7e5fc3 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:20:26.799433   832 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29a1f6ed8add4ddeb88b7832dad139ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.801559   829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00fda6319cef4443a37dc908e5245c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.802935   828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef1618bc7c94ff28a5312b7bb7e5fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.804958   831 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e1e3ade8a3474da6f42ae3abe03d41. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:26.888352   641 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:26.898411   686 tablet_service.cc:1511] Processing CreateTablet for tablet 9f4e57df4d0d4f8793bd68db29de1067 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:20:26.898247   685 tablet_service.cc:1511] Processing CreateTablet for tablet 53e3310bec7842c1ad43b21aa1128e00 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:20:26.910843   828 tablet_service.cc:1511] Processing CreateTablet for tablet d3e563cf261d49c9bb1d4ad63dc6500e (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:20:26.911871   684 tablet_service.cc:1511] Processing CreateTablet for tablet ddb09687dc34423aadf5b8bebf5f34cb (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:20:26.925263   686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4e57df4d0d4f8793bd68db29de1067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.931471   887 tablet_bootstrap.cc:492] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c: Bootstrap starting.
I20260709 12:20:26.930402   683 tablet_service.cc:1511] Processing CreateTablet for tablet 29a1f6ed8add4ddeb88b7832dad139ca (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:20:26.911943   679 tablet_service.cc:1511] Processing CreateTablet for tablet 3ef1618bc7c94ff28a5312b7bb7e5fc3 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:20:26.922624   680 tablet_service.cc:1511] Processing CreateTablet for tablet 00fda6319cef4443a37dc908e5245c54 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:20:26.937804   678 tablet_service.cc:1511] Processing CreateTablet for tablet d3e563cf261d49c9bb1d4ad63dc6500e (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:20:26.926692   682 tablet_service.cc:1511] Processing CreateTablet for tablet 47e1e3ade8a3474da6f42ae3abe03d41 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:20:26.934772   681 tablet_service.cc:1511] Processing CreateTablet for tablet 55d9233a4c174bf6ac4f1608d6444455 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:20:26.937413   828 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e563cf261d49c9bb1d4ad63dc6500e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.946822   684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb09687dc34423aadf5b8bebf5f34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.948284   683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29a1f6ed8add4ddeb88b7832dad139ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.950006   682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e1e3ade8a3474da6f42ae3abe03d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.954128   681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d9233a4c174bf6ac4f1608d6444455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.958074   685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e3310bec7842c1ad43b21aa1128e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.963945   679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef1618bc7c94ff28a5312b7bb7e5fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.966382   678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e563cf261d49c9bb1d4ad63dc6500e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.968726   680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00fda6319cef4443a37dc908e5245c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:26.990198   761 tablet_service.cc:1511] Processing CreateTablet for tablet 9f4e57df4d0d4f8793bd68db29de1067 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:20:26.991976   759 tablet_service.cc:1511] Processing CreateTablet for tablet ddb09687dc34423aadf5b8bebf5f34cb (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:20:26.993079   758 tablet_service.cc:1511] Processing CreateTablet for tablet 29a1f6ed8add4ddeb88b7832dad139ca (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:20:26.995795   760 tablet_service.cc:1511] Processing CreateTablet for tablet 53e3310bec7842c1ad43b21aa1128e00 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:20:27.004846   757 tablet_service.cc:1511] Processing CreateTablet for tablet 47e1e3ade8a3474da6f42ae3abe03d41 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:20:27.016197   761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f4e57df4d0d4f8793bd68db29de1067. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.016467   887 tablet_bootstrap.cc:654] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.019029   756 tablet_service.cc:1511] Processing CreateTablet for tablet 55d9233a4c174bf6ac4f1608d6444455 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:20:27.021138   760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53e3310bec7842c1ad43b21aa1128e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.026957   755 tablet_service.cc:1511] Processing CreateTablet for tablet 00fda6319cef4443a37dc908e5245c54 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:20:27.039673   757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47e1e3ade8a3474da6f42ae3abe03d41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.045339   754 tablet_service.cc:1511] Processing CreateTablet for tablet 3ef1618bc7c94ff28a5312b7bb7e5fc3 (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:20:27.049232   755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00fda6319cef4443a37dc908e5245c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.051097   753 tablet_service.cc:1511] Processing CreateTablet for tablet d3e563cf261d49c9bb1d4ad63dc6500e (DEFAULT_TABLE table=test-workload [id=4bc570fbf60f4ecd8f95a35674922ea9]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:20:27.054925   756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55d9233a4c174bf6ac4f1608d6444455. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.057796   758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29a1f6ed8add4ddeb88b7832dad139ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.060650   753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3e563cf261d49c9bb1d4ad63dc6500e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.066480   754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef1618bc7c94ff28a5312b7bb7e5fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.077164   759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddb09687dc34423aadf5b8bebf5f34cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:27.099095   887 tablet_bootstrap.cc:492] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c: No bootstrap required, opened a new log
I20260709 12:20:27.099706   887 ts_tablet_manager.cc:1403] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c: Time spent bootstrapping tablet: real 0.174s	user 0.023s	sys 0.025s
I20260709 12:20:27.104907   887 raft_consensus.cc:359] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.105835   887 raft_consensus.cc:385] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.106199   887 raft_consensus.cc:740] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.113206   889 tablet_bootstrap.cc:492] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113: Bootstrap starting.
I20260709 12:20:27.145612   889 tablet_bootstrap.cc:654] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.167357   887 consensus_queue.cc:260] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.189350   890 tablet_bootstrap.cc:492] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6: Bootstrap starting.
I20260709 12:20:27.199221   889 tablet_bootstrap.cc:492] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113: No bootstrap required, opened a new log
I20260709 12:20:27.199906   889 ts_tablet_manager.cc:1403] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113: Time spent bootstrapping tablet: real 0.092s	user 0.009s	sys 0.012s
I20260709 12:20:27.204392   887 ts_tablet_manager.cc:1434] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c: Time spent starting tablet: real 0.104s	user 0.016s	sys 0.018s
I20260709 12:20:27.205452   870 heartbeater.cc:499] Master 127.0.140.126:46575 was elected leader, sending a full tablet report...
I20260709 12:20:27.207779   721 heartbeater.cc:499] Master 127.0.140.126:46575 was elected leader, sending a full tablet report...
I20260709 12:20:27.210036   887 tablet_bootstrap.cc:492] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c: Bootstrap starting.
I20260709 12:20:27.209311   889 raft_consensus.cc:359] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.210985   890 tablet_bootstrap.cc:654] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.210577   889 raft_consensus.cc:385] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.214427   889 raft_consensus.cc:740] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.215595   889 consensus_queue.cc:260] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.231276   887 tablet_bootstrap.cc:654] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.233302   889 ts_tablet_manager.cc:1434] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113: Time spent starting tablet: real 0.031s	user 0.013s	sys 0.003s
I20260709 12:20:27.237912   890 tablet_bootstrap.cc:492] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6: No bootstrap required, opened a new log
I20260709 12:20:27.238111   889 tablet_bootstrap.cc:492] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113: Bootstrap starting.
I20260709 12:20:27.238651   890 ts_tablet_manager.cc:1403] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6: Time spent bootstrapping tablet: real 0.050s	user 0.006s	sys 0.012s
I20260709 12:20:27.242358   890 raft_consensus.cc:359] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.243348   890 raft_consensus.cc:385] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.243750   890 raft_consensus.cc:740] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.244938   890 consensus_queue.cc:260] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.250154   889 tablet_bootstrap.cc:654] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.280073   887 tablet_bootstrap.cc:492] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c: No bootstrap required, opened a new log
I20260709 12:20:27.280290   889 tablet_bootstrap.cc:492] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113: No bootstrap required, opened a new log
I20260709 12:20:27.280629   887 ts_tablet_manager.cc:1403] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c: Time spent bootstrapping tablet: real 0.074s	user 0.026s	sys 0.008s
I20260709 12:20:27.280701   889 ts_tablet_manager.cc:1403] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113: Time spent bootstrapping tablet: real 0.046s	user 0.002s	sys 0.014s
I20260709 12:20:27.283833   889 raft_consensus.cc:359] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.283733   887 raft_consensus.cc:359] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.284576   889 raft_consensus.cc:385] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.284636   887 raft_consensus.cc:385] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.284994   889 raft_consensus.cc:740] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.285082   887 raft_consensus.cc:740] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.285837   889 consensus_queue.cc:260] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.285835   887 consensus_queue.cc:260] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.288178   889 ts_tablet_manager.cc:1434] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 12:20:27.288391   887 ts_tablet_manager.cc:1434] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:27.289158   887 tablet_bootstrap.cc:492] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c: Bootstrap starting.
I20260709 12:20:27.294853   796 heartbeater.cc:499] Master 127.0.140.126:46575 was elected leader, sending a full tablet report...
I20260709 12:20:27.295832   887 tablet_bootstrap.cc:654] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.289158   889 tablet_bootstrap.cc:492] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113: Bootstrap starting.
I20260709 12:20:27.294853   890 ts_tablet_manager.cc:1434] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6: Time spent starting tablet: real 0.056s	user 0.023s	sys 0.031s
I20260709 12:20:27.300979   890 tablet_bootstrap.cc:492] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6: Bootstrap starting.
I20260709 12:20:27.308265   889 tablet_bootstrap.cc:654] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.308851   890 tablet_bootstrap.cc:654] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.312467   887 tablet_bootstrap.cc:492] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c: No bootstrap required, opened a new log
W20260709 12:20:27.312649   871 tablet.cc:2406] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:20:27.313104   887 ts_tablet_manager.cc:1403] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c: Time spent bootstrapping tablet: real 0.024s	user 0.005s	sys 0.016s
I20260709 12:20:27.316526   887 raft_consensus.cc:359] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.317386   887 raft_consensus.cc:385] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.317678   887 raft_consensus.cc:740] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.318403   887 consensus_queue.cc:260] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.320852   887 ts_tablet_manager.cc:1434] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:27.321868   887 tablet_bootstrap.cc:492] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c: Bootstrap starting.
I20260709 12:20:27.323940   890 tablet_bootstrap.cc:492] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6: No bootstrap required, opened a new log
I20260709 12:20:27.324496   890 ts_tablet_manager.cc:1403] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6: Time spent bootstrapping tablet: real 0.024s	user 0.018s	sys 0.000s
I20260709 12:20:27.328011   890 raft_consensus.cc:359] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.328826   890 raft_consensus.cc:385] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.329126   890 raft_consensus.cc:740] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.329949   889 tablet_bootstrap.cc:492] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113: No bootstrap required, opened a new log
I20260709 12:20:27.329892   890 consensus_queue.cc:260] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.330502   889 ts_tablet_manager.cc:1403] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113: Time spent bootstrapping tablet: real 0.042s	user 0.018s	sys 0.000s
I20260709 12:20:27.330659   887 tablet_bootstrap.cc:654] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.332204   890 ts_tablet_manager.cc:1434] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:27.333400   890 tablet_bootstrap.cc:492] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6: Bootstrap starting.
I20260709 12:20:27.333729   889 raft_consensus.cc:359] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.334664   889 raft_consensus.cc:385] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.335049   889 raft_consensus.cc:740] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.335338   894 raft_consensus.cc:493] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.335824   894 raft_consensus.cc:515] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.338925   894 leader_election.cc:290] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:27.342020   890 tablet_bootstrap.cc:654] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.335844   889 consensus_queue.cc:260] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.345903   889 ts_tablet_manager.cc:1434] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20260709 12:20:27.351150   887 tablet_bootstrap.cc:492] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c: No bootstrap required, opened a new log
I20260709 12:20:27.349386   889 tablet_bootstrap.cc:492] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113: Bootstrap starting.
I20260709 12:20:27.358955   887 ts_tablet_manager.cc:1403] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c: Time spent bootstrapping tablet: real 0.037s	user 0.012s	sys 0.007s
I20260709 12:20:27.367856   887 raft_consensus.cc:359] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.368609   887 raft_consensus.cc:385] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.368899   887 raft_consensus.cc:740] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.371312   887 consensus_queue.cc:260] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.375978   887 ts_tablet_manager.cc:1434] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.003s
I20260709 12:20:27.377003   887 tablet_bootstrap.cc:492] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c: Bootstrap starting.
I20260709 12:20:27.383409   890 tablet_bootstrap.cc:492] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6: No bootstrap required, opened a new log
I20260709 12:20:27.384373   890 ts_tablet_manager.cc:1403] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6: Time spent bootstrapping tablet: real 0.051s	user 0.018s	sys 0.006s
I20260709 12:20:27.388712   889 tablet_bootstrap.cc:654] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.389081   890 raft_consensus.cc:359] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.390933   890 raft_consensus.cc:385] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.391271   890 raft_consensus.cc:740] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.395262   890 consensus_queue.cc:260] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.398551   894 raft_consensus.cc:493] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.399025   890 ts_tablet_manager.cc:1434] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.003s
I20260709 12:20:27.399158   894 raft_consensus.cc:515] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.400431   890 tablet_bootstrap.cc:492] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6: Bootstrap starting.
I20260709 12:20:27.403491   894 leader_election.cc:290] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:27.404222   894 raft_consensus.cc:493] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.404798   894 raft_consensus.cc:515] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.409780   894 leader_election.cc:290] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:27.414932   887 tablet_bootstrap.cc:654] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.417272   771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
I20260709 12:20:27.417836   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
I20260709 12:20:27.419505   769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
W20260709 12:20:27.423544   661 leader_election.cc:343] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.426344   889 tablet_bootstrap.cc:492] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113: No bootstrap required, opened a new log
W20260709 12:20:27.426225   661 leader_election.cc:343] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.426947   889 ts_tablet_manager.cc:1403] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113: Time spent bootstrapping tablet: real 0.079s	user 0.017s	sys 0.007s
I20260709 12:20:27.432076   890 tablet_bootstrap.cc:654] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:27.429138   661 leader_election.cc:343] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.438999   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:27.440341   844 raft_consensus.cc:2468] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 0.
I20260709 12:20:27.440562   889 raft_consensus.cc:359] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.441712   889 raft_consensus.cc:385] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.442011   843 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:27.442564   661 leader_election.cc:304] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:27.443814   907 raft_consensus.cc:2804] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:20:27.444118   661 leader_election.cc:343] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.444471   907 raft_consensus.cc:493] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.444634   661 leader_election.cc:304] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113; no voters: 2851d9518eb34a4c94883642f0fd37b6, f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:27.445067   907 raft_consensus.cc:3060] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.445443   894 raft_consensus.cc:2749] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.442132   889 raft_consensus.cc:740] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.439126   845 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:27.448431   887 tablet_bootstrap.cc:492] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c: No bootstrap required, opened a new log
I20260709 12:20:27.454484   845 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 0.
I20260709 12:20:27.454532   896 raft_consensus.cc:493] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.454990   887 ts_tablet_manager.cc:1403] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c: Time spent bootstrapping tablet: real 0.078s	user 0.023s	sys 0.001s
I20260709 12:20:27.454644   889 consensus_queue.cc:260] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.455359   896 raft_consensus.cc:515] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.457459   889 ts_tablet_manager.cc:1434] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.000s
I20260709 12:20:27.451005   890 tablet_bootstrap.cc:492] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6: No bootstrap required, opened a new log
I20260709 12:20:27.459213   890 ts_tablet_manager.cc:1403] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6: Time spent bootstrapping tablet: real 0.059s	user 0.014s	sys 0.008s
I20260709 12:20:27.453329   907 raft_consensus.cc:515] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.459563   896 leader_election.cc:290] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:27.461899   907 leader_election.cc:290] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:27.462373   890 raft_consensus.cc:359] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.463924   890 raft_consensus.cc:385] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.464283   890 raft_consensus.cc:740] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.458356   887 raft_consensus.cc:359] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.465101   887 raft_consensus.cc:385] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.465438   887 raft_consensus.cc:740] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.466372   891 raft_consensus.cc:493] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.466555   887 consensus_queue.cc:260] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.467044   891 raft_consensus.cc:515] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.468426   769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:27.470258   661 leader_election.cc:304] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:27.470590   891 leader_election.cc:290] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:27.471501   907 raft_consensus.cc:2804] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.471938   907 raft_consensus.cc:493] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.472337   907 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.472379   845 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:27.473021   845 raft_consensus.cc:3060] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.476464   889 tablet_bootstrap.cc:492] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113: Bootstrap starting.
I20260709 12:20:27.469261   890 consensus_queue.cc:260] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.481076   845 raft_consensus.cc:2468] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 1.
W20260709 12:20:27.483645   661 leader_election.cc:343] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.485383   890 ts_tablet_manager.cc:1434] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.000s
I20260709 12:20:27.486580   890 tablet_bootstrap.cc:492] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6: Bootstrap starting.
I20260709 12:20:27.493031   907 raft_consensus.cc:515] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.493521   661 leader_election.cc:304] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:27.502556   894 raft_consensus.cc:2804] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.512480   890 tablet_bootstrap.cc:654] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.525053   889 tablet_bootstrap.cc:654] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.525043   907 leader_election.cc:290] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:27.529492   896 raft_consensus.cc:493] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.533128   896 raft_consensus.cc:515] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.539320   907 raft_consensus.cc:493] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.540400   907 raft_consensus.cc:515] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.538937   896 leader_election.cc:290] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:27.554517   907 leader_election.cc:290] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:27.550568   887 ts_tablet_manager.cc:1434] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c: Time spent starting tablet: real 0.095s	user 0.012s	sys 0.016s
I20260709 12:20:27.556023   894 raft_consensus.cc:697] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:27.558687   894 consensus_queue.cc:237] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.559230   887 tablet_bootstrap.cc:492] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c: Bootstrap starting.
I20260709 12:20:27.594956   889 tablet_bootstrap.cc:492] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113: No bootstrap required, opened a new log
I20260709 12:20:27.595594   889 ts_tablet_manager.cc:1403] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113: Time spent bootstrapping tablet: real 0.119s	user 0.016s	sys 0.007s
I20260709 12:20:27.608827   896 raft_consensus.cc:493] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.609027   889 raft_consensus.cc:359] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.609449   896 raft_consensus.cc:515] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.610949   890 tablet_bootstrap.cc:492] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6: No bootstrap required, opened a new log
I20260709 12:20:27.611524   890 ts_tablet_manager.cc:1403] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6: Time spent bootstrapping tablet: real 0.125s	user 0.012s	sys 0.020s
I20260709 12:20:27.609802   889 raft_consensus.cc:385] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.611821   769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
I20260709 12:20:27.611656   896 leader_election.cc:290] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293), be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195)
I20260709 12:20:27.614322   889 raft_consensus.cc:740] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Initialized, Role: FOLLOWER
W20260709 12:20:27.620869   810 leader_election.cc:343] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.621552   769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:27.623000   769 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.631202   845 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:27.632014   845 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.646588   843 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
W20260709 12:20:27.650099   661 leader_election.cc:343] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.650229   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
I20260709 12:20:27.651032   770 raft_consensus.cc:2468] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 0.
I20260709 12:20:27.657444   661 leader_election.cc:304] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:27.621721   889 consensus_queue.cc:260] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.660009   894 raft_consensus.cc:2804] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.661865   845 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 1.
I20260709 12:20:27.662892   894 raft_consensus.cc:493] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.663757   843 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:27.663654   694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55d9233a4c174bf6ac4f1608d6444455" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:27.659828   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:27.665431   889 ts_tablet_manager.cc:1434] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113: Time spent starting tablet: real 0.070s	user 0.010s	sys 0.000s
I20260709 12:20:27.665264   894 raft_consensus.cc:3060] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.653689   593 catalog_manager.cc:5697] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 0 to 1, leader changed from <none> to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:27.662835   696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:27.666491   890 raft_consensus.cc:359] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.668882   842 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:27.670188   694 raft_consensus.cc:2468] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 0.
I20260709 12:20:27.670053   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55d9233a4c174bf6ac4f1608d6444455" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:27.670859   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
W20260709 12:20:27.673341   810 leader_election.cc:343] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.671356   887 tablet_bootstrap.cc:654] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.673149   844 raft_consensus.cc:2468] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 0.
I20260709 12:20:27.676494   810 leader_election.cc:304] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f40ae05748a5432199f3d7ca95a3311c; no voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113
I20260709 12:20:27.670933   661 leader_election.cc:304] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:27.679147   912 raft_consensus.cc:2749] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.670739   890 raft_consensus.cc:385] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.680284   918 raft_consensus.cc:2804] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:20:27.680707   736 leader_election.cc:343] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.685400   918 raft_consensus.cc:697] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:27.686097   736 leader_election.cc:304] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:27.687016   896 raft_consensus.cc:2804] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.687428   896 raft_consensus.cc:493] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.686625   918 consensus_queue.cc:237] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.687786   896 raft_consensus.cc:3060] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.681339   889 tablet_bootstrap.cc:492] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113: Bootstrap starting.
I20260709 12:20:27.691921   890 raft_consensus.cc:740] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.692961   890 consensus_queue.cc:260] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
W20260709 12:20:27.693827   736 leader_election.cc:343] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.684366   894 raft_consensus.cc:515] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.694360   736 leader_election.cc:304] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6; no voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:27.695919   693 raft_consensus.cc:2393] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2851d9518eb34a4c94883642f0fd37b6 in current term 1: Already voted for candidate be75e7f361dd41209a39df0af0d6a113 in this term.
I20260709 12:20:27.683449   769 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 1.
I20260709 12:20:27.698357   894 leader_election.cc:290] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Requested vote from peers f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:27.701661   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:27.702330   770 raft_consensus.cc:3060] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.700304   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
W20260709 12:20:27.704090   661 leader_election.cc:343] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.708775   896 raft_consensus.cc:515] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.712579   770 raft_consensus.cc:2468] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 1.
I20260709 12:20:27.714251   661 leader_election.cc:304] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:27.715457   918 raft_consensus.cc:2804] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.717933   890 ts_tablet_manager.cc:1434] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6: Time spent starting tablet: real 0.106s	user 0.014s	sys 0.005s
I20260709 12:20:27.719333   924 raft_consensus.cc:2749] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.718286   918 raft_consensus.cc:697] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:27.720553   887 tablet_bootstrap.cc:492] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c: No bootstrap required, opened a new log
I20260709 12:20:27.721447   887 ts_tablet_manager.cc:1403] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c: Time spent bootstrapping tablet: real 0.163s	user 0.020s	sys 0.012s
I20260709 12:20:27.721879   890 tablet_bootstrap.cc:492] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6: Bootstrap starting.
I20260709 12:20:27.723793   918 consensus_queue.cc:237] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.727919   887 raft_consensus.cc:359] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.720772   896 leader_election.cc:290] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:27.728809   887 raft_consensus.cc:385] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.729717   887 raft_consensus.cc:740] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.730940   887 consensus_queue.cc:260] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.734172   887 ts_tablet_manager.cc:1434] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.002s
I20260709 12:20:27.735299   887 tablet_bootstrap.cc:492] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c: Bootstrap starting.
I20260709 12:20:27.738381   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55d9233a4c174bf6ac4f1608d6444455" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:27.739481   693 raft_consensus.cc:3060] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:27.742271   736 leader_election.cc:343] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.751848   889 tablet_bootstrap.cc:654] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.754266   736 leader_election.cc:304] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6; no voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:27.755972   896 raft_consensus.cc:2749] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.756237   890 tablet_bootstrap.cc:654] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.759740   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55d9233a4c174bf6ac4f1608d6444455" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:27.760658   844 raft_consensus.cc:3060] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.764253   693 raft_consensus.cc:2468] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 1.
I20260709 12:20:27.765731   736 leader_election.cc:304] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:27.766693   896 raft_consensus.cc:2804] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.767557   896 raft_consensus.cc:697] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:27.767017   887 tablet_bootstrap.cc:654] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.770887   896 consensus_queue.cc:237] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.778445   889 tablet_bootstrap.cc:492] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113: No bootstrap required, opened a new log
I20260709 12:20:27.781102   844 raft_consensus.cc:2468] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 1.
I20260709 12:20:27.792718   889 ts_tablet_manager.cc:1403] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113: Time spent bootstrapping tablet: real 0.109s	user 0.018s	sys 0.001s
I20260709 12:20:27.800845   887 tablet_bootstrap.cc:492] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c: No bootstrap required, opened a new log
I20260709 12:20:27.801512   887 ts_tablet_manager.cc:1403] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c: Time spent bootstrapping tablet: real 0.066s	user 0.008s	sys 0.010s
I20260709 12:20:27.805160   887 raft_consensus.cc:359] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.806408   887 raft_consensus.cc:385] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.806833   887 raft_consensus.cc:740] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.807744   887 consensus_queue.cc:260] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.810494   887 ts_tablet_manager.cc:1434] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:20:27.813419   890 tablet_bootstrap.cc:492] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6: No bootstrap required, opened a new log
I20260709 12:20:27.814178   890 ts_tablet_manager.cc:1403] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6: Time spent bootstrapping tablet: real 0.093s	user 0.012s	sys 0.006s
I20260709 12:20:27.796439   889 raft_consensus.cc:359] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.817399   889 raft_consensus.cc:385] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.817878   889 raft_consensus.cc:740] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.818933   889 consensus_queue.cc:260] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.819834   887 tablet_bootstrap.cc:492] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c: Bootstrap starting.
I20260709 12:20:27.822202   889 ts_tablet_manager.cc:1434] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260709 12:20:27.823907   889 tablet_bootstrap.cc:492] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113: Bootstrap starting.
I20260709 12:20:27.826856   890 raft_consensus.cc:359] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.828459   890 raft_consensus.cc:385] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.829257   890 raft_consensus.cc:740] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.830161   890 consensus_queue.cc:260] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.833010   890 ts_tablet_manager.cc:1434] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.003s
I20260709 12:20:27.834345   890 tablet_bootstrap.cc:492] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6: Bootstrap starting.
I20260709 12:20:27.837903   889 tablet_bootstrap.cc:654] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.827966   591 catalog_manager.cc:5697] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:27.839497   887 tablet_bootstrap.cc:654] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.842717   890 tablet_bootstrap.cc:654] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.850534   889 tablet_bootstrap.cc:492] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113: No bootstrap required, opened a new log
I20260709 12:20:27.851217   889 ts_tablet_manager.cc:1403] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.003s
I20260709 12:20:27.860185   593 catalog_manager.cc:5697] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 0 to 1, leader changed from <none> to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:27.861389   889 raft_consensus.cc:359] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.862231   889 raft_consensus.cc:385] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.862761   889 raft_consensus.cc:740] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.866006   889 consensus_queue.cc:260] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.869169   889 ts_tablet_manager.cc:1434] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.004s
I20260709 12:20:27.870662   889 tablet_bootstrap.cc:492] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113: Bootstrap starting.
I20260709 12:20:27.863664   593 catalog_manager.cc:5697] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 0 to 1, leader changed from <none> to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:27.877905   889 tablet_bootstrap.cc:654] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.885848   890 tablet_bootstrap.cc:492] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6: No bootstrap required, opened a new log
I20260709 12:20:27.886509   890 ts_tablet_manager.cc:1403] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6: Time spent bootstrapping tablet: real 0.052s	user 0.015s	sys 0.003s
I20260709 12:20:27.887518   891 raft_consensus.cc:493] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.888089   891 raft_consensus.cc:515] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:27.889655   890 raft_consensus.cc:359] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.890257   891 leader_election.cc:290] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:27.890412   890 raft_consensus.cc:385] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.890872   891 raft_consensus.cc:493] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.891008   890 raft_consensus.cc:740] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.891373   891 raft_consensus.cc:515] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.891952   890 consensus_queue.cc:260] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.893514   891 leader_election.cc:290] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
W20260709 12:20:27.893850   641 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:27.894840   890 ts_tablet_manager.cc:1434] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:27.895576   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
I20260709 12:20:27.895934   890 tablet_bootstrap.cc:492] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6: Bootstrap starting.
I20260709 12:20:27.896412   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:27.897238   769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
I20260709 12:20:27.897783   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:27.898401   695 raft_consensus.cc:2468] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 0.
W20260709 12:20:27.898639   810 leader_election.cc:343] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.896433   770 raft_consensus.cc:2393] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f40ae05748a5432199f3d7ca95a3311c in current term 1: Already voted for candidate be75e7f361dd41209a39df0af0d6a113 in this term.
I20260709 12:20:27.904258   810 leader_election.cc:304] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f40ae05748a5432199f3d7ca95a3311c; no voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113
I20260709 12:20:27.905578   810 leader_election.cc:304] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:27.906205   891 raft_consensus.cc:3060] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.907013   912 raft_consensus.cc:2804] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:27.907620   912 raft_consensus.cc:493] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:27.907992   912 raft_consensus.cc:3060] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.922560   887 tablet_bootstrap.cc:492] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c: No bootstrap required, opened a new log
I20260709 12:20:27.927870   887 ts_tablet_manager.cc:1403] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c: Time spent bootstrapping tablet: real 0.108s	user 0.026s	sys 0.026s
I20260709 12:20:27.936434   912 raft_consensus.cc:515] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.944392   891 raft_consensus.cc:2749] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:27.931705   887 raft_consensus.cc:359] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.945282   887 raft_consensus.cc:385] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.945398   912 leader_election.cc:290] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:27.945601   887 raft_consensus.cc:740] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.946260   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:27.946924   695 raft_consensus.cc:3060] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:27.946426   887 consensus_queue.cc:260] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.947345   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:27.948824   887 ts_tablet_manager.cc:1434] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.000s
W20260709 12:20:27.949489   810 leader_election.cc:343] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:27.949781   887 tablet_bootstrap.cc:492] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c: Bootstrap starting.
I20260709 12:20:27.950628   889 tablet_bootstrap.cc:492] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113: No bootstrap required, opened a new log
I20260709 12:20:27.951169   889 ts_tablet_manager.cc:1403] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113: Time spent bootstrapping tablet: real 0.081s	user 0.008s	sys 0.044s
I20260709 12:20:27.955551   889 raft_consensus.cc:359] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.956915   889 raft_consensus.cc:385] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.957093   890 tablet_bootstrap.cc:654] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.956513   887 tablet_bootstrap.cc:654] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.957191   889 raft_consensus.cc:740] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.961149   889 consensus_queue.cc:260] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.964543   889 ts_tablet_manager.cc:1434] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.001s
I20260709 12:20:27.965366   695 raft_consensus.cc:2468] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 1.
I20260709 12:20:27.965631   889 tablet_bootstrap.cc:492] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113: Bootstrap starting.
I20260709 12:20:27.967801   810 leader_election.cc:304] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:27.972138   912 raft_consensus.cc:2804] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:27.973618   912 raft_consensus.cc:697] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Becoming Leader. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:27.974599   912 consensus_queue.cc:237] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.983862   890 tablet_bootstrap.cc:492] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6: No bootstrap required, opened a new log
I20260709 12:20:27.984380   890 ts_tablet_manager.cc:1403] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6: Time spent bootstrapping tablet: real 0.089s	user 0.012s	sys 0.003s
I20260709 12:20:27.985327   887 tablet_bootstrap.cc:492] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c: No bootstrap required, opened a new log
I20260709 12:20:27.985754   887 ts_tablet_manager.cc:1403] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.009s
I20260709 12:20:27.988232   890 raft_consensus.cc:359] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.988894   890 raft_consensus.cc:385] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:27.989115   890 raft_consensus.cc:740] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Initialized, Role: FOLLOWER
I20260709 12:20:27.989854   890 consensus_queue.cc:260] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.989952   896 raft_consensus.cc:493] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:27.993311   890 ts_tablet_manager.cc:1434] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 12:20:27.993593   896 raft_consensus.cc:515] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:27.994638   889 tablet_bootstrap.cc:654] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:27.994661   890 tablet_bootstrap.cc:492] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6: Bootstrap starting.
I20260709 12:20:27.995944   896 leader_election.cc:290] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:27.997443   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:27.997611   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:27.999110   844 raft_consensus.cc:2393] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2851d9518eb34a4c94883642f0fd37b6 in current term 1: Already voted for candidate be75e7f361dd41209a39df0af0d6a113 in this term.
I20260709 12:20:28.000463   887 raft_consensus.cc:359] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.001194   887 raft_consensus.cc:385] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.001438   887 raft_consensus.cc:740] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.002269   887 consensus_queue.cc:260] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.004782   887 ts_tablet_manager.cc:1434] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.001s
I20260709 12:20:28.006554   736 leader_election.cc:304] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6; no voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:28.007211   770 raft_consensus.cc:3060] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.010574   890 tablet_bootstrap.cc:654] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:28.016659   889 tablet_bootstrap.cc:492] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113: No bootstrap required, opened a new log
I20260709 12:20:28.017405   889 ts_tablet_manager.cc:1403] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113: Time spent bootstrapping tablet: real 0.052s	user 0.017s	sys 0.000s
I20260709 12:20:28.013080   590 catalog_manager.cc:5697] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c reported cstate change: term changed from 0 to 1, leader changed from <none> to f40ae05748a5432199f3d7ca95a3311c (127.0.140.67). New cstate: current_term: 1 leader_uuid: "f40ae05748a5432199f3d7ca95a3311c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:28.022899   889 raft_consensus.cc:359] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.023757   889 raft_consensus.cc:385] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.024138   889 raft_consensus.cc:740] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.025321   896 raft_consensus.cc:2749] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.025068   889 consensus_queue.cc:260] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.027791   889 ts_tablet_manager.cc:1434] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:20:28.029414   918 consensus_queue.cc:1048] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.059295   890 tablet_bootstrap.cc:492] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6: No bootstrap required, opened a new log
I20260709 12:20:28.059868   890 ts_tablet_manager.cc:1403] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6: Time spent bootstrapping tablet: real 0.065s	user 0.027s	sys 0.009s
I20260709 12:20:28.061267   894 raft_consensus.cc:493] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.061830   894 raft_consensus.cc:515] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.063025   890 raft_consensus.cc:359] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.063791   890 raft_consensus.cc:385] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:28.064085   890 raft_consensus.cc:740] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Initialized, Role: FOLLOWER
I20260709 12:20:28.064205   896 raft_consensus.cc:493] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.064667   896 raft_consensus.cc:515] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.064816   890 consensus_queue.cc:260] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.063803   894 leader_election.cc:290] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.067070   890 ts_tablet_manager.cc:1434] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:28.070555   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:28.071421   695 raft_consensus.cc:2468] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 0.
I20260709 12:20:28.072264   769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
I20260709 12:20:28.072664   896 leader_election.cc:290] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.072678   736 leader_election.cc:304] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:28.073367   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:28.073978   844 raft_consensus.cc:2468] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 0.
I20260709 12:20:28.076720   661 leader_election.cc:304] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:28.076989   842 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:28.077656   842 raft_consensus.cc:2468] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 0.
I20260709 12:20:28.077736   894 raft_consensus.cc:2804] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.078234   924 raft_consensus.cc:493] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.078485   894 raft_consensus.cc:493] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.079033   894 raft_consensus.cc:3060] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.078711   924 raft_consensus.cc:515] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.072906   769 raft_consensus.cc:2468] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 0.
I20260709 12:20:28.082155   896 raft_consensus.cc:2804] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.082429   924 leader_election.cc:290] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.083211   896 raft_consensus.cc:493] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.084570   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:28.084789   842 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:28.085304   695 raft_consensus.cc:2468] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 0.
I20260709 12:20:28.085479   842 raft_consensus.cc:2468] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 0.
I20260709 12:20:28.086644   736 leader_election.cc:304] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:28.089541   924 raft_consensus.cc:2804] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.089910   924 raft_consensus.cc:493] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.090248   924 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.092072   894 raft_consensus.cc:515] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.092701   914 consensus_queue.cc:1048] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:28.094475   894 leader_election.cc:290] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.096179   843 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:28.097920   924 raft_consensus.cc:515] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.096750   843 raft_consensus.cc:3060] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.100862   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:28.101465   695 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.104362   769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:28.108836   695 raft_consensus.cc:2468] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 1.
I20260709 12:20:28.110103   736 leader_election.cc:304] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:28.111330   925 raft_consensus.cc:2804] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.112707   845 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:28.113338   845 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.083643   896 raft_consensus.cc:3060] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.121632   845 raft_consensus.cc:2468] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 1.
I20260709 12:20:28.123653   924 leader_election.cc:290] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.125285   924 raft_consensus.cc:493] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.125739   924 raft_consensus.cc:515] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.128885   843 raft_consensus.cc:2468] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 1.
I20260709 12:20:28.129277   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:28.130528   661 leader_election.cc:304] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:28.131878   918 raft_consensus.cc:2804] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.132565   918 raft_consensus.cc:697] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:28.133015   845 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:28.134850   918 consensus_queue.cc:237] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.133518   924 leader_election.cc:290] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.130766   695 raft_consensus.cc:2393] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2851d9518eb34a4c94883642f0fd37b6 in current term 1: Already voted for candidate f40ae05748a5432199f3d7ca95a3311c in this term.
I20260709 12:20:28.137197   925 raft_consensus.cc:697] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 1 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:28.137038   896 raft_consensus.cc:515] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.138337   769 raft_consensus.cc:2393] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate be75e7f361dd41209a39df0af0d6a113 in current term 1: Already voted for candidate 2851d9518eb34a4c94883642f0fd37b6 in this term.
I20260709 12:20:28.138077   736 leader_election.cc:304] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6; no voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:28.138193   925 consensus_queue.cc:237] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.141530   949 raft_consensus.cc:3060] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.155525   845 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:28.156414   845 raft_consensus.cc:2393] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2851d9518eb34a4c94883642f0fd37b6 in current term 1: Already voted for candidate be75e7f361dd41209a39df0af0d6a113 in this term.
I20260709 12:20:28.156438   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:28.161481   736 leader_election.cc:304] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6; no voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:28.161935   590 catalog_manager.cc:5697] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 0 to 1, leader changed from <none> to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.163232   896 leader_election.cc:290] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 1 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.163604   949 raft_consensus.cc:2749] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.164103   896 raft_consensus.cc:2749] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:28.166013   918 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:28.167064   589 catalog_manager.cc:5697] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.178566   912 raft_consensus.cc:493] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.179234   912 raft_consensus.cc:515] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.180480   918 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:28.181674   912 leader_election.cc:290] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:28.183437   769 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
I20260709 12:20:28.184167   769 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 0.
I20260709 12:20:28.185309   810 leader_election.cc:304] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:28.186281   912 raft_consensus.cc:2804] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.186707   912 raft_consensus.cc:493] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.187129   912 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.187361   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113" is_pre_election: true
I20260709 12:20:28.203075   695 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 0.
I20260709 12:20:28.195185   912 raft_consensus.cc:515] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.187582   918 consensus_queue.cc:1048] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 12:20:28.206990   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:28.205410   912 leader_election.cc:290] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:28.207823   693 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.214031   918 consensus_queue.cc:1048] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.215070   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:28.216074   770 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.221520   693 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 1.
I20260709 12:20:28.223038   810 leader_election.cc:304] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:28.224220   912 raft_consensus.cc:2804] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.225128   912 raft_consensus.cc:697] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Becoming Leader. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:28.226258   912 consensus_queue.cc:237] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.233613   770 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 1.
I20260709 12:20:28.273955   589 catalog_manager.cc:5697] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c reported cstate change: term changed from 0 to 1, leader changed from <none> to f40ae05748a5432199f3d7ca95a3311c (127.0.140.67). New cstate: current_term: 1 leader_uuid: "f40ae05748a5432199f3d7ca95a3311c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:28.280014   918 raft_consensus.cc:493] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:28.280592   918 raft_consensus.cc:515] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.282923   918 leader_election.cc:290] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.283880   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6" is_pre_election: true
I20260709 12:20:28.284793   770 raft_consensus.cc:2468] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 0.
I20260709 12:20:28.285512   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c" is_pre_election: true
I20260709 12:20:28.286217   661 leader_election.cc:304] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:28.286239   844 raft_consensus.cc:2468] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 0.
I20260709 12:20:28.287133   918 raft_consensus.cc:2804] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:28.287544   918 raft_consensus.cc:493] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:28.287923   918 raft_consensus.cc:3060] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.295274   918 raft_consensus.cc:515] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.297228   918 leader_election.cc:290] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.298378   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:28.298355   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:28.299177   844 raft_consensus.cc:3060] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.299214   770 raft_consensus.cc:3060] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:28.307282   770 raft_consensus.cc:2468] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 1.
I20260709 12:20:28.307282   844 raft_consensus.cc:2468] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 1.
I20260709 12:20:28.309015   661 leader_election.cc:304] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:28.309783   918 raft_consensus.cc:2804] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:28.310401   918 raft_consensus.cc:697] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:28.311350   918 consensus_queue.cc:237] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.321509   896 consensus_queue.cc:1048] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.321895   589 catalog_manager.cc:5697] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 0 to 1, leader changed from <none> to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.340191   950 consensus_queue.cc:1048] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.553195   912 consensus_queue.cc:1048] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.564054   845 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00"
dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
mode: GRACEFUL
new_leader_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:36542
I20260709 12:20:28.564867   845 raft_consensus.cc:606] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Received request to transfer leadership to TS be75e7f361dd41209a39df0af0d6a113
I20260709 12:20:28.571022   891 consensus_queue.cc:1048] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.573097   845 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb"
dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
mode: GRACEFUL
new_leader_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:36542
I20260709 12:20:28.573997   845 raft_consensus.cc:606] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Received request to transfer leadership to TS be75e7f361dd41209a39df0af0d6a113
I20260709 12:20:28.586052   924 consensus_queue.cc:1048] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:28.600291   770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
mode: GRACEFUL
new_leader_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:34624
I20260709 12:20:28.600971   770 raft_consensus.cc:606] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 1 LEADER]: Received request to transfer leadership to TS be75e7f361dd41209a39df0af0d6a113
I20260709 12:20:28.611645   949 consensus_queue.cc:1048] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.614635   950 raft_consensus.cc:993] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6: : Instructing follower be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:28.615239   950 raft_consensus.cc:1081] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 1 LEADER]: Signalling peer be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:28.622196   695 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:46374
I20260709 12:20:28.623689   695 raft_consensus.cc:493] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:28.624886   695 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:28.634801   693 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
mode: GRACEFUL
new_leader_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:46406
I20260709 12:20:28.635474   693 raft_consensus.cc:606] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Received request to transfer leadership to TS 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:28.636941   695 raft_consensus.cc:515] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.639185   695 leader_election.cc:290] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 2 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.641346   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:28.642154   770 raft_consensus.cc:3055] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:28.642531   770 raft_consensus.cc:740] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:28.643359   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:28.643543   770 consensus_queue.cc:260] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.649493   770 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:28.646003   695 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
mode: GRACEFUL
new_leader_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:46406
I20260709 12:20:28.654507   695 raft_consensus.cc:606] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Received request to transfer leadership to TS 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:28.656581   770 raft_consensus.cc:2468] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 2.
I20260709 12:20:28.658921   661 leader_election.cc:304] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:28.660779   918 raft_consensus.cc:2804] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:28.661373   918 raft_consensus.cc:697] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:28.663081   695 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
mode: GRACEFUL
new_leader_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:46406
I20260709 12:20:28.662312   918 consensus_queue.cc:237] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.663740   695 raft_consensus.cc:606] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Received request to transfer leadership to TS 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:28.669188   695 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
mode: GRACEFUL
new_leader_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:46406
I20260709 12:20:28.669785   695 raft_consensus.cc:606] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Received request to transfer leadership to TS f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:28.672024   591 catalog_manager.cc:5697] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 1 to 2, leader changed from 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66) to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.676105   891 consensus_queue.cc:1048] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:28.692426   912 consensus_queue.cc:1048] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.707352   918 consensus_queue.cc:1048] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:28.712338   891 raft_consensus.cc:993] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c: : Instructing follower be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:28.712877   891 raft_consensus.cc:1081] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Signalling peer be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:28.715050   695 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:46390
I20260709 12:20:28.716965   695 raft_consensus.cc:493] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:28.717545   695 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:28.721375   918 consensus_queue.cc:1048] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:28.724612   695 raft_consensus.cc:515] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.727101   695 leader_election.cc:290] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 2 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.728127   771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:28.728271   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:28.728943   771 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:28.729059   844 raft_consensus.cc:3055] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:28.729496   844 raft_consensus.cc:740] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:28.730273   844 consensus_queue.cc:260] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.731251   844 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:28.737462   771 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 2.
I20260709 12:20:28.739464   661 leader_election.cc:304] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:28.741691   918 raft_consensus.cc:2804] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:28.742726   918 raft_consensus.cc:697] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:28.743772   918 consensus_queue.cc:237] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.757565   844 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 2.
I20260709 12:20:28.759092   914 raft_consensus.cc:993] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113: : Instructing follower f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:28.760694   914 raft_consensus.cc:1081] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Signalling peer f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:28.764729   843 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e"
dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:36538
I20260709 12:20:28.765399   843 raft_consensus.cc:493] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:28.765817   843 raft_consensus.cc:3060] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:28.769595   914 consensus_queue.cc:1048] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:28.774080   843 raft_consensus.cc:515] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.777503   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:28.778153   695 raft_consensus.cc:3055] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:28.778469   695 raft_consensus.cc:740] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:28.779453   844 raft_consensus.cc:1240] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Rejecting Update request from peer be75e7f361dd41209a39df0af0d6a113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:28.779539   695 consensus_queue.cc:260] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.781525   695 raft_consensus.cc:3060] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:28.787768   771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d3e563cf261d49c9bb1d4ad63dc6500e" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:28.788488   771 raft_consensus.cc:3060] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:28.781340   918 consensus_queue.cc:1175] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f40ae05748a5432199f3d7ca95a3311c" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer be75e7f361dd41209a39df0af0d6a113 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:20:28.791874   590 catalog_manager.cc:5697] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 1 to 2, leader changed from f40ae05748a5432199f3d7ca95a3311c (127.0.140.67) to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.799937   771 raft_consensus.cc:2468] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 2.
I20260709 12:20:28.800850   843 leader_election.cc:290] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 2 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:28.803355   810 leader_election.cc:304] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:28.804291   912 raft_consensus.cc:2804] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:28.810241   912 raft_consensus.cc:697] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 2 LEADER]: Becoming Leader. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:28.811197   912 consensus_queue.cc:237] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.811636   695 raft_consensus.cc:2468] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 2.
I20260709 12:20:28.813108   894 raft_consensus.cc:993] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113: : Instructing follower 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:28.813563   894 raft_consensus.cc:1081] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Signalling peer 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:28.820205   589 catalog_manager.cc:5697] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c reported cstate change: term changed from 1 to 2, leader changed from be75e7f361dd41209a39df0af0d6a113 (127.0.140.65) to f40ae05748a5432199f3d7ca95a3311c (127.0.140.67). New cstate: current_term: 2 leader_uuid: "f40ae05748a5432199f3d7ca95a3311c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:28.821973   771 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:34600
I20260709 12:20:28.822636   771 raft_consensus.cc:493] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:28.823081   771 raft_consensus.cc:3060] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:28.827728   918 consensus_queue.cc:1048] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:28.830538   771 raft_consensus.cc:515] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.834069   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:28.834816   695 raft_consensus.cc:3055] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:28.835153   695 raft_consensus.cc:740] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:28.835963   770 raft_consensus.cc:1240] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Rejecting Update request from peer be75e7f361dd41209a39df0af0d6a113 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:28.838479   695 consensus_queue.cc:260] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.839920   695 raft_consensus.cc:3060] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:28.842291   771 leader_election.cc:290] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 2 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:28.843510   843 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:28.849167   695 raft_consensus.cc:2468] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 2.
I20260709 12:20:28.850304   736 leader_election.cc:304] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:28.851060   896 raft_consensus.cc:2804] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:28.851521   896 raft_consensus.cc:697] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:28.852340   896 consensus_queue.cc:237] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:28.861881   589 catalog_manager.cc:5697] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 1 to 2, leader changed from be75e7f361dd41209a39df0af0d6a113 (127.0.140.65) to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.989881   970 raft_consensus.cc:993] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c: : Instructing follower be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:28.990459   970 raft_consensus.cc:1081] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Signalling peer be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:28.992554   695 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:46390
I20260709 12:20:28.993209   695 raft_consensus.cc:493] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:28.993579   695 raft_consensus.cc:3060] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.001117   695 raft_consensus.cc:515] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:29.003520   695 leader_election.cc:290] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 2 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:29.004554   771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:29.004806   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:29.005261   771 raft_consensus.cc:3060] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.005333   844 raft_consensus.cc:3055] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:29.005741   844 raft_consensus.cc:740] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:29.006604   844 consensus_queue.cc:260] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:29.007864   844 raft_consensus.cc:3060] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.013468   771 raft_consensus.cc:2468] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 2.
I20260709 12:20:29.014500   844 raft_consensus.cc:2468] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 2.
I20260709 12:20:29.014856   661 leader_election.cc:304] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:29.015741   918 raft_consensus.cc:2804] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:29.016206   918 raft_consensus.cc:697] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:29.017149   894 raft_consensus.cc:993] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113: : Instructing follower 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:29.017014   918 consensus_queue.cc:237] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:29.017676   894 raft_consensus.cc:1081] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Signalling peer 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:29.019697   771 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:34600
I20260709 12:20:29.020380   771 raft_consensus.cc:493] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:29.020944   771 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.026394   771 raft_consensus.cc:515] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:29.025938   589 catalog_manager.cc:5697] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 1 to 2, leader changed from f40ae05748a5432199f3d7ca95a3311c (127.0.140.67) to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:29.028620   771 leader_election.cc:290] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 2 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:29.030232   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:29.030930   695 raft_consensus.cc:3055] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:29.030232   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:29.031335   695 raft_consensus.cc:740] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:29.031539   844 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.032174   695 consensus_queue.cc:260] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:29.033675   695 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.039690   844 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 2.
I20260709 12:20:29.041123   736 leader_election.cc:304] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:29.042100   896 raft_consensus.cc:2804] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:29.042469   894 raft_consensus.cc:993] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113: : Instructing follower 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:29.042958   894 raft_consensus.cc:1081] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Signalling peer 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:29.042625   896 raft_consensus.cc:697] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:29.045001   770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:34600
I20260709 12:20:29.044611   896 consensus_queue.cc:237] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:29.045353   695 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 2.
I20260709 12:20:29.045614   770 raft_consensus.cc:493] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:29.046170   770 raft_consensus.cc:3060] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.052443   770 raft_consensus.cc:515] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:29.055790   770 leader_election.cc:290] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 2 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:29.054937   589 catalog_manager.cc:5697] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 1 to 2, leader changed from be75e7f361dd41209a39df0af0d6a113 (127.0.140.65) to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:29.057715   843 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.056821   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:29.058473   695 raft_consensus.cc:3055] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:29.059129   695 raft_consensus.cc:740] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:29.057104   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:29.060065   844 raft_consensus.cc:3060] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.065303   843 raft_consensus.cc:1275] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.066165   695 consensus_queue.cc:260] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:29.067819   695 raft_consensus.cc:3060] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.067889   918 consensus_queue.cc:1048] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:29.071911   844 raft_consensus.cc:2468] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 2.
I20260709 12:20:29.073269   736 leader_election.cc:304] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:29.074229   896 raft_consensus.cc:2804] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:29.074756   896 raft_consensus.cc:697] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:29.076464   896 consensus_queue.cc:237] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:29.076423   695 raft_consensus.cc:2468] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 2.
I20260709 12:20:29.085881   770 raft_consensus.cc:1275] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.087596   918 consensus_queue.cc:1048] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:29.090407   591 catalog_manager.cc:5697] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 1 to 2, leader changed from be75e7f361dd41209a39df0af0d6a113 (127.0.140.65) to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:29.218575   843 raft_consensus.cc:1275] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.220821   918 consensus_queue.cc:1048] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:29.234453   770 raft_consensus.cc:1275] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.236186   914 consensus_queue.cc:1048] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:29.267139   695 raft_consensus.cc:1275] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.268759   896 consensus_queue.cc:1048] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:29.286765   843 raft_consensus.cc:3060] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:29.295171   843 raft_consensus.cc:1275] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.298203   693 raft_consensus.cc:1275] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.297979   949 consensus_queue.cc:1048] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:29.299844   970 consensus_queue.cc:1048] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:29.315403   770 raft_consensus.cc:1275] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.317670   912 consensus_queue.cc:1048] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:29.530359   843 raft_consensus.cc:1275] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.531843   949 consensus_queue.cc:1048] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:29.541412   693 raft_consensus.cc:1275] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.543269   896 consensus_queue.cc:1048] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:29.556734   843 raft_consensus.cc:1275] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.558482   896 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:29.568694   693 raft_consensus.cc:1275] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.571027   976 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:29.599668   770 raft_consensus.cc:1275] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.601145   918 consensus_queue.cc:1048] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:29.611568   843 raft_consensus.cc:1275] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:29.613157   918 consensus_queue.cc:1048] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:31.705794   561 auto_leader_rebalancer-test.cc:294] The leader distribution is 
I20260709 12:20:31.706707   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:31.725337   770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
mode: GRACEFUL
new_leader_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:51726
I20260709 12:20:31.726014   770 raft_consensus.cc:606] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Received request to transfer leadership to TS f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:31.738116   693 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
mode: GRACEFUL
new_leader_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:43888
I20260709 12:20:31.738760   693 raft_consensus.cc:606] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Received request to transfer leadership to TS f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:31.740458   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:20:31.741107   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:32.027961   989 raft_consensus.cc:993] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113: : Instructing follower f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:32.028473   985 raft_consensus.cc:1081] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Signalling peer f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:32.030656   844 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067"
dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:36538
I20260709 12:20:32.031313   844 raft_consensus.cc:493] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:32.031625   844 raft_consensus.cc:3060] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:32.037922   844 raft_consensus.cc:515] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:32.040313   844 leader_election.cc:290] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 2 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:32.041178   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:32.041555   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:32.041960   695 raft_consensus.cc:3055] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:32.042260   770 raft_consensus.cc:3060] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:32.042450   695 raft_consensus.cc:740] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:32.043565   695 consensus_queue.cc:260] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:32.044930   695 raft_consensus.cc:3060] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:32.049489   770 raft_consensus.cc:2468] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 2.
I20260709 12:20:32.050911   810 leader_election.cc:304] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:32.051702   984 raft_consensus.cc:2804] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:32.051729   695 raft_consensus.cc:2468] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 2.
I20260709 12:20:32.053593   984 raft_consensus.cc:697] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 2 LEADER]: Becoming Leader. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:32.054504   984 consensus_queue.cc:237] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:32.062479   590 catalog_manager.cc:5697] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c reported cstate change: term changed from 1 to 2, leader changed from be75e7f361dd41209a39df0af0d6a113 (127.0.140.65) to f40ae05748a5432199f3d7ca95a3311c (127.0.140.67). New cstate: current_term: 2 leader_uuid: "f40ae05748a5432199f3d7ca95a3311c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:32.234426   896 raft_consensus.cc:993] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6: : Instructing follower f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:32.234947   896 raft_consensus.cc:1081] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Signalling peer f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:32.236709   844 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41"
dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:36540
I20260709 12:20:32.237244   844 raft_consensus.cc:493] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:32.237587   844 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:32.243039   844 raft_consensus.cc:515] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:32.244771   844 leader_election.cc:290] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 3 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:32.245867   695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:32.246464   695 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:32.246421   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:32.247283   770 raft_consensus.cc:3055] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:32.247643   770 raft_consensus.cc:740] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:32.248443   770 consensus_queue.cc:260] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:32.249459   770 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:32.252445   695 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 3.
I20260709 12:20:32.253769   810 leader_election.cc:304] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:32.254696   984 raft_consensus.cc:2804] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:32.255107   770 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 3.
I20260709 12:20:32.255281   984 raft_consensus.cc:697] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 3 LEADER]: Becoming Leader. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:32.256259   984 consensus_queue.cc:237] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:32.263689   590 catalog_manager.cc:5697] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c reported cstate change: term changed from 2 to 3, leader changed from 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66) to f40ae05748a5432199f3d7ca95a3311c (127.0.140.67). New cstate: current_term: 3 leader_uuid: "f40ae05748a5432199f3d7ca95a3311c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:32.462071   770 raft_consensus.cc:1275] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:32.463666   984 consensus_queue.cc:1048] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:32.479815   695 raft_consensus.cc:1275] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:32.481563   990 consensus_queue.cc:1048] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:32.742338   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:32.747838   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:32.748272   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:32.776996   695 raft_consensus.cc:1275] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:32.778637   984 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.783265   770 raft_consensus.cc:1275] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:32.785555   990 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:33.749194   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:33.754122   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:33.754501   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:34.755740   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:34.761982   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:34.762506   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:35.763764   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:35.768953   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:35.769395   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:37.172411   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:37.177635   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:37.178148   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:20:38.021230   866 debug-util.cc:398] Leaking SignalData structure 0x7b080016c060 after lost signal to thread 564
W20260709 12:20:38.022286   866 debug-util.cc:398] Leaking SignalData structure 0x7b080016c740 after lost signal to thread 627
W20260709 12:20:38.023319   866 debug-util.cc:398] Leaking SignalData structure 0x7b08001e1ca0 after lost signal to thread 720
W20260709 12:20:38.024125   866 debug-util.cc:398] Leaking SignalData structure 0x7b0800197440 after lost signal to thread 795
W20260709 12:20:38.025024   866 debug-util.cc:398] Leaking SignalData structure 0x7b0800190b20 after lost signal to thread 869
I20260709 12:20:38.179276   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:38.185966   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:38.186565   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:39.187754   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:39.193146   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:39.193666   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:40.195050   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:40.202528   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:40.203083   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:41.204152   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:41.210513   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:41.211148   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:42.247771   561 auto_leader_rebalancer-test.cc:313] The leader distribution is 
I20260709 12:20:42.253389   844 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067"
dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
mode: GRACEFUL
new_leader_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:36542
I20260709 12:20:42.254097   844 raft_consensus.cc:606] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 2 LEADER]: Received request to transfer leadership to TS 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:42.260471   693 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
mode: GRACEFUL
new_leader_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:46406
I20260709 12:20:42.261075   693 raft_consensus.cc:606] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Received request to transfer leadership to TS 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:42.267086   693 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
mode: GRACEFUL
new_leader_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:46406
I20260709 12:20:42.267632   693 raft_consensus.cc:606] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Received request to transfer leadership to TS 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:42.273474   693 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
mode: GRACEFUL
new_leader_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:46406
I20260709 12:20:42.273999   693 raft_consensus.cc:606] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Received request to transfer leadership to TS 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:42.280073   844 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41"
dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
mode: GRACEFUL
new_leader_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:36542
I20260709 12:20:42.280618   844 raft_consensus.cc:606] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 3 LEADER]: Received request to transfer leadership to TS 2851d9518eb34a4c94883642f0fd37b6
I20260709 12:20:42.441653  1018 raft_consensus.cc:993] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113: : Instructing follower 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.442221  1022 raft_consensus.cc:1081] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Signalling peer 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.444017   770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:34600
I20260709 12:20:42.444437   770 raft_consensus.cc:493] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:42.444721   770 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.449381   770 raft_consensus.cc:515] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.451126   770 leader_election.cc:290] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 3 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:42.452337   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:42.452435   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:42.453157   693 raft_consensus.cc:3055] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:42.453312   844 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.453501   693 raft_consensus.cc:740] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:42.454378   693 consensus_queue.cc:260] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.455354   693 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.459386   844 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 3.
I20260709 12:20:42.459997   693 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 3.
I20260709 12:20:42.460944   736 leader_election.cc:304] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:42.461933  1020 raft_consensus.cc:2804] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:42.463112  1020 raft_consensus.cc:697] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:42.464051  1020 consensus_queue.cc:237] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.474308   589 catalog_manager.cc:5697] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 2 to 3, leader changed from be75e7f361dd41209a39df0af0d6a113 (127.0.140.65) to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:42.517318  1016 raft_consensus.cc:993] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c: : Instructing follower 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.518067  1021 raft_consensus.cc:1081] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 3 LEADER]: Signalling peer 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.520088   770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:34612
I20260709 12:20:42.520619   770 raft_consensus.cc:493] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:42.520952   770 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:42.525537   770 raft_consensus.cc:515] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.527349   770 leader_election.cc:290] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 4 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:42.528287   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:42.528445   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:42.528942   693 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:42.528942   844 raft_consensus.cc:3055] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:42.529526   844 raft_consensus.cc:740] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:42.530329   844 consensus_queue.cc:260] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.531653   844 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:42.535985   693 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 4.
I20260709 12:20:42.537338   736 leader_election.cc:304] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:42.538285  1020 raft_consensus.cc:2804] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:42.538311   844 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 4.
I20260709 12:20:42.539113  1020 raft_consensus.cc:697] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:42.540097  1020 consensus_queue.cc:237] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.548344   589 catalog_manager.cc:5697] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 3 to 4, leader changed from f40ae05748a5432199f3d7ca95a3311c (127.0.140.67) to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 4 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:42.604971  1016 raft_consensus.cc:993] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c: : Instructing follower 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.605573  1021 raft_consensus.cc:1081] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 2 LEADER]: Signalling peer 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.607254   770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:34612
I20260709 12:20:42.607775   770 raft_consensus.cc:493] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:42.608098   770 raft_consensus.cc:3060] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.611872   770 raft_consensus.cc:515] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:42.613425   770 leader_election.cc:290] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 3 election: Requested vote from peers f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293), be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195)
I20260709 12:20:42.614461   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:42.615021   693 raft_consensus.cc:3060] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.615068   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9f4e57df4d0d4f8793bd68db29de1067" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:42.615728   844 raft_consensus.cc:3055] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:42.615999   844 raft_consensus.cc:740] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:42.616880   844 consensus_queue.cc:260] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:42.617965   844 raft_consensus.cc:3060] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.619673   693 raft_consensus.cc:2468] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 3.
I20260709 12:20:42.620857   736 leader_election.cc:304] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, be75e7f361dd41209a39df0af0d6a113; no voters: 
I20260709 12:20:42.621846  1023 raft_consensus.cc:2804] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:42.622360  1023 raft_consensus.cc:697] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:42.623615  1023 consensus_queue.cc:237] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } }
I20260709 12:20:42.625105   844 raft_consensus.cc:2468] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 3.
I20260709 12:20:42.632040   589 catalog_manager.cc:5697] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 2 to 3, leader changed from f40ae05748a5432199f3d7ca95a3311c (127.0.140.67) to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:42.772011  1018 raft_consensus.cc:993] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113: : Instructing follower 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.772524  1018 raft_consensus.cc:1081] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Signalling peer 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.774278   770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:34600
I20260709 12:20:42.774690   770 raft_consensus.cc:493] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:42.774966   770 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.778988   770 raft_consensus.cc:515] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.780658   770 leader_election.cc:290] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 3 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:42.781759   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:42.781870   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:42.782320   844 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.782543   693 raft_consensus.cc:3055] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:42.783016   693 raft_consensus.cc:740] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:42.783710   693 consensus_queue.cc:260] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.784632   693 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.787029   844 raft_consensus.cc:2468] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 3.
I20260709 12:20:42.788059   736 leader_election.cc:304] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:42.788772  1023 raft_consensus.cc:2804] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:42.789287  1023 raft_consensus.cc:697] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:42.790287   693 raft_consensus.cc:2468] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 3.
I20260709 12:20:42.790233  1023 consensus_queue.cc:237] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.798375   589 catalog_manager.cc:5697] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 2 to 3, leader changed from be75e7f361dd41209a39df0af0d6a113 (127.0.140.65) to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:42.800719  1022 raft_consensus.cc:993] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113: : Instructing follower 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.801260  1018 raft_consensus.cc:1081] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Signalling peer 2851d9518eb34a4c94883642f0fd37b6 to start an election
I20260709 12:20:42.803321   770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
 from {username='slave'} at 127.0.0.1:34600
I20260709 12:20:42.803876   770 raft_consensus.cc:493] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:42.804338   770 raft_consensus.cc:3060] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.810137   770 raft_consensus.cc:515] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.811671   770 leader_election.cc:290] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 3 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:42.812758   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:42.813066   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ddb09687dc34423aadf5b8bebf5f34cb" candidate_uuid: "2851d9518eb34a4c94883642f0fd37b6" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:42.813321   693 raft_consensus.cc:3055] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:42.813709   844 raft_consensus.cc:3060] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.813858   693 raft_consensus.cc:740] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:42.814723   693 consensus_queue.cc:260] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.815728   693 raft_consensus.cc:3060] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:42.818348   844 raft_consensus.cc:2468] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 3.
I20260709 12:20:42.819624   736 leader_election.cc:304] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2851d9518eb34a4c94883642f0fd37b6, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:42.820415  1023 raft_consensus.cc:2804] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:42.820956  1023 raft_consensus.cc:697] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Becoming Leader. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:42.821159   693 raft_consensus.cc:2468] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2851d9518eb34a4c94883642f0fd37b6 in term 3.
I20260709 12:20:42.821975  1023 consensus_queue.cc:237] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:42.830649   589 catalog_manager.cc:5697] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 reported cstate change: term changed from 2 to 3, leader changed from be75e7f361dd41209a39df0af0d6a113 (127.0.140.65) to 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "2851d9518eb34a4c94883642f0fd37b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:42.889156   844 raft_consensus.cc:1275] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:42.890749  1020 consensus_queue.cc:1048] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:42.904376   693 raft_consensus.cc:1275] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:42.906818  1036 consensus_queue.cc:1048] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.973567   844 raft_consensus.cc:1275] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:42.974974  1036 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:42.985739   693 raft_consensus.cc:1275] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:42.988350  1036 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:43.049474   844 raft_consensus.cc:1275] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:43.051183  1036 consensus_queue.cc:1048] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:43.062152   693 raft_consensus.cc:1275] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:43.066859  1036 consensus_queue.cc:1048] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:43.217355   693 raft_consensus.cc:1275] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:43.218979  1036 consensus_queue.cc:1048] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:43.228933   844 raft_consensus.cc:1275] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:43.231033  1040 consensus_queue.cc:1048] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:43.302120   693 raft_consensus.cc:1275] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:43.303696  1020 consensus_queue.cc:1048] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:43.305269   844 raft_consensus.cc:1275] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Refusing update from remote peer 2851d9518eb34a4c94883642f0fd37b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:43.306540  1020 consensus_queue.cc:1048] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:45.330760   561 auto_leader_rebalancer-test.cc:328] The leader distribution is 
I20260709 12:20:45.332407   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:45.342607   770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
mode: GRACEFUL
new_leader_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:51726
I20260709 12:20:45.343348   770 raft_consensus.cc:606] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Received request to transfer leadership to TS f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:45.346565   770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
mode: GRACEFUL
new_leader_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:51726
I20260709 12:20:45.347005   770 raft_consensus.cc:606] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Received request to transfer leadership to TS f40ae05748a5432199f3d7ca95a3311c
I20260709 12:20:45.349645   770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
mode: GRACEFUL
new_leader_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:51726
I20260709 12:20:45.350188   770 raft_consensus.cc:606] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Received request to transfer leadership to TS be75e7f361dd41209a39df0af0d6a113
I20260709 12:20:45.352550   770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
mode: GRACEFUL
new_leader_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:51726
I20260709 12:20:45.353017   770 raft_consensus.cc:606] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 LEADER]: Received request to transfer leadership to TS be75e7f361dd41209a39df0af0d6a113
I20260709 12:20:45.355602   770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00"
dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
mode: GRACEFUL
new_leader_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:51726
I20260709 12:20:45.356174   770 raft_consensus.cc:606] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Received request to transfer leadership to TS be75e7f361dd41209a39df0af0d6a113
I20260709 12:20:45.357460   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:45.357959   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:45.451148  1050 raft_consensus.cc:993] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6: : Instructing follower f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:45.451717  1050 raft_consensus.cc:1081] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Signalling peer f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:45.453711   844 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca"
dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:36540
I20260709 12:20:45.454350   844 raft_consensus.cc:493] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:45.454698   844 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:45.461858   844 raft_consensus.cc:515] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.464304   844 leader_election.cc:290] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 4 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:45.465425   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:45.465605   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "29a1f6ed8add4ddeb88b7832dad139ca" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:45.466290   693 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:45.466300   770 raft_consensus.cc:3055] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:45.466948   770 raft_consensus.cc:740] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:45.467900   770 consensus_queue.cc:260] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.469027   770 raft_consensus.cc:3060] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:45.471482   693 raft_consensus.cc:2468] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 4.
I20260709 12:20:45.472666   810 leader_election.cc:304] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:45.473455  1021 raft_consensus.cc:2804] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:45.474464  1021 raft_consensus.cc:697] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 4 LEADER]: Becoming Leader. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:45.474852   770 raft_consensus.cc:2468] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 4.
I20260709 12:20:45.475509  1021 consensus_queue.cc:237] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.483243   589 catalog_manager.cc:5697] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c reported cstate change: term changed from 3 to 4, leader changed from 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66) to f40ae05748a5432199f3d7ca95a3311c (127.0.140.67). New cstate: current_term: 4 leader_uuid: "f40ae05748a5432199f3d7ca95a3311c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:45.559999  1051 raft_consensus.cc:993] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6: : Instructing follower be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:45.560674  1050 raft_consensus.cc:1081] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 LEADER]: Signalling peer be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:45.562565   693 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:46374
I20260709 12:20:45.563088   693 raft_consensus.cc:493] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:45.563351   693 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:45.568014   693 raft_consensus.cc:515] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.570103   693 leader_election.cc:290] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 5 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:45.571681   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:45.572379   770 raft_consensus.cc:3055] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:20:45.572290   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47e1e3ade8a3474da6f42ae3abe03d41" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:45.572760   770 raft_consensus.cc:740] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:45.573140   844 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:45.573632   770 consensus_queue.cc:260] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.574680   770 raft_consensus.cc:3060] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:45.577926   844 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 5.
I20260709 12:20:45.579438   661 leader_election.cc:304] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:45.580626   770 raft_consensus.cc:2468] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 5.
I20260709 12:20:45.581516  1058 raft_consensus.cc:2804] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:20:45.582904  1058 raft_consensus.cc:697] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 5 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:45.583791  1058 consensus_queue.cc:237] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.591818   589 catalog_manager.cc:5697] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 4 to 5, leader changed from 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66) to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 5 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:45.754326  1051 raft_consensus.cc:993] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6: : Instructing follower be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:45.754901  1050 raft_consensus.cc:1081] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Signalling peer be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:45.756786   693 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:46374
I20260709 12:20:45.757251   693 raft_consensus.cc:493] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:45.757539   693 raft_consensus.cc:3060] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:45.761917   693 raft_consensus.cc:515] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.764305   693 leader_election.cc:290] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 3 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:45.765457   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:45.765686   843 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef1618bc7c94ff28a5312b7bb7e5fc3" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:45.766666   770 raft_consensus.cc:3055] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:45.766806   843 raft_consensus.cc:3060] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:45.767315   770 raft_consensus.cc:740] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:45.767562  1051 raft_consensus.cc:993] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6: : Instructing follower f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:45.768051  1051 raft_consensus.cc:1081] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Signalling peer f40ae05748a5432199f3d7ca95a3311c to start an election
I20260709 12:20:45.768280   770 consensus_queue.cc:260] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.769573   770 raft_consensus.cc:3060] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:45.769950   844 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54"
dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
 from {username='slave'} at 127.0.0.1:36540
I20260709 12:20:45.770493   844 raft_consensus.cc:493] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:45.770877   844 raft_consensus.cc:3060] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:45.775905   843 raft_consensus.cc:2468] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 3.
I20260709 12:20:45.776563   770 raft_consensus.cc:2468] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 3.
I20260709 12:20:45.777693   661 leader_election.cc:304] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:45.777858   844 raft_consensus.cc:515] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.778692  1058 raft_consensus.cc:2804] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:45.779455  1058 raft_consensus.cc:697] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 3 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:45.780447  1058 consensus_queue.cc:237] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.781318   844 leader_election.cc:290] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 3 election: Requested vote from peers be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195), 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033)
I20260709 12:20:45.782501   693 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
I20260709 12:20:45.782521   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00fda6319cef4443a37dc908e5245c54" candidate_uuid: "f40ae05748a5432199f3d7ca95a3311c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:45.783553   693 raft_consensus.cc:3060] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:45.783641   770 raft_consensus.cc:3055] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:45.784132   770 raft_consensus.cc:740] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:45.785194   770 consensus_queue.cc:260] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.787050   770 raft_consensus.cc:3060] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:45.790537   693 raft_consensus.cc:2468] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 3.
I20260709 12:20:45.791985   810 leader_election.cc:304] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:45.792944  1021 raft_consensus.cc:2804] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:45.792725   589 catalog_manager.cc:5697] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 2 to 3, leader changed from 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66) to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 3 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:45.793479   770 raft_consensus.cc:2468] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f40ae05748a5432199f3d7ca95a3311c in term 3.
I20260709 12:20:45.793594  1021 raft_consensus.cc:697] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 3 LEADER]: Becoming Leader. State: Replica: f40ae05748a5432199f3d7ca95a3311c, State: Running, Role: LEADER
I20260709 12:20:45.794656  1021 consensus_queue.cc:237] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.804189   590 catalog_manager.cc:5697] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c reported cstate change: term changed from 2 to 3, leader changed from 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66) to f40ae05748a5432199f3d7ca95a3311c (127.0.140.67). New cstate: current_term: 3 leader_uuid: "f40ae05748a5432199f3d7ca95a3311c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:45.842582  1050 raft_consensus.cc:993] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6: : Instructing follower be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:45.843261  1051 raft_consensus.cc:1081] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Signalling peer be75e7f361dd41209a39df0af0d6a113 to start an election
I20260709 12:20:45.845078   693 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00"
dest_uuid: "be75e7f361dd41209a39df0af0d6a113"
 from {username='slave'} at 127.0.0.1:46374
I20260709 12:20:45.845620   693 raft_consensus.cc:493] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:45.845886   693 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:45.850294   693 raft_consensus.cc:515] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.852177   693 leader_election.cc:290] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 4 election: Requested vote from peers 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033), f40ae05748a5432199f3d7ca95a3311c (127.0.140.67:34293)
I20260709 12:20:45.853379   770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2851d9518eb34a4c94883642f0fd37b6"
I20260709 12:20:45.853552   844 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53e3310bec7842c1ad43b21aa1128e00" candidate_uuid: "be75e7f361dd41209a39df0af0d6a113" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "f40ae05748a5432199f3d7ca95a3311c"
I20260709 12:20:45.853960   844 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:45.853961   770 raft_consensus.cc:3055] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:45.854497   770 raft_consensus.cc:740] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2851d9518eb34a4c94883642f0fd37b6, State: Running, Role: LEADER
I20260709 12:20:45.855382   770 consensus_queue.cc:260] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.856493   770 raft_consensus.cc:3060] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:45.860635   844 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 4.
I20260709 12:20:45.861850   661 leader_election.cc:304] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be75e7f361dd41209a39df0af0d6a113, f40ae05748a5432199f3d7ca95a3311c; no voters: 
I20260709 12:20:45.862727   770 raft_consensus.cc:2468] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be75e7f361dd41209a39df0af0d6a113 in term 4.
I20260709 12:20:45.862703  1058 raft_consensus.cc:2804] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:45.863619  1058 raft_consensus.cc:697] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 4 LEADER]: Becoming Leader. State: Replica: be75e7f361dd41209a39df0af0d6a113, State: Running, Role: LEADER
I20260709 12:20:45.864584  1058 consensus_queue.cc:237] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } }
I20260709 12:20:45.872748   590 catalog_manager.cc:5697] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 reported cstate change: term changed from 3 to 4, leader changed from 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66) to be75e7f361dd41209a39df0af0d6a113 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "be75e7f361dd41209a39df0af0d6a113" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:45.988472   844 raft_consensus.cc:1275] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 5 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:20:45.990017  1058 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:46.003227   770 raft_consensus.cc:1275] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 5 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:20:46.004837  1059 consensus_queue.cc:1048] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:46.020694   693 raft_consensus.cc:1275] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:46.022109  1021 consensus_queue.cc:1048] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.035315   770 raft_consensus.cc:1275] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:46.036847  1021 consensus_queue.cc:1048] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.212934   844 raft_consensus.cc:1275] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:46.214551  1059 consensus_queue.cc:1048] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.227197   770 raft_consensus.cc:1275] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:46.229070  1059 consensus_queue.cc:1048] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:46.304852   693 raft_consensus.cc:1275] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:46.306563  1021 consensus_queue.cc:1048] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be75e7f361dd41209a39df0af0d6a113" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 37195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.318385   770 raft_consensus.cc:1275] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Refusing update from remote peer f40ae05748a5432199f3d7ca95a3311c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:46.320237  1052 consensus_queue.cc:1048] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:46.359381   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:46.365032   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:46.365561   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:46.370184   770 raft_consensus.cc:1275] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:46.371732  1059 consensus_queue.cc:1048] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2851d9518eb34a4c94883642f0fd37b6" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.381939   844 raft_consensus.cc:1275] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Refusing update from remote peer be75e7f361dd41209a39df0af0d6a113: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:46.383565  1078 consensus_queue.cc:1048] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40ae05748a5432199f3d7ca95a3311c" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:47.366838   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:47.373005   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:47.373668   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:48.374985   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:48.379848   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:48.380309   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:49.381479   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:49.386729   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:49.387224   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:50.388195   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:50.394433   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:50.394920   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:51.395957   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:51.402837   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:51.403352   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:52.404624   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:52.410189   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:52.410722   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:53.411834   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:53.416970   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:53.417445   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:54.418540   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:54.423717   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:54.424196   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:55.461373   561 auto_leader_rebalancer-test.cc:340] The leader distribution is 
I20260709 12:20:55.462035   561 tablet_server.cc:179] TabletServer@127.0.140.65:0 shutting down...
I20260709 12:20:55.505664   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:55.506712   561 tablet_replica.cc:333] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113: stopping tablet replica
I20260709 12:20:55.507623   561 raft_consensus.cc:2243] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.508348   561 raft_consensus.cc:2272] T 29a1f6ed8add4ddeb88b7832dad139ca P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.511178   561 tablet_replica.cc:333] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113: stopping tablet replica
I20260709 12:20:55.511755   561 raft_consensus.cc:2243] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.512269   561 raft_consensus.cc:2272] T 00fda6319cef4443a37dc908e5245c54 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.514231   561 tablet_replica.cc:333] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113: stopping tablet replica
I20260709 12:20:55.514853   561 raft_consensus.cc:2243] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.515306   561 raft_consensus.cc:2272] T d3e563cf261d49c9bb1d4ad63dc6500e P be75e7f361dd41209a39df0af0d6a113 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.517201   561 tablet_replica.cc:333] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113: stopping tablet replica
I20260709 12:20:55.517791   561 raft_consensus.cc:2243] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.518287   561 raft_consensus.cc:2272] T 9f4e57df4d0d4f8793bd68db29de1067 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.520109   561 tablet_replica.cc:333] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113: stopping tablet replica
I20260709 12:20:55.520797   561 raft_consensus.cc:2243] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:55.521715   561 raft_consensus.cc:2272] T 53e3310bec7842c1ad43b21aa1128e00 P be75e7f361dd41209a39df0af0d6a113 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.523566   561 tablet_replica.cc:333] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113: stopping tablet replica
I20260709 12:20:55.524101   561 raft_consensus.cc:2243] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:55.525013   561 raft_consensus.cc:2272] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.526898   561 tablet_replica.cc:333] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113: stopping tablet replica
I20260709 12:20:55.527448   561 raft_consensus.cc:2243] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.527881   561 raft_consensus.cc:2272] T ddb09687dc34423aadf5b8bebf5f34cb P be75e7f361dd41209a39df0af0d6a113 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.529826   561 tablet_replica.cc:333] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113: stopping tablet replica
I20260709 12:20:55.530335   561 raft_consensus.cc:2243] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.530759   561 raft_consensus.cc:2272] T 55d9233a4c174bf6ac4f1608d6444455 P be75e7f361dd41209a39df0af0d6a113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.532634   561 tablet_replica.cc:333] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113: stopping tablet replica
I20260709 12:20:55.533140   561 raft_consensus.cc:2243] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:20:55.534016   561 raft_consensus.cc:2272] T 47e1e3ade8a3474da6f42ae3abe03d41 P be75e7f361dd41209a39df0af0d6a113 [term 5 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:55.550407   810 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:20:55.650411   561 tablet_server.cc:196] TabletServer@127.0.140.65:0 shutdown complete.
W20260709 12:20:55.651472   810 consensus_peers.cc:597] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c -> Peer be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195): Couldn't send request to peer be75e7f361dd41209a39df0af0d6a113. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:37195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:55.656267   810 consensus_peers.cc:597] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c -> Peer be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195): Couldn't send request to peer be75e7f361dd41209a39df0af0d6a113. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:37195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:55.673540   736 consensus_peers.cc:597] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 -> Peer be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195): Couldn't send request to peer be75e7f361dd41209a39df0af0d6a113. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:37195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:55.676661   561 tablet_server.cc:179] TabletServer@127.0.140.66:0 shutting down...
W20260709 12:20:55.682297   810 consensus_peers.cc:597] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c -> Peer 2851d9518eb34a4c94883642f0fd37b6 (127.0.140.66:45033): Couldn't send request to peer 2851d9518eb34a4c94883642f0fd37b6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:55.708218   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:55.709088   561 tablet_replica.cc:333] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6: stopping tablet replica
I20260709 12:20:55.709832   561 raft_consensus.cc:2243] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.710443   561 raft_consensus.cc:2272] T 29a1f6ed8add4ddeb88b7832dad139ca P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.712667   561 tablet_replica.cc:333] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6: stopping tablet replica
I20260709 12:20:55.713413   561 raft_consensus.cc:2243] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.713954   561 raft_consensus.cc:2272] T 00fda6319cef4443a37dc908e5245c54 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.716029   561 tablet_replica.cc:333] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6: stopping tablet replica
I20260709 12:20:55.716969   561 raft_consensus.cc:2243] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.717484   561 raft_consensus.cc:2272] T d3e563cf261d49c9bb1d4ad63dc6500e P 2851d9518eb34a4c94883642f0fd37b6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.720124   561 tablet_replica.cc:333] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6: stopping tablet replica
I20260709 12:20:55.720924   561 raft_consensus.cc:2243] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:55.721933   561 raft_consensus.cc:2272] T 9f4e57df4d0d4f8793bd68db29de1067 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:55.722215   736 consensus_peers.cc:597] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 -> Peer be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195): Couldn't send request to peer be75e7f361dd41209a39df0af0d6a113. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:37195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:55.724403   561 tablet_replica.cc:333] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6: stopping tablet replica
I20260709 12:20:55.725013   561 raft_consensus.cc:2243] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:55.725888   561 raft_consensus.cc:2272] T ddb09687dc34423aadf5b8bebf5f34cb P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.727782   561 tablet_replica.cc:333] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6: stopping tablet replica
I20260709 12:20:55.728312   561 raft_consensus.cc:2243] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.728741   561 raft_consensus.cc:2272] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P 2851d9518eb34a4c94883642f0fd37b6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.730443   561 tablet_replica.cc:333] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6: stopping tablet replica
I20260709 12:20:55.731019   561 raft_consensus.cc:2243] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.731443   561 raft_consensus.cc:2272] T 53e3310bec7842c1ad43b21aa1128e00 P 2851d9518eb34a4c94883642f0fd37b6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.733245   561 tablet_replica.cc:333] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6: stopping tablet replica
I20260709 12:20:55.733743   561 raft_consensus.cc:2243] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.734170   561 raft_consensus.cc:2272] T 47e1e3ade8a3474da6f42ae3abe03d41 P 2851d9518eb34a4c94883642f0fd37b6 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.736107   561 tablet_replica.cc:333] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6: stopping tablet replica
I20260709 12:20:55.736640   561 raft_consensus.cc:2243] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:55.737561   561 raft_consensus.cc:2272] T 55d9233a4c174bf6ac4f1608d6444455 P 2851d9518eb34a4c94883642f0fd37b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.770275   561 tablet_server.cc:196] TabletServer@127.0.140.66:0 shutdown complete.
I20260709 12:20:55.793709   561 tablet_server.cc:179] TabletServer@127.0.140.67:0 shutting down...
I20260709 12:20:55.825279   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:55.826768   561 tablet_replica.cc:333] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c: stopping tablet replica
I20260709 12:20:55.827925   561 raft_consensus.cc:2243] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:20:55.828424   810 consensus_peers.cc:597] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c -> Peer be75e7f361dd41209a39df0af0d6a113 (127.0.140.65:37195): Couldn't send request to peer be75e7f361dd41209a39df0af0d6a113. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:37195: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:55.828967   561 raft_consensus.cc:2272] T d3e563cf261d49c9bb1d4ad63dc6500e P f40ae05748a5432199f3d7ca95a3311c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.831019   561 tablet_replica.cc:333] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c: stopping tablet replica
I20260709 12:20:55.831581   561 raft_consensus.cc:2243] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.832043   561 raft_consensus.cc:2272] T 9f4e57df4d0d4f8793bd68db29de1067 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.833904   561 tablet_replica.cc:333] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c: stopping tablet replica
I20260709 12:20:55.834444   561 raft_consensus.cc:2243] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:55.835322   561 raft_consensus.cc:2272] T 29a1f6ed8add4ddeb88b7832dad139ca P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.837291   561 tablet_replica.cc:333] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c: stopping tablet replica
I20260709 12:20:55.837872   561 raft_consensus.cc:2243] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.838376   561 raft_consensus.cc:2272] T 53e3310bec7842c1ad43b21aa1128e00 P f40ae05748a5432199f3d7ca95a3311c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.840085   561 tablet_replica.cc:333] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c: stopping tablet replica
I20260709 12:20:55.840595   561 raft_consensus.cc:2243] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.841058   561 raft_consensus.cc:2272] T 3ef1618bc7c94ff28a5312b7bb7e5fc3 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.842774   561 tablet_replica.cc:333] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c: stopping tablet replica
I20260709 12:20:55.843351   561 raft_consensus.cc:2243] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.843794   561 raft_consensus.cc:2272] T ddb09687dc34423aadf5b8bebf5f34cb P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.845438   561 tablet_replica.cc:333] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c: stopping tablet replica
I20260709 12:20:55.845919   561 raft_consensus.cc:2243] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:55.846673   561 raft_consensus.cc:2272] T 00fda6319cef4443a37dc908e5245c54 P f40ae05748a5432199f3d7ca95a3311c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.848448   561 tablet_replica.cc:333] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c: stopping tablet replica
I20260709 12:20:55.848919   561 raft_consensus.cc:2243] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.849289   561 raft_consensus.cc:2272] T 55d9233a4c174bf6ac4f1608d6444455 P f40ae05748a5432199f3d7ca95a3311c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.850911   561 tablet_replica.cc:333] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c: stopping tablet replica
I20260709 12:20:55.851352   561 raft_consensus.cc:2243] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:55.851765   561 raft_consensus.cc:2272] T 47e1e3ade8a3474da6f42ae3abe03d41 P f40ae05748a5432199f3d7ca95a3311c [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:55.900130   561 tablet_server.cc:196] TabletServer@127.0.140.67:0 shutdown complete.
I20260709 12:20:55.923547   561 master.cc:562] Master@127.0.140.126:46575 shutting down...
I20260709 12:20:56.097690   561 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:56.098362   561 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:56.098706   561 tablet_replica.cc:333] T 00000000000000000000000000000000 P b0b5c3d3ac00431bbf5e9232d0839896: stopping tablet replica
I20260709 12:20:56.120241   561 master.cc:584] Master@127.0.140.126:46575 shutdown complete.
2851d9518eb34a4c94883642f0fd37b6  4
be75e7f361dd41209a39df0af0d6a113  4
f40ae05748a5432199f3d7ca95a3311c  1
2851d9518eb34a4c94883642f0fd37b6  3
be75e7f361dd41209a39df0af0d6a113  3
f40ae05748a5432199f3d7ca95a3311c  3
2851d9518eb34a4c94883642f0fd37b6  8
be75e7f361dd41209a39df0af0d6a113  0
f40ae05748a5432199f3d7ca95a3311c  1
2851d9518eb34a4c94883642f0fd37b6  3
be75e7f361dd41209a39df0af0d6a113  3
f40ae05748a5432199f3d7ca95a3311c  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (31852 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 12:20:56.167541   561 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.140.126:46231
I20260709 12:20:56.168905   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:56.175449  1094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:56.175640  1095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:56.177000  1097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:56.178660   561 server_base.cc:1061] running on GCE node
I20260709 12:20:56.179898   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:56.180097   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:56.180300   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599656180284 us; error 0 us; skew 500 ppm
I20260709 12:20:56.180917   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:56.184178   561 webserver.cc:533] Webserver started at http://127.0.140.126:38753/ using document root <none> and password file <none>
I20260709 12:20:56.184741   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:56.184906   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:56.185281   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:56.186558   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/master-0-root/instance:
uuid: "a9dc3a7296d44a2da1c7ca62a3003025"
format_stamp: "Formatted at 2026-07-09 12:20:56 on dist-test-slave-7mcn"
I20260709 12:20:56.191748   561 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:20:56.195713  1102 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.196775   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.004s
I20260709 12:20:56.197165   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/master-0-root
uuid: "a9dc3a7296d44a2da1c7ca62a3003025"
format_stamp: "Formatted at 2026-07-09 12:20:56 on dist-test-slave-7mcn"
I20260709 12:20:56.197539   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:56.210270   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:56.211959   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:56.260185   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.126:46231
I20260709 12:20:56.260273  1153 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.126:46231 every 8 connection(s)
I20260709 12:20:56.264742  1154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.274272  1154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025: Bootstrap starting.
I20260709 12:20:56.279042  1154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:56.284116  1154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025: No bootstrap required, opened a new log
I20260709 12:20:56.286374  1154 raft_consensus.cc:359] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc3a7296d44a2da1c7ca62a3003025" member_type: VOTER }
I20260709 12:20:56.286880  1154 raft_consensus.cc:385] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:56.287150  1154 raft_consensus.cc:740] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9dc3a7296d44a2da1c7ca62a3003025, State: Initialized, Role: FOLLOWER
I20260709 12:20:56.287835  1154 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc3a7296d44a2da1c7ca62a3003025" member_type: VOTER }
I20260709 12:20:56.288322  1154 raft_consensus.cc:399] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:20:56.288570  1154 raft_consensus.cc:493] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:20:56.288837  1154 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:56.292727  1154 raft_consensus.cc:515] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc3a7296d44a2da1c7ca62a3003025" member_type: VOTER }
I20260709 12:20:56.293303  1154 leader_election.cc:304] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9dc3a7296d44a2da1c7ca62a3003025; no voters: 
I20260709 12:20:56.294582  1154 leader_election.cc:290] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:20:56.295214  1157 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:56.296892  1157 raft_consensus.cc:697] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 1 LEADER]: Becoming Leader. State: Replica: a9dc3a7296d44a2da1c7ca62a3003025, State: Running, Role: LEADER
I20260709 12:20:56.297742  1154 sys_catalog.cc:565] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:20:56.297742  1157 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc3a7296d44a2da1c7ca62a3003025" member_type: VOTER }
I20260709 12:20:56.303509  1159 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9dc3a7296d44a2da1c7ca62a3003025. Latest consensus state: current_term: 1 leader_uuid: "a9dc3a7296d44a2da1c7ca62a3003025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc3a7296d44a2da1c7ca62a3003025" member_type: VOTER } }
I20260709 12:20:56.304365  1159 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:56.304358  1158 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9dc3a7296d44a2da1c7ca62a3003025" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9dc3a7296d44a2da1c7ca62a3003025" member_type: VOTER } }
I20260709 12:20:56.305626  1158 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:56.311496  1166 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:20:56.319859  1166 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:20:56.320967   561 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:20:56.330484  1166 catalog_manager.cc:1383] Generated new cluster ID: 59821a570cf546609db574cd44339ca8
I20260709 12:20:56.330905  1166 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:20:56.352967  1166 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:20:56.354393  1166 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:20:56.372337  1166 catalog_manager.cc:6070] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025: Generated new TSK 0
I20260709 12:20:56.373190  1166 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:20:56.387965   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:56.395004  1175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:56.396654  1176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:56.398665  1178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:56.399067   561 server_base.cc:1061] running on GCE node
I20260709 12:20:56.400341   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:56.400647   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:56.400846   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599656400829 us; error 0 us; skew 500 ppm
I20260709 12:20:56.401444   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:56.404358   561 webserver.cc:533] Webserver started at http://127.0.140.65:43215/ using document root <none> and password file <none>
I20260709 12:20:56.404922   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:56.405148   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:56.405452   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:56.406740   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-0-root/instance:
uuid: "9e058a4b582f4811851ac79b25015652"
format_stamp: "Formatted at 2026-07-09 12:20:56 on dist-test-slave-7mcn"
I20260709 12:20:56.412132   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.002s	sys 0.003s
I20260709 12:20:56.416498  1183 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.417671   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:20:56.418004   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "9e058a4b582f4811851ac79b25015652"
format_stamp: "Formatted at 2026-07-09 12:20:56 on dist-test-slave-7mcn"
I20260709 12:20:56.418320   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:56.434986   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:56.436475   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:56.438141   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:56.440892   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:56.441083   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.441413   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:56.441596   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.490985   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:42805
I20260709 12:20:56.491168  1245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:42805 every 8 connection(s)
I20260709 12:20:56.496518   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:56.506374  1250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:56.506584  1251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:56.510983  1253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:56.512254   561 server_base.cc:1061] running on GCE node
I20260709 12:20:56.514287  1246 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46231
I20260709 12:20:56.514528   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:20:56.514760  1246 heartbeater.cc:461] Registering TS with master...
W20260709 12:20:56.514883   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:56.515292   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599656515267 us; error 0 us; skew 500 ppm
I20260709 12:20:56.515885   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:56.515867  1246 heartbeater.cc:507] Master 127.0.140.126:46231 requested a full tablet report, sending...
I20260709 12:20:56.518451  1119 ts_manager.cc:194] Registered new tserver with Master: 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805)
I20260709 12:20:56.519448   561 webserver.cc:533] Webserver started at http://127.0.140.66:39729/ using document root <none> and password file <none>
I20260709 12:20:56.520008   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:56.520237   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:56.520555   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:56.520752  1119 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41880
I20260709 12:20:56.522270   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-1-root/instance:
uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
format_stamp: "Formatted at 2026-07-09 12:20:56 on dist-test-slave-7mcn"
I20260709 12:20:56.527302   561 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.003s	sys 0.000s
I20260709 12:20:56.531383  1258 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.532265   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20260709 12:20:56.532596   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-1-root
uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
format_stamp: "Formatted at 2026-07-09 12:20:56 on dist-test-slave-7mcn"
I20260709 12:20:56.532918   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:56.557212   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:56.558678   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:56.560638   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:56.563182   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:56.563432   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.563678   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:56.563854   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.610890   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.66:36271
I20260709 12:20:56.611088  1320 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.66:36271 every 8 connection(s)
I20260709 12:20:56.616339   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:56.625141  1324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:56.625614  1325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:56.628795  1327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:56.629558   561 server_base.cc:1061] running on GCE node
I20260709 12:20:56.629704  1321 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46231
I20260709 12:20:56.630249  1321 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:56.630610   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:56.630894   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:56.631080   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599656631049 us; error 0 us; skew 500 ppm
I20260709 12:20:56.631115  1321 heartbeater.cc:507] Master 127.0.140.126:46231 requested a full tablet report, sending...
I20260709 12:20:56.631810   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:56.633930  1119 ts_manager.cc:194] Registered new tserver with Master: 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:20:56.635150   561 webserver.cc:533] Webserver started at http://127.0.140.67:34149/ using document root <none> and password file <none>
I20260709 12:20:56.635887   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:56.636154   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:56.636086  1119 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41894
I20260709 12:20:56.636603   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:56.638186   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-2-root/instance:
uuid: "73c8315376d94914993022aa0c732bef"
format_stamp: "Formatted at 2026-07-09 12:20:56 on dist-test-slave-7mcn"
I20260709 12:20:56.643410   561 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.001s
I20260709 12:20:56.647082  1332 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.648003   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20260709 12:20:56.648347   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-2-root
uuid: "73c8315376d94914993022aa0c732bef"
format_stamp: "Formatted at 2026-07-09 12:20:56 on dist-test-slave-7mcn"
I20260709 12:20:56.648665   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599624140507-561-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:56.665604   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:56.667080   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:56.668721   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:56.671131   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:56.671363   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.671603   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:56.671782   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:56.721011   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.67:41011
I20260709 12:20:56.721127  1394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.67:41011 every 8 connection(s)
I20260709 12:20:56.736071  1395 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46231
I20260709 12:20:56.736478  1395 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:56.737315  1395 heartbeater.cc:507] Master 127.0.140.126:46231 requested a full tablet report, sending...
I20260709 12:20:56.739598  1119 ts_manager.cc:194] Registered new tserver with Master: 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:56.740621   561 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016081355s
I20260709 12:20:56.741058  1119 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41910
I20260709 12:20:56.769903  1119 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41924:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:20:56.773409  1119 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:20:56.857146  1211 tablet_service.cc:1511] Processing CreateTablet for tablet 353ca46a3b43489e84044d61904b29dd (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:20:56.858772  1211 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 353ca46a3b43489e84044d61904b29dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.859040  1210 tablet_service.cc:1511] Processing CreateTablet for tablet f511268e183a4bf3b84194db30a596fe (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:20:56.860340  1210 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f511268e183a4bf3b84194db30a596fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.861893  1286 tablet_service.cc:1511] Processing CreateTablet for tablet 353ca46a3b43489e84044d61904b29dd (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:20:56.862673  1358 tablet_service.cc:1511] Processing CreateTablet for tablet 3f9eda1852b54150b09e1959360cca59 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:20:56.864229  1358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f9eda1852b54150b09e1959360cca59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.864045  1285 tablet_service.cc:1511] Processing CreateTablet for tablet f511268e183a4bf3b84194db30a596fe (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:20:56.866042  1284 tablet_service.cc:1511] Processing CreateTablet for tablet 3f9eda1852b54150b09e1959360cca59 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:20:56.866499  1359 tablet_service.cc:1511] Processing CreateTablet for tablet f511268e183a4bf3b84194db30a596fe (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:20:56.867998  1359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f511268e183a4bf3b84194db30a596fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.869989  1208 tablet_service.cc:1511] Processing CreateTablet for tablet a4ae46406a784fe48853b862ee551087 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:20:56.872107  1360 tablet_service.cc:1511] Processing CreateTablet for tablet 353ca46a3b43489e84044d61904b29dd (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:20:56.872612  1282 tablet_service.cc:1511] Processing CreateTablet for tablet b4f5893c97dd4d098bf8ef33f33e252b (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:20:56.857227  1209 tablet_service.cc:1511] Processing CreateTablet for tablet 3f9eda1852b54150b09e1959360cca59 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:20:56.873646  1360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 353ca46a3b43489e84044d61904b29dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.875638  1207 tablet_service.cc:1511] Processing CreateTablet for tablet b4f5893c97dd4d098bf8ef33f33e252b (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:20:56.868053  1283 tablet_service.cc:1511] Processing CreateTablet for tablet a4ae46406a784fe48853b862ee551087 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:20:56.877228  1208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4ae46406a784fe48853b862ee551087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.878171  1206 tablet_service.cc:1511] Processing CreateTablet for tablet 5a0db964250b4b9a9cead728caf6f10d (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:20:56.879462  1206 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0db964250b4b9a9cead728caf6f10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.882491  1281 tablet_service.cc:1511] Processing CreateTablet for tablet 5a0db964250b4b9a9cead728caf6f10d (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:20:56.883313  1286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 353ca46a3b43489e84044d61904b29dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.888214  1205 tablet_service.cc:1511] Processing CreateTablet for tablet c90d162778ba4ebc8b912b0c075a67a8 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:20:56.889585  1205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c90d162778ba4ebc8b912b0c075a67a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.892064  1280 tablet_service.cc:1511] Processing CreateTablet for tablet c90d162778ba4ebc8b912b0c075a67a8 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:20:56.893728  1281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0db964250b4b9a9cead728caf6f10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.896581  1357 tablet_service.cc:1511] Processing CreateTablet for tablet a4ae46406a784fe48853b862ee551087 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:20:56.898025  1357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4ae46406a784fe48853b862ee551087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.897744  1355 tablet_service.cc:1511] Processing CreateTablet for tablet 5a0db964250b4b9a9cead728caf6f10d (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:20:56.899062  1355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a0db964250b4b9a9cead728caf6f10d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.900038  1356 tablet_service.cc:1511] Processing CreateTablet for tablet b4f5893c97dd4d098bf8ef33f33e252b (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:20:56.901166  1353 tablet_service.cc:1511] Processing CreateTablet for tablet cef72d2249014732b0a6e8257a63b64f (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:20:56.902365  1279 tablet_service.cc:1511] Processing CreateTablet for tablet cef72d2249014732b0a6e8257a63b64f (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:20:56.904675  1278 tablet_service.cc:1511] Processing CreateTablet for tablet bbc02fd29b234b3c90ee57f0231e83d8 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:20:56.905778  1354 tablet_service.cc:1511] Processing CreateTablet for tablet c90d162778ba4ebc8b912b0c075a67a8 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:20:56.896584  1204 tablet_service.cc:1511] Processing CreateTablet for tablet cef72d2249014732b0a6e8257a63b64f (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:20:56.907126  1354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c90d162778ba4ebc8b912b0c075a67a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.908290  1356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f5893c97dd4d098bf8ef33f33e252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.909406  1352 tablet_service.cc:1511] Processing CreateTablet for tablet bbc02fd29b234b3c90ee57f0231e83d8 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:20:56.909772  1277 tablet_service.cc:1511] Processing CreateTablet for tablet d01c4e21ba1741f4b9ce3589503b7c1c (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:20:56.911016  1353 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef72d2249014732b0a6e8257a63b64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.915588  1351 tablet_service.cc:1511] Processing CreateTablet for tablet d01c4e21ba1741f4b9ce3589503b7c1c (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:20:56.917258  1351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d01c4e21ba1741f4b9ce3589503b7c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.919234  1352 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc02fd29b234b3c90ee57f0231e83d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.924875  1203 tablet_service.cc:1511] Processing CreateTablet for tablet bbc02fd29b234b3c90ee57f0231e83d8 (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:20:56.919234  1204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef72d2249014732b0a6e8257a63b64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.926462  1203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc02fd29b234b3c90ee57f0231e83d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.936726  1202 tablet_service.cc:1511] Processing CreateTablet for tablet d01c4e21ba1741f4b9ce3589503b7c1c (DEFAULT_TABLE table=test-workload [id=c1bc5f2357ef44bb8ef27dec3ca8d57c]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:20:56.937525  1209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f9eda1852b54150b09e1959360cca59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.939174  1207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f5893c97dd4d098bf8ef33f33e252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.943275  1202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d01c4e21ba1741f4b9ce3589503b7c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.970868  1410 tablet_bootstrap.cc:492] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
I20260709 12:20:56.991251  1283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4ae46406a784fe48853b862ee551087. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.992869  1279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef72d2249014732b0a6e8257a63b64f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.994426  1278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc02fd29b234b3c90ee57f0231e83d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.995956  1284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f9eda1852b54150b09e1959360cca59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.997422  1280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c90d162778ba4ebc8b912b0c075a67a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:56.998859  1285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f511268e183a4bf3b84194db30a596fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:57.001770  1282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4f5893c97dd4d098bf8ef33f33e252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:57.024070  1277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d01c4e21ba1741f4b9ce3589503b7c1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:57.045030  1411 tablet_bootstrap.cc:492] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.061302  1412 tablet_bootstrap.cc:492] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.060063  1410 tablet_bootstrap.cc:654] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.096101  1412 tablet_bootstrap.cc:654] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.100083  1411 tablet_bootstrap.cc:654] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.101591  1410 tablet_bootstrap.cc:492] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.102174  1410 ts_tablet_manager.cc:1403] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.137s	user 0.016s	sys 0.017s
I20260709 12:20:57.103386  1412 tablet_bootstrap.cc:492] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.103940  1412 ts_tablet_manager.cc:1403] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.043s	user 0.007s	sys 0.005s
I20260709 12:20:57.105279  1410 raft_consensus.cc:359] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.106045  1410 raft_consensus.cc:385] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.106359  1410 raft_consensus.cc:740] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.106719  1411 tablet_bootstrap.cc:492] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.106860  1412 raft_consensus.cc:359] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.107249  1411 ts_tablet_manager.cc:1403] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.063s	user 0.011s	sys 0.004s
I20260709 12:20:57.107648  1412 raft_consensus.cc:385] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.107172  1410 consensus_queue.cc:260] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.107980  1412 raft_consensus.cc:740] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.108794  1412 consensus_queue.cc:260] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.110055  1395 heartbeater.cc:499] Master 127.0.140.126:46231 was elected leader, sending a full tablet report...
I20260709 12:20:57.110347  1410 ts_tablet_manager.cc:1434] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 12:20:57.110749  1411 raft_consensus.cc:359] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.111400  1410 tablet_bootstrap.cc:492] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
I20260709 12:20:57.111447  1411 raft_consensus.cc:385] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.111833  1411 raft_consensus.cc:740] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.112836  1411 consensus_queue.cc:260] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.119072  1410 tablet_bootstrap.cc:654] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.122085  1411 ts_tablet_manager.cc:1434] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.014s	user 0.002s	sys 0.011s
I20260709 12:20:57.123075  1246 heartbeater.cc:499] Master 127.0.140.126:46231 was elected leader, sending a full tablet report...
I20260709 12:20:57.132156  1411 tablet_bootstrap.cc:492] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.141429  1412 ts_tablet_manager.cc:1434] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.037s	user 0.007s	sys 0.024s
I20260709 12:20:57.142541  1321 heartbeater.cc:499] Master 127.0.140.126:46231 was elected leader, sending a full tablet report...
I20260709 12:20:57.143705  1412 tablet_bootstrap.cc:492] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.151086  1412 tablet_bootstrap.cc:654] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.152281  1411 tablet_bootstrap.cc:654] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.160558  1417 raft_consensus.cc:493] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.161267  1417 raft_consensus.cc:515] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.161998  1411 tablet_bootstrap.cc:492] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.162475  1411 ts_tablet_manager.cc:1403] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.033s	user 0.011s	sys 0.007s
I20260709 12:20:57.164846  1411 raft_consensus.cc:359] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.165621  1411 raft_consensus.cc:385] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.165939  1411 raft_consensus.cc:740] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.161998  1410 tablet_bootstrap.cc:492] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.167281  1410 ts_tablet_manager.cc:1403] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.056s	user 0.010s	sys 0.020s
I20260709 12:20:57.170739  1410 raft_consensus.cc:359] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.171706  1410 raft_consensus.cc:385] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.171957  1417 leader_election.cc:290] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 73c8315376d94914993022aa0c732bef (127.0.140.67:41011), 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805)
I20260709 12:20:57.172096  1410 raft_consensus.cc:740] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.173343  1410 consensus_queue.cc:260] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.166702  1411 consensus_queue.cc:260] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.177682  1410 ts_tablet_manager.cc:1434] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20260709 12:20:57.178709  1410 tablet_bootstrap.cc:492] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
I20260709 12:20:57.184965  1410 tablet_bootstrap.cc:654] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.188109  1411 ts_tablet_manager.cc:1434] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.025s	user 0.003s	sys 0.003s
I20260709 12:20:57.189208  1411 tablet_bootstrap.cc:492] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.194605  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef" is_pre_election: true
W20260709 12:20:57.196547  1261 leader_election.cc:343] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c8315376d94914993022aa0c732bef (127.0.140.67:41011): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.196889  1411 tablet_bootstrap.cc:654] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.198681  1412 tablet_bootstrap.cc:492] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.199443  1412 ts_tablet_manager.cc:1403] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.057s	user 0.012s	sys 0.005s
I20260709 12:20:57.203440  1412 raft_consensus.cc:359] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.204430  1412 raft_consensus.cc:385] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.204784  1412 raft_consensus.cc:740] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.205835  1412 consensus_queue.cc:260] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.207399  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
I20260709 12:20:57.208781  1412 ts_tablet_manager.cc:1434] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.009s	user 0.000s	sys 0.007s
I20260709 12:20:57.209928  1412 tablet_bootstrap.cc:492] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.210337  1410 tablet_bootstrap.cc:492] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.211107  1410 ts_tablet_manager.cc:1403] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.002s
W20260709 12:20:57.213694  1259 leader_election.cc:343] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.214336  1259 leader_election.cc:304] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2; no voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652
I20260709 12:20:57.215341  1417 raft_consensus.cc:2749] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:57.217027  1411 tablet_bootstrap.cc:492] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.217701  1411 ts_tablet_manager.cc:1403] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.029s	user 0.006s	sys 0.014s
I20260709 12:20:57.217790  1412 tablet_bootstrap.cc:654] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.222558  1411 raft_consensus.cc:359] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.223361  1411 raft_consensus.cc:385] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.223670  1411 raft_consensus.cc:740] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.224792  1412 tablet_bootstrap.cc:492] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.224581  1411 consensus_queue.cc:260] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.225402  1412 ts_tablet_manager.cc:1403] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.016s	user 0.006s	sys 0.008s
I20260709 12:20:57.228456  1418 raft_consensus.cc:493] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.228493  1412 raft_consensus.cc:359] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.228730  1410 raft_consensus.cc:359] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.229101  1418 raft_consensus.cc:515] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.229494  1412 raft_consensus.cc:385] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.229663  1410 raft_consensus.cc:385] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.230077  1412 raft_consensus.cc:740] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.230103  1410 raft_consensus.cc:740] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.231292  1412 consensus_queue.cc:260] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.232014  1418 leader_election.cc:290] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.232805  1416 raft_consensus.cc:493] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.233347  1416 raft_consensus.cc:515] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.234253  1412 ts_tablet_manager.cc:1434] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:57.235554  1412 tablet_bootstrap.cc:492] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.242345  1411 ts_tablet_manager.cc:1434] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.024s	user 0.002s	sys 0.007s
I20260709 12:20:57.243212  1416 leader_election.cc:290] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:20:57.243610  1412 tablet_bootstrap.cc:654] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.245946  1411 tablet_bootstrap.cc:492] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.231523  1410 consensus_queue.cc:260] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.262833  1410 ts_tablet_manager.cc:1434] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.051s	user 0.007s	sys 0.001s
I20260709 12:20:57.263906  1411 tablet_bootstrap.cc:654] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.264302  1410 tablet_bootstrap.cc:492] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
I20260709 12:20:57.288949  1411 tablet_bootstrap.cc:492] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.289641  1411 ts_tablet_manager.cc:1403] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.000s
I20260709 12:20:57.293465  1411 raft_consensus.cc:359] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.294464  1411 raft_consensus.cc:385] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.294855  1411 raft_consensus.cc:740] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.295836  1411 consensus_queue.cc:260] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.300347  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" is_pre_election: true
I20260709 12:20:57.300581  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef" is_pre_election: true
W20260709 12:20:57.302438  1186 leader_election.cc:343] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c8315376d94914993022aa0c732bef (127.0.140.67:41011): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:57.303165  1187 leader_election.cc:343] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.303834  1187 leader_election.cc:304] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e058a4b582f4811851ac79b25015652; no voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef
I20260709 12:20:57.306301  1418 raft_consensus.cc:2749] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:57.308131  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" is_pre_election: true
I20260709 12:20:57.308840  1296 raft_consensus.cc:2468] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 0.
I20260709 12:20:57.311197  1412 tablet_bootstrap.cc:492] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.311386  1410 tablet_bootstrap.cc:654] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.311453  1336 leader_election.cc:304] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:20:57.311849  1412 ts_tablet_manager.cc:1403] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.076s	user 0.029s	sys 0.042s
I20260709 12:20:57.312498  1416 raft_consensus.cc:2804] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.312975  1416 raft_consensus.cc:493] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:57.313189  1167 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:57.313360  1416 raft_consensus.cc:3060] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.311336  1411 ts_tablet_manager.cc:1434] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.001s
I20260709 12:20:57.315485  1412 raft_consensus.cc:359] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.316311  1412 raft_consensus.cc:385] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.316668  1412 raft_consensus.cc:740] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.316900  1411 tablet_bootstrap.cc:492] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.317617  1412 consensus_queue.cc:260] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.320192  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
W20260709 12:20:57.322618  1333 leader_election.cc:343] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.323823  1412 ts_tablet_manager.cc:1434] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.002s
I20260709 12:20:57.324986  1412 tablet_bootstrap.cc:492] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.328123  1416 raft_consensus.cc:515] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.331405  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:57.331501  1412 tablet_bootstrap.cc:654] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.331501  1411 tablet_bootstrap.cc:654] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:57.333513  1333 leader_election.cc:343] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.335829  1410 tablet_bootstrap.cc:492] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.336529  1410 ts_tablet_manager.cc:1403] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.073s	user 0.018s	sys 0.001s
I20260709 12:20:57.339464  1416 leader_election.cc:290] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:20:57.340343  1418 raft_consensus.cc:493] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.340080  1410 raft_consensus.cc:359] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.340833  1410 raft_consensus.cc:385] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.340811  1418 raft_consensus.cc:515] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.341280  1410 raft_consensus.cc:740] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.342021  1410 consensus_queue.cc:260] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.343202  1418 leader_election.cc:290] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.344867  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f511268e183a4bf3b84194db30a596fe" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef" is_pre_election: true
I20260709 12:20:57.344866  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f511268e183a4bf3b84194db30a596fe" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" is_pre_election: true
I20260709 12:20:57.345675  1370 raft_consensus.cc:2468] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 0.
I20260709 12:20:57.345753  1411 tablet_bootstrap.cc:492] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.345753  1412 tablet_bootstrap.cc:492] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.346544  1411 ts_tablet_manager.cc:1403] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.030s	user 0.015s	sys 0.005s
I20260709 12:20:57.346603  1412 ts_tablet_manager.cc:1403] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.022s	user 0.018s	sys 0.000s
W20260709 12:20:57.346592  1187 leader_election.cc:343] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.347571  1186 leader_election.cc:304] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 4bcdfb75c762440bb4e7852ba6c37ec2
I20260709 12:20:57.348359  1418 raft_consensus.cc:2804] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.348824  1418 raft_consensus.cc:493] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.349344  1418 raft_consensus.cc:3060] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.349936  1412 raft_consensus.cc:359] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.349856  1411 raft_consensus.cc:359] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.350718  1412 raft_consensus.cc:385] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.350800  1411 raft_consensus.cc:385] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.351176  1412 raft_consensus.cc:740] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.351246  1411 raft_consensus.cc:740] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.351455  1410 ts_tablet_manager.cc:1434] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.001s
I20260709 12:20:57.352072  1411 consensus_queue.cc:260] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.352219  1412 consensus_queue.cc:260] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.349671  1416 raft_consensus.cc:493] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.353737  1416 raft_consensus.cc:515] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.356482  1410 tablet_bootstrap.cc:492] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
I20260709 12:20:57.356981  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
I20260709 12:20:57.357676  1412 ts_tablet_manager.cc:1434] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
W20260709 12:20:57.358562  1333 leader_election.cc:343] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.358664  1412 tablet_bootstrap.cc:492] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.359648  1416 leader_election.cc:290] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805)
W20260709 12:20:57.366499  1322 tablet.cc:2406] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:20:57.368163  1412 tablet_bootstrap.cc:654] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.368435  1410 tablet_bootstrap.cc:654] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.371881  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:20:57.372668  1296 raft_consensus.cc:3060] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.373626  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" is_pre_election: true
I20260709 12:20:57.374354  1295 raft_consensus.cc:2468] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 0.
I20260709 12:20:57.376133  1411 ts_tablet_manager.cc:1434] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.029s	user 0.010s	sys 0.008s
I20260709 12:20:57.377475  1411 tablet_bootstrap.cc:492] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.384341  1435 raft_consensus.cc:493] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.384415  1336 leader_election.cc:304] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 9e058a4b582f4811851ac79b25015652
I20260709 12:20:57.384989  1435 raft_consensus.cc:515] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.385982  1416 raft_consensus.cc:2804] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.386386  1416 raft_consensus.cc:493] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.386740  1416 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.387888  1411 tablet_bootstrap.cc:654] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.388417  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc02fd29b234b3c90ee57f0231e83d8" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
W20260709 12:20:57.390021  1333 leader_election.cc:343] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.391109  1435 leader_election.cc:290] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:20:57.395954  1416 raft_consensus.cc:515] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.396121  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc02fd29b234b3c90ee57f0231e83d8" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" is_pre_election: true
I20260709 12:20:57.398065  1416 leader_election.cc:290] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805)
I20260709 12:20:57.399400  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:57.401367  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:20:57.402388  1295 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.403563  1296 raft_consensus.cc:2468] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 1.
I20260709 12:20:57.407439  1410 tablet_bootstrap.cc:492] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.408610  1410 ts_tablet_manager.cc:1403] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.052s	user 0.008s	sys 0.006s
W20260709 12:20:57.404918  1333 leader_election.cc:343] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.410355  1418 raft_consensus.cc:515] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.412138  1410 raft_consensus.cc:359] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.412995  1410 raft_consensus.cc:385] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.413637  1410 raft_consensus.cc:740] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
W20260709 12:20:57.412966  1336 leader_election.cc:343] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.414484  1336 leader_election.cc:304] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73c8315376d94914993022aa0c732bef; no voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652
I20260709 12:20:57.414575  1410 consensus_queue.cc:260] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.415756  1435 raft_consensus.cc:2749] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:57.417294  1412 tablet_bootstrap.cc:492] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.418056  1412 ts_tablet_manager.cc:1403] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.060s	user 0.019s	sys 0.019s
I20260709 12:20:57.418066  1410 ts_tablet_manager.cc:1434] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:20:57.419545  1410 tablet_bootstrap.cc:492] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
I20260709 12:20:57.420522  1295 raft_consensus.cc:2468] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 1.
I20260709 12:20:57.420720  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f511268e183a4bf3b84194db30a596fe" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:20:57.421614  1411 tablet_bootstrap.cc:492] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.421535  1296 raft_consensus.cc:3060] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.423332  1336 leader_election.cc:304] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 9e058a4b582f4811851ac79b25015652
I20260709 12:20:57.423447  1412 raft_consensus.cc:359] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.424307  1412 raft_consensus.cc:385] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.424611  1412 raft_consensus.cc:740] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.424764  1336 leader_election.cc:304] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 9e058a4b582f4811851ac79b25015652
I20260709 12:20:57.425647  1435 raft_consensus.cc:2804] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.426180  1416 raft_consensus.cc:2804] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.425629  1412 consensus_queue.cc:260] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.428845  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f511268e183a4bf3b84194db30a596fe" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:57.429342  1296 raft_consensus.cc:2468] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 1.
I20260709 12:20:57.429481  1370 raft_consensus.cc:3060] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.429805  1418 leader_election.cc:290] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.432199  1412 ts_tablet_manager.cc:1434] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.014s	user 0.010s	sys 0.000s
I20260709 12:20:57.433403  1412 tablet_bootstrap.cc:492] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.437289  1410 tablet_bootstrap.cc:654] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.440765  1412 tablet_bootstrap.cc:654] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.446866  1435 raft_consensus.cc:697] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Becoming Leader. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:20:57.448271  1435 consensus_queue.cc:237] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.450771  1411 ts_tablet_manager.cc:1403] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.074s	user 0.024s	sys 0.016s
I20260709 12:20:57.455219  1411 raft_consensus.cc:359] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.456319  1411 raft_consensus.cc:385] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.456723  1411 raft_consensus.cc:740] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.457674  1411 consensus_queue.cc:260] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.458601  1410 tablet_bootstrap.cc:492] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.459506  1410 ts_tablet_manager.cc:1403] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.040s	user 0.013s	sys 0.000s
I20260709 12:20:57.460076  1370 raft_consensus.cc:2468] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 1.
I20260709 12:20:57.462812  1411 ts_tablet_manager.cc:1434] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:20:57.463881  1411 tablet_bootstrap.cc:492] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.464658  1416 raft_consensus.cc:697] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Becoming Leader. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:20:57.461885  1187 leader_election.cc:304] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:57.466094  1418 raft_consensus.cc:2804] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.466614  1418 raft_consensus.cc:697] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 1 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:20:57.467754  1418 consensus_queue.cc:237] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.465670  1416 consensus_queue.cc:237] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.470311  1417 raft_consensus.cc:493] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.470870  1417 raft_consensus.cc:515] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.473222  1411 tablet_bootstrap.cc:654] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.475715  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
I20260709 12:20:57.476492  1221 raft_consensus.cc:2468] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 0.
I20260709 12:20:57.478344  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef" is_pre_election: true
I20260709 12:20:57.480257  1259 leader_election.cc:304] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:57.479292  1119 catalog_manager.cc:5697] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c8315376d94914993022aa0c732bef (127.0.140.67). New cstate: current_term: 1 leader_uuid: "73c8315376d94914993022aa0c732bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: HEALTHY } } }
W20260709 12:20:57.482379  1261 leader_election.cc:343] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 73c8315376d94914993022aa0c732bef (127.0.140.67:41011): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.483037  1417 leader_election.cc:290] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.486084  1410 raft_consensus.cc:359] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.487303  1410 raft_consensus.cc:385] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.488502  1410 raft_consensus.cc:740] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.482728  1449 raft_consensus.cc:2804] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.489599  1449 raft_consensus.cc:493] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.490057  1449 raft_consensus.cc:3060] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.491127  1412 tablet_bootstrap.cc:492] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.491674  1412 ts_tablet_manager.cc:1403] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.059s	user 0.023s	sys 0.006s
I20260709 12:20:57.490638  1410 consensus_queue.cc:260] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.496737  1410 ts_tablet_manager.cc:1434] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.033s	user 0.005s	sys 0.002s
I20260709 12:20:57.506757  1410 tablet_bootstrap.cc:492] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
I20260709 12:20:57.510815  1118 catalog_manager.cc:5697] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:57.511432  1412 raft_consensus.cc:359] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.513156  1412 raft_consensus.cc:385] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.513481  1412 raft_consensus.cc:740] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.514465  1412 consensus_queue.cc:260] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.517273  1412 ts_tablet_manager.cc:1434] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.003s
I20260709 12:20:57.518440  1412 tablet_bootstrap.cc:492] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.521906  1411 tablet_bootstrap.cc:492] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.523512  1410 tablet_bootstrap.cc:654] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.524946  1411 ts_tablet_manager.cc:1403] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.061s	user 0.018s	sys 0.002s
I20260709 12:20:57.528812  1412 tablet_bootstrap.cc:654] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.531276  1416 raft_consensus.cc:493] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.531916  1416 raft_consensus.cc:515] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.533725  1416 leader_election.cc:290] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:20:57.535039  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d01c4e21ba1741f4b9ce3589503b7c1c" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
I20260709 12:20:57.535130  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d01c4e21ba1741f4b9ce3589503b7c1c" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" is_pre_election: true
I20260709 12:20:57.535841  1221 raft_consensus.cc:2468] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 0.
I20260709 12:20:57.536176  1410 tablet_bootstrap.cc:492] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.536755  1410 ts_tablet_manager.cc:1403] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.030s	user 0.013s	sys 0.002s
W20260709 12:20:57.536824  1336 leader_election.cc:343] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.534093  1411 raft_consensus.cc:359] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.537690  1411 raft_consensus.cc:385] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.538054  1411 raft_consensus.cc:740] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.538949  1411 consensus_queue.cc:260] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.537521  1333 leader_election.cc:304] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 4bcdfb75c762440bb4e7852ba6c37ec2
I20260709 12:20:57.530177  1449 raft_consensus.cc:515] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.540125  1410 raft_consensus.cc:359] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.541071  1410 raft_consensus.cc:385] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.541438  1411 ts_tablet_manager.cc:1434] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 12:20:57.541390  1416 raft_consensus.cc:2804] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.541560  1410 raft_consensus.cc:740] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.542084  1416 raft_consensus.cc:493] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.542481  1416 raft_consensus.cc:3060] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.542850  1411 tablet_bootstrap.cc:492] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.543115  1410 consensus_queue.cc:260] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.543541  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:57.544238  1221 raft_consensus.cc:3060] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.544479  1449 leader_election.cc:290] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.548008  1410 ts_tablet_manager.cc:1434] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260709 12:20:57.547791  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:57.549291  1410 tablet_bootstrap.cc:492] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
W20260709 12:20:57.549700  1261 leader_election.cc:343] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 73c8315376d94914993022aa0c732bef (127.0.140.67:41011): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.550266  1411 tablet_bootstrap.cc:654] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.553143  1221 raft_consensus.cc:2468] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 1.
I20260709 12:20:57.554894  1259 leader_election.cc:304] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 73c8315376d94914993022aa0c732bef
I20260709 12:20:57.556284  1449 raft_consensus.cc:2804] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.556806  1449 raft_consensus.cc:697] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Becoming Leader. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:20:57.557018  1410 tablet_bootstrap.cc:654] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.557755  1449 consensus_queue.cc:237] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.561617  1416 raft_consensus.cc:515] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.563079  1412 tablet_bootstrap.cc:492] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.563617  1412 ts_tablet_manager.cc:1403] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.045s	user 0.005s	sys 0.008s
I20260709 12:20:57.564977  1417 raft_consensus.cc:493] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.565099  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d01c4e21ba1741f4b9ce3589503b7c1c" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:57.565474  1417 raft_consensus.cc:515] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.566133  1221 raft_consensus.cc:3060] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.566663  1412 raft_consensus.cc:359] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.567390  1412 raft_consensus.cc:385] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.567692  1412 raft_consensus.cc:740] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.568183  1417 leader_election.cc:290] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.571563  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef" is_pre_election: true
I20260709 12:20:57.571619  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
I20260709 12:20:57.572538  1370 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 0.
I20260709 12:20:57.572973  1220 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 0.
I20260709 12:20:57.573853  1417 raft_consensus.cc:493] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.576138  1417 raft_consensus.cc:515] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.578707  1417 leader_election.cc:290] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.573894  1412 consensus_queue.cc:260] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.575177  1261 leader_election.cc:304] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:20:57.580947  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef" is_pre_election: true
I20260709 12:20:57.581799  1370 raft_consensus.cc:2468] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 0.
I20260709 12:20:57.583525  1261 leader_election.cc:304] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:20:57.586400  1417 raft_consensus.cc:2804] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.586944  1417 raft_consensus.cc:493] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.587394  1417 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.589957  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
I20260709 12:20:57.591045  1220 raft_consensus.cc:2468] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 0.
I20260709 12:20:57.592288  1416 leader_election.cc:290] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:20:57.593544  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d01c4e21ba1741f4b9ce3589503b7c1c" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
W20260709 12:20:57.595798  1336 leader_election.cc:343] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.597600  1449 raft_consensus.cc:2804] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.598260  1449 raft_consensus.cc:493] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.598831  1449 raft_consensus.cc:3060] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.607589  1412 ts_tablet_manager.cc:1434] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.044s	user 0.009s	sys 0.002s
I20260709 12:20:57.612389  1458 raft_consensus.cc:493] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.613217  1458 raft_consensus.cc:515] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.617405  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
W20260709 12:20:57.619380  1259 leader_election.cc:343] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:57.623435  1417 raft_consensus.cc:515] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.627216  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:57.627591  1410 tablet_bootstrap.cc:492] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.627616  1411 tablet_bootstrap.cc:492] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.627875  1220 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.628414  1221 raft_consensus.cc:2468] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 1.
I20260709 12:20:57.628722  1410 ts_tablet_manager.cc:1403] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.080s	user 0.024s	sys 0.054s
I20260709 12:20:57.629987  1333 leader_election.cc:304] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 4bcdfb75c762440bb4e7852ba6c37ec2
I20260709 12:20:57.630355  1449 raft_consensus.cc:515] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.631417  1416 raft_consensus.cc:2804] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.631942  1416 raft_consensus.cc:697] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Becoming Leader. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:20:57.632176  1410 raft_consensus.cc:359] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.632743  1449 leader_election.cc:290] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.633153  1410 raft_consensus.cc:385] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.632900  1416 consensus_queue.cc:237] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.633565  1410 raft_consensus.cc:740] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.634591  1410 consensus_queue.cc:260] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.635200  1412 tablet_bootstrap.cc:492] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.635725  1411 ts_tablet_manager.cc:1403] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.093s	user 0.035s	sys 0.010s
I20260709 12:20:57.636226  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:57.636873  1370 raft_consensus.cc:3060] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.637298  1410 ts_tablet_manager.cc:1434] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:20:57.638388  1410 tablet_bootstrap.cc:492] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
I20260709 12:20:57.638744  1411 raft_consensus.cc:359] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.639496  1411 raft_consensus.cc:385] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.639780  1411 raft_consensus.cc:740] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.642335  1220 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 1.
I20260709 12:20:57.643639  1259 leader_election.cc:304] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:57.644655  1411 consensus_queue.cc:260] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.645231  1449 raft_consensus.cc:2804] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.645669  1410 tablet_bootstrap.cc:654] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.646554  1449 raft_consensus.cc:697] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Becoming Leader. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:20:57.647210  1458 leader_election.cc:290] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.647851  1369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef" is_pre_election: true
I20260709 12:20:57.647531  1449 consensus_queue.cc:237] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.648527  1369 raft_consensus.cc:2468] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 0.
I20260709 12:20:57.650115  1261 leader_election.cc:304] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 9e058a4b582f4811851ac79b25015652
I20260709 12:20:57.651202  1115 catalog_manager.cc:5697] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:57.655079  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:57.655913  1221 raft_consensus.cc:3060] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.663910  1460 raft_consensus.cc:2804] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.664589  1460 raft_consensus.cc:493] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.665030  1412 tablet_bootstrap.cc:654] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.665086  1460 raft_consensus.cc:3060] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.670233  1411 ts_tablet_manager.cc:1434] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.034s	user 0.003s	sys 0.005s
I20260709 12:20:57.673265  1411 tablet_bootstrap.cc:492] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.680907  1418 raft_consensus.cc:493] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.681660  1418 raft_consensus.cc:515] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.683729  1411 tablet_bootstrap.cc:654] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.666808  1370 raft_consensus.cc:2468] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 1.
I20260709 12:20:57.685763  1418 leader_election.cc:290] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.690150  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" is_pre_election: true
I20260709 12:20:57.690456  1369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef" is_pre_election: true
I20260709 12:20:57.691040  1296 raft_consensus.cc:2393] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e058a4b582f4811851ac79b25015652 in current term 1: Already voted for candidate 73c8315376d94914993022aa0c732bef in this term.
I20260709 12:20:57.692822  1187 leader_election.cc:304] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9e058a4b582f4811851ac79b25015652; no voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef
I20260709 12:20:57.694514  1418 raft_consensus.cc:3060] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.695964  1261 leader_election.cc:304] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:20:57.697604  1417 leader_election.cc:290] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.703446  1221 raft_consensus.cc:2468] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 1.
I20260709 12:20:57.703979  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:57.704906  1370 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.701395  1417 raft_consensus.cc:2804] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.717931  1370 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 1.
I20260709 12:20:57.721880  1417 raft_consensus.cc:697] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Becoming Leader. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:20:57.729056  1460 raft_consensus.cc:515] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.729701  1410 tablet_bootstrap.cc:492] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.730286  1410 ts_tablet_manager.cc:1403] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.092s	user 0.032s	sys 0.043s
I20260709 12:20:57.731746  1417 consensus_queue.cc:237] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.732465  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:57.733098  1221 raft_consensus.cc:3060] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.732309  1119 catalog_manager.cc:5697] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c8315376d94914993022aa0c732bef (127.0.140.67). New cstate: current_term: 1 leader_uuid: "73c8315376d94914993022aa0c732bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:57.733274  1410 raft_consensus.cc:359] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.734016  1410 raft_consensus.cc:385] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.734295  1410 raft_consensus.cc:740] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.735272  1410 consensus_queue.cc:260] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.736984  1410 ts_tablet_manager.cc:1434] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:20:57.738160  1410 tablet_bootstrap.cc:492] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef: Bootstrap starting.
I20260709 12:20:57.740206  1412 tablet_bootstrap.cc:492] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.741588  1418 raft_consensus.cc:2749] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:57.741466  1221 raft_consensus.cc:2468] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 1.
I20260709 12:20:57.743891  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:57.744779  1410 tablet_bootstrap.cc:654] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.744743  1412 ts_tablet_manager.cc:1403] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.112s	user 0.013s	sys 0.002s
I20260709 12:20:57.745850  1259 leader_election.cc:304] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:57.744678  1370 raft_consensus.cc:3060] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.747275  1449 raft_consensus.cc:2804] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.747907  1449 raft_consensus.cc:697] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Becoming Leader. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:20:57.749151  1449 consensus_queue.cc:237] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.753062  1412 raft_consensus.cc:359] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.753867  1412 raft_consensus.cc:385] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.758965  1460 leader_election.cc:290] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.759074  1412 raft_consensus.cc:740] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.734560  1119 catalog_manager.cc:5697] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c8315376d94914993022aa0c732bef (127.0.140.67). New cstate: current_term: 1 leader_uuid: "73c8315376d94914993022aa0c732bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:57.761687  1412 consensus_queue.cc:260] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.764719  1370 raft_consensus.cc:2468] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 1.
I20260709 12:20:57.766098  1411 tablet_bootstrap.cc:492] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.766824  1411 ts_tablet_manager.cc:1403] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.094s	user 0.016s	sys 0.005s
I20260709 12:20:57.769912  1411 raft_consensus.cc:359] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.770483  1411 raft_consensus.cc:385] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.770660  1411 raft_consensus.cc:740] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.771198  1412 ts_tablet_manager.cc:1434] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.001s
I20260709 12:20:57.771533  1411 consensus_queue.cc:260] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.772235  1412 tablet_bootstrap.cc:492] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2: Bootstrap starting.
I20260709 12:20:57.776559  1460 raft_consensus.cc:493] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.777375  1460 raft_consensus.cc:515] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.779754  1412 tablet_bootstrap.cc:654] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.780920  1410 tablet_bootstrap.cc:492] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef: No bootstrap required, opened a new log
I20260709 12:20:57.780699  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc02fd29b234b3c90ee57f0231e83d8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
I20260709 12:20:57.781581  1460 leader_election.cc:290] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.781371  1221 raft_consensus.cc:2468] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 0.
I20260709 12:20:57.781375  1410 ts_tablet_manager.cc:1403] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef: Time spent bootstrapping tablet: real 0.044s	user 0.015s	sys 0.026s
I20260709 12:20:57.782724  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc02fd29b234b3c90ee57f0231e83d8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef" is_pre_election: true
I20260709 12:20:57.783524  1370 raft_consensus.cc:2468] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 0.
I20260709 12:20:57.783268  1259 leader_election.cc:304] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:57.784614  1460 raft_consensus.cc:2804] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.785076  1460 raft_consensus.cc:493] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.785459  1460 raft_consensus.cc:3060] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.785758  1410 raft_consensus.cc:359] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.786468  1410 raft_consensus.cc:385] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.786733  1410 raft_consensus.cc:740] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.787460  1411 ts_tablet_manager.cc:1434] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 12:20:57.787484  1410 consensus_queue.cc:260] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.789048  1411 tablet_bootstrap.cc:492] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652: Bootstrap starting.
I20260709 12:20:57.790102  1410 ts_tablet_manager.cc:1434] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:20:57.792897  1460 raft_consensus.cc:515] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.795804  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc02fd29b234b3c90ee57f0231e83d8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:57.796414  1221 raft_consensus.cc:3060] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.797789  1416 raft_consensus.cc:493] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:57.798275  1416 raft_consensus.cc:515] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.800505  1416 leader_election.cc:290] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:20:57.802398  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652" is_pre_election: true
I20260709 12:20:57.803056  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" is_pre_election: true
I20260709 12:20:57.803324  1220 raft_consensus.cc:2468] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 0.
I20260709 12:20:57.803876  1296 raft_consensus.cc:2468] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 0.
I20260709 12:20:57.805492  1411 tablet_bootstrap.cc:654] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:57.805441  1336 leader_election.cc:304] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:20:57.806813  1416 raft_consensus.cc:2804] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:57.806172  1118 catalog_manager.cc:5697] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:57.807889  1416 raft_consensus.cc:493] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:57.808720  1416 raft_consensus.cc:3060] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.809213  1221 raft_consensus.cc:2468] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 1.
I20260709 12:20:57.803028  1412 tablet_bootstrap.cc:492] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2: No bootstrap required, opened a new log
I20260709 12:20:57.811221  1412 ts_tablet_manager.cc:1403] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent bootstrapping tablet: real 0.039s	user 0.015s	sys 0.004s
I20260709 12:20:57.811232  1460 leader_election.cc:290] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:57.813072  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc02fd29b234b3c90ee57f0231e83d8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:57.813401  1259 leader_election.cc:304] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:57.813781  1370 raft_consensus.cc:3060] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.808916  1118 catalog_manager.cc:5697] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:57.816709  1412 raft_consensus.cc:359] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.817628  1412 raft_consensus.cc:385] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.817929  1412 raft_consensus.cc:740] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.818673  1412 consensus_queue.cc:260] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.821177  1416 raft_consensus.cc:515] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.822331  1412 ts_tablet_manager.cc:1434] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:20:57.824513  1370 raft_consensus.cc:2468] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 1.
I20260709 12:20:57.826994  1460 raft_consensus.cc:2804] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.827996  1460 raft_consensus.cc:697] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Becoming Leader. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:20:57.828994  1460 consensus_queue.cc:237] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.833957  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:57.835035  1221 raft_consensus.cc:3060] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.836011  1416 leader_election.cc:290] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:20:57.836953  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:20:57.837580  1296 raft_consensus.cc:3060] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.843264  1221 raft_consensus.cc:2468] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 1.
I20260709 12:20:57.844784  1333 leader_election.cc:304] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:57.845860  1435 raft_consensus.cc:2804] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:57.846750  1435 raft_consensus.cc:697] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Becoming Leader. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:20:57.847183  1296 raft_consensus.cc:2468] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 1.
I20260709 12:20:57.848232  1435 consensus_queue.cc:237] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:57.851836  1411 tablet_bootstrap.cc:492] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652: No bootstrap required, opened a new log
I20260709 12:20:57.852435  1411 ts_tablet_manager.cc:1403] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652: Time spent bootstrapping tablet: real 0.064s	user 0.012s	sys 0.011s
I20260709 12:20:57.854938  1411 raft_consensus.cc:359] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.855624  1411 raft_consensus.cc:385] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:57.855942  1411 raft_consensus.cc:740] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Initialized, Role: FOLLOWER
I20260709 12:20:57.856971  1411 consensus_queue.cc:260] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:57.859794  1411 ts_tablet_manager.cc:1434] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:57.859935  1119 catalog_manager.cc:5697] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:57.862995  1119 catalog_manager.cc:5697] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:57.875083  1118 catalog_manager.cc:5697] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef reported cstate change: term changed from 0 to 1, leader changed from <none> to 73c8315376d94914993022aa0c732bef (127.0.140.67). New cstate: current_term: 1 leader_uuid: "73c8315376d94914993022aa0c732bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:57.879513  1435 consensus_queue.cc:1048] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:57.884266  1418 consensus_queue.cc:1048] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:57.902190  1221 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.912161  1435 consensus_queue.cc:1048] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:57.928315  1481 consensus_queue.cc:1048] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:57.932802  1435 consensus_queue.cc:1048] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:57.948979  1460 consensus_queue.cc:1048] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:57.957248  1443 consensus_queue.cc:1048] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:57.973253  1370 raft_consensus.cc:3060] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:57.987767  1458 consensus_queue.cc:1048] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:58.006749  1370 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd"
dest_uuid: "73c8315376d94914993022aa0c732bef"
mode: GRACEFUL
new_leader_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:42512
I20260709 12:20:58.007548  1370 raft_consensus.cc:606] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Received request to transfer leadership to TS 9e058a4b582f4811851ac79b25015652
I20260709 12:20:58.017621  1370 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59"
dest_uuid: "73c8315376d94914993022aa0c732bef"
mode: GRACEFUL
new_leader_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:42512
I20260709 12:20:58.018220  1370 raft_consensus.cc:606] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Received request to transfer leadership to TS 9e058a4b582f4811851ac79b25015652
I20260709 12:20:58.037995  1295 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a4ae46406a784fe48853b862ee551087"
dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
mode: GRACEFUL
new_leader_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:40204
I20260709 12:20:58.038564  1295 raft_consensus.cc:606] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Received request to transfer leadership to TS 9e058a4b582f4811851ac79b25015652
I20260709 12:20:58.044600  1295 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b"
dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
mode: GRACEFUL
new_leader_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:40204
I20260709 12:20:58.045230  1295 raft_consensus.cc:606] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Received request to transfer leadership to TS 9e058a4b582f4811851ac79b25015652
I20260709 12:20:58.058187  1370 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f"
dest_uuid: "73c8315376d94914993022aa0c732bef"
mode: GRACEFUL
new_leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
 from {username='slave'} at 127.0.0.1:42512
I20260709 12:20:58.058694  1370 raft_consensus.cc:606] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Received request to transfer leadership to TS 4bcdfb75c762440bb4e7852ba6c37ec2
I20260709 12:20:58.141125  1435 consensus_queue.cc:1048] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:58.151993  1295 raft_consensus.cc:3060] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:58.160460  1458 consensus_queue.cc:1048] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.162669  1443 consensus_queue.cc:1048] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:58.169891  1458 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:58.175417  1417 consensus_queue.cc:1048] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.196834  1460 raft_consensus.cc:993] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2: : Instructing follower 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:20:58.197458  1449 raft_consensus.cc:1081] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Signalling peer 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:20:58.200057  1220 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a4ae46406a784fe48853b862ee551087"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:46790
I20260709 12:20:58.202553  1220 raft_consensus.cc:493] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:58.203133  1220 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.203483  1493 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.212826  1220 raft_consensus.cc:515] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.221020  1220 leader_election.cc:290] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:58.221961  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:20:58.222621  1295 raft_consensus.cc:3055] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.223021  1295 raft_consensus.cc:740] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:20:58.224071  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:58.223960  1295 consensus_queue.cc:260] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.224679  1370 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.225399  1295 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.233363  1370 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:20:58.233561  1295 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:20:58.235054  1186 leader_election.cc:304] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:58.236125  1481 raft_consensus.cc:2804] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:58.236672  1481 raft_consensus.cc:697] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:20:58.237437  1481 consensus_queue.cc:237] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.245433  1116 catalog_manager.cc:5697] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 1 to 2, leader changed from 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66) to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.309785  1493 consensus_queue.cc:1048] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:58.319496  1458 raft_consensus.cc:993] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2: : Instructing follower 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:20:58.320571  1458 raft_consensus.cc:1081] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Signalling peer 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:20:58.324936  1221 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:46790
I20260709 12:20:58.327158  1221 raft_consensus.cc:493] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:58.327702  1221 raft_consensus.cc:3060] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.328717  1449 consensus_queue.cc:1048] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.338105  1443 consensus_queue.cc:1048] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:58.337657  1221 raft_consensus.cc:515] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.345551  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:58.346163  1370 raft_consensus.cc:3060] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.347344  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:20:58.347965  1295 raft_consensus.cc:3055] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.348268  1295 raft_consensus.cc:740] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:20:58.348619  1221 leader_election.cc:290] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:58.349072  1295 consensus_queue.cc:260] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.350308  1295 raft_consensus.cc:3060] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.355683  1370 raft_consensus.cc:2468] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:20:58.358937  1186 leader_election.cc:304] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:58.359859  1479 raft_consensus.cc:2804] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:58.360303  1479 raft_consensus.cc:697] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:20:58.361167  1479 consensus_queue.cc:237] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.367802  1295 raft_consensus.cc:2468] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:20:58.373590  1435 consensus_queue.cc:1048] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:58.373788  1493 consensus_queue.cc:1048] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:58.382699  1416 raft_consensus.cc:993] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef: : Instructing follower 4bcdfb75c762440bb4e7852ba6c37ec2 to start an election
I20260709 12:20:58.383203  1416 raft_consensus.cc:1081] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Signalling peer 4bcdfb75c762440bb4e7852ba6c37ec2 to start an election
I20260709 12:20:58.385031  1296 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f"
dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:20:58.385759  1296 raft_consensus.cc:493] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:58.386219  1296 raft_consensus.cc:3060] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.392037  1493 consensus_queue.cc:1048] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:58.394048  1296 raft_consensus.cc:515] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.395545  1119 catalog_manager.cc:5697] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 1 to 2, leader changed from 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66) to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.396492  1296 leader_election.cc:290] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 2 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:58.397687  1369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:58.398142  1369 raft_consensus.cc:3055] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.398548  1369 raft_consensus.cc:740] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:20:58.399654  1369 consensus_queue.cc:260] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.400164  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:20:58.401371  1369 raft_consensus.cc:3060] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.401526  1220 raft_consensus.cc:3060] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.410521  1443 raft_consensus.cc:993] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef: : Instructing follower 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:20:58.410964  1220 raft_consensus.cc:2468] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 2.
I20260709 12:20:58.411089  1443 raft_consensus.cc:1081] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Signalling peer 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:20:58.412920  1259 leader_election.cc:304] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:58.413755  1369 raft_consensus.cc:2468] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 2.
I20260709 12:20:58.413803  1220 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:46794
I20260709 12:20:58.414237  1493 raft_consensus.cc:2804] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:58.414642  1220 raft_consensus.cc:493] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:58.415211  1220 raft_consensus.cc:3060] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.415159  1493 raft_consensus.cc:697] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 LEADER]: Becoming Leader. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:20:58.421274  1493 consensus_queue.cc:237] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.423669  1220 raft_consensus.cc:515] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.428049  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:20:58.428691  1296 raft_consensus.cc:3060] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.430019  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f9eda1852b54150b09e1959360cca59" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:58.430693  1370 raft_consensus.cc:3055] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.431113  1370 raft_consensus.cc:740] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:20:58.432135  1370 consensus_queue.cc:260] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.433449  1370 raft_consensus.cc:3060] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.434965  1220 leader_election.cc:290] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:58.437440  1296 raft_consensus.cc:2468] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:20:58.436743  1119 catalog_manager.cc:5697] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 reported cstate change: term changed from 1 to 2, leader changed from 73c8315376d94914993022aa0c732bef (127.0.140.67) to 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.439625  1187 leader_election.cc:304] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:58.440531  1479 raft_consensus.cc:2804] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:58.441089  1479 raft_consensus.cc:697] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:20:58.441910  1479 consensus_queue.cc:237] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:20:58.442540  1370 raft_consensus.cc:2468] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:20:58.449599  1119 catalog_manager.cc:5697] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 1 to 2, leader changed from 73c8315376d94914993022aa0c732bef (127.0.140.67) to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:58.454476  1435 raft_consensus.cc:993] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef: : Instructing follower 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:20:58.454977  1435 raft_consensus.cc:1081] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Signalling peer 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:20:58.456708  1220 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:46794
I20260709 12:20:58.457543  1220 raft_consensus.cc:493] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:58.458014  1220 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.463914  1220 raft_consensus.cc:515] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:58.465867  1220 leader_election.cc:290] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:20:58.467137  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:20:58.467861  1370 raft_consensus.cc:3055] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.467753  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:20:58.468261  1370 raft_consensus.cc:740] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:20:58.468518  1296 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.469341  1370 consensus_queue.cc:260] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:58.471508  1370 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.477754  1296 raft_consensus.cc:2468] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:20:58.478789  1370 raft_consensus.cc:2468] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:20:58.479126  1187 leader_election.cc:304] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:20:58.479969  1479 raft_consensus.cc:2804] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:58.480388  1479 raft_consensus.cc:697] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:20:58.481050  1479 consensus_queue.cc:237] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:20:58.487336  1119 catalog_manager.cc:5697] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 1 to 2, leader changed from 73c8315376d94914993022aa0c732bef (127.0.140.67) to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:58.635339  1296 raft_consensus.cc:1275] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.636682  1479 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.644013  1370 raft_consensus.cc:1275] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.646308  1481 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.784962  1370 raft_consensus.cc:1275] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.786468  1481 consensus_queue.cc:1048] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.795017  1296 raft_consensus.cc:1275] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.796715  1418 consensus_queue.cc:1048] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.860298  1370 raft_consensus.cc:1275] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.861812  1418 consensus_queue.cc:1048] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.870703  1296 raft_consensus.cc:1275] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.872239  1479 consensus_queue.cc:1048] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.929132  1220 raft_consensus.cc:1275] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Refusing update from remote peer 4bcdfb75c762440bb4e7852ba6c37ec2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.930966  1460 consensus_queue.cc:1048] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.938422  1370 raft_consensus.cc:1275] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Refusing update from remote peer 4bcdfb75c762440bb4e7852ba6c37ec2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.940164  1460 consensus_queue.cc:1048] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.949519  1370 raft_consensus.cc:1275] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.951110  1503 consensus_queue.cc:1048] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.961205  1296 raft_consensus.cc:1275] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.963701  1503 consensus_queue.cc:1048] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:01.103520   561 auto_leader_rebalancer-test.cc:365] The leader distribution is 
I20260709 12:21:01.104341   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:01.127401  1220 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
mode: GRACEFUL
new_leader_uuid: "73c8315376d94914993022aa0c732bef"
 from {username='slave'} at 127.0.0.1:44386
I20260709 12:21:01.127990  1220 raft_consensus.cc:606] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Received request to transfer leadership to TS 73c8315376d94914993022aa0c732bef
I20260709 12:21:01.130714  1220 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a4ae46406a784fe48853b862ee551087"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
mode: GRACEFUL
new_leader_uuid: "73c8315376d94914993022aa0c732bef"
 from {username='slave'} at 127.0.0.1:44386
I20260709 12:21:01.131289  1220 raft_consensus.cc:606] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Received request to transfer leadership to TS 73c8315376d94914993022aa0c732bef
I20260709 12:21:01.132371   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:21:01.132853   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:01.158226  1503 raft_consensus.cc:993] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652: : Instructing follower 73c8315376d94914993022aa0c732bef to start an election
I20260709 12:21:01.159293  1511 raft_consensus.cc:1081] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Signalling peer 73c8315376d94914993022aa0c732bef to start an election
I20260709 12:21:01.160457  1509 raft_consensus.cc:993] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652: : Instructing follower 73c8315376d94914993022aa0c732bef to start an election
I20260709 12:21:01.161059  1509 raft_consensus.cc:1081] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Signalling peer 73c8315376d94914993022aa0c732bef to start an election
I20260709 12:21:01.161442  1369 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a4ae46406a784fe48853b862ee551087"
dest_uuid: "73c8315376d94914993022aa0c732bef"
 from {username='slave'} at 127.0.0.1:42510
I20260709 12:21:01.162048  1369 raft_consensus.cc:493] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:01.162372  1369 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.163062  1370 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd"
dest_uuid: "73c8315376d94914993022aa0c732bef"
 from {username='slave'} at 127.0.0.1:42510
I20260709 12:21:01.163677  1370 raft_consensus.cc:493] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:01.164044  1370 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.167712  1369 raft_consensus.cc:515] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:01.168987  1370 raft_consensus.cc:515] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:21:01.170488  1369 leader_election.cc:290] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 3 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:21:01.171118  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:21:01.171840  1220 raft_consensus.cc:3055] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:01.172169  1220 raft_consensus.cc:740] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:01.172180  1296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:21:01.172860  1296 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.173307  1220 consensus_queue.cc:260] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:01.174211  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:21:01.174752  1220 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.174853  1221 raft_consensus.cc:3055] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:01.175344  1221 raft_consensus.cc:740] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:01.175499  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:21:01.176028  1295 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.176211  1221 consensus_queue.cc:260] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:21:01.177322  1221 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.182566  1296 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 3.
I20260709 12:21:01.183768  1220 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 3.
I20260709 12:21:01.183631  1295 raft_consensus.cc:2468] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 3.
I20260709 12:21:01.184142  1221 raft_consensus.cc:2468] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 3.
I20260709 12:21:01.184319  1336 leader_election.cc:304] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:21:01.173804  1370 leader_election.cc:290] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 3 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805)
I20260709 12:21:01.185585  1506 raft_consensus.cc:2804] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:01.185873  1333 leader_election.cc:304] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:21:01.187019  1506 raft_consensus.cc:697] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Becoming Leader. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:21:01.187865  1513 raft_consensus.cc:2804] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:01.188446  1513 raft_consensus.cc:697] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Becoming Leader. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:21:01.188066  1506 consensus_queue.cc:237] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:01.189190  1513 consensus_queue.cc:237] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:21:01.196806  1116 catalog_manager.cc:5697] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef reported cstate change: term changed from 2 to 3, leader changed from 9e058a4b582f4811851ac79b25015652 (127.0.140.65) to 73c8315376d94914993022aa0c732bef (127.0.140.67). New cstate: current_term: 3 leader_uuid: "73c8315376d94914993022aa0c732bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:01.213981  1116 catalog_manager.cc:5697] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef reported cstate change: term changed from 2 to 3, leader changed from 9e058a4b582f4811851ac79b25015652 (127.0.140.65) to 73c8315376d94914993022aa0c732bef (127.0.140.67). New cstate: current_term: 3 leader_uuid: "73c8315376d94914993022aa0c732bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:01.585379  1296 raft_consensus.cc:1275] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Refusing update from remote peer 73c8315376d94914993022aa0c732bef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:01.586998  1513 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:01.597783  1220 raft_consensus.cc:1275] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Refusing update from remote peer 73c8315376d94914993022aa0c732bef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:01.599649  1513 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:01.688902  1220 raft_consensus.cc:1275] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Refusing update from remote peer 73c8315376d94914993022aa0c732bef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:01.690315  1506 consensus_queue.cc:1048] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:01.699668  1296 raft_consensus.cc:1275] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Refusing update from remote peer 73c8315376d94914993022aa0c732bef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:01.701773  1506 consensus_queue.cc:1048] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:02.133963   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:02.139986   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:02.140471   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:03.141672   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:03.149683   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:03.150323   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:04.151489   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:04.157203   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:04.157753   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:05.159166   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:05.167153   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:05.167778   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:06.169051   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:06.174625   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:06.175132   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:07.176254   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:07.181715   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:07.182224   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:08.183390   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:08.188992   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:08.189580   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:09.190697   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:09.196750   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:09.197364   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:10.198609   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:10.204423   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:10.204996   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:11.244975   561 auto_leader_rebalancer-test.cc:384] The leader distribution is 
I20260709 12:21:11.250324  1370 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd"
dest_uuid: "73c8315376d94914993022aa0c732bef"
mode: GRACEFUL
new_leader_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:42512
I20260709 12:21:11.251144  1370 raft_consensus.cc:606] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Received request to transfer leadership to TS 9e058a4b582f4811851ac79b25015652
I20260709 12:21:11.262756  1370 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a4ae46406a784fe48853b862ee551087"
dest_uuid: "73c8315376d94914993022aa0c732bef"
mode: GRACEFUL
new_leader_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:42512
I20260709 12:21:11.263381  1370 raft_consensus.cc:606] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Received request to transfer leadership to TS 9e058a4b582f4811851ac79b25015652
I20260709 12:21:11.272053  1296 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d"
dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
mode: GRACEFUL
new_leader_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:40204
I20260709 12:21:11.272662  1296 raft_consensus.cc:606] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Received request to transfer leadership to TS 9e058a4b582f4811851ac79b25015652
I20260709 12:21:11.278959  1296 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8"
dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
mode: GRACEFUL
new_leader_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:40204
I20260709 12:21:11.279491  1296 raft_consensus.cc:606] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Received request to transfer leadership to TS 9e058a4b582f4811851ac79b25015652
I20260709 12:21:11.284839  1296 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f"
dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
mode: GRACEFUL
new_leader_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:40204
I20260709 12:21:11.285378  1296 raft_consensus.cc:606] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 LEADER]: Received request to transfer leadership to TS 9e058a4b582f4811851ac79b25015652
I20260709 12:21:11.373530  1538 raft_consensus.cc:993] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2: : Instructing follower 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.374222  1540 raft_consensus.cc:1081] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Signalling peer 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.375991  1220 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:46790
I20260709 12:21:11.376507  1220 raft_consensus.cc:493] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:11.376820  1220 raft_consensus.cc:3060] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:11.383178  1220 raft_consensus.cc:515] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.385031  1220 leader_election.cc:290] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:21:11.386629  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:21:11.386634  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:21:11.387590  1370 raft_consensus.cc:3060] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:11.387796  1295 raft_consensus.cc:3055] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:11.388239  1295 raft_consensus.cc:740] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:21:11.389213  1295 consensus_queue.cc:260] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.390403  1295 raft_consensus.cc:3060] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:11.393216  1370 raft_consensus.cc:2468] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:21:11.394857  1186 leader_election.cc:304] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:21:11.395993  1539 raft_consensus.cc:2804] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:11.396440  1539 raft_consensus.cc:697] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:11.397329  1295 raft_consensus.cc:2468] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:21:11.397487  1539 consensus_queue.cc:237] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.406030  1116 catalog_manager.cc:5697] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 1 to 2, leader changed from 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66) to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.437657  1537 raft_consensus.cc:993] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef: : Instructing follower 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.438144  1537 raft_consensus.cc:1081] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Signalling peer 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.439916  1220 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:46794
I20260709 12:21:11.440555  1220 raft_consensus.cc:493] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:11.440820  1220 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.446439  1220 raft_consensus.cc:515] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:21:11.448266  1220 leader_election.cc:290] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 4 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:21:11.449433  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:21:11.449414  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "353ca46a3b43489e84044d61904b29dd" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:21:11.449986  1370 raft_consensus.cc:3055] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:11.450322  1295 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.450341  1370 raft_consensus.cc:740] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:21:11.451821  1370 consensus_queue.cc:260] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:21:11.452903  1370 raft_consensus.cc:3060] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.456310  1295 raft_consensus.cc:2468] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 4.
I20260709 12:21:11.457871  1187 leader_election.cc:304] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:21:11.458623  1370 raft_consensus.cc:2468] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 4.
I20260709 12:21:11.458884  1539 raft_consensus.cc:2804] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:11.459558  1539 raft_consensus.cc:697] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 4 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:11.460562  1539 consensus_queue.cc:237] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } }
I20260709 12:21:11.469010  1116 catalog_manager.cc:5697] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 3 to 4, leader changed from 73c8315376d94914993022aa0c732bef (127.0.140.67) to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:11.515679  1541 raft_consensus.cc:993] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2: : Instructing follower 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.516269  1540 raft_consensus.cc:1081] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Signalling peer 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.518141  1220 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:46790
I20260709 12:21:11.518893  1220 raft_consensus.cc:493] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:11.519236  1220 raft_consensus.cc:3060] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:11.524974  1220 raft_consensus.cc:515] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.526903  1220 leader_election.cc:290] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:21:11.527860  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:21:11.527868  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:21:11.528693  1295 raft_consensus.cc:3055] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:11.528800  1370 raft_consensus.cc:3060] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:11.529070  1295 raft_consensus.cc:740] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:21:11.530105  1295 consensus_queue.cc:260] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.531327  1295 raft_consensus.cc:3060] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:11.534610  1370 raft_consensus.cc:2468] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:21:11.536159  1547 raft_consensus.cc:993] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef: : Instructing follower 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.535970  1186 leader_election.cc:304] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:21:11.536664  1537 raft_consensus.cc:1081] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Signalling peer 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.536974  1539 raft_consensus.cc:2804] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:11.537659  1539 raft_consensus.cc:697] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:11.538569  1220 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a4ae46406a784fe48853b862ee551087"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:46794
I20260709 12:21:11.538528  1539 consensus_queue.cc:237] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.539291  1220 raft_consensus.cc:493] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:11.539701  1220 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.540040  1295 raft_consensus.cc:2468] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 2.
I20260709 12:21:11.546917  1220 raft_consensus.cc:515] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.547384  1116 catalog_manager.cc:5697] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 1 to 2, leader changed from 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66) to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.548671  1220 leader_election.cc:290] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 4 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:21:11.549604  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:21:11.549912  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:21:11.550256  1295 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.550444  1370 raft_consensus.cc:3055] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:11.550815  1370 raft_consensus.cc:740] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:21:11.551729  1370 consensus_queue.cc:260] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.552990  1370 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.558060  1295 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 4.
I20260709 12:21:11.559454  1187 leader_election.cc:304] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:21:11.560137  1370 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 4.
I20260709 12:21:11.560240  1539 raft_consensus.cc:2804] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:11.560838  1539 raft_consensus.cc:697] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 4 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:11.561748  1539 consensus_queue.cc:237] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.569872  1116 catalog_manager.cc:5697] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 3 to 4, leader changed from 73c8315376d94914993022aa0c732bef (127.0.140.67) to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.644240  1541 raft_consensus.cc:993] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2: : Instructing follower 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.644822  1540 raft_consensus.cc:1081] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 LEADER]: Signalling peer 9e058a4b582f4811851ac79b25015652 to start an election
I20260709 12:21:11.646725  1220 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
 from {username='slave'} at 127.0.0.1:46790
I20260709 12:21:11.647385  1220 raft_consensus.cc:493] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:11.647699  1220 raft_consensus.cc:3060] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.653563  1220 raft_consensus.cc:515] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.655678  1220 leader_election.cc:290] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 3 election: Requested vote from peers 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:21:11.656762  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:21:11.656991  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef72d2249014732b0a6e8257a63b64f" candidate_uuid: "9e058a4b582f4811851ac79b25015652" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:21:11.657454  1370 raft_consensus.cc:3060] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.657485  1295 raft_consensus.cc:3055] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:11.658003  1295 raft_consensus.cc:740] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:21:11.659047  1295 consensus_queue.cc:260] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.660413  1295 raft_consensus.cc:3060] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.665691  1370 raft_consensus.cc:2468] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 3.
I20260709 12:21:11.667147  1186 leader_election.cc:304] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73c8315376d94914993022aa0c732bef, 9e058a4b582f4811851ac79b25015652; no voters: 
I20260709 12:21:11.668056  1295 raft_consensus.cc:2468] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e058a4b582f4811851ac79b25015652 in term 3.
I20260709 12:21:11.668126  1539 raft_consensus.cc:2804] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:11.668835  1539 raft_consensus.cc:697] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 3 LEADER]: Becoming Leader. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:11.669907  1539 consensus_queue.cc:237] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:11.678078  1116 catalog_manager.cc:5697] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 reported cstate change: term changed from 2 to 3, leader changed from 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66) to 9e058a4b582f4811851ac79b25015652 (127.0.140.65). New cstate: current_term: 3 leader_uuid: "9e058a4b582f4811851ac79b25015652" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:11.844360  1295 raft_consensus.cc:1275] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:11.846105  1539 consensus_queue.cc:1048] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:11.858283  1370 raft_consensus.cc:1275] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:11.863226  1555 consensus_queue.cc:1048] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:11.889441  1370 raft_consensus.cc:1275] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 4 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:11.890919  1555 consensus_queue.cc:1048] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:11.901315  1295 raft_consensus.cc:1275] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 4 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:11.902904  1539 consensus_queue.cc:1048] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:12.042254  1370 raft_consensus.cc:1275] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:12.043788  1539 consensus_queue.cc:1048] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:12.050177  1295 raft_consensus.cc:1275] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 4 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:12.052419  1509 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:12.057813  1296 raft_consensus.cc:1275] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:12.059820  1555 consensus_queue.cc:1048] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:12.068574  1370 raft_consensus.cc:1275] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 4 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:12.070463  1509 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:12.073669  1370 raft_consensus.cc:1275] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:12.075846  1561 consensus_queue.cc:1048] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.089541  1295 raft_consensus.cc:1275] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Refusing update from remote peer 9e058a4b582f4811851ac79b25015652: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:12.091922  1539 consensus_queue.cc:1048] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:14.328680   561 auto_leader_rebalancer-test.cc:399] The leader distribution is 
I20260709 12:21:14.329425   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:14.337838  1220 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
mode: GRACEFUL
new_leader_uuid: "73c8315376d94914993022aa0c732bef"
 from {username='slave'} at 127.0.0.1:44386
I20260709 12:21:14.338336  1220 raft_consensus.cc:606] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Received request to transfer leadership to TS 73c8315376d94914993022aa0c732bef
I20260709 12:21:14.341241  1220 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a4ae46406a784fe48853b862ee551087"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
mode: GRACEFUL
new_leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
 from {username='slave'} at 127.0.0.1:44386
I20260709 12:21:14.341773  1220 raft_consensus.cc:606] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 4 LEADER]: Received request to transfer leadership to TS 4bcdfb75c762440bb4e7852ba6c37ec2
I20260709 12:21:14.344342  1220 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
mode: GRACEFUL
new_leader_uuid: "73c8315376d94914993022aa0c732bef"
 from {username='slave'} at 127.0.0.1:44386
I20260709 12:21:14.344862  1220 raft_consensus.cc:606] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Received request to transfer leadership to TS 73c8315376d94914993022aa0c732bef
I20260709 12:21:14.347610  1220 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8"
dest_uuid: "9e058a4b582f4811851ac79b25015652"
mode: GRACEFUL
new_leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
 from {username='slave'} at 127.0.0.1:44386
I20260709 12:21:14.348274  1220 raft_consensus.cc:606] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Received request to transfer leadership to TS 4bcdfb75c762440bb4e7852ba6c37ec2
I20260709 12:21:14.349617   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:21:14.350167   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:14.690348  1569 raft_consensus.cc:993] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652: : Instructing follower 4bcdfb75c762440bb4e7852ba6c37ec2 to start an election
I20260709 12:21:14.692229  1555 raft_consensus.cc:1081] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 4 LEADER]: Signalling peer 4bcdfb75c762440bb4e7852ba6c37ec2 to start an election
I20260709 12:21:14.694499  1295 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a4ae46406a784fe48853b862ee551087"
dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
 from {username='slave'} at 127.0.0.1:40196
I20260709 12:21:14.695393  1295 raft_consensus.cc:493] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:14.695753  1295 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:14.703109  1295 raft_consensus.cc:515] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.707273  1295 leader_election.cc:290] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 5 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:21:14.709134  1221 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:21:14.709095  1370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4ae46406a784fe48853b862ee551087" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:21:14.710218  1221 raft_consensus.cc:3055] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:14.710387  1370 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:14.710767  1221 raft_consensus.cc:740] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:14.712162  1221 consensus_queue.cc:260] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.714104  1221 raft_consensus.cc:3060] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:14.720440  1370 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 5.
I20260709 12:21:14.722141  1261 leader_election.cc:304] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:21:14.723521  1540 raft_consensus.cc:2804] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:14.723573  1221 raft_consensus.cc:2468] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 5.
I20260709 12:21:14.724815  1540 raft_consensus.cc:697] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 5 LEADER]: Becoming Leader. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:21:14.726667  1540 consensus_queue.cc:237] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.734732  1116 catalog_manager.cc:5697] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 reported cstate change: term changed from 4 to 5, leader changed from 9e058a4b582f4811851ac79b25015652 (127.0.140.65) to 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66). New cstate: current_term: 5 leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:14.779939  1569 raft_consensus.cc:993] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652: : Instructing follower 73c8315376d94914993022aa0c732bef to start an election
I20260709 12:21:14.780527  1555 raft_consensus.cc:1081] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Signalling peer 73c8315376d94914993022aa0c732bef to start an election
I20260709 12:21:14.782431  1370 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d"
dest_uuid: "73c8315376d94914993022aa0c732bef"
 from {username='slave'} at 127.0.0.1:42510
I20260709 12:21:14.783110  1370 raft_consensus.cc:493] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:14.783406  1370 raft_consensus.cc:3060] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.788899  1370 raft_consensus.cc:515] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.791836  1370 leader_election.cc:290] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 3 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:21:14.794039  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:21:14.794121  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5a0db964250b4b9a9cead728caf6f10d" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:21:14.794889  1295 raft_consensus.cc:3060] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.794891  1220 raft_consensus.cc:3055] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:14.795441  1220 raft_consensus.cc:740] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:14.795975  1569 raft_consensus.cc:993] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652: : Instructing follower 73c8315376d94914993022aa0c732bef to start an election
I20260709 12:21:14.796761  1555 raft_consensus.cc:1081] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Signalling peer 73c8315376d94914993022aa0c732bef to start an election
I20260709 12:21:14.796267  1220 consensus_queue.cc:260] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.797863  1220 raft_consensus.cc:3060] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.798920  1369 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b"
dest_uuid: "73c8315376d94914993022aa0c732bef"
 from {username='slave'} at 127.0.0.1:42510
I20260709 12:21:14.799544  1369 raft_consensus.cc:493] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:14.799901  1369 raft_consensus.cc:3060] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.801882  1295 raft_consensus.cc:2468] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 3.
I20260709 12:21:14.803606  1336 leader_election.cc:304] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:21:14.804895  1537 raft_consensus.cc:2804] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:14.805730  1220 raft_consensus.cc:2468] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 3.
I20260709 12:21:14.806442  1537 raft_consensus.cc:697] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Becoming Leader. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:21:14.807639  1537 consensus_queue.cc:237] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.808048  1369 raft_consensus.cc:515] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.810882  1369 leader_election.cc:290] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 3 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271)
I20260709 12:21:14.811147  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:21:14.812171  1295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4f5893c97dd4d098bf8ef33f33e252b" candidate_uuid: "73c8315376d94914993022aa0c732bef" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
I20260709 12:21:14.812875  1295 raft_consensus.cc:3060] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.816466  1220 raft_consensus.cc:3055] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:14.817313  1220 raft_consensus.cc:740] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:14.818320  1220 consensus_queue.cc:260] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.819407  1555 raft_consensus.cc:993] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652: : Instructing follower 4bcdfb75c762440bb4e7852ba6c37ec2 to start an election
I20260709 12:21:14.820065  1555 raft_consensus.cc:1081] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Signalling peer 4bcdfb75c762440bb4e7852ba6c37ec2 to start an election
I20260709 12:21:14.819239  1116 catalog_manager.cc:5697] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef reported cstate change: term changed from 2 to 3, leader changed from 9e058a4b582f4811851ac79b25015652 (127.0.140.65) to 73c8315376d94914993022aa0c732bef (127.0.140.67). New cstate: current_term: 3 leader_uuid: "73c8315376d94914993022aa0c732bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.819922  1220 raft_consensus.cc:3060] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.821555  1295 raft_consensus.cc:2468] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 3.
I20260709 12:21:14.822391  1296 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8"
dest_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2"
 from {username='slave'} at 127.0.0.1:40196
I20260709 12:21:14.822973  1296 raft_consensus.cc:493] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:21:14.823328  1296 raft_consensus.cc:3060] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.823109  1336 leader_election.cc:304] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:21:14.824216  1537 raft_consensus.cc:2804] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:14.824932  1537 raft_consensus.cc:697] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Becoming Leader. State: Replica: 73c8315376d94914993022aa0c732bef, State: Running, Role: LEADER
I20260709 12:21:14.826110  1537 consensus_queue.cc:237] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.829295  1220 raft_consensus.cc:2468] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73c8315376d94914993022aa0c732bef in term 3.
I20260709 12:21:14.830708  1296 raft_consensus.cc:515] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.833249  1296 leader_election.cc:290] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 3 election: Requested vote from peers 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805), 73c8315376d94914993022aa0c732bef (127.0.140.67:41011)
I20260709 12:21:14.833930  1369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "73c8315376d94914993022aa0c732bef"
I20260709 12:21:14.834715  1369 raft_consensus.cc:3060] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.835752  1220 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c90d162778ba4ebc8b912b0c075a67a8" candidate_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9e058a4b582f4811851ac79b25015652"
I20260709 12:21:14.836386  1220 raft_consensus.cc:3055] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:14.836675  1220 raft_consensus.cc:740] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9e058a4b582f4811851ac79b25015652, State: Running, Role: LEADER
I20260709 12:21:14.837471  1220 consensus_queue.cc:260] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.838747  1220 raft_consensus.cc:3060] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.840754  1116 catalog_manager.cc:5697] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef reported cstate change: term changed from 2 to 3, leader changed from 9e058a4b582f4811851ac79b25015652 (127.0.140.65) to 73c8315376d94914993022aa0c732bef (127.0.140.67). New cstate: current_term: 3 leader_uuid: "73c8315376d94914993022aa0c732bef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.842321  1369 raft_consensus.cc:2468] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 3.
I20260709 12:21:14.843746  1261 leader_election.cc:304] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bcdfb75c762440bb4e7852ba6c37ec2, 73c8315376d94914993022aa0c732bef; no voters: 
I20260709 12:21:14.844619  1540 raft_consensus.cc:2804] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:14.845093  1540 raft_consensus.cc:697] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 LEADER]: Becoming Leader. State: Replica: 4bcdfb75c762440bb4e7852ba6c37ec2, State: Running, Role: LEADER
I20260709 12:21:14.845441  1220 raft_consensus.cc:2468] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bcdfb75c762440bb4e7852ba6c37ec2 in term 3.
I20260709 12:21:14.846227  1540 consensus_queue.cc:237] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } }
I20260709 12:21:14.855989  1116 catalog_manager.cc:5697] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 reported cstate change: term changed from 2 to 3, leader changed from 9e058a4b582f4811851ac79b25015652 (127.0.140.65) to 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:15.211570  1369 raft_consensus.cc:1275] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 5 FOLLOWER]: Refusing update from remote peer 4bcdfb75c762440bb4e7852ba6c37ec2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:15.213274  1540 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:15.226908  1220 raft_consensus.cc:1275] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 5 FOLLOWER]: Refusing update from remote peer 4bcdfb75c762440bb4e7852ba6c37ec2: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:15.228420  1221 raft_consensus.cc:1275] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Refusing update from remote peer 73c8315376d94914993022aa0c732bef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:15.228895  1583 consensus_queue.cc:1048] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:15.230250  1537 consensus_queue.cc:1048] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:15.247219  1296 raft_consensus.cc:1275] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Refusing update from remote peer 73c8315376d94914993022aa0c732bef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:15.247540  1220 raft_consensus.cc:1275] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Refusing update from remote peer 4bcdfb75c762440bb4e7852ba6c37ec2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:15.249266  1577 consensus_queue.cc:1048] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:15.250839  1583 consensus_queue.cc:1048] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:15.265739  1369 raft_consensus.cc:1275] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Refusing update from remote peer 4bcdfb75c762440bb4e7852ba6c37ec2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:15.267356  1571 consensus_queue.cc:1048] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c8315376d94914993022aa0c732bef" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:15.307240  1296 raft_consensus.cc:1275] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Refusing update from remote peer 73c8315376d94914993022aa0c732bef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:15.308801  1537 consensus_queue.cc:1048] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bcdfb75c762440bb4e7852ba6c37ec2" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:15.319787  1220 raft_consensus.cc:1275] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Refusing update from remote peer 73c8315376d94914993022aa0c732bef: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:15.321846  1577 consensus_queue.cc:1048] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e058a4b582f4811851ac79b25015652" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 42805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:15.351663   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:15.357869   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:15.358415   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:16.359691   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:16.365295   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:16.365841   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:17.367004   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:17.372606   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:17.373199   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:18.374426   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:18.379679   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:18.380137   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:19.381160   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:19.386611   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:19.387130   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:20.388098   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:20.394580   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:20.395078   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:21.396199   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:21.403235   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:21.403770   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:22.404956   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:22.412521   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:22.413249   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:23.414716   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:23.422209   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:21:23.422760   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:24.458254   561 auto_leader_rebalancer-test.cc:410] The leader distribution is 
I20260709 12:21:24.459092   561 tablet_server.cc:179] TabletServer@127.0.140.65:0 shutting down...
I20260709 12:21:24.500030   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:24.501034   561 tablet_replica.cc:333] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.501812   561 raft_consensus.cc:2243] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.502939   561 raft_consensus.cc:2272] T 353ca46a3b43489e84044d61904b29dd P 9e058a4b582f4811851ac79b25015652 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.505170   561 tablet_replica.cc:333] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.505851   561 raft_consensus.cc:2243] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.506291   561 raft_consensus.cc:2272] T b4f5893c97dd4d098bf8ef33f33e252b P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.508152   561 tablet_replica.cc:333] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.508685   561 raft_consensus.cc:2243] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.509509   561 raft_consensus.cc:2272] T 3f9eda1852b54150b09e1959360cca59 P 9e058a4b582f4811851ac79b25015652 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.511514   561 tablet_replica.cc:333] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.512104   561 raft_consensus.cc:2243] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.512581   561 raft_consensus.cc:2272] T 5a0db964250b4b9a9cead728caf6f10d P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.514433   561 tablet_replica.cc:333] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.515066   561 raft_consensus.cc:2243] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.515560   561 raft_consensus.cc:2272] T bbc02fd29b234b3c90ee57f0231e83d8 P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.517418   561 tablet_replica.cc:333] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.517959   561 raft_consensus.cc:2243] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.518384   561 raft_consensus.cc:2272] T a4ae46406a784fe48853b862ee551087 P 9e058a4b582f4811851ac79b25015652 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.520287   561 tablet_replica.cc:333] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.520825   561 raft_consensus.cc:2243] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.521235   561 raft_consensus.cc:2272] T c90d162778ba4ebc8b912b0c075a67a8 P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.523056   561 tablet_replica.cc:333] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.523610   561 raft_consensus.cc:2243] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.524462   561 raft_consensus.cc:2272] T f511268e183a4bf3b84194db30a596fe P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.526304   561 tablet_replica.cc:333] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.526744   561 raft_consensus.cc:2243] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.527173   561 raft_consensus.cc:2272] T d01c4e21ba1741f4b9ce3589503b7c1c P 9e058a4b582f4811851ac79b25015652 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.528776   561 tablet_replica.cc:333] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652: stopping tablet replica
I20260709 12:21:24.529225   561 raft_consensus.cc:2243] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.529991   561 raft_consensus.cc:2272] T cef72d2249014732b0a6e8257a63b64f P 9e058a4b582f4811851ac79b25015652 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.586259   561 tablet_server.cc:196] TabletServer@127.0.140.65:0 shutdown complete.
I20260709 12:21:24.624338   561 tablet_server.cc:179] TabletServer@127.0.140.66:0 shutting down...
I20260709 12:21:24.666647   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:24.667788   561 tablet_replica.cc:333] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.668926   561 raft_consensus.cc:2243] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.669677   561 raft_consensus.cc:2272] T d01c4e21ba1741f4b9ce3589503b7c1c P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.672364   561 tablet_replica.cc:333] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.673053   561 raft_consensus.cc:2243] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.673688   561 raft_consensus.cc:2272] T cef72d2249014732b0a6e8257a63b64f P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.676036   561 tablet_replica.cc:333] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.676717   561 raft_consensus.cc:2243] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.677840   561 raft_consensus.cc:2272] T bbc02fd29b234b3c90ee57f0231e83d8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.680080   561 tablet_replica.cc:333] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.680742   561 raft_consensus.cc:2243] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.682891   561 raft_consensus.cc:2272] T a4ae46406a784fe48853b862ee551087 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 5 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:24.685205  1336 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 5 similar messages]
I20260709 12:21:24.686403   561 tablet_replica.cc:333] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.687422   561 raft_consensus.cc:2243] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.688395   561 raft_consensus.cc:2272] T b4f5893c97dd4d098bf8ef33f33e252b P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:24.689859  1336 consensus_peers.cc:597] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef -> Peer 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271): Couldn't send request to peer 4bcdfb75c762440bb4e7852ba6c37ec2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:24.691368   561 tablet_replica.cc:333] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.691942   561 raft_consensus.cc:2243] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.693178   561 raft_consensus.cc:2272] T c90d162778ba4ebc8b912b0c075a67a8 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.695516   561 tablet_replica.cc:333] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.696017   561 raft_consensus.cc:2243] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.696487   561 raft_consensus.cc:2272] T f511268e183a4bf3b84194db30a596fe P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.698405   561 tablet_replica.cc:333] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.699119   561 raft_consensus.cc:2243] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.699898   561 raft_consensus.cc:2272] T 5a0db964250b4b9a9cead728caf6f10d P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.702056   561 tablet_replica.cc:333] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.702554   561 raft_consensus.cc:2243] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.703140   561 raft_consensus.cc:2272] T 3f9eda1852b54150b09e1959360cca59 P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.705118   561 tablet_replica.cc:333] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2: stopping tablet replica
I20260709 12:21:24.705564   561 raft_consensus.cc:2243] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.706112   561 raft_consensus.cc:2272] T 353ca46a3b43489e84044d61904b29dd P 4bcdfb75c762440bb4e7852ba6c37ec2 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:24.729660  1333 consensus_peers.cc:597] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef -> Peer 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805): Couldn't send request to peer 9e058a4b582f4811851ac79b25015652. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:24.741545   561 tablet_server.cc:196] TabletServer@127.0.140.66:0 shutdown complete.
W20260709 12:21:24.747196  1336 consensus_peers.cc:597] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef -> Peer 4bcdfb75c762440bb4e7852ba6c37ec2 (127.0.140.66:36271): Couldn't send request to peer 4bcdfb75c762440bb4e7852ba6c37ec2. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:36271: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:24.748157  1333 consensus_peers.cc:597] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef -> Peer 9e058a4b582f4811851ac79b25015652 (127.0.140.65:42805): Couldn't send request to peer 9e058a4b582f4811851ac79b25015652. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:42805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:24.770264   561 tablet_server.cc:179] TabletServer@127.0.140.67:0 shutting down...
I20260709 12:21:24.815634   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:21:24.816732   561 tablet_replica.cc:333] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.817514   561 raft_consensus.cc:2243] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.818521   561 raft_consensus.cc:2272] T 5a0db964250b4b9a9cead728caf6f10d P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.820703   561 tablet_replica.cc:333] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.821321   561 raft_consensus.cc:2243] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.822364   561 raft_consensus.cc:2272] T b4f5893c97dd4d098bf8ef33f33e252b P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.824421   561 tablet_replica.cc:333] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.825085   561 raft_consensus.cc:2243] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.825668   561 raft_consensus.cc:2272] T c90d162778ba4ebc8b912b0c075a67a8 P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.827620   561 tablet_replica.cc:333] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.828192   561 raft_consensus.cc:2243] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.828830   561 raft_consensus.cc:2272] T cef72d2249014732b0a6e8257a63b64f P 73c8315376d94914993022aa0c732bef [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.830740   561 tablet_replica.cc:333] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.831383   561 raft_consensus.cc:2243] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.832463   561 raft_consensus.cc:2272] T d01c4e21ba1741f4b9ce3589503b7c1c P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.834501   561 tablet_replica.cc:333] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.835107   561 raft_consensus.cc:2243] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.835733   561 raft_consensus.cc:2272] T f511268e183a4bf3b84194db30a596fe P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.837952   561 tablet_replica.cc:333] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.838419   561 raft_consensus.cc:2243] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.838950   561 raft_consensus.cc:2272] T a4ae46406a784fe48853b862ee551087 P 73c8315376d94914993022aa0c732bef [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.840692   561 tablet_replica.cc:333] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.841138   561 raft_consensus.cc:2243] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.841540   561 raft_consensus.cc:2272] T bbc02fd29b234b3c90ee57f0231e83d8 P 73c8315376d94914993022aa0c732bef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.843842   561 tablet_replica.cc:333] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.844349   561 raft_consensus.cc:2243] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.844903   561 raft_consensus.cc:2272] T 353ca46a3b43489e84044d61904b29dd P 73c8315376d94914993022aa0c732bef [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.847142   561 tablet_replica.cc:333] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef: stopping tablet replica
I20260709 12:21:24.847581   561 raft_consensus.cc:2243] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:24.848038   561 raft_consensus.cc:2272] T 3f9eda1852b54150b09e1959360cca59 P 73c8315376d94914993022aa0c732bef [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.881042   561 tablet_server.cc:196] TabletServer@127.0.140.67:0 shutdown complete.
I20260709 12:21:24.907171   561 master.cc:562] Master@127.0.140.126:46231 shutting down...
I20260709 12:21:24.980355   561 raft_consensus.cc:2243] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:21:24.980984   561 raft_consensus.cc:2272] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:24.981284   561 tablet_replica.cc:333] T 00000000000000000000000000000000 P a9dc3a7296d44a2da1c7ca62a3003025: stopping tablet replica
W20260709 12:21:25.948786  1150 debug-util.cc:398] Leaking SignalData structure 0x7b0800052260 after lost signal to thread 564
W20260709 12:21:26.000839   561 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1150)
W20260709 12:21:27.001469   561 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1150)
I20260709 12:21:27.636776   561 master.cc:584] Master@127.0.140.126:46231 shutdown complete.
4bcdfb75c762440bb4e7852ba6c37ec2  4
73c8315376d94914993022aa0c732bef  1
9e058a4b582f4811851ac79b25015652  5
4bcdfb75c762440bb4e7852ba6c37ec2  4
73c8315376d94914993022aa0c732bef  3
9e058a4b582f4811851ac79b25015652  3
4bcdfb75c762440bb4e7852ba6c37ec2  1
73c8315376d94914993022aa0c732bef  1
9e058a4b582f4811851ac79b25015652  8
4bcdfb75c762440bb4e7852ba6c37ec2  3
73c8315376d94914993022aa0c732bef  3
9e058a4b582f4811851ac79b25015652  4
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (31522 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 12:21:27.690353   561 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.140.126:43371
I20260709 12:21:27.691794   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:27.699064  1605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:27.700416  1606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:27.701273  1608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:27.701690   561 server_base.cc:1061] running on GCE node
I20260709 12:21:27.703187   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:27.703400   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:27.703635   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599687703609 us; error 0 us; skew 500 ppm
I20260709 12:21:27.704281   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:27.707604   561 webserver.cc:533] Webserver started at http://127.0.140.126:33701/ using document root <none> and password file <none>
I20260709 12:21:27.708225   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:27.708447   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:27.708806   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:27.710155   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/master-0-root/instance:
uuid: "4002daa3304c4fb4be053abd4780c195"
format_stamp: "Formatted at 2026-07-09 12:21:27 on dist-test-slave-7mcn"
I20260709 12:21:27.715880   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:21:27.720194  1613 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:27.721340   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:21:27.721680   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/master-0-root
uuid: "4002daa3304c4fb4be053abd4780c195"
format_stamp: "Formatted at 2026-07-09 12:21:27 on dist-test-slave-7mcn"
I20260709 12:21:27.721988   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:27.743831   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:27.745500   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:27.803403   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.126:43371
I20260709 12:21:27.803526  1664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.126:43371 every 8 connection(s)
I20260709 12:21:27.808909  1665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:27.819732  1665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Bootstrap starting.
I20260709 12:21:27.824888  1665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:27.830987  1665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: No bootstrap required, opened a new log
I20260709 12:21:27.833526  1665 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002daa3304c4fb4be053abd4780c195" member_type: VOTER }
I20260709 12:21:27.834038  1665 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:27.834265  1665 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4002daa3304c4fb4be053abd4780c195, State: Initialized, Role: FOLLOWER
I20260709 12:21:27.834971  1665 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002daa3304c4fb4be053abd4780c195" member_type: VOTER }
I20260709 12:21:27.835484  1665 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:21:27.835722  1665 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:21:27.835974  1665 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:27.840741  1665 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002daa3304c4fb4be053abd4780c195" member_type: VOTER }
I20260709 12:21:27.841320  1665 leader_election.cc:304] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4002daa3304c4fb4be053abd4780c195; no voters: 
I20260709 12:21:27.842701  1665 leader_election.cc:290] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:21:27.843456  1668 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:27.845217  1668 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 1 LEADER]: Becoming Leader. State: Replica: 4002daa3304c4fb4be053abd4780c195, State: Running, Role: LEADER
I20260709 12:21:27.846184  1668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002daa3304c4fb4be053abd4780c195" member_type: VOTER }
I20260709 12:21:27.846427  1665 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:21:27.854348  1669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4002daa3304c4fb4be053abd4780c195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002daa3304c4fb4be053abd4780c195" member_type: VOTER } }
I20260709 12:21:27.854182  1670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4002daa3304c4fb4be053abd4780c195. Latest consensus state: current_term: 1 leader_uuid: "4002daa3304c4fb4be053abd4780c195" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4002daa3304c4fb4be053abd4780c195" member_type: VOTER } }
I20260709 12:21:27.855496  1670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [sys.catalog]: This master's current role is: LEADER
I20260709 12:21:27.855479  1669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [sys.catalog]: This master's current role is: LEADER
I20260709 12:21:27.864466  1677 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:21:27.874166  1677 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:21:27.875428   561 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:21:27.885226  1677 catalog_manager.cc:1383] Generated new cluster ID: 0fee4f3ab9794763b9fc441046b17f64
I20260709 12:21:27.885613  1677 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:21:27.923637  1677 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:21:27.925006  1677 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:21:27.945034  1677 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Generated new TSK 0
I20260709 12:21:27.945972  1677 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:21:28.007551   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:28.015056  1686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:28.015980  1687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:28.018131  1689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:28.022102   561 server_base.cc:1061] running on GCE node
I20260709 12:21:28.023327   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:28.023552   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:28.023731   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599688023710 us; error 0 us; skew 500 ppm
I20260709 12:21:28.024379   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:28.027644   561 webserver.cc:533] Webserver started at http://127.0.140.65:39105/ using document root <none> and password file <none>
I20260709 12:21:28.028242   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:28.028484   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:28.028780   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:28.030057   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/instance:
uuid: "075704438992440697732181e49ccd09"
format_stamp: "Formatted at 2026-07-09 12:21:28 on dist-test-slave-7mcn"
I20260709 12:21:28.036821   561 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:21:28.041847  1694 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:28.043432   561 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20260709 12:21:28.043828   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "075704438992440697732181e49ccd09"
format_stamp: "Formatted at 2026-07-09 12:21:28 on dist-test-slave-7mcn"
I20260709 12:21:28.044195   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:28.063766   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:28.065158   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:28.067239   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:28.069965   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:28.070262   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:28.070669   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:28.071074   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:21:28.120640   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:35067
I20260709 12:21:28.120792  1756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:35067 every 8 connection(s)
I20260709 12:21:28.126376   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:28.134328  1761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:28.138597  1762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:28.139659  1764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:28.140625   561 server_base.cc:1061] running on GCE node
I20260709 12:21:28.142519   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:28.142848   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:28.143095   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599688143071 us; error 0 us; skew 500 ppm
I20260709 12:21:28.143836   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:28.147953   561 webserver.cc:533] Webserver started at http://127.0.140.66:41679/ using document root <none> and password file <none>
I20260709 12:21:28.148044  1757 heartbeater.cc:344] Connected to a master server at 127.0.140.126:43371
I20260709 12:21:28.148612  1757 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:28.148681   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:28.149115   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:28.149427   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:28.149659  1757 heartbeater.cc:507] Master 127.0.140.126:43371 requested a full tablet report, sending...
I20260709 12:21:28.150872   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/instance:
uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
format_stamp: "Formatted at 2026-07-09 12:21:28 on dist-test-slave-7mcn"
I20260709 12:21:28.152348  1630 ts_manager.cc:194] Registered new tserver with Master: 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:21:28.155169  1630 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52630
I20260709 12:21:28.158026   561 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:21:28.162671  1769 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:28.163831   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:21:28.164270   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root
uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
format_stamp: "Formatted at 2026-07-09 12:21:28 on dist-test-slave-7mcn"
I20260709 12:21:28.164625   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:28.192970   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:28.194425   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:28.196281   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:28.198992   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:28.199280   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:28.199558   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:28.199748   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:28.255028   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.66:40361
I20260709 12:21:28.255162  1831 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.66:40361 every 8 connection(s)
I20260709 12:21:28.261936   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:28.271015  1835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:28.272696  1836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:28.275398   561 server_base.cc:1061] running on GCE node
W20260709 12:21:28.276499  1839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:28.277887   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:28.278264   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:28.278527   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599688278501 us; error 0 us; skew 500 ppm
I20260709 12:21:28.278604  1832 heartbeater.cc:344] Connected to a master server at 127.0.140.126:43371
I20260709 12:21:28.279083  1832 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:28.279510   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:28.280359  1832 heartbeater.cc:507] Master 127.0.140.126:43371 requested a full tablet report, sending...
I20260709 12:21:28.282984   561 webserver.cc:533] Webserver started at http://127.0.140.67:34027/ using document root <none> and password file <none>
I20260709 12:21:28.283304  1630 ts_manager.cc:194] Registered new tserver with Master: 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:28.283732   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:28.284044   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:28.284411   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:28.285326  1630 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52636
I20260709 12:21:28.286170   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/instance:
uuid: "a20e99e125b84513a44e88860667ed61"
format_stamp: "Formatted at 2026-07-09 12:21:28 on dist-test-slave-7mcn"
I20260709 12:21:28.291797   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20260709 12:21:28.296159  1843 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:28.297302   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:21:28.297624   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root
uuid: "a20e99e125b84513a44e88860667ed61"
format_stamp: "Formatted at 2026-07-09 12:21:28 on dist-test-slave-7mcn"
I20260709 12:21:28.298214   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:28.322012   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:28.323526   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:28.325623   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:28.328833   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:28.329120   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:28.329427   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:28.329690   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:28.381596   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.67:35761
I20260709 12:21:28.381795  1905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.67:35761 every 8 connection(s)
I20260709 12:21:28.404896  1906 heartbeater.cc:344] Connected to a master server at 127.0.140.126:43371
I20260709 12:21:28.405426  1906 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:28.406421  1906 heartbeater.cc:507] Master 127.0.140.126:43371 requested a full tablet report, sending...
I20260709 12:21:28.409135  1630 ts_manager.cc:194] Registered new tserver with Master: a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:28.409906   561 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.024626201s
I20260709 12:21:28.411131  1630 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52644
I20260709 12:21:28.444440  1630 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52658:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:21:28.452821  1630 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:21:28.672688  1871 tablet_service.cc:1511] Processing CreateTablet for tablet 9585ba97eaf14e069dcd2dcaf726a09b (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:21:28.674206  1871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9585ba97eaf14e069dcd2dcaf726a09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.677374  1722 tablet_service.cc:1511] Processing CreateTablet for tablet 9585ba97eaf14e069dcd2dcaf726a09b (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:21:28.677556  1721 tablet_service.cc:1511] Processing CreateTablet for tablet 65799458c3fb44e7a162099338c0bb62 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:21:28.679288  1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9585ba97eaf14e069dcd2dcaf726a09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.685148  1868 tablet_service.cc:1511] Processing CreateTablet for tablet 76f6bc3c4cce45d7855e6b3429341da2 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:21:28.681147  1720 tablet_service.cc:1511] Processing CreateTablet for tablet d891ed30ca204d09b7b083088037b63d (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:21:28.686846  1868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f6bc3c4cce45d7855e6b3429341da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.684001  1719 tablet_service.cc:1511] Processing CreateTablet for tablet 76f6bc3c4cce45d7855e6b3429341da2 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:21:28.684340  1869 tablet_service.cc:1511] Processing CreateTablet for tablet d891ed30ca204d09b7b083088037b63d (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:21:28.688752  1869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d891ed30ca204d09b7b083088037b63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.681063  1870 tablet_service.cc:1511] Processing CreateTablet for tablet 65799458c3fb44e7a162099338c0bb62 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:21:28.691695  1870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65799458c3fb44e7a162099338c0bb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.697209  1867 tablet_service.cc:1511] Processing CreateTablet for tablet 12cfc1bb936b41fbbc9cdfafc0cd9470 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:21:28.686892  1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d891ed30ca204d09b7b083088037b63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.699515  1867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12cfc1bb936b41fbbc9cdfafc0cd9470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.703167  1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f6bc3c4cce45d7855e6b3429341da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.707268  1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65799458c3fb44e7a162099338c0bb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.709899  1866 tablet_service.cc:1511] Processing CreateTablet for tablet 034c167cd4e24138be60353036ce8092 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:21:28.710237  1718 tablet_service.cc:1511] Processing CreateTablet for tablet 12cfc1bb936b41fbbc9cdfafc0cd9470 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:21:28.713181  1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12cfc1bb936b41fbbc9cdfafc0cd9470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.713951  1717 tablet_service.cc:1511] Processing CreateTablet for tablet 034c167cd4e24138be60353036ce8092 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:21:28.718371  1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034c167cd4e24138be60353036ce8092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.716606  1866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034c167cd4e24138be60353036ce8092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.729789  1865 tablet_service.cc:1511] Processing CreateTablet for tablet 908089442ca44109875257bfbc9d06c2 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:21:28.731009  1716 tablet_service.cc:1511] Processing CreateTablet for tablet 908089442ca44109875257bfbc9d06c2 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:21:28.732154  1865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908089442ca44109875257bfbc9d06c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.732494  1716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908089442ca44109875257bfbc9d06c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.736558  1715 tablet_service.cc:1511] Processing CreateTablet for tablet 9875eb35aaed48d7b2670b5ea57e3948 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:21:28.737980  1715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9875eb35aaed48d7b2670b5ea57e3948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.745411  1714 tablet_service.cc:1511] Processing CreateTablet for tablet bd91edf20ef54a30a2f5af0c0b682350 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:21:28.748631  1713 tablet_service.cc:1511] Processing CreateTablet for tablet b3034677490f4f0fac2a4ed3e2a8142b (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:21:28.751596  1714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd91edf20ef54a30a2f5af0c0b682350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.760445  1713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3034677490f4f0fac2a4ed3e2a8142b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.777711  1864 tablet_service.cc:1511] Processing CreateTablet for tablet 9875eb35aaed48d7b2670b5ea57e3948 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:21:28.779306  1864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9875eb35aaed48d7b2670b5ea57e3948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.789516  1863 tablet_service.cc:1511] Processing CreateTablet for tablet bd91edf20ef54a30a2f5af0c0b682350 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:21:28.790382  1862 tablet_service.cc:1511] Processing CreateTablet for tablet b3034677490f4f0fac2a4ed3e2a8142b (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:21:28.791249  1863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd91edf20ef54a30a2f5af0c0b682350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.792001  1862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3034677490f4f0fac2a4ed3e2a8142b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.808869  1864 tablet_service.cc:1511] Processing CreateTablet for tablet efbb8c187a4e410aa09e81ceb9ebecc3 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:21:28.854682  1864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efbb8c187a4e410aa09e81ceb9ebecc3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:21:28.861708  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:28.878723  1868 tablet_service.cc:1511] Processing CreateTablet for tablet bc103cf2bf8545cf8f5f5f34e38ad0e1 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:21:28.884549  1797 tablet_service.cc:1511] Processing CreateTablet for tablet 9585ba97eaf14e069dcd2dcaf726a09b (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:21:28.887593  1797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9585ba97eaf14e069dcd2dcaf726a09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.888410  1795 tablet_service.cc:1511] Processing CreateTablet for tablet d891ed30ca204d09b7b083088037b63d (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:21:28.890012  1795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d891ed30ca204d09b7b083088037b63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.885845  1796 tablet_service.cc:1511] Processing CreateTablet for tablet 65799458c3fb44e7a162099338c0bb62 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:21:28.890908  1865 tablet_service.cc:1511] Processing CreateTablet for tablet a4c45f93428e4cc68f85f17ea3ec3ad0 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:21:28.892118  1796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65799458c3fb44e7a162099338c0bb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.896705  1863 tablet_service.cc:1511] Processing CreateTablet for tablet c072b820694347cabf7868a1b3805f76 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:21:28.898262  1863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c072b820694347cabf7868a1b3805f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.902900  1794 tablet_service.cc:1511] Processing CreateTablet for tablet 76f6bc3c4cce45d7855e6b3429341da2 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:21:28.903776  1793 tablet_service.cc:1511] Processing CreateTablet for tablet 12cfc1bb936b41fbbc9cdfafc0cd9470 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:21:28.904613  1794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f6bc3c4cce45d7855e6b3429341da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.904524  1792 tablet_service.cc:1511] Processing CreateTablet for tablet 034c167cd4e24138be60353036ce8092 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:21:28.917994  1789 tablet_service.cc:1511] Processing CreateTablet for tablet bd91edf20ef54a30a2f5af0c0b682350 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:21:28.919281  1791 tablet_service.cc:1511] Processing CreateTablet for tablet 908089442ca44109875257bfbc9d06c2 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:21:28.920091  1788 tablet_service.cc:1511] Processing CreateTablet for tablet b3034677490f4f0fac2a4ed3e2a8142b (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:21:28.928593  1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd91edf20ef54a30a2f5af0c0b682350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.929672  1714 tablet_service.cc:1511] Processing CreateTablet for tablet a4c45f93428e4cc68f85f17ea3ec3ad0 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:21:28.921300  1924 tablet_bootstrap.cc:492] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:28.921144  1790 tablet_service.cc:1511] Processing CreateTablet for tablet 9875eb35aaed48d7b2670b5ea57e3948 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:21:28.914683  1868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc103cf2bf8545cf8f5f5f34e38ad0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.932791  1791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908089442ca44109875257bfbc9d06c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.917824  1721 tablet_service.cc:1511] Processing CreateTablet for tablet efbb8c187a4e410aa09e81ceb9ebecc3 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:21:28.931033  1714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c45f93428e4cc68f85f17ea3ec3ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.937290  1790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9875eb35aaed48d7b2670b5ea57e3948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.939133  1719 tablet_service.cc:1511] Processing CreateTablet for tablet bc103cf2bf8545cf8f5f5f34e38ad0e1 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:21:28.944746  1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034c167cd4e24138be60353036ce8092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.945436  1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12cfc1bb936b41fbbc9cdfafc0cd9470. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.945968  1720 tablet_service.cc:1511] Processing CreateTablet for tablet 1b33fb206f534e1e890a4546d3d43e84 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:21:28.947172  1788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3034677490f4f0fac2a4ed3e2a8142b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.948230  1871 tablet_service.cc:1511] Processing CreateTablet for tablet 8bb813e4fab14a61b810a24b56899586 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:21:28.926213  1866 tablet_service.cc:1511] Processing CreateTablet for tablet 1b33fb206f534e1e890a4546d3d43e84 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:21:28.951159  1862 tablet_service.cc:1511] Processing CreateTablet for tablet 62f288182b21420f8ef28e25fc10cd70 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:21:28.952836  1870 tablet_service.cc:1511] Processing CreateTablet for tablet 3e3f88cbac3d4fa4bdb3a23367a0e160 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:21:28.953852  1867 tablet_service.cc:1511] Processing CreateTablet for tablet c22ef2bc934c4b7ea57aad5853ab6122 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:21:28.959321  1862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f288182b21420f8ef28e25fc10cd70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.960798  1871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb813e4fab14a61b810a24b56899586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:28.976473  1716 tablet_service.cc:1511] Processing CreateTablet for tablet 8bb813e4fab14a61b810a24b56899586 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:21:28.977326  1924 tablet_bootstrap.cc:654] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:28.979851  1869 tablet_service.cc:1511] Processing CreateTablet for tablet a534e4794d9548638a2781e4dc451bc6 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:21:28.983842  1715 tablet_service.cc:1511] Processing CreateTablet for tablet c072b820694347cabf7868a1b3805f76 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:21:28.999199  1867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22ef2bc934c4b7ea57aad5853ab6122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.000303  1870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e3f88cbac3d4fa4bdb3a23367a0e160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.003916  1865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c45f93428e4cc68f85f17ea3ec3ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.007303  1869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a534e4794d9548638a2781e4dc451bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.009433  1925 tablet_bootstrap.cc:492] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:29.011030  1713 tablet_service.cc:1511] Processing CreateTablet for tablet c22ef2bc934c4b7ea57aad5853ab6122 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:21:29.018947  1722 tablet_service.cc:1511] Processing CreateTablet for tablet 3e3f88cbac3d4fa4bdb3a23367a0e160 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:21:29.012374  1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efbb8c187a4e410aa09e81ceb9ebecc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.019291  1717 tablet_service.cc:1511] Processing CreateTablet for tablet a534e4794d9548638a2781e4dc451bc6 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:21:29.020831  1716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb813e4fab14a61b810a24b56899586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.014750  1718 tablet_service.cc:1511] Processing CreateTablet for tablet 62f288182b21420f8ef28e25fc10cd70 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:21:29.021521  1713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22ef2bc934c4b7ea57aad5853ab6122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.032476  1866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b33fb206f534e1e890a4546d3d43e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.051857  1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc103cf2bf8545cf8f5f5f34e38ad0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.055847  1795 tablet_service.cc:1511] Processing CreateTablet for tablet efbb8c187a4e410aa09e81ceb9ebecc3 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:21:29.061553  1795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efbb8c187a4e410aa09e81ceb9ebecc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.065587  1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e3f88cbac3d4fa4bdb3a23367a0e160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.066807  1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a534e4794d9548638a2781e4dc451bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.083622  1789 tablet_service.cc:1511] Processing CreateTablet for tablet a4c45f93428e4cc68f85f17ea3ec3ad0 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:21:29.085222  1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c45f93428e4cc68f85f17ea3ec3ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.109441  1927 tablet_bootstrap.cc:492] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:29.104372  1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f288182b21420f8ef28e25fc10cd70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.115563  1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b33fb206f534e1e890a4546d3d43e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.123216  1715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c072b820694347cabf7868a1b3805f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.125502  1796 tablet_service.cc:1511] Processing CreateTablet for tablet c072b820694347cabf7868a1b3805f76 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:21:29.136855  1796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c072b820694347cabf7868a1b3805f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.137814  1924 tablet_bootstrap.cc:492] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:29.138407  1924 ts_tablet_manager.cc:1403] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.218s	user 0.012s	sys 0.014s
I20260709 12:21:29.143934  1721 tablet_service.cc:1511] Processing CreateTablet for tablet 8f285ce2457a47ebbd05920b42b35fba (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:21:29.145498  1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f285ce2457a47ebbd05920b42b35fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.145738  1924 raft_consensus.cc:359] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:29.151621  1797 tablet_service.cc:1511] Processing CreateTablet for tablet bc103cf2bf8545cf8f5f5f34e38ad0e1 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:21:29.153169  1797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc103cf2bf8545cf8f5f5f34e38ad0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.155856  1864 tablet_service.cc:1511] Processing CreateTablet for tablet 513580259d0a4e6d96fb16084d5a90dc (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:21:29.157341  1864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513580259d0a4e6d96fb16084d5a90dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.158766  1757 heartbeater.cc:499] Master 127.0.140.126:43371 was elected leader, sending a full tablet report...
I20260709 12:21:29.160436  1794 tablet_service.cc:1511] Processing CreateTablet for tablet 8bb813e4fab14a61b810a24b56899586 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:21:29.146577  1924 raft_consensus.cc:385] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:29.175194  1925 tablet_bootstrap.cc:654] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:29.166970  1924 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:29.163663  1788 tablet_service.cc:1511] Processing CreateTablet for tablet 1b33fb206f534e1e890a4546d3d43e84 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:21:29.180613  1927 tablet_bootstrap.cc:654] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:29.195988  1863 tablet_service.cc:1511] Processing CreateTablet for tablet 36f1f37b7d544bb29cd9cace84f6e946 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:21:29.197284  1863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f1f37b7d544bb29cd9cace84f6e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.200583  1924 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:29.194613  1791 tablet_service.cc:1511] Processing CreateTablet for tablet c22ef2bc934c4b7ea57aad5853ab6122 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:21:29.206023  1714 tablet_service.cc:1511] Processing CreateTablet for tablet 513580259d0a4e6d96fb16084d5a90dc (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:21:29.247670  1714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513580259d0a4e6d96fb16084d5a90dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.213899  1794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb813e4fab14a61b810a24b56899586. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.231169  1906 heartbeater.cc:499] Master 127.0.140.126:43371 was elected leader, sending a full tablet report...
I20260709 12:21:29.251662  1791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22ef2bc934c4b7ea57aad5853ab6122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.254977  1867 tablet_service.cc:1511] Processing CreateTablet for tablet 8f285ce2457a47ebbd05920b42b35fba (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:21:29.256556  1867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f285ce2457a47ebbd05920b42b35fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.257929  1788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1b33fb206f534e1e890a4546d3d43e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.244994  1792 tablet_service.cc:1511] Processing CreateTablet for tablet 3e3f88cbac3d4fa4bdb3a23367a0e160 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:21:29.262640  1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e3f88cbac3d4fa4bdb3a23367a0e160. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.264739  1869 tablet_service.cc:1511] Processing CreateTablet for tablet aca584e2f1114809bcadeaf6faad34f8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:21:29.266450  1869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aca584e2f1114809bcadeaf6faad34f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.245672  1924 ts_tablet_manager.cc:1434] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.107s	user 0.006s	sys 0.006s
I20260709 12:21:29.268034  1924 tablet_bootstrap.cc:492] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:29.267832  1795 tablet_service.cc:1511] Processing CreateTablet for tablet 513580259d0a4e6d96fb16084d5a90dc (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:21:29.269256  1795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513580259d0a4e6d96fb16084d5a90dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.281960  1924 tablet_bootstrap.cc:654] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:29.284636  1871 tablet_service.cc:1511] Processing CreateTablet for tablet bc4fe4ebd84a44b9bee2535ccd62ba22 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:21:29.286139  1871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc4fe4ebd84a44b9bee2535ccd62ba22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.250429  1790 tablet_service.cc:1511] Processing CreateTablet for tablet a534e4794d9548638a2781e4dc451bc6 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:21:29.286973  1789 tablet_service.cc:1511] Processing CreateTablet for tablet 36f1f37b7d544bb29cd9cace84f6e946 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:21:29.287896  1790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a534e4794d9548638a2781e4dc451bc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.288656  1832 heartbeater.cc:499] Master 127.0.140.126:43371 was elected leader, sending a full tablet report...
I20260709 12:21:29.291337  1722 tablet_service.cc:1511] Processing CreateTablet for tablet 5280e62b42a141359d53bd4f241659ef (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:21:29.288640  1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f1f37b7d544bb29cd9cace84f6e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.299993  1719 tablet_service.cc:1511] Processing CreateTablet for tablet 36f1f37b7d544bb29cd9cace84f6e946 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:21:29.303013  1865 tablet_service.cc:1511] Processing CreateTablet for tablet 8c92163822f64868a647b688599f49bd (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:21:29.304623  1865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c92163822f64868a647b688599f49bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.307627  1793 tablet_service.cc:1511] Processing CreateTablet for tablet 62f288182b21420f8ef28e25fc10cd70 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:21:29.318147  1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f288182b21420f8ef28e25fc10cd70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.317795  1870 tablet_service.cc:1511] Processing CreateTablet for tablet 6f86194bda894372ab64d2c6a9937a5b (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:21:29.319180  1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5280e62b42a141359d53bd4f241659ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.319500  1870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f86194bda894372ab64d2c6a9937a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.322028  1868 tablet_service.cc:1511] Processing CreateTablet for tablet 5280e62b42a141359d53bd4f241659ef (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:21:29.323658  1868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5280e62b42a141359d53bd4f241659ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.324172  1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f1f37b7d544bb29cd9cace84f6e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.332634  1866 tablet_service.cc:1511] Processing CreateTablet for tablet ea5d8f652fc448d6af846ff6850004b8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:21:29.334235  1866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea5d8f652fc448d6af846ff6850004b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.336259  1716 tablet_service.cc:1511] Processing CreateTablet for tablet ea5d8f652fc448d6af846ff6850004b8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:21:29.337323  1718 tablet_service.cc:1511] Processing CreateTablet for tablet aca584e2f1114809bcadeaf6faad34f8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:21:29.355412  1863 tablet_service.cc:1511] Processing CreateTablet for tablet 80ed6fcb806a4023b94ecd7b266b421e (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:21:29.364245  1863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80ed6fcb806a4023b94ecd7b266b421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.368772  1862 tablet_service.cc:1511] Processing CreateTablet for tablet 9b8cae5d30dd4a68a4fba5846e83873e (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:21:29.370493  1862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b8cae5d30dd4a68a4fba5846e83873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.374110  1869 tablet_service.cc:1511] Processing CreateTablet for tablet f15e08b3d9804a00b9836e5223042266 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:21:29.376498  1925 tablet_bootstrap.cc:492] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:29.377337  1927 tablet_bootstrap.cc:492] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:29.394685  1927 ts_tablet_manager.cc:1403] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.286s	user 0.010s	sys 0.010s
I20260709 12:21:29.378377  1791 tablet_service.cc:1511] Processing CreateTablet for tablet 8f285ce2457a47ebbd05920b42b35fba (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
W20260709 12:21:29.390278  1907 tablet.cc:2406] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:21:29.391968  1720 tablet_service.cc:1511] Processing CreateTablet for tablet bc4fe4ebd84a44b9bee2535ccd62ba22 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:21:29.398584  1927 raft_consensus.cc:359] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.400645  1927 raft_consensus.cc:385] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:29.401180  1927 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:29.401646  1925 ts_tablet_manager.cc:1403] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.393s	user 0.005s	sys 0.019s
I20260709 12:21:29.403108  1927 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.398626  1791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f285ce2457a47ebbd05920b42b35fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.409557  1715 tablet_service.cc:1511] Processing CreateTablet for tablet 8c92163822f64868a647b688599f49bd (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:21:29.412156  1925 raft_consensus.cc:359] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.413233  1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aca584e2f1114809bcadeaf6faad34f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.409848  1717 tablet_service.cc:1511] Processing CreateTablet for tablet 6f86194bda894372ab64d2c6a9937a5b (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:21:29.416396  1869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f15e08b3d9804a00b9836e5223042266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.421766  1792 tablet_service.cc:1511] Processing CreateTablet for tablet 5280e62b42a141359d53bd4f241659ef (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:21:29.416766  1925 raft_consensus.cc:385] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:29.425406  1928 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.459978  1925 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:29.460155  1928 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:29.460896  1925 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.423144  1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc4fe4ebd84a44b9bee2535ccd62ba22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.457916  1721 tablet_service.cc:1511] Processing CreateTablet for tablet 9b8cae5d30dd4a68a4fba5846e83873e (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:21:29.443135  1924 tablet_bootstrap.cc:492] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:29.440516  1713 tablet_service.cc:1511] Processing CreateTablet for tablet 80ed6fcb806a4023b94ecd7b266b421e (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:21:29.450770  1927 ts_tablet_manager.cc:1434] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.056s	user 0.008s	sys 0.001s
I20260709 12:21:29.422652  1864 tablet_service.cc:1511] Processing CreateTablet for tablet bbbcc16cebc244769e7f1a1b6e29594c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:21:29.454730  1796 tablet_service.cc:1511] Processing CreateTablet for tablet ea5d8f652fc448d6af846ff6850004b8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:21:29.428589  1793 tablet_service.cc:1511] Processing CreateTablet for tablet aca584e2f1114809bcadeaf6faad34f8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:21:29.439002  1867 tablet_service.cc:1511] Processing CreateTablet for tablet b1cb2a27e8444930b93f1e21c2deb19c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:21:29.460865  1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5280e62b42a141359d53bd4f241659ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.471763  1928 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:29.478734  1924 ts_tablet_manager.cc:1403] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.211s	user 0.005s	sys 0.025s
I20260709 12:21:29.478803  1715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c92163822f64868a647b688599f49bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.511227  1933 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.516283  1933 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.471792  1790 tablet_service.cc:1511] Processing CreateTablet for tablet 8c92163822f64868a647b688599f49bd (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:21:29.520475  1927 tablet_bootstrap.cc:492] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:29.524807  1796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea5d8f652fc448d6af846ff6850004b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.524832  1797 tablet_service.cc:1511] Processing CreateTablet for tablet 6f86194bda894372ab64d2c6a9937a5b (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:21:29.526000  1790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c92163822f64868a647b688599f49bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.526206  1933 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:29.526603  1797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f86194bda894372ab64d2c6a9937a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.520496  1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f86194bda894372ab64d2c6a9937a5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.529088  1864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbbcc16cebc244769e7f1a1b6e29594c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.497443  1794 tablet_service.cc:1511] Processing CreateTablet for tablet bc4fe4ebd84a44b9bee2535ccd62ba22 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:21:29.534725  1925 ts_tablet_manager.cc:1434] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.133s	user 0.013s	sys 0.019s
I20260709 12:21:29.566511  1722 tablet_service.cc:1511] Processing CreateTablet for tablet b1cb2a27e8444930b93f1e21c2deb19c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:21:29.573817  1791 tablet_service.cc:1511] Processing CreateTablet for tablet 9b8cae5d30dd4a68a4fba5846e83873e (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:21:29.560581  1713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80ed6fcb806a4023b94ecd7b266b421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.563630  1867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cb2a27e8444930b93f1e21c2deb19c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.588534  1925 tablet_bootstrap.cc:492] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:29.589833  1789 tablet_service.cc:1511] Processing CreateTablet for tablet f15e08b3d9804a00b9836e5223042266 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:21:29.592574  1795 tablet_service.cc:1511] Processing CreateTablet for tablet bbbcc16cebc244769e7f1a1b6e29594c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:21:29.596546  1925 tablet_bootstrap.cc:654] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:29.607528  1936 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.607918  1719 tablet_service.cc:1511] Processing CreateTablet for tablet bbbcc16cebc244769e7f1a1b6e29594c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:21:29.608404  1936 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.611768  1794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc4fe4ebd84a44b9bee2535ccd62ba22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.614267  1788 tablet_service.cc:1511] Processing CreateTablet for tablet 80ed6fcb806a4023b94ecd7b266b421e (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:21:29.561857  1714 tablet_service.cc:1511] Processing CreateTablet for tablet f15e08b3d9804a00b9836e5223042266 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:21:29.616657  1924 raft_consensus.cc:359] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.617542  1924 raft_consensus.cc:385] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:29.617859  1924 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:29.570570  1868 tablet_service.cc:1511] Processing CreateTablet for tablet b9c190a9868442e4a7e6d84d752329f3 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:21:29.618624  1924 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.619589  1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aca584e2f1114809bcadeaf6faad34f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.619874  1868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c190a9868442e4a7e6d84d752329f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.607553  1716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea5d8f652fc448d6af846ff6850004b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.632125  1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b8cae5d30dd4a68a4fba5846e83873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.633708  1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbbcc16cebc244769e7f1a1b6e29594c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.636828  1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cb2a27e8444930b93f1e21c2deb19c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.638154  1924 ts_tablet_manager.cc:1434] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.147s	user 0.010s	sys 0.000s
I20260709 12:21:29.663465  1924 tablet_bootstrap.cc:492] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:29.667241  1714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f15e08b3d9804a00b9836e5223042266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.668619  1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f15e08b3d9804a00b9836e5223042266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.639384  1871 tablet_service.cc:1511] Processing CreateTablet for tablet def0036446324543ba07e132e885133a (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:21:29.715008  1936 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:29.780027  1718 tablet_service.cc:1511] Processing CreateTablet for tablet b9c190a9868442e4a7e6d84d752329f3 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:21:29.759004  1788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80ed6fcb806a4023b94ecd7b266b421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.769691  1927 tablet_bootstrap.cc:700] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent opening tablet: real 0.151s	user 0.005s	sys 0.001s
I20260709 12:21:29.795711  1791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b8cae5d30dd4a68a4fba5846e83873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.796042  1927 tablet_bootstrap.cc:654] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:29.784279  1862 tablet_service.cc:1511] Processing CreateTablet for tablet bb48ef23a7434834929994987a36f360 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:21:29.785759  1871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet def0036446324543ba07e132e885133a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.798075  1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c190a9868442e4a7e6d84d752329f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.788137  1870 tablet_service.cc:1511] Processing CreateTablet for tablet 82eacf0c2a8c4d8db4052327304a8793 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:21:29.791990  1865 tablet_service.cc:1511] Processing CreateTablet for tablet 1bd4f4834ed2497290417394b678edc0 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:21:29.809656  1795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbbcc16cebc244769e7f1a1b6e29594c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:21:29.835428  1614 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45790.6us
I20260709 12:21:29.837644  1862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb48ef23a7434834929994987a36f360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.843178  1865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd4f4834ed2497290417394b678edc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.847142  1870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82eacf0c2a8c4d8db4052327304a8793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.855242  1925 tablet_bootstrap.cc:492] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:29.855854  1925 ts_tablet_manager.cc:1403] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.268s	user 0.028s	sys 0.017s
I20260709 12:21:29.837801  1720 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 26) took 1016 ms. Trace:
W20260709 12:21:29.902310  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:29.902761  1866 tablet_service.cc:1511] Processing CreateTablet for tablet 66d3405528d44edb8c0be3b296ef72be (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:21:29.904951  1866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d3405528d44edb8c0be3b296ef72be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.911689  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:21:29.914022  1845 leader_election.cc:343] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:29.919067  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:29.921263  1771 leader_election.cc:343] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:29.927740  1863 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 29) took 1063 ms. Trace:
I20260709 12:21:29.862257  1869 tablet_service.cc:1511] Processing CreateTablet for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:21:29.933069  1869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:29.931512  1863 rpcz_store.cc:276] 0709 12:21:28.863941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.346771 (+482830us) service_pool.cc:224] Handling call
0709 12:21:29.363735 (+ 16964us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.363841 (+   106us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.631617 (+267776us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:29.922889 (+291272us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:29.927655 (+  4766us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:29.857251  1720 rpcz_store.cc:276] 0709 12:21:28.821777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.342308 (+520531us) service_pool.cc:224] Handling call
0709 12:21:29.403439 (+ 61131us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0e0c831 00007f16b0de9ac3 00007f16b0d98400 00007f16aa21daf4 00007f16aa21da82 00007f16aa21d9b1
0709 12:21:29.406717 (+  3278us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.416585 (+  9868us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.608768 (+192183us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:29.837221 (+228453us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:29.837729 (+   508us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4027264}
I20260709 12:21:29.950348  1863 tablet_service.cc:1511] Processing CreateTablet for tablet 79f5c05e64b945cbacc02914f31c5762 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:21:29.922250  1925 raft_consensus.cc:359] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.957284  1925 raft_consensus.cc:385] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:29.957760  1925 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:29.958765  1925 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.972510  1936 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.972970  1936 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:29.979076  1720 tablet_service.cc:1511] Processing CreateTablet for tablet def0036446324543ba07e132e885133a (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:21:29.967734  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:29.988348  1792 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 23) took 1062 ms. Trace:
I20260709 12:21:29.989645  1924 tablet_bootstrap.cc:654] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:29.999831  1844 leader_election.cc:343] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.003125  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:30.004212  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:29.997678  1792 rpcz_store.cc:276] 0709 12:21:28.925532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.420506 (+494974us) service_pool.cc:224] Handling call
0709 12:21:29.460185 (+ 39679us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.460376 (+   191us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.644818 (+184442us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:29.987636 (+342818us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:29.988275 (+   639us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.005113  1807 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:30.006054  1928 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:30.008529  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:30.009869  1934 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:30.010296  1934 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.010651  1934 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:29.968233  1927 tablet_bootstrap.cc:492] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:30.021373  1927 ts_tablet_manager.cc:1403] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.520s	user 0.018s	sys 0.012s
I20260709 12:21:30.023365  1722 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 33) took 1155 ms. Trace:
I20260709 12:21:30.023947  1722 rpcz_store.cc:276] 0709 12:21:28.868313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.565316 (+697003us) service_pool.cc:224] Handling call
0709 12:21:29.572485 (+  7169us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.572576 (+    91us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.975009 (+402433us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.022680 (+ 47671us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.023288 (+   608us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3328}
I20260709 12:21:29.964190  1796 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 25) took 1038 ms. Trace:
I20260709 12:21:30.030323  1796 rpcz_store.cc:276] 0709 12:21:28.926150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.453605 (+527455us) service_pool.cc:224] Handling call
0709 12:21:29.493916 (+ 40311us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.499691 (+  5775us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.843459 (+343768us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:29.963643 (+120184us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:29.964125 (+   482us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.032747  1796 tablet_service.cc:1511] Processing CreateTablet for tablet def0036446324543ba07e132e885133a (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:21:30.047250  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:30.048061  1795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 32) took 1036 ms. Trace:
I20260709 12:21:30.048491  1795 rpcz_store.cc:276] 0709 12:21:29.011321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.591433 (+580112us) service_pool.cc:224] Handling call
0709 12:21:29.653101 (+ 61668us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.661731 (+  8630us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.964848 (+303117us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.047474 (+ 82626us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.047987 (+   513us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.053256  1713 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 29) took 1230 ms. Trace:
I20260709 12:21:30.053905  1713 rpcz_store.cc:276] 0709 12:21:28.866834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.438376 (+571542us) service_pool.cc:224] Handling call
0709 12:21:29.479290 (+ 40914us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.479401 (+   111us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.849305 (+369904us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.052333 (+203028us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.053124 (+   791us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.059093  1713 tablet_service.cc:1511] Processing CreateTablet for tablet 82eacf0c2a8c4d8db4052327304a8793 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:21:30.062031  1713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82eacf0c2a8c4d8db4052327304a8793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.064198  1927 raft_consensus.cc:359] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:30.065040  1927 raft_consensus.cc:385] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.065366  1927 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.006484  1792 tablet_service.cc:1511] Processing CreateTablet for tablet b9c190a9868442e4a7e6d84d752329f3 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:21:29.950690  1788 tablet_service.cc:1511] Processing CreateTablet for tablet b1cb2a27e8444930b93f1e21c2deb19c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:21:30.037313  1796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet def0036446324543ba07e132e885133a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.066215  1927 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:30.080294  1789 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 31) took 1069 ms. Trace:
I20260709 12:21:30.086699  1925 ts_tablet_manager.cc:1434] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.231s	user 0.009s	sys 0.002s
I20260709 12:21:30.086674  1789 rpcz_store.cc:276] 0709 12:21:29.011021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.588409 (+577388us) service_pool.cc:224] Handling call
0709 12:21:29.612169 (+ 23760us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.638592 (+ 26423us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.960961 (+322369us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.074417 (+113456us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.080199 (+  5782us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.088107  1925 tablet_bootstrap.cc:492] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:30.089646  1864 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 32) took 1221 ms. Trace:
I20260709 12:21:30.100239  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:30.098875  1789 tablet_service.cc:1511] Processing CreateTablet for tablet 82eacf0c2a8c4d8db4052327304a8793 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:21:30.101399  1863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f5c05e64b945cbacc02914f31c5762. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:21:30.101706  1696 leader_election.cc:343] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.102129  1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c190a9868442e4a7e6d84d752329f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.100159  1864 rpcz_store.cc:276] 0709 12:21:28.867985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.421590 (+553605us) service_pool.cc:224] Handling call
0709 12:21:29.492454 (+ 70864us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.498422 (+  5968us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.961463 (+463041us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.074508 (+113045us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.089562 (+ 15054us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1412}
I20260709 12:21:30.104564  1788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1cb2a27e8444930b93f1e21c2deb19c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.105342  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:30.106204  1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82eacf0c2a8c4d8db4052327304a8793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.106751  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:30.105285  1864 tablet_service.cc:1511] Processing CreateTablet for tablet 6388b1a5243d463589750794e8ac5511 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:21:30.108376  1864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6388b1a5243d463589750794e8ac5511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.092406  1867 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 33) took 1223 ms. Trace:
I20260709 12:21:30.113344  1867 rpcz_store.cc:276] 0709 12:21:28.869282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.437668 (+568386us) service_pool.cc:224] Handling call
0709 12:21:29.562870 (+125202us) spinlock_profiling.cc:243] Waited 5.68 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0e0c831 00007f16b0de9ac3 00007f16b0d98400 00007f16aa21daf4 00007f16aa21da82 00007f16aa21d9b1
0709 12:21:29.563047 (+   177us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.563163 (+   116us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.675385 (+112222us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.091905 (+416520us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.092339 (+   434us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13067136}
I20260709 12:21:30.050670  1795 tablet_service.cc:1511] Processing CreateTablet for tablet bb48ef23a7434834929994987a36f360 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:21:30.115535  1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet def0036446324543ba07e132e885133a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.116470  1795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb48ef23a7434834929994987a36f360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.095314  1722 tablet_service.cc:1511] Processing CreateTablet for tablet bb48ef23a7434834929994987a36f360 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:21:30.119710  1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb48ef23a7434834929994987a36f360. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.096007  1868 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 34) took 1223 ms. Trace:
I20260709 12:21:30.096320  1924 tablet_bootstrap.cc:492] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:30.123325  1924 ts_tablet_manager.cc:1403] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.460s	user 0.023s	sys 0.054s
I20260709 12:21:30.123030  1866 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 40) took 1165 ms. Trace:
I20260709 12:21:30.123579  1718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 34) took 1255 ms. Trace:
I20260709 12:21:30.123914  1866 rpcz_store.cc:276] 0709 12:21:28.958019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.865435 (+907416us) service_pool.cc:224] Handling call
0709 12:21:29.904081 (+ 38646us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.904246 (+   165us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.964736 (+ 60490us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.045127 (+ 80391us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.122954 (+ 77827us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.099717  1928 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:30.108223  1696 leader_election.cc:343] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.108564  1936 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:30.122988  1868 rpcz_store.cc:276] 0709 12:21:28.872595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.483981 (+611386us) service_pool.cc:224] Handling call
0709 12:21:29.619357 (+135376us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.619458 (+   101us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.859887 (+240429us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.095430 (+235543us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.095947 (+   517us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.128924  1928 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.126683  1924 raft_consensus.cc:359] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.093469  1716 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 25) took 1272 ms. Trace:
I20260709 12:21:30.130672  1924 raft_consensus.cc:385] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.111055  1934 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.130841  1880 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:30.131263  1924 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.135843  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:30.136652  1731 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:30.136346  1924 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.137796  1925 tablet_bootstrap.cc:654] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:30.139590  1844 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:30.100960  1719 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 32) took 1233 ms. Trace:
I20260709 12:21:30.131840  1732 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 1: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:21:30.132110  1717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 28) took 1309 ms. Trace:
I20260709 12:21:30.140444  1717 rpcz_store.cc:276] 0709 12:21:28.822463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.408545 (+586082us) service_pool.cc:224] Handling call
0709 12:21:29.436043 (+ 27498us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.436159 (+   116us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.009087 (+572928us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.054755 (+ 45668us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.132049 (+ 77294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.142026  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:30.142349  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:30.143505  1794 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 26) took 1217 ms. Trace:
I20260709 12:21:30.143751  1696 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:30.143963  1717 tablet_service.cc:1511] Processing CreateTablet for tablet 1bd4f4834ed2497290417394b678edc0 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:21:30.145390  1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd4f4834ed2497290417394b678edc0. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:21:30.145473  1696 leader_election.cc:343] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.143989  1794 rpcz_store.cc:276] 0709 12:21:28.926457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.465464 (+539007us) service_pool.cc:224] Handling call
0709 12:21:29.568373 (+102909us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.570127 (+  1754us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.925248 (+355121us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.114431 (+189183us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.143441 (+ 29010us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.090292  1714 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 31) took 1222 ms. Trace:
I20260709 12:21:30.133466  1934 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:30.135293  1928 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:30.123942  1718 rpcz_store.cc:276] 0709 12:21:28.868641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.778960 (+910319us) service_pool.cc:224] Handling call
0709 12:21:29.797586 (+ 18626us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.797691 (+   105us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.006198 (+208507us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.122987 (+116789us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.123524 (+   537us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.135072  1871 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 35) took 1258 ms. Trace:
I20260709 12:21:30.110240  1862 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 36) took 1232 ms. Trace:
I20260709 12:21:30.153524  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:30.163781  1806 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:30.151494  1719 rpcz_store.cc:276] 0709 12:21:28.867990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.514618 (+646628us) service_pool.cc:224] Handling call
0709 12:21:29.609568 (+ 94950us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.609726 (+   158us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.968885 (+359159us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.100401 (+131516us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.100877 (+   476us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.131064  1716 rpcz_store.cc:276] 0709 12:21:28.821438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.335156 (+513718us) service_pool.cc:224] Handling call
0709 12:21:29.354954 (+ 19798us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.404537 (+ 49583us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.925248 (+520711us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.092923 (+167675us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.093401 (+   478us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.153609  1928 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:30.168336  1716 tablet_service.cc:1511] Processing CreateTablet for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:21:30.168908  1928 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.169376  1928 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:30.169872  1716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.170763  1869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 39) took 1290 ms. Trace:
I20260709 12:21:30.153957  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:30.172039  1790 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 27) took 1245 ms. Trace:
W20260709 12:21:30.172952  1845 leader_election.cc:343] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.173425  1718 tablet_service.cc:1511] Processing CreateTablet for tablet 66d3405528d44edb8c0be3b296ef72be (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:21:30.175009  1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d3405528d44edb8c0be3b296ef72be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.172469  1790 rpcz_store.cc:276] 0709 12:21:28.926749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.470424 (+543675us) service_pool.cc:224] Handling call
0709 12:21:29.525472 (+ 55048us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.525575 (+   103us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.633313 (+107738us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.099737 (+466424us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.171977 (+ 72240us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46432}
I20260709 12:21:30.161409  1862 rpcz_store.cc:276] 0709 12:21:28.877609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.783168 (+905559us) service_pool.cc:224] Handling call
0709 12:21:29.798058 (+ 14890us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.798278 (+   220us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.926532 (+128254us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.109704 (+183172us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.110174 (+   470us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.180598  1790 tablet_service.cc:1511] Processing CreateTablet for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:21:30.182329  1790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.184270  1865 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 38) took 1304 ms. Trace:
I20260709 12:21:30.184809  1865 rpcz_store.cc:276] 0709 12:21:28.879666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.790630 (+910964us) service_pool.cc:224] Handling call
0709 12:21:29.806618 (+ 15988us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.806740 (+   122us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.092141 (+285401us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.183707 (+ 91566us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.184201 (+   494us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.186916  1793 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 24) took 1261 ms. Trace:
I20260709 12:21:30.187893  1715 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 27) took 1365 ms. Trace:
I20260709 12:21:30.187510  1793 rpcz_store.cc:276] 0709 12:21:28.925857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.427598 (+501741us) service_pool.cc:224] Handling call
0709 12:21:29.562655 (+135057us) spinlock_profiling.cc:243] Waited 9.34 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0e0c831 00007f16b0de9ac3 00007f16b0d98400 00007f16aa21daf4 00007f16aa21da82 00007f16aa21d9b1
0709 12:21:29.586835 (+ 24180us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.593966 (+  7131us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.850062 (+256096us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.088418 (+238356us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.186848 (+ 98430us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35691,"spinlock_wait_cycles":21487360}
I20260709 12:21:30.188477  1715 rpcz_store.cc:276] 0709 12:21:28.822128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.408223 (+586095us) service_pool.cc:224] Handling call
0709 12:21:29.421910 (+ 13687us) spinlock_profiling.cc:243] Waited 2.97 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0e0c831 00007f16b0de9ac3 00007f16b0d98400 00007f16aa21daf4 00007f16aa21da82 00007f16aa21d9b1
0709 12:21:29.448750 (+ 26840us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.448866 (+   116us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.953221 (+504355us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.146629 (+193408us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.187827 (+ 41198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6829952}
I20260709 12:21:30.190979  1793 tablet_service.cc:1511] Processing CreateTablet for tablet 66d3405528d44edb8c0be3b296ef72be (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:21:30.192459  1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d3405528d44edb8c0be3b296ef72be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.194631  1721 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 30) took 1327 ms. Trace:
I20260709 12:21:30.155264  1713 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 37) took 1285 ms. Trace:
I20260709 12:21:30.200809  1713 rpcz_store.cc:276] 0709 12:21:28.869672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.057244 (+1187572us) service_pool.cc:224] Handling call
0709 12:21:30.061100 (+  3856us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.061311 (+   211us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.077777 (+ 16466us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.154695 (+ 76918us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.155200 (+   505us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.203027  1713 tablet_service.cc:1511] Processing CreateTablet for tablet 6388b1a5243d463589750794e8ac5511 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:21:30.204422  1713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6388b1a5243d463589750794e8ac5511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.160709  1871 rpcz_store.cc:276] 0709 12:21:28.876264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.590376 (+714112us) service_pool.cc:224] Handling call
0709 12:21:29.785100 (+194724us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.785200 (+   100us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.913464 (+128264us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.045995 (+132531us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.134992 (+ 88997us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":428}
I20260709 12:21:30.207209  1792 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 34) took 1195 ms. Trace:
I20260709 12:21:30.207733  1792 rpcz_store.cc:276] 0709 12:21:29.011889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.005465 (+993576us) service_pool.cc:224] Handling call
0709 12:21:30.084509 (+ 79044us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.101712 (+ 17203us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.157534 (+ 55822us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.201834 (+ 44300us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.207128 (+  5294us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4737,"spinlock_wait_cycles":1665280}
I20260709 12:21:30.208748  1871 tablet_service.cc:1511] Processing CreateTablet for tablet 82c4d9259f1a44119e802a40207d64b5 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:21:30.210233  1871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c4d9259f1a44119e802a40207d64b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.171650  1869 rpcz_store.cc:276] 0709 12:21:28.880036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.841760 (+961724us) service_pool.cc:224] Handling call
0709 12:21:29.932422 (+ 90662us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.932568 (+   146us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.027074 (+ 94506us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.170195 (+143121us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.170700 (+   505us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.210592  1792 tablet_service.cc:1511] Processing CreateTablet for tablet 79f5c05e64b945cbacc02914f31c5762 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:21:30.212972  1869 tablet_service.cc:1511] Processing CreateTablet for tablet 4a57b66dec794db09a7f9048a3d0de10 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:21:30.213673  1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f5c05e64b945cbacc02914f31c5762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.214439  1869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a57b66dec794db09a7f9048a3d0de10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.214354  1790 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 39) took 1200 ms. Trace:
I20260709 12:21:30.153666  1936 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:30.149971  1714 rpcz_store.cc:276] 0709 12:21:28.867625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.493542 (+625917us) service_pool.cc:224] Handling call
0709 12:21:29.616671 (+123129us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.616782 (+   111us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:29.954823 (+338041us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.089711 (+134888us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.090224 (+   513us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.232067  1936 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.232574  1936 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:30.233503  1714 tablet_service.cc:1511] Processing CreateTablet for tablet c5f3d785e05841fbb0efa210c3d9675c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:21:30.235071  1714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f3d785e05841fbb0efa210c3d9675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.219069  1791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 30) took 1208 ms. Trace:
I20260709 12:21:30.163604  1868 tablet_service.cc:1511] Processing CreateTablet for tablet c5f3d785e05841fbb0efa210c3d9675c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:21:30.239787  1868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f3d785e05841fbb0efa210c3d9675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.241425  1806 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:30.192505  1715 tablet_service.cc:1511] Processing CreateTablet for tablet 79f5c05e64b945cbacc02914f31c5762 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:21:30.243034  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:30.195549  1797 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 28) took 1259 ms. Trace:
I20260709 12:21:30.127311  1866 tablet_service.cc:1511] Processing CreateTablet for tablet ae23688e76624153b5a7089586669bb0 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:21:30.244026  1934 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:30.150002  1794 tablet_service.cc:1511] Processing CreateTablet for tablet 1bd4f4834ed2497290417394b678edc0 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:21:30.244051  1715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f5c05e64b945cbacc02914f31c5762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.199326  1862 tablet_service.cc:1511] Processing CreateTablet for tablet 2e89c188ae524548ae35a90336c0c92c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:21:30.260330  1864 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 42) took 1298 ms. Trace:
I20260709 12:21:30.260871  1722 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 36) took 1391 ms. Trace:
I20260709 12:21:30.261276  1789 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 37) took 1248 ms. Trace:
I20260709 12:21:30.244230  1797 rpcz_store.cc:276] 0709 12:21:28.967094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.505400 (+538306us) service_pool.cc:224] Handling call
0709 12:21:29.525902 (+ 20502us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.525996 (+    94us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.063596 (+537600us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.195004 (+131408us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.195487 (+   483us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:30.245308  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 105821us
I20260709 12:21:30.195188  1721 rpcz_store.cc:276] 0709 12:21:28.867278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.456813 (+589535us) service_pool.cc:224] Handling call
0709 12:21:29.476300 (+ 19487us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.476456 (+   156us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.107784 (+631328us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.193980 (+ 86196us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.194566 (+   586us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.272176  1866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae23688e76624153b5a7089586669bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.268872  1864 rpcz_store.cc:276] 0709 12:21:28.962052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.104011 (+1141959us) service_pool.cc:224] Handling call
0709 12:21:30.107874 (+  3863us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.107976 (+   102us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.190746 (+ 82770us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.259727 (+ 68981us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.260242 (+   515us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.285193  1864 tablet_service.cc:1511] Processing CreateTablet for tablet 2b797b0728dc4e01976ab61ef164fa90 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:21:30.271452  1789 rpcz_store.cc:276] 0709 12:21:29.012794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.090239 (+1077445us) service_pool.cc:224] Handling call
0709 12:21:30.101712 (+ 11473us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.101830 (+   118us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.167609 (+ 65779us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.260368 (+ 92759us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.261168 (+   800us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.288906  1793 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 40) took 1275 ms. Trace:
I20260709 12:21:30.289269  1789 tablet_service.cc:1511] Processing CreateTablet for tablet 6388b1a5243d463589750794e8ac5511 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:21:30.289330  1793 rpcz_store.cc:276] 0709 12:21:29.013867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.189832 (+1175965us) service_pool.cc:224] Handling call
0709 12:21:30.191914 (+  2082us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.192028 (+   114us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.241017 (+ 48989us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.288339 (+ 47322us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.288840 (+   501us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.282232  1794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd4f4834ed2497290417394b678edc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.291122  1927 ts_tablet_manager.cc:1434] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.269s	user 0.016s	sys 0.001s
I20260709 12:21:30.291502  1793 tablet_service.cc:1511] Processing CreateTablet for tablet c5f3d785e05841fbb0efa210c3d9675c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:21:30.292461  1927 tablet_bootstrap.cc:492] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:30.292115  1721 tablet_service.cc:1511] Processing CreateTablet for tablet 2e89c188ae524548ae35a90336c0c92c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:21:30.293120  1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f3d785e05841fbb0efa210c3d9675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.293529  1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e89c188ae524548ae35a90336c0c92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.299626  1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6388b1a5243d463589750794e8ac5511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.270663  1722 rpcz_store.cc:276] 0709 12:21:28.869296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.046610 (+1177314us) service_pool.cc:224] Handling call
0709 12:21:30.119201 (+ 72591us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.119311 (+   110us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.206600 (+ 87289us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.238818 (+ 32218us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.260807 (+ 21989us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.230657  1790 rpcz_store.cc:276] 0709 12:21:29.013566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.179023 (+1165457us) service_pool.cc:224] Handling call
0709 12:21:30.181616 (+  2593us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.181737 (+   121us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.205229 (+ 23492us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.213767 (+  8538us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.214280 (+   513us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.308194  1864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b797b0728dc4e01976ab61ef164fa90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.309340  1790 tablet_service.cc:1511] Processing CreateTablet for tablet 2e89c188ae524548ae35a90336c0c92c (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:21:30.313272  1719 tablet_service.cc:1511] Processing CreateTablet for tablet ae23688e76624153b5a7089586669bb0 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:21:30.330952  1954 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:30.245002  1791 rpcz_store.cc:276] 0709 12:21:29.010637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.572638 (+562001us) service_pool.cc:224] Handling call
0709 12:21:29.641183 (+ 68545us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.653243 (+ 12060us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.075661 (+422418us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.126496 (+ 50835us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.218994 (+ 92498us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38583}
I20260709 12:21:30.333830  1717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 38) took 1463 ms. Trace:
I20260709 12:21:30.306339  1722 tablet_service.cc:1511] Processing CreateTablet for tablet 36bc2f66a73947468d61113c226ed318 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:21:30.334981  1862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e89c188ae524548ae35a90336c0c92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.345384  1717 rpcz_store.cc:276] 0709 12:21:28.870022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.142815 (+1272793us) service_pool.cc:224] Handling call
0709 12:21:30.144899 (+  2084us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.145000 (+   101us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.206092 (+ 61092us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.333251 (+127159us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.333759 (+   508us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.335954  1867 tablet_service.cc:1511] Processing CreateTablet for tablet 84ce9896327e4c9d9888e80e8dbeeb0d (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:21:30.340299  1796 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 35) took 1327 ms. Trace:
I20260709 12:21:30.340692  1951 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.354017  1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae23688e76624153b5a7089586669bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.360638  1865 tablet_service.cc:1511] Processing CreateTablet for tablet 36bc2f66a73947468d61113c226ed318 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:21:30.366878  1790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e89c188ae524548ae35a90336c0c92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.365047  1934 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:30.357885  1928 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.379396  1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bc2f66a73947468d61113c226ed318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.375339  1865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bc2f66a73947468d61113c226ed318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.382485  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:30.357825  1951 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:30.382488  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:21:30.396579  1845 leader_election.cc:343] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.361768  1797 tablet_service.cc:1511] Processing CreateTablet for tablet 36bc2f66a73947468d61113c226ed318 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:21:30.355631  1796 rpcz_store.cc:276] 0709 12:21:29.012149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.031533 (+1019384us) service_pool.cc:224] Handling call
0709 12:21:30.033632 (+  2099us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.033737 (+   105us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.130667 (+ 96930us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.336806 (+206139us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.339743 (+  2937us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.384416  1928 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:30.401790  1717 tablet_service.cc:1511] Processing CreateTablet for tablet 84ce9896327e4c9d9888e80e8dbeeb0d (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:21:30.403366  1951 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:21:30.404567  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:30.401873  1791 tablet_service.cc:1511] Processing CreateTablet for tablet ae23688e76624153b5a7089586669bb0 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:21:30.407869  1927 tablet_bootstrap.cc:654] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:30.408083  1925 tablet_bootstrap.cc:492] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:30.408461  1716 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 39) took 1538 ms. Trace:
I20260709 12:21:30.408730  1797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bc2f66a73947468d61113c226ed318. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.434263  1716 rpcz_store.cc:276] 0709 12:21:28.870354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.167174 (+1296820us) service_pool.cc:224] Handling call
0709 12:21:30.169312 (+  2138us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.169430 (+   118us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.237823 (+ 68393us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.376570 (+138747us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.408384 (+ 31814us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.409422  1870 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 37) took 1530 ms. Trace:
I20260709 12:21:30.415433  1880 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:30.436655  1716 tablet_service.cc:1511] Processing CreateTablet for tablet 82c4d9259f1a44119e802a40207d64b5 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:21:30.438195  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:30.422461  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:30.439680  1718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 40) took 1569 ms. Trace:
W20260709 12:21:30.440361  1770 leader_election.cc:343] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.440737  1793 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 43) took 1426 ms. Trace:
I20260709 12:21:30.441143  1793 rpcz_store.cc:276] 0709 12:21:29.014729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.290444 (+1275715us) service_pool.cc:224] Handling call
0709 12:21:30.292578 (+  2134us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.292688 (+   110us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.302077 (+  9389us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.430772 (+128695us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.440669 (+  9897us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.441690  1951 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:30.442261  1951 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.442607  1951 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:30.442821  1792 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 41) took 1428 ms. Trace:
I20260709 12:21:30.443229  1791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae23688e76624153b5a7089586669bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.444504  1793 tablet_service.cc:1511] Processing CreateTablet for tablet 2b797b0728dc4e01976ab61ef164fa90 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:21:30.427978  1866 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 46) took 1446 ms. Trace:
I20260709 12:21:30.446480  1866 rpcz_store.cc:276] 0709 12:21:28.981127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.126233 (+1145106us) service_pool.cc:224] Handling call
0709 12:21:30.244535 (+118302us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.244658 (+   123us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.341195 (+ 96537us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.427373 (+ 86178us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.427892 (+   519us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.427975  1717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ce9896327e4c9d9888e80e8dbeeb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.429407  1789 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 42) took 1415 ms. Trace:
I20260709 12:21:30.440110  1718 rpcz_store.cc:276] 0709 12:21:28.870660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.172258 (+1301598us) service_pool.cc:224] Handling call
0709 12:21:30.174473 (+  2215us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.174582 (+   109us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.420872 (+246290us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.439079 (+ 18207us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.439609 (+   530us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.448833  1866 tablet_service.cc:1511] Processing CreateTablet for tablet 7b550688b5e84ddda5fcf049010ac1e8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:21:30.419682  1720 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 35) took 1549 ms. Trace:
I20260709 12:21:30.443271  1792 rpcz_store.cc:276] 0709 12:21:29.014147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.209196 (+1195049us) service_pool.cc:224] Handling call
0709 12:21:30.213101 (+  3905us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.213244 (+   143us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.260456 (+ 47212us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.435260 (+174804us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.442720 (+  7460us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.451257  1720 rpcz_store.cc:276] 0709 12:21:28.868960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.977755 (+1108795us) service_pool.cc:224] Handling call
0709 12:21:30.026577 (+ 48822us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.037565 (+ 10988us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.177635 (+140070us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.384614 (+206979us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.418060 (+ 33446us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":170}
I20260709 12:21:30.454030  1720 tablet_service.cc:1511] Processing CreateTablet for tablet 7b550688b5e84ddda5fcf049010ac1e8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:21:30.451148  1789 rpcz_store.cc:276] 0709 12:21:29.014436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.288180 (+1273744us) service_pool.cc:224] Handling call
0709 12:21:30.290262 (+  2082us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.290373 (+   111us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.337570 (+ 47197us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.426464 (+ 88894us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.429005 (+  2541us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb41a2 00007f16b1bf5889 00007f16b0d26d88 00007f16b0ded973 00007f16b0de970f 00007f16b0de9ecd 00007f16b0d98400 00007f16aa21daf4 00007f16aa21da82
0709 12:21:30.429336 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3820416}
I20260709 12:21:30.455809  1718 tablet_service.cc:1511] Processing CreateTablet for tablet 4a57b66dec794db09a7f9048a3d0de10 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:21:30.456777  1721 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 44) took 1554 ms. Trace:
I20260709 12:21:30.433413  1936 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.447103  1925 ts_tablet_manager.cc:1403] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.359s	user 0.028s	sys 0.028s
I20260709 12:21:30.417315  1867 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ce9896327e4c9d9888e80e8dbeeb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.461253  1716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c4d9259f1a44119e802a40207d64b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.453758  1792 tablet_service.cc:1511] Processing CreateTablet for tablet 34263595c9ed4e0a9eba4bfe41c0702e (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:21:30.462224  1928 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.457428  1924 ts_tablet_manager.cc:1434] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.334s	user 0.008s	sys 0.029s
I20260709 12:21:30.476354  1928 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.477626  1732 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:21:30.478833  1928 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:30.480089  1844 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:30.481981  1928 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:21:30.483165  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:30.483706  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:21:30.485001  1696 leader_election.cc:343] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.485306  1866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b550688b5e84ddda5fcf049010ac1e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:21:30.485553  1845 leader_election.cc:343] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.487816  1796 tablet_service.cc:1511] Processing CreateTablet for tablet 7295d6494bad4378b63658f8274dfde8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:21:30.447268  1793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b797b0728dc4e01976ab61ef164fa90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.458581  1934 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.486754  1924 tablet_bootstrap.cc:492] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:30.456585  1789 tablet_service.cc:1511] Processing CreateTablet for tablet 84ce9896327e4c9d9888e80e8dbeeb0d (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:21:30.495288  1792 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34263595c9ed4e0a9eba4bfe41c0702e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.500957  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:30.457352  1721 rpcz_store.cc:276] 0709 12:21:28.902556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.291073 (+1388517us) service_pool.cc:224] Handling call
0709 12:21:30.293008 (+  1935us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.293115 (+   107us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.420427 (+127312us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.454269 (+ 33842us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.456702 (+  2433us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.501679  1795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 36) took 1489 ms. Trace:
I20260709 12:21:30.501976  1880 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:21:30.502113  1795 rpcz_store.cc:276] 0709 12:21:29.012471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.049551 (+1037080us) service_pool.cc:224] Handling call
0709 12:21:30.115758 (+ 66207us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.115901 (+   143us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.422810 (+306909us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.501090 (+ 78280us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.501610 (+   520us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.469699  1925 raft_consensus.cc:359] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.470969  1869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 49) took 1486 ms. Trace:
I20260709 12:21:30.457587  1870 rpcz_store.cc:276] 0709 12:21:28.879261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.786771 (+907510us) service_pool.cc:224] Handling call
0709 12:21:29.801889 (+ 15118us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:29.802045 (+   156us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.218380 (+416335us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.393702 (+175322us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.409361 (+ 15659us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.476110  1788 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 33) took 1463 ms. Trace:
I20260709 12:21:30.510236  1788 rpcz_store.cc:276] 0709 12:21:29.011619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.949653 (+938034us) service_pool.cc:224] Handling call
0709 12:21:30.104082 (+154429us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.104194 (+   112us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.170076 (+ 65882us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.472147 (+302071us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.474882 (+  2735us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.511698  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:30.511770  1870 tablet_service.cc:1511] Processing CreateTablet for tablet 7295d6494bad4378b63658f8274dfde8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
W20260709 12:21:30.513607  1844 leader_election.cc:343] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.513932  1870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7295d6494bad4378b63658f8274dfde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.514161  1844 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:30.515338  1789 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ce9896327e4c9d9888e80e8dbeeb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.515591  1928 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:30.517048  1796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7295d6494bad4378b63658f8274dfde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.508159  1696 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:30.518210  1957 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:21:30.511914  1718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a57b66dec794db09a7f9048a3d0de10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.504626  1925 raft_consensus.cc:385] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.516510  1951 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:30.522626  1720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b550688b5e84ddda5fcf049010ac1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.508229  1721 tablet_service.cc:1511] Processing CreateTablet for tablet 23caea311a774ba3b29b7749bd3f1cd6 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:21:30.540988  1788 tablet_service.cc:1511] Processing CreateTablet for tablet 4a57b66dec794db09a7f9048a3d0de10 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:21:30.535187  1925 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.542629  1788 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a57b66dec794db09a7f9048a3d0de10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.505625  1869 rpcz_store.cc:276] 0709 12:21:28.984137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.211679 (+1227542us) service_pool.cc:224] Handling call
0709 12:21:30.213905 (+  2226us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.214008 (+   103us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.341095 (+127087us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.470249 (+129154us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.470886 (+   637us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.504298  1936 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:30.560071  1721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23caea311a774ba3b29b7749bd3f1cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.573772  1927 tablet_bootstrap.cc:492] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:30.574426  1927 ts_tablet_manager.cc:1403] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.282s	user 0.010s	sys 0.017s
I20260709 12:21:30.578698  1630 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:30.578083  1927 raft_consensus.cc:359] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.548043  1925 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.591995  1794 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 38) took 1578 ms. Trace:
I20260709 12:21:30.585026  1927 raft_consensus.cc:385] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.592888  1927 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.592609  1794 rpcz_store.cc:276] 0709 12:21:29.013234 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.148982 (+1135748us) service_pool.cc:224] Handling call
0709 12:21:30.273047 (+124065us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.278603 (+  5556us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.399310 (+120707us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.591251 (+191941us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.591906 (+   655us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.594051  1927 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.551602  1869 tablet_service.cc:1511] Processing CreateTablet for tablet 23caea311a774ba3b29b7749bd3f1cd6 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:21:30.599426  1869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23caea311a774ba3b29b7749bd3f1cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.601019  1714 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 43) took 1695 ms. Trace:
I20260709 12:21:30.601600  1714 rpcz_store.cc:276] 0709 12:21:28.902219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.232420 (+1330201us) service_pool.cc:224] Handling call
0709 12:21:30.234446 (+  2026us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.234585 (+   139us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.336140 (+101555us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.564639 (+228499us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.597512 (+ 32873us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.602942  1795 tablet_service.cc:1511] Processing CreateTablet for tablet 82c4d9259f1a44119e802a40207d64b5 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:21:30.603783  1714 tablet_service.cc:1511] Processing CreateTablet for tablet 2b797b0728dc4e01976ab61ef164fa90 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:21:30.604425  1795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82c4d9259f1a44119e802a40207d64b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.605053  1714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b797b0728dc4e01976ab61ef164fa90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.608357  1927 ts_tablet_manager.cc:1434] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.034s	user 0.010s	sys 0.001s
I20260709 12:21:30.608654  1863 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 41) took 1648 ms. Trace:
I20260709 12:21:30.609009  1863 rpcz_store.cc:276] 0709 12:21:28.959921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.949163 (+989242us) service_pool.cc:224] Handling call
0709 12:21:30.035354 (+ 86191us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.080527 (+ 45173us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.414034 (+333507us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.608066 (+194032us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.608583 (+   517us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.613794  1924 tablet_bootstrap.cc:654] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:30.615513  1868 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 43) took 1637 ms. Trace:
I20260709 12:21:30.615917  1868 rpcz_store.cc:276] 0709 12:21:28.978546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.162178 (+1183632us) service_pool.cc:224] Handling call
0709 12:21:30.239235 (+ 77057us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.239352 (+   117us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.334413 (+ 95061us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.584349 (+249936us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.615452 (+ 31103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.616691  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:30.617413  1880 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:30.620334  1951 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:21:30.621750  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:21:30.623389  1770 leader_election.cc:343] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.627216  1715 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 41) took 1727 ms. Trace:
I20260709 12:21:30.627683  1715 rpcz_store.cc:276] 0709 12:21:28.900332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.191398 (+1291066us) service_pool.cc:224] Handling call
0709 12:21:30.243531 (+ 52133us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.243642 (+   111us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.414228 (+170586us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.626569 (+212341us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.627118 (+   549us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.629858  1715 tablet_service.cc:1511] Processing CreateTablet for tablet 7295d6494bad4378b63658f8274dfde8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:21:30.609527  1927 tablet_bootstrap.cc:492] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:30.614539  1722 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 45) took 1711 ms. Trace:
I20260709 12:21:30.631407  1864 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 52) took 1644 ms. Trace:
I20260709 12:21:30.643252  1925 ts_tablet_manager.cc:1434] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.177s	user 0.002s	sys 0.007s
I20260709 12:21:30.644526  1925 tablet_bootstrap.cc:492] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:30.641067  1864 rpcz_store.cc:276] 0709 12:21:28.987106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.284078 (+1296972us) service_pool.cc:224] Handling call
0709 12:21:30.286159 (+  2081us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.286255 (+    96us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.476999 (+190744us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.595988 (+118989us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.631334 (+ 35346us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.648708  1864 tablet_service.cc:1511] Processing CreateTablet for tablet 84a31d3e62ae470d9c01e48ea29fbaee (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:21:30.650216  1864 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a31d3e62ae470d9c01e48ea29fbaee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.618391  1868 tablet_service.cc:1511] Processing CreateTablet for tablet 4f27ba9b299f46b7a89ef2d660780f8d (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:21:30.665827  1868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f27ba9b299f46b7a89ef2d660780f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.659088  1715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7295d6494bad4378b63658f8274dfde8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.667734  1790 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 44) took 1636 ms. Trace:
I20260709 12:21:30.668305  1713 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 42) took 1767 ms. Trace:
I20260709 12:21:30.668231  1790 rpcz_store.cc:276] 0709 12:21:29.031344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.308109 (+1276765us) service_pool.cc:224] Handling call
0709 12:21:30.310516 (+  2407us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.310664 (+   148us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.508202 (+197538us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.662510 (+154308us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.667659 (+  5149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.669004  1794 tablet_service.cc:1511] Processing CreateTablet for tablet 7b550688b5e84ddda5fcf049010ac1e8 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:21:30.662093  1719 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 46) took 1758 ms. Trace:
I20260709 12:21:30.670828  1871 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 48) took 1687 ms. Trace:
I20260709 12:21:30.675102  1793 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 52) took 1618 ms. Trace:
I20260709 12:21:30.675315  1862 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 44) took 1696 ms. Trace:
I20260709 12:21:30.637307  1722 rpcz_store.cc:276] 0709 12:21:28.902936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.305248 (+1402312us) service_pool.cc:224] Handling call
0709 12:21:30.346808 (+ 41560us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.346931 (+   123us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.491118 (+144187us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.613974 (+122856us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.614468 (+   494us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.676086  1863 tablet_service.cc:1511] Processing CreateTablet for tablet 34263595c9ed4e0a9eba4bfe41c0702e (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:21:30.658517  1796 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 53) took 1601 ms. Trace:
I20260709 12:21:30.687549  1936 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.701604  1797 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 45) took 1669 ms. Trace:
I20260709 12:21:30.702271  1936 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.702203  1797 rpcz_store.cc:276] 0709 12:21:29.031714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.356297 (+1324583us) service_pool.cc:224] Handling call
0709 12:21:30.408266 (+ 51969us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.408364 (+    98us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.532072 (+123708us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.700284 (+168212us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.701533 (+  1249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":789}
I20260709 12:21:30.704631  1936 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:30.706871  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:30.683498  1871 rpcz_store.cc:276] 0709 12:21:28.983152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.207683 (+1224531us) service_pool.cc:224] Handling call
0709 12:21:30.209685 (+  2002us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.209796 (+   111us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.468521 (+258725us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.670116 (+201595us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.670738 (+   622us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.707224  1797 tablet_service.cc:1511] Processing CreateTablet for tablet 4f27ba9b299f46b7a89ef2d660780f8d (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:21:30.682711  1713 rpcz_store.cc:276] 0709 12:21:28.901441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.201925 (+1300484us) service_pool.cc:224] Handling call
0709 12:21:30.203889 (+  1964us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.203984 (+    95us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.580500 (+376516us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.667618 (+ 87118us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.668237 (+   619us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.709299  1871 tablet_service.cc:1511] Processing CreateTablet for tablet 5e573c99734f4d91bb30d1fe72b56cf7 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:21:30.710399  1713 tablet_service.cc:1511] Processing CreateTablet for tablet 4f27ba9b299f46b7a89ef2d660780f8d (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:21:30.684458  1862 rpcz_store.cc:276] 0709 12:21:28.979083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.181468 (+1202385us) service_pool.cc:224] Handling call
0709 12:21:30.277587 (+ 96119us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.283021 (+  5434us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.464762 (+181741us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.670124 (+205362us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.675243 (+  5119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23221}
W20260709 12:21:30.711691  1696 leader_election.cc:343] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.712121  1713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f27ba9b299f46b7a89ef2d660780f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.712771  1862 tablet_service.cc:1511] Processing CreateTablet for tablet ae12891834034d968900107cc58fa821 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:21:30.713459  1715 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 53) took 1762 ms. Trace:
I20260709 12:21:30.714176  1715 rpcz_store.cc:276] 0709 12:21:28.951408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.628760 (+1677352us) service_pool.cc:224] Handling call
0709 12:21:30.633710 (+  4950us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.637732 (+  4022us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.706027 (+ 68295us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.712926 (+  6899us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.713394 (+   468us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.682176  1719 rpcz_store.cc:276] 0709 12:21:28.903288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.311658 (+1408370us) service_pool.cc:224] Handling call
0709 12:21:30.344062 (+ 32404us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.344205 (+   143us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.500887 (+156682us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.660003 (+159116us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.662015 (+  2012us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.716521  1715 tablet_service.cc:1511] Processing CreateTablet for tablet 84a31d3e62ae470d9c01e48ea29fbaee (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:21:30.716670  1719 tablet_service.cc:1511] Processing CreateTablet for tablet 5e573c99734f4d91bb30d1fe72b56cf7 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:21:30.717929  1715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a31d3e62ae470d9c01e48ea29fbaee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.690917  1863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34263595c9ed4e0a9eba4bfe41c0702e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.719422  1862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae12891834034d968900107cc58fa821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.723136  1871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e573c99734f4d91bb30d1fe72b56cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.723270  1866 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 50) took 1738 ms. Trace:
I20260709 12:21:30.723901  1866 rpcz_store.cc:276] 0709 12:21:28.985226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.447638 (+1462412us) service_pool.cc:224] Handling call
0709 12:21:30.450573 (+  2935us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.450664 (+    91us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.649338 (+198674us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.722360 (+ 73022us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.723189 (+   829us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.735618  1719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e573c99734f4d91bb30d1fe72b56cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.736147  1868 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 55) took 1708 ms. Trace:
I20260709 12:21:30.736703  1868 rpcz_store.cc:276] 0709 12:21:29.029622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.617050 (+1587428us) service_pool.cc:224] Handling call
0709 12:21:30.665291 (+ 48241us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.665404 (+   113us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.720238 (+ 54834us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.735590 (+ 15352us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.736077 (+   487us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.738863  1869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 51) took 1752 ms. Trace:
I20260709 12:21:30.739125  1715 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 56) took 1683 ms. Trace:
I20260709 12:21:30.739432  1869 rpcz_store.cc:276] 0709 12:21:28.986128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.549948 (+1563820us) service_pool.cc:224] Handling call
0709 12:21:30.598689 (+ 48741us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.598858 (+   169us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.702860 (+104002us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.738243 (+ 35383us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.738753 (+   510us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.691810  1927 tablet_bootstrap.cc:654] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:30.691936  1865 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 45) took 1711 ms. Trace:
I20260709 12:21:30.696766  1722 tablet_service.cc:1511] Processing CreateTablet for tablet 34263595c9ed4e0a9eba4bfe41c0702e (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:21:30.743826  1722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34263595c9ed4e0a9eba4bfe41c0702e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.714922  1796 rpcz_store.cc:276] 0709 12:21:29.057501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.418489 (+1360988us) service_pool.cc:224] Handling call
0709 12:21:30.488789 (+ 70300us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.489053 (+   264us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.547313 (+ 58260us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.657918 (+110605us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.658440 (+   522us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.697647  1794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b550688b5e84ddda5fcf049010ac1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.684598  1880 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:30.698261  1925 tablet_bootstrap.cc:654] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:30.747920  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:30.748405  1797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f27ba9b299f46b7a89ef2d660780f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.749581  1951 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:30.750130  1951 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:30.699148  1790 tablet_service.cc:1511] Processing CreateTablet for tablet 23caea311a774ba3b29b7749bd3f1cd6 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:21:30.751470  1790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23caea311a774ba3b29b7749bd3f1cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.751088  1951 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:30.699664  1867 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 47) took 1717 ms. Trace:
I20260709 12:21:30.700775  1791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 46) took 1668 ms. Trace:
I20260709 12:21:30.700979  1788 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 49) took 1644 ms. Trace:
I20260709 12:21:30.754887  1716 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 48) took 1850 ms. Trace:
I20260709 12:21:30.701395  1795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 48) took 1645 ms. Trace:
I20260709 12:21:30.755125  1791 rpcz_store.cc:276] 0709 12:21:29.032010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.367487 (+1335477us) service_pool.cc:224] Handling call
0709 12:21:30.428516 (+ 61029us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.428647 (+   131us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.534925 (+106278us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.700058 (+165133us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.700716 (+   658us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4459,"spinlock_wait_cycles":17664}
I20260709 12:21:30.706693  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:30.754611  1862 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 58) took 1567 ms. Trace:
I20260709 12:21:30.759326  1862 rpcz_store.cc:276] 0709 12:21:29.190685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.711711 (+1521026us) service_pool.cc:224] Handling call
0709 12:21:30.713984 (+  2273us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.714105 (+   121us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.744450 (+ 30345us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.753987 (+  9537us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.754528 (+   541us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.757874  1795 rpcz_store.cc:276] 0709 12:21:29.055746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.522437 (+1466691us) service_pool.cc:224] Handling call
0709 12:21:30.603887 (+ 81450us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.604000 (+   113us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.692929 (+ 88929us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.700182 (+  7253us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.701335 (+  1153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1002}
I20260709 12:21:30.755002  1867 rpcz_store.cc:276] 0709 12:21:28.982205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.197554 (+1215349us) service_pool.cc:224] Handling call
0709 12:21:30.363589 (+166035us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.366246 (+  2657us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.652458 (+286212us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.699072 (+ 46614us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.699601 (+   529us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.762928  1720 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 50) took 1858 ms. Trace:
I20260709 12:21:30.739610  1715 rpcz_store.cc:276] 0709 12:21:29.055525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.715452 (+1659927us) service_pool.cc:224] Handling call
0709 12:21:30.717425 (+  1973us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.717530 (+   105us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.731631 (+ 14101us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.738594 (+  6963us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.739070 (+   476us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.683900  1793 rpcz_store.cc:276] 0709 12:21:29.057127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.443468 (+1386341us) service_pool.cc:224] Handling call
0709 12:21:30.446699 (+  3231us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.446839 (+   140us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.576855 (+130016us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.662408 (+ 85553us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.675035 (+ 12627us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.690729  1924 tablet_bootstrap.cc:492] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:30.755226  1788 rpcz_store.cc:276] 0709 12:21:29.056161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.539866 (+1483705us) service_pool.cc:224] Handling call
0709 12:21:30.542087 (+  2221us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.542195 (+   108us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.578259 (+ 36064us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.699641 (+121382us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.700911 (+  1270us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":740}
W20260709 12:21:30.759985  1696 leader_election.cc:343] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.775561  1924 ts_tablet_manager.cc:1403] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.289s	user 0.011s	sys 0.005s
I20260709 12:21:30.787127  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:30.756300  1865 rpcz_store.cc:276] 0709 12:21:28.980059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.186848 (+1206789us) service_pool.cc:224] Handling call
0709 12:21:30.366340 (+179492us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.366446 (+   106us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.542917 (+176471us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.691307 (+148390us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.691858 (+   551us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17576}
I20260709 12:21:30.789383  1936 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:30.701604  1717 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 47) took 1798 ms. Trace:
I20260709 12:21:30.795059  1717 rpcz_store.cc:276] 0709 12:21:28.903728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.349519 (+1445791us) service_pool.cc:224] Handling call
0709 12:21:30.427199 (+ 77680us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0e0c831 00007f16b0de9ac3 00007f16b0d98400 00007f16aa21daf4 00007f16aa21da82 00007f16aa21d9b1
0709 12:21:30.427362 (+   163us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.427470 (+   108us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.498107 (+ 70637us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.700978 (+202871us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.701530 (+   552us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4859776}
I20260709 12:21:30.796725  1870 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 53) took 1807 ms. Trace:
I20260709 12:21:30.780701  1795 tablet_service.cc:1511] Processing CreateTablet for tablet ae12891834034d968900107cc58fa821 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:21:30.797222  1870 rpcz_store.cc:276] 0709 12:21:28.989784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.510359 (+1520575us) service_pool.cc:224] Handling call
0709 12:21:30.513179 (+  2820us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.513338 (+   159us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.690970 (+177632us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.795982 (+105012us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.796643 (+   661us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.798913  1795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae12891834034d968900107cc58fa821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.755347  1716 rpcz_store.cc:276] 0709 12:21:28.904087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.435521 (+1531434us) service_pool.cc:224] Handling call
0709 12:21:30.437556 (+  2035us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.437668 (+   112us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.659202 (+221534us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.754348 (+ 95146us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.754820 (+   472us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.784816  1791 tablet_service.cc:1511] Processing CreateTablet for tablet 5e573c99734f4d91bb30d1fe72b56cf7 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:21:30.802397  1716 tablet_service.cc:1511] Processing CreateTablet for tablet ae12891834034d968900107cc58fa821 (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:21:30.803217  1791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e573c99734f4d91bb30d1fe72b56cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.803932  1716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae12891834034d968900107cc58fa821. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.763547  1720 rpcz_store.cc:276] 0709 12:21:28.904857 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.452922 (+1548065us) service_pool.cc:224] Handling call
0709 12:21:30.521825 (+ 68903us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.522011 (+   186us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.652748 (+130737us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.762116 (+109368us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.762827 (+   711us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.809528  1794 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 50) took 1753 ms. Trace:
I20260709 12:21:30.814397  1794 rpcz_store.cc:276] 0709 12:21:29.056477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.594254 (+1537777us) service_pool.cc:224] Handling call
0709 12:21:30.682038 (+ 87784us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.683279 (+  1241us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.791116 (+107837us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.808949 (+ 17833us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.809452 (+   503us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.829005  1925 tablet_bootstrap.cc:492] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:30.829559  1925 ts_tablet_manager.cc:1403] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.185s	user 0.020s	sys 0.013s
I20260709 12:21:30.834015  1795 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 58) took 1646 ms. Trace:
I20260709 12:21:30.834453  1795 rpcz_store.cc:276] 0709 12:21:29.190578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.777557 (+1586979us) service_pool.cc:224] Handling call
0709 12:21:30.798184 (+ 20627us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.798330 (+   146us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.826459 (+ 28129us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.833493 (+  7034us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.833949 (+   456us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.836349  1925 raft_consensus.cc:359] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.837114  1925 raft_consensus.cc:385] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.837416  1925 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.838649  1871 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 57) took 1688 ms. Trace:
I20260709 12:21:30.839181  1871 rpcz_store.cc:276] 0709 12:21:29.150156 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.708226 (+1558070us) service_pool.cc:224] Handling call
0709 12:21:30.710194 (+  1968us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.710309 (+   115us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.825147 (+114838us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.838079 (+ 12932us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.838584 (+   505us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.838680  1925 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.843765  1925 ts_tablet_manager.cc:1434] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.003s
I20260709 12:21:30.844846  1925 tablet_bootstrap.cc:492] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:30.850085  1927 tablet_bootstrap.cc:492] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:30.851815  1927 ts_tablet_manager.cc:1403] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.243s	user 0.020s	sys 0.014s
I20260709 12:21:30.856328  1925 tablet_bootstrap.cc:654] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:30.856094  1927 raft_consensus.cc:359] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.856863  1927 raft_consensus.cc:385] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.864013  1722 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 54) took 1875 ms. Trace:
I20260709 12:21:30.864454  1722 rpcz_store.cc:276] 0709 12:21:28.988384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.695691 (+1707307us) service_pool.cc:224] Handling call
0709 12:21:30.743245 (+ 47554us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.743359 (+   114us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.795318 (+ 51959us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.863507 (+ 68189us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.863946 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.865995  1864 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 56) took 1756 ms. Trace:
I20260709 12:21:30.866489  1864 rpcz_store.cc:276] 0709 12:21:29.110071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.647626 (+1537555us) service_pool.cc:224] Handling call
0709 12:21:30.649684 (+  2058us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.649782 (+    98us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.791340 (+141558us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.865489 (+ 74149us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.865933 (+   444us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.867245  1718 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 49) took 1962 ms. Trace:
I20260709 12:21:30.867959  1796 tablet_service.cc:1511] Processing CreateTablet for tablet 84a31d3e62ae470d9c01e48ea29fbaee (DEFAULT_TABLE table=test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:21:30.868414  1718 rpcz_store.cc:276] 0709 12:21:28.904434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.451747 (+1547313us) service_pool.cc:224] Handling call
0709 12:21:30.473760 (+ 22013us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.511377 (+ 37617us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.644357 (+132980us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.866639 (+222282us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.867191 (+   552us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.869307  1796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a31d3e62ae470d9c01e48ea29fbaee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:30.871592  1628 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:30.879657  1713 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 55) took 1875 ms. Trace:
I20260709 12:21:30.880096  1713 rpcz_store.cc:276] 0709 12:21:29.004059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.709296 (+1705237us) service_pool.cc:224] Handling call
0709 12:21:30.711570 (+  2274us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.711676 (+   106us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.829272 (+117596us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.879086 (+ 49814us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.879588 (+   502us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.883046  1721 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 51) took 1977 ms. Trace:
I20260709 12:21:30.883387  1863 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:43056 (request call id 54) took 1889 ms. Trace:
I20260709 12:21:30.883898  1863 rpcz_store.cc:276] 0709 12:21:28.994183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.610077 (+1615894us) service_pool.cc:224] Handling call
0709 12:21:30.690124 (+ 80047us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.690337 (+   213us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.862225 (+171888us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.882998 (+ 20773us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.883338 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.883694  1721 rpcz_store.cc:276] 0709 12:21:28.908774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.505055 (+1596281us) service_pool.cc:224] Handling call
0709 12:21:30.539881 (+ 34826us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.539991 (+   110us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.653157 (+113166us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.882253 (+229096us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.882929 (+   676us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.858098  1927 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.875998  1924 raft_consensus.cc:359] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.887578  1927 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.888124  1924 raft_consensus.cc:385] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.888453  1924 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.889457  1924 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.890185  1927 ts_tablet_manager.cc:1434] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.037s	user 0.004s	sys 0.004s
I20260709 12:21:30.890862  1792 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 54) took 1818 ms. Trace:
I20260709 12:21:30.891245  1792 rpcz_store.cc:276] 0709 12:21:29.057833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.452785 (+1394952us) service_pool.cc:224] Handling call
0709 12:21:30.476362 (+ 23577us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.476469 (+   107us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.647218 (+170749us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.861639 (+214421us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.876215 (+ 14576us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.900708  1951 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.901590  1951 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.902567  1719 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 57) took 1765 ms. Trace:
I20260709 12:21:30.903046  1719 rpcz_store.cc:276] 0709 12:21:29.137508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.715598 (+1578090us) service_pool.cc:224] Handling call
0709 12:21:30.717647 (+  2049us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.717753 (+   106us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.895333 (+177580us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.902027 (+  6694us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.902497 (+   470us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.904863  1789 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 47) took 1849 ms. Trace:
W20260709 12:21:30.905048  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:30.905269  1789 rpcz_store.cc:276] 0709 12:21:29.055329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.455591 (+1400262us) service_pool.cc:224] Handling call
0709 12:21:30.495436 (+ 39845us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.495550 (+   114us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.677023 (+181473us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.904346 (+227323us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.904790 (+   444us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.907284  1927 tablet_bootstrap.cc:492] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:30.905187  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:21:30.909541  1770 leader_election.cc:343] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.911898  1924 ts_tablet_manager.cc:1434] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.125s	user 0.008s	sys 0.000s
I20260709 12:21:30.912904  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:30.913591  1927 tablet_bootstrap.cc:654] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:30.915830  1771 leader_election.cc:343] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:30.916539  1771 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:30.917604  1954 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:30.919684  1951 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:30.913039  1924 tablet_bootstrap.cc:492] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:30.926350  1797 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 55) took 1868 ms. Trace:
I20260709 12:21:30.926852  1797 rpcz_store.cc:276] 0709 12:21:29.058157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.704404 (+1646247us) service_pool.cc:224] Handling call
0709 12:21:30.708179 (+  3775us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.708296 (+   117us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.838265 (+129969us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.925766 (+ 87501us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.926284 (+   518us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.929625  1927 tablet_bootstrap.cc:492] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:30.929929  1790 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 51) took 1873 ms. Trace:
I20260709 12:21:30.930393  1927 ts_tablet_manager.cc:1403] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.023s	user 0.005s	sys 0.012s
I20260709 12:21:30.933003  1925 tablet_bootstrap.cc:492] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:30.933794  1925 ts_tablet_manager.cc:1403] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.089s	user 0.034s	sys 0.033s
I20260709 12:21:30.934198  1927 raft_consensus.cc:359] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.935238  1927 raft_consensus.cc:385] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.935560  1927 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.936553  1927 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.937464  1925 raft_consensus.cc:359] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.938113  1716 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 58) took 1763 ms. Trace:
I20260709 12:21:30.938570  1925 raft_consensus.cc:385] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.938961  1925 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.938720  1716 rpcz_store.cc:276] 0709 12:21:29.174640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.801234 (+1626594us) service_pool.cc:224] Handling call
0709 12:21:30.803353 (+  2119us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.803470 (+   117us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.914744 (+111274us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.937118 (+ 22374us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.937738 (+   620us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.939847  1925 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.941074  1924 tablet_bootstrap.cc:654] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:30.941936  1925 ts_tablet_manager.cc:1434] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:21:30.942894  1925 tablet_bootstrap.cc:492] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:30.943558  1714 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48982 (request call id 52) took 2033 ms. Trace:
I20260709 12:21:30.944190  1927 ts_tablet_manager.cc:1434] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260709 12:21:30.930403  1790 rpcz_store.cc:276] 0709 12:21:29.056803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.695042 (+1638239us) service_pool.cc:224] Handling call
0709 12:21:30.750952 (+ 55910us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.751068 (+   116us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.848975 (+ 97907us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.929277 (+ 80302us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.929853 (+   576us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.945360  1927 tablet_bootstrap.cc:492] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:30.944387  1714 rpcz_store.cc:276] 0709 12:21:28.911582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.602692 (+1691110us) service_pool.cc:224] Handling call
0709 12:21:30.604590 (+  1898us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.604687 (+    97us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.921331 (+316644us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.942963 (+ 21632us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.943487 (+   524us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.949940  1925 tablet_bootstrap.cc:654] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:30.959082  1927 tablet_bootstrap.cc:654] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:30.965824  1791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 57) took 1814 ms. Trace:
I20260709 12:21:30.966580  1791 rpcz_store.cc:276] 0709 12:21:29.151215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.783386 (+1632171us) service_pool.cc:224] Handling call
0709 12:21:30.802579 (+ 19193us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.802705 (+   126us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.902244 (+ 99539us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.964816 (+ 62572us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.965713 (+   897us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.970131  1925 tablet_bootstrap.cc:492] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:30.970738  1925 ts_tablet_manager.cc:1403] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.028s	user 0.017s	sys 0.007s
I20260709 12:21:30.971182  1796 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44522 (request call id 56) took 1854 ms. Trace:
I20260709 12:21:30.971590  1796 rpcz_store.cc:276] 0709 12:21:29.117261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.764406 (+1647145us) service_pool.cc:224] Handling call
0709 12:21:30.868830 (+104424us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0709 12:21:30.868924 (+    94us) ts_tablet_manager.cc:682] Creating new metadata...
0709 12:21:30.947134 (+ 78210us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:30.970643 (+ 23509us) tablet_replica.cc:206] Creating consensus instance
0709 12:21:30.971124 (+   481us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.979149  1924 tablet_bootstrap.cc:492] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:30.979673  1924 ts_tablet_manager.cc:1403] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.067s	user 0.008s	sys 0.012s
I20260709 12:21:30.975001  1925 raft_consensus.cc:359] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.981684  1925 raft_consensus.cc:385] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.982098  1925 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.983090  1925 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.982613  1924 raft_consensus.cc:359] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.984159  1924 raft_consensus.cc:385] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:30.984477  1924 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:30.985320  1924 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:30.986002  1925 ts_tablet_manager.cc:1434] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:21:30.987161  1925 tablet_bootstrap.cc:492] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:30.991466  1924 ts_tablet_manager.cc:1434] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:21:30.992648  1924 tablet_bootstrap.cc:492] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:30.994232  1925 tablet_bootstrap.cc:654] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.000227  1927 tablet_bootstrap.cc:492] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.000777  1927 ts_tablet_manager.cc:1403] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.056s	user 0.006s	sys 0.021s
I20260709 12:21:31.003952  1924 tablet_bootstrap.cc:654] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.008817  1925 tablet_bootstrap.cc:492] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:31.009439  1925 ts_tablet_manager.cc:1403] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.000s
I20260709 12:21:31.011910  1924 tablet_bootstrap.cc:492] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:31.012579  1924 ts_tablet_manager.cc:1403] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.020s	user 0.010s	sys 0.002s
I20260709 12:21:31.013420  1925 raft_consensus.cc:359] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.014274  1925 raft_consensus.cc:385] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.013724  1927 raft_consensus.cc:359] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.014673  1925 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.015039  1927 raft_consensus.cc:385] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.015769  1927 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.015686  1925 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.016855  1924 raft_consensus.cc:359] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.017699  1924 raft_consensus.cc:385] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.018087  1924 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.018412  1925 ts_tablet_manager.cc:1434] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20260709 12:21:31.019671  1925 tablet_bootstrap.cc:492] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:31.019232  1924 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.022716  1924 ts_tablet_manager.cc:1434] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.001s
I20260709 12:21:31.024025  1924 tablet_bootstrap.cc:492] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:31.028172  1925 tablet_bootstrap.cc:654] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.031646  1924 tablet_bootstrap.cc:654] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.017253  1927 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.036559  1928 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.037101  1927 ts_tablet_manager.cc:1434] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.036s	user 0.010s	sys 0.000s
I20260709 12:21:31.037317  1928 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.038223  1927 tablet_bootstrap.cc:492] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.041546  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.042117  1928 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:31.042316  1732 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:31.044138  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:31.045367  1927 tablet_bootstrap.cc:654] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.045789  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:31.046479  1807 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:31.048834  1928 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.049469  1928 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.050031  1928 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.053700  1924 tablet_bootstrap.cc:492] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:31.054548  1924 ts_tablet_manager.cc:1403] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.031s	user 0.010s	sys 0.008s
I20260709 12:21:31.059109  1924 raft_consensus.cc:359] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.060216  1924 raft_consensus.cc:385] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.062044  1924 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.065675  1924 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.066673  1928 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:31.070503  1696 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:31.071467  1928 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:31.074319  1927 tablet_bootstrap.cc:492] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.075198  1927 ts_tablet_manager.cc:1403] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.037s	user 0.016s	sys 0.010s
I20260709 12:21:31.076260  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:31.076901  1732 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.079274  1927 raft_consensus.cc:359] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.079457  1936 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:31.080153  1927 raft_consensus.cc:385] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.080504  1927 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.081352  1927 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.083602  1925 tablet_bootstrap.cc:492] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:31.084158  1925 ts_tablet_manager.cc:1403] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.065s	user 0.032s	sys 0.029s
I20260709 12:21:31.085191  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:31.086194  1807 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.086865  1927 ts_tablet_manager.cc:1434] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.001s
I20260709 12:21:31.087924  1927 tablet_bootstrap.cc:492] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.094864  1927 tablet_bootstrap.cc:654] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.095259  1732 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:31.097312  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:31.098381  1928 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.099025  1928 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:31.100163  1925 raft_consensus.cc:359] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.103716  1925 raft_consensus.cc:385] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.104142  1925 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.100149  1928 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.105208  1925 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.107158  1807 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:31.108213  1925 ts_tablet_manager.cc:1434] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:21:31.111457  1924 ts_tablet_manager.cc:1434] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.056s	user 0.019s	sys 0.011s
I20260709 12:21:31.112948  1924 tablet_bootstrap.cc:492] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:31.116603  1951 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.117177  1951 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.118090  1927 tablet_bootstrap.cc:492] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.118417  1954 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.118753  1927 ts_tablet_manager.cc:1403] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.031s	user 0.017s	sys 0.001s
I20260709 12:21:31.118985  1954 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.121004  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:31.121282  1954 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.121740  1880 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.121930  1927 raft_consensus.cc:359] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.122090  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.122648  1927 raft_consensus.cc:385] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.123193  1925 tablet_bootstrap.cc:492] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:31.122984  1731 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.123646  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:31.125029  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:31.125381  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:31.126190  1880 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.119477  1951 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.127364  1951 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.127740  1951 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.128078  1951 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.129092  1924 tablet_bootstrap.cc:654] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.131004  1954 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.131520  1954 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.131872  1954 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.130870  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.133015  1732 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.135214  1925 tablet_bootstrap.cc:654] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.123013  1927 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.147439  1925 tablet_bootstrap.cc:492] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:31.147933  1925 ts_tablet_manager.cc:1403] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.025s	user 0.007s	sys 0.004s
I20260709 12:21:31.148275  1927 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.151796  1924 tablet_bootstrap.cc:492] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:31.152354  1924 ts_tablet_manager.cc:1403] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.040s	user 0.021s	sys 0.009s
I20260709 12:21:31.154156  1925 raft_consensus.cc:359] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:31.154863  1925 raft_consensus.cc:385] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.155128  1925 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.155280  1924 raft_consensus.cc:359] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.156152  1924 raft_consensus.cc:385] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.156574  1924 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.157713  1924 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.158393  1925 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:21:31.160339  1924 ts_tablet_manager.cc:1434] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:21:31.160823  1927 ts_tablet_manager.cc:1434] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.042s	user 0.004s	sys 0.003s
I20260709 12:21:31.162247  1924 tablet_bootstrap.cc:492] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:31.168984  1627 catalog_manager.cc:5697] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:31.170594  1924 tablet_bootstrap.cc:654] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.176262  1925 ts_tablet_manager.cc:1434] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260709 12:21:31.176307  1954 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.178689  1954 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.176266  1951 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.179884  1927 tablet_bootstrap.cc:492] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.180832  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:31.181452  1951 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.181529  1880 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.183182  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:31.183790  1732 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.188722  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:31.189652  1731 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.192708  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:31.193594  1881 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.199312  1924 tablet_bootstrap.cc:492] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:31.199883  1924 ts_tablet_manager.cc:1403] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.038s	user 0.007s	sys 0.006s
I20260709 12:21:31.202924  1924 raft_consensus.cc:359] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.203698  1924 raft_consensus.cc:385] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.203984  1924 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.205646  1925 tablet_bootstrap.cc:492] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:31.211165  1924 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.213380  1928 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.216665  1928 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.219066  1925 tablet_bootstrap.cc:654] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.224706  1927 tablet_bootstrap.cc:654] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.224502  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:31.225297  1806 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:31.225870  1928 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:31.226392  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.227934  1730 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:31.229521  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:31.230342  1731 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.231189  1928 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.231673  1928 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.213953  1732 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.231837  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:31.232098  1928 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.233126  1880 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.233736  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:31.234301  1954 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.234596  1951 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.236305  1984 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.236967  1924 ts_tablet_manager.cc:1434] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.035s	user 0.008s	sys 0.000s
I20260709 12:21:31.237927  1924 tablet_bootstrap.cc:492] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:31.236987  1951 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.234994  1881 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.245699  1951 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.248500  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:31.254065  1880 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.255015  1924 tablet_bootstrap.cc:654] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.255292  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:31.258080  1925 tablet_bootstrap.cc:492] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:31.258531  1925 ts_tablet_manager.cc:1403] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.055s	user 0.010s	sys 0.013s
I20260709 12:21:31.256170  1951 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.265311  1925 raft_consensus.cc:359] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.258224  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.256474  1954 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:31.267359  1984 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:31.266963  1925 raft_consensus.cc:385] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.271380  1925 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.267067  1951 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.272817  1925 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.272481  1731 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.273422  1951 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.275632  1925 ts_tablet_manager.cc:1434] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.002s
I20260709 12:21:31.276628  1925 tablet_bootstrap.cc:492] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:31.270720  1954 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.282902  1925 tablet_bootstrap.cc:654] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.270773  1984 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.286873  1924 tablet_bootstrap.cc:492] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:31.287685  1924 ts_tablet_manager.cc:1403] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.050s	user 0.010s	sys 0.008s
I20260709 12:21:31.289345  1927 tablet_bootstrap.cc:492] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.289829  1927 ts_tablet_manager.cc:1403] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.110s	user 0.012s	sys 0.003s
I20260709 12:21:31.286549  1951 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.292878  1924 raft_consensus.cc:359] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.293769  1924 raft_consensus.cc:385] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.294090  1924 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.279549  1928 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.294004  1927 raft_consensus.cc:359] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.295583  1927 raft_consensus.cc:385] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.295463  1924 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.296046  1927 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.298058  1924 ts_tablet_manager.cc:1434] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:21:31.297487  1927 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.299846  1927 ts_tablet_manager.cc:1434] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.004s
I20260709 12:21:31.301031  1927 tablet_bootstrap.cc:492] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.301748  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:31.302662  1731 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.303989  1924 tablet_bootstrap.cc:492] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:31.315894  1928 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:31.317334  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:31.318102  1807 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.326071  1925 tablet_bootstrap.cc:492] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:31.326406  1807 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:31.326653  1925 ts_tablet_manager.cc:1403] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.050s	user 0.023s	sys 0.018s
I20260709 12:21:31.326968  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:31.327642  1732 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.328015  1845 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:31.329131  1928 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.329860  1928 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:31.330013  1925 raft_consensus.cc:359] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.330767  1925 raft_consensus.cc:385] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.331168  1925 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.330854  1928 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.332010  1925 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.333760  1994 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:31.335263  1730 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.335409  1924 tablet_bootstrap.cc:654] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.339490  1951 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.340413  1925 ts_tablet_manager.cc:1434] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 12:21:31.341506  1925 tablet_bootstrap.cc:492] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:31.342406  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:31.343146  1881 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.348207  1925 tablet_bootstrap.cc:654] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.354003  1927 tablet_bootstrap.cc:654] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.365044  1881 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.368464  1630 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.370750  1954 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:31.371090  1731 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:31.378749  1732 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.384272  1983 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.397995  1983 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.401136  1983 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.385033  1927 tablet_bootstrap.cc:492] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.402681  1927 ts_tablet_manager.cc:1403] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.102s	user 0.010s	sys 0.005s
I20260709 12:21:31.404274  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:31.404994  1807 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:31.406023  1927 raft_consensus.cc:359] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.406731  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:31.407213  1927 raft_consensus.cc:385] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.407708  1927 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.408334  1987 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.408908  1987 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.409580  1987 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.410434  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:31.411757  1954 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.408833  1927 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.413558  1954 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:31.415652  1927 ts_tablet_manager.cc:1434] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:21:31.417065  1927 tablet_bootstrap.cc:492] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.426268  1927 tablet_bootstrap.cc:654] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.439002  1927 tablet_bootstrap.cc:492] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.439571  1927 ts_tablet_manager.cc:1403] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.023s	user 0.010s	sys 0.011s
I20260709 12:21:31.440053  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:31.441090  1880 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:31.443312  1927 raft_consensus.cc:359] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.414878  1954 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.444367  1927 raft_consensus.cc:385] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.444821  1927 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.446182  1927 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.449559  1927 ts_tablet_manager.cc:1434] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:21:31.450613  1927 tablet_bootstrap.cc:492] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.457646  1927 tablet_bootstrap.cc:654] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.459906  1951 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.460646  1951 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.466673  1951 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.467052  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.468272  1732 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 1: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:21:31.475598  1927 tablet_bootstrap.cc:492] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.476405  1927 ts_tablet_manager.cc:1403] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.007s
I20260709 12:21:31.480158  1927 raft_consensus.cc:359] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.483238  1927 raft_consensus.cc:385] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.483659  1927 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.484596  1927 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.487485  1927 ts_tablet_manager.cc:1434] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:21:31.488879  1927 tablet_bootstrap.cc:492] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.486637  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:31.496212  1880 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:21:31.497234  1927 tablet_bootstrap.cc:654] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.510448  1927 tablet_bootstrap.cc:492] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.511252  1927 ts_tablet_manager.cc:1403] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.023s	user 0.017s	sys 0.004s
I20260709 12:21:31.515796  1927 raft_consensus.cc:359] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.516811  1927 raft_consensus.cc:385] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.517210  1927 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.518290  1927 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.521417  1927 ts_tablet_manager.cc:1434] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:21:31.522691  1927 tablet_bootstrap.cc:492] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.438069  1627 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:31.533212  1927 tablet_bootstrap.cc:654] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.533053  1928 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:31.537767  1951 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.538444  1951 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.539206  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:31.540176  1954 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.543170  1951 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.545871  1924 tablet_bootstrap.cc:492] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:31.546502  1924 ts_tablet_manager.cc:1403] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.243s	user 0.080s	sys 0.094s
I20260709 12:21:31.548811  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.554318  1731 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.556001  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:31.557087  1951 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.557525  1951 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.557919  1951 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.559301  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:31.566356  1925 tablet_bootstrap.cc:492] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:31.562741  1928 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:21:31.567552  1925 ts_tablet_manager.cc:1403] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.226s	user 0.069s	sys 0.138s
I20260709 12:21:31.570686  1925 raft_consensus.cc:359] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.565871  1951 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.571556  1925 raft_consensus.cc:385] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.571890  1925 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.571203  1628 catalog_manager.cc:5697] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.562651  1924 raft_consensus.cc:359] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.572563  1925 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.573417  1924 raft_consensus.cc:385] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.573117  1628 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.574156  1924 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.574288  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:31.575026  1732 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.576354  1925 ts_tablet_manager.cc:1434] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:21:31.577422  1925 tablet_bootstrap.cc:492] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:31.581928  1983 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.583204  1924 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.566872  1987 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.586611  1880 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.588400  1925 tablet_bootstrap.cc:654] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.589083  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:31.589814  1807 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.592399  1924 ts_tablet_manager.cc:1434] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.036s	user 0.008s	sys 0.000s
I20260709 12:21:31.594221  1951 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.594599  1924 tablet_bootstrap.cc:492] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:31.591445  1983 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.600602  1983 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.603636  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:31.606077  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:21:31.606648  1696 leader_election.cc:343] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:31.608150  1927 tablet_bootstrap.cc:492] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.607316  1806 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:31.612623  1951 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.615728  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:31.616436  1880 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.616309  1951 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.618597  1951 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.618688  1987 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.620430  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.566810  1954 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:31.621142  1730 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.621106  1732 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.622720  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:31.623158  1770 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:31.623482  1881 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.624221  1951 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.624603  1951 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.624966  1951 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.625669  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:31.627478  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:31.628075  1881 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.633723  1927 ts_tablet_manager.cc:1403] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.111s	user 0.011s	sys 0.017s
I20260709 12:21:31.636139  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:31.640386  1879 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.643608  1925 tablet_bootstrap.cc:492] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:31.644167  1925 ts_tablet_manager.cc:1403] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.067s	user 0.025s	sys 0.002s
I20260709 12:21:31.647159  1880 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.645669  1881 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:31.646106  1807 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:31.649533  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:31.650399  1983 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.651031  1983 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:31.651777  1983 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.644603  1987 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.646304  1954 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.658315  1951 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.654974  1987 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.663177  1987 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.664652  1951 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.666049  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:31.666759  1881 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.672262  1954 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:31.673331  1954 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.673868  1987 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.675350  1951 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:31.667095  1927 raft_consensus.cc:359] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.678287  1927 raft_consensus.cc:385] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.678721  1927 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.680066  1927 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.667958  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:31.685848  1732 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.688169  1927 ts_tablet_manager.cc:1434] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.054s	user 0.004s	sys 0.004s
I20260709 12:21:31.692339  1925 raft_consensus.cc:359] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.693318  1925 raft_consensus.cc:385] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.693637  1925 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.694343  1925 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.695425  1927 tablet_bootstrap.cc:492] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.702584  1924 tablet_bootstrap.cc:654] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.713712  1925 ts_tablet_manager.cc:1434] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.069s	user 0.009s	sys 0.001s
I20260709 12:21:31.717303  1925 tablet_bootstrap.cc:492] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:31.718576  1732 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.721340  1770 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:31.723384  1951 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.724098  1951 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:31.727108  1881 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.727730  1951 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.731268  1630 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.731048  1987 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.735368  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:31.736104  1807 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:31.736433  2017 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.003s
I20260709 12:21:31.740526  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:21:31.742525  1696 leader_election.cc:343] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:31.759774  1807 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:31.761510  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:31.763926  1927 tablet_bootstrap.cc:654] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.764293  1987 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.777832  1630 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.781450  1924 tablet_bootstrap.cc:492] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:31.781875  1924 ts_tablet_manager.cc:1403] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.188s	user 0.019s	sys 0.006s
I20260709 12:21:31.781981  1925 tablet_bootstrap.cc:654] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.785004  1924 raft_consensus.cc:359] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.786085  1924 raft_consensus.cc:385] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.786343  1924 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.779476  1987 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:31.787190  1924 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.791373  1924 ts_tablet_manager.cc:1434] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:21:31.792693  1924 tablet_bootstrap.cc:492] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:31.798408  1927 tablet_bootstrap.cc:492] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.799032  1927 ts_tablet_manager.cc:1403] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.104s	user 0.026s	sys 0.024s
I20260709 12:21:31.802043  1927 raft_consensus.cc:359] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.802769  1927 raft_consensus.cc:385] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.803061  1927 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.803046  1984 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:31.804097  1927 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.804600  1627 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.806902  1927 ts_tablet_manager.cc:1434] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:21:31.807914  1927 tablet_bootstrap.cc:492] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.807976  1987 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.819162  2016 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:31.820973  1924 tablet_bootstrap.cc:654] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.823769  1927 tablet_bootstrap.cc:654] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.830323  1925 tablet_bootstrap.cc:492] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:31.830828  1925 ts_tablet_manager.cc:1403] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.114s	user 0.009s	sys 0.007s
I20260709 12:21:31.834007  1925 raft_consensus.cc:359] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.834683  1925 raft_consensus.cc:385] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.835219  1951 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.836004  1951 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.838012  1924 tablet_bootstrap.cc:492] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:31.838502  1924 ts_tablet_manager.cc:1403] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.046s	user 0.011s	sys 0.002s
I20260709 12:21:31.835227  1925 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.840408  1925 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.841876  1925 ts_tablet_manager.cc:1434] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.000s
I20260709 12:21:31.842602  1925 tablet_bootstrap.cc:492] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:31.844363  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.838757  1951 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.845335  1984 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:21:31.846999  1770 leader_election.cc:343] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:31.847784  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:31.851886  1924 raft_consensus.cc:359] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.852432  1924 raft_consensus.cc:385] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.852654  1924 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.854176  1924 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.855329  1879 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.856124  1927 tablet_bootstrap.cc:492] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.856573  1924 ts_tablet_manager.cc:1434] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.000s
I20260709 12:21:31.856659  1927 ts_tablet_manager.cc:1403] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.049s	user 0.019s	sys 0.003s
I20260709 12:21:31.863926  1924 tablet_bootstrap.cc:492] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:31.857156  1771 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:31.867172  1984 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.867683  1984 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.868024  1984 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.864940  1630 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.882207  1979 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:31.882381  1927 raft_consensus.cc:359] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.883258  1924 tablet_bootstrap.cc:654] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.884905  1984 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.883273  1927 raft_consensus.cc:385] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.888311  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:31.888890  1927 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.888924  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:31.889621  1879 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:31.890064  1770 leader_election.cc:343] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:31.896586  1951 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:31.890094  1984 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.889803  1927 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.900415  1927 ts_tablet_manager.cc:1434] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.037s	user 0.003s	sys 0.004s
I20260709 12:21:31.901475  1927 tablet_bootstrap.cc:492] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.912004  1879 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:31.913983  1771 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:31.914959  1994 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:31.915426  1994 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:31.916349  1994 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.918071  1925 tablet_bootstrap.cc:654] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:31.921379  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:31.928876  1927 tablet_bootstrap.cc:654] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.941250  1925 tablet_bootstrap.cc:492] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:31.941730  1925 ts_tablet_manager.cc:1403] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.099s	user 0.005s	sys 0.009s
I20260709 12:21:31.944424  1924 tablet_bootstrap.cc:492] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:31.944985  1924 ts_tablet_manager.cc:1403] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.081s	user 0.024s	sys 0.021s
I20260709 12:21:31.945792  1925 raft_consensus.cc:359] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.946539  1925 raft_consensus.cc:385] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.946823  1925 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.947464  1925 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.948165  1924 raft_consensus.cc:359] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.949153  1924 raft_consensus.cc:385] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.949491  1924 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.949844  1927 tablet_bootstrap.cc:492] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:31.950312  1927 ts_tablet_manager.cc:1403] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.049s	user 0.013s	sys 0.000s
I20260709 12:21:31.950964  1924 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.952582  1925 ts_tablet_manager.cc:1434] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.001s
I20260709 12:21:31.952474  1630 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:31.954919  1924 ts_tablet_manager.cc:1434] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:21:31.955662  1925 tablet_bootstrap.cc:492] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:31.958683  1924 tablet_bootstrap.cc:492] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:31.962461  1927 raft_consensus.cc:359] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.963167  1927 raft_consensus.cc:385] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:31.963438  1927 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:31.964098  1927 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.964501  1951 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.965157  1951 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:31.966215  1927 ts_tablet_manager.cc:1434] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.016s	user 0.003s	sys 0.000s
I20260709 12:21:31.967365  1927 tablet_bootstrap.cc:492] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:31.967600  1925 tablet_bootstrap.cc:654] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.967712  1951 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:31.970361  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:31.973001  1771 leader_election.cc:343] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:31.974431  1924 tablet_bootstrap.cc:654] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:31.977169  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:31.978114  1732 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:31.979980  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:31.981395  1951 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:31.981743  1951 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.982075  1951 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:31.984098  1994 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:31.996824  1927 tablet_bootstrap.cc:654] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.001242  1951 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.002732  1994 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.007364  1994 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.010318  1994 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.011446  1994 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:32.013037  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:32.005062  1927 tablet_bootstrap.cc:492] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:32.014233  1927 ts_tablet_manager.cc:1403] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.047s	user 0.015s	sys 0.000s
I20260709 12:21:32.017673  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:32.018402  1731 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.020481  1927 raft_consensus.cc:359] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.021552  1927 raft_consensus.cc:385] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.021904  1927 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.015628  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:32.016731  1770 leader_election.cc:343] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.024996  1925 tablet_bootstrap.cc:492] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:32.025517  1925 ts_tablet_manager.cc:1403] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.070s	user 0.011s	sys 0.015s
I20260709 12:21:32.026208  1951 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.026392  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:32.026916  1979 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:32.025382  1771 leader_election.cc:343] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.027371  1927 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.027801  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:32.027479  1979 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.028939  1951 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:32.031348  1924 tablet_bootstrap.cc:492] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:32.031713  1927 ts_tablet_manager.cc:1434] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:21:32.031329  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:32.032215  1924 ts_tablet_manager.cc:1403] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.074s	user 0.020s	sys 0.011s
I20260709 12:21:32.032740  1927 tablet_bootstrap.cc:492] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:32.033274  1925 raft_consensus.cc:359] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:32.033679  1844 leader_election.cc:343] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.035283  1979 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:21:32.036082  1771 leader_election.cc:343] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.035069  1924 raft_consensus.cc:359] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.034134  1925 raft_consensus.cc:385] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.036773  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:32.038187  1925 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.038398  1807 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:32.039517  1987 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.039222  1925 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.040109  1987 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.040338  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:32.041308  1979 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:32.041764  1979 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.041873  1925 ts_tablet_manager.cc:1434] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.003s
I20260709 12:21:32.042266  1979 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.042562  1987 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.043112  1925 tablet_bootstrap.cc:492] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:32.044327  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:32.039687  1927 tablet_bootstrap.cc:654] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.050400  1979 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:32.052465  1696 leader_election.cc:343] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.053475  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:21:32.054935  1844 leader_election.cc:343] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.053388  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:32.055256  1731 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:32.056141  1925 tablet_bootstrap.cc:654] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.056815  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:32.057595  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:32.057696  1951 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:32.058511  1807 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.058701  1951 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:32.059594  1951 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.063185  1979 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:32.067027  1807 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:32.068732  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:32.069753  1979 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:32.070734  1979 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:32.071575  1979 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.037963  1924 raft_consensus.cc:385] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.072948  1924 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.074244  1924 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.075507  1987 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:32.075701  1881 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:32.077350  1924 ts_tablet_manager.cc:1434] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.000s
I20260709 12:21:32.079811  1924 tablet_bootstrap.cc:492] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:32.081877  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:32.086838  1983 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:32.087389  1983 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.087793  1983 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.143329  1983 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.145372  1983 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.147312  1630 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:32.149931  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:32.151134  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:32.151714  1879 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.152760  1994 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.153319  1994 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:32.156471  1696 leader_election.cc:343] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.155166  1954 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:21:32.160017  1994 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.162227  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:32.163733  1927 tablet_bootstrap.cc:492] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:32.164489  1927 ts_tablet_manager.cc:1403] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.132s	user 0.015s	sys 0.014s
I20260709 12:21:32.166049  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:32.166890  1878 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:32.168493  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:21:32.169637  1770 leader_election.cc:343] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.170948  1994 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:32.171509  1994 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.171828  1994 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.176822  1879 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:32.178727  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:32.179986  1983 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:32.179592  1627 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:32.180727  1983 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:32.181602  1983 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.190120  1954 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:32.189217  1924 tablet_bootstrap.cc:654] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.201623  1927 raft_consensus.cc:359] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.202425  1927 raft_consensus.cc:385] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.202703  1927 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.210305  1936 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:32.211517  1994 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.236848  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:21:32.239964  1770 leader_election.cc:343] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.240876  1994 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.252781  1987 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:21:32.259994  1696 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:32.259526  1927 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.262204  1927 ts_tablet_manager.cc:1434] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.097s	user 0.010s	sys 0.003s
I20260709 12:21:32.263135  1927 tablet_bootstrap.cc:492] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:32.268731  1951 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:32.269729  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:32.270231  1925 tablet_bootstrap.cc:492] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:32.270395  1878 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.270823  1925 ts_tablet_manager.cc:1403] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.228s	user 0.061s	sys 0.063s
I20260709 12:21:32.271667  1979 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.276746  1994 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.272183  1979 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.277297  1994 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.264765  1627 catalog_manager.cc:5697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:32.282636  1979 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:32.288892  1994 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.291004  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:32.291340  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:32.292191  1731 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
W20260709 12:21:32.292866  1771 leader_election.cc:343] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.295492  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:32.296495  1732 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:32.298314  1770 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:32.300012  1994 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:32.300407  1994 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.300743  1994 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.306668  1924 tablet_bootstrap.cc:492] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:32.307255  1924 ts_tablet_manager.cc:1403] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.229s	user 0.016s	sys 0.012s
I20260709 12:21:32.308831  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:32.310663  1844 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:32.311100  1805 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:32.313987  1987 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.314343  1925 raft_consensus.cc:359] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.315169  1925 raft_consensus.cc:385] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.314988  1987 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.315518  1925 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.316443  1925 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.318487  1925 ts_tablet_manager.cc:1434] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.047s	user 0.005s	sys 0.000s
I20260709 12:21:32.319444  1925 tablet_bootstrap.cc:492] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:32.319144  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:32.319995  1805 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:32.321743  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:32.324745  1696 leader_election.cc:343] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.326403  1979 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:32.326982  1979 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.326431  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:32.327438  1979 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.326470  1878 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:32.335613  2048 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:32.343070  2048 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.343804  2048 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.329560  1987 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.395648  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:32.403025  1984 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:32.403606  1984 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:32.404780  1984 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.401181  1927 tablet_bootstrap.cc:654] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.394376  1979 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.424713  1925 tablet_bootstrap.cc:654] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.426646  1994 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.428030  2055 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:32.431005  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:32.431668  1732 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.446542  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:32.447646  1731 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.453455  1924 raft_consensus.cc:359] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.455411  1979 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:32.454660  1924 raft_consensus.cc:385] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.457425  1924 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.458498  1924 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.469776  2048 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.470196  1731 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:21:32.453078  1994 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.474892  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:32.485411  1806 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:32.476440  1924 ts_tablet_manager.cc:1434] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.169s	user 0.010s	sys 0.000s
I20260709 12:21:32.487589  1924 tablet_bootstrap.cc:492] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
W20260709 12:21:32.475970  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 80434.5us
I20260709 12:21:32.499114  1844 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:32.501142  1979 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:32.504819  2048 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.505559  1927 tablet_bootstrap.cc:492] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:32.514696  1979 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:32.515872  1927 ts_tablet_manager.cc:1403] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.253s	user 0.010s	sys 0.008s
I20260709 12:21:32.507306  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:32.508167  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:32.508646  1806 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:21:32.498478  1732 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:32.511900  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:32.518545  1807 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.519011  1979 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:32.519558  1696 leader_election.cc:343] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.526082  1770 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:32.528364  2057 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:32.528590  1925 tablet_bootstrap.cc:492] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:32.528775  1994 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:32.529378  1925 ts_tablet_manager.cc:1403] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.210s	user 0.033s	sys 0.032s
I20260709 12:21:32.529044  2057 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:32.531936  1927 raft_consensus.cc:359] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.533003  1927 raft_consensus.cc:385] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.533252  1927 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.535777  2057 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.537034  1927 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.538317  1925 raft_consensus.cc:359] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.539130  1925 raft_consensus.cc:385] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.539463  1925 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.533861  1629 catalog_manager.cc:5697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:32.540266  1807 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:32.540231  1925 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.541718  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:32.542609  1927 ts_tablet_manager.cc:1434] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.026s	user 0.004s	sys 0.002s
I20260709 12:21:32.543257  2048 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:32.543754  1927 tablet_bootstrap.cc:492] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:32.543864  2048 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:32.545351  2048 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:32.539937  1771 leader_election.cc:343] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.542609  1925 ts_tablet_manager.cc:1434] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.001s
I20260709 12:21:32.554546  1925 tablet_bootstrap.cc:492] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:32.558307  1924 tablet_bootstrap.cc:654] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.577402  1925 tablet_bootstrap.cc:654] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.587471  1984 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.591750  1984 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.595120  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:32.595788  1730 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:32.597278  1927 tablet_bootstrap.cc:654] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.597313  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:32.589843  1630 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:32.598505  1994 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:32.602248  1994 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.601949  2061 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:21:32.602631  1994 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.606865  2048 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.607470  2048 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.609922  2048 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.610891  1629 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 2, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:32.612463  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:32.613222  1807 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:21:32.613677  1770 consensus_peers.cc:597] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:32.616844  1730 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.619015  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:32.619215  1627 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:32.620508  1696 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:21:32.622370  1771 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:32.624543  1984 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.616892  1994 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.631596  1924 tablet_bootstrap.cc:492] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:32.632292  1924 ts_tablet_manager.cc:1403] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.145s	user 0.020s	sys 0.012s
I20260709 12:21:32.630843  1984 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:21:32.639106  2048 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:32.640009  1994 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.641871  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:32.643098  1732 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.648739  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:32.649650  1879 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:32.660352  2061 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:32.657495  1732 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:32.665278  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:32.667639  1924 raft_consensus.cc:359] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.668370  1924 raft_consensus.cc:385] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.668634  1924 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.668948  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:32.669281  1924 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.669627  1879 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.671183  1927 tablet_bootstrap.cc:492] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:32.671455  1924 ts_tablet_manager.cc:1434] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260709 12:21:32.671730  1927 ts_tablet_manager.cc:1403] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.128s	user 0.018s	sys 0.005s
I20260709 12:21:32.672358  1924 tablet_bootstrap.cc:492] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:32.677485  1927 raft_consensus.cc:359] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.678143  1927 raft_consensus.cc:385] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.678505  1927 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.676626  1984 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:32.684132  1927 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.686194  1927 ts_tablet_manager.cc:1434] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.000s
I20260709 12:21:32.683302  1984 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:32.711997  1984 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.721522  1925 tablet_bootstrap.cc:492] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:32.722416  1925 ts_tablet_manager.cc:1403] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.168s	user 0.044s	sys 0.009s
I20260709 12:21:32.722633  1927 tablet_bootstrap.cc:492] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:32.723368  1924 tablet_bootstrap.cc:654] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.718586  1879 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:32.726383  1925 raft_consensus.cc:359] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.727439  1925 raft_consensus.cc:385] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.727842  1925 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.729403  1927 tablet_bootstrap.cc:654] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.728938  1925 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.738030  1925 ts_tablet_manager.cc:1434] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:21:32.739539  1925 tablet_bootstrap.cc:492] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:32.757638  1927 tablet_bootstrap.cc:492] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:32.759389  1927 ts_tablet_manager.cc:1403] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.037s	user 0.010s	sys 0.007s
I20260709 12:21:32.762738  1983 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:32.777626  1627 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:32.783455  1927 raft_consensus.cc:359] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.784320  1927 raft_consensus.cc:385] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.784667  1927 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.785422  1927 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.787817  1927 ts_tablet_manager.cc:1434] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.028s	user 0.008s	sys 0.000s
I20260709 12:21:32.788892  1927 tablet_bootstrap.cc:492] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:32.797026  1925 tablet_bootstrap.cc:654] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.797978  1924 tablet_bootstrap.cc:492] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:32.798534  1924 ts_tablet_manager.cc:1403] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.126s	user 0.014s	sys 0.009s
I20260709 12:21:32.805868  1927 tablet_bootstrap.cc:654] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.807041  1924 raft_consensus.cc:359] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.807842  1924 raft_consensus.cc:385] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.808151  1924 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.808993  1924 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.813282  1924 ts_tablet_manager.cc:1434] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.014s	user 0.005s	sys 0.001s
I20260709 12:21:32.814363  1924 tablet_bootstrap.cc:492] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:32.814426  1925 tablet_bootstrap.cc:492] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:32.815114  1925 ts_tablet_manager.cc:1403] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.076s	user 0.019s	sys 0.000s
I20260709 12:21:32.815127  1927 tablet_bootstrap.cc:492] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:32.816126  1927 ts_tablet_manager.cc:1403] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.000s
I20260709 12:21:32.817155  1806 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.818305  1925 raft_consensus.cc:359] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.819136  1925 raft_consensus.cc:385] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.818979  1927 raft_consensus.cc:359] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.819722  1927 raft_consensus.cc:385] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.820051  1927 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.819591  1925 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.820897  1925 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.820945  1927 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.825752  1925 ts_tablet_manager.cc:1434] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 12:21:32.826907  1925 tablet_bootstrap.cc:492] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:32.830160  1924 tablet_bootstrap.cc:654] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.833616  1925 tablet_bootstrap.cc:654] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.837255  1927 ts_tablet_manager.cc:1434] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260709 12:21:32.838335  1927 tablet_bootstrap.cc:492] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:32.840159  1984 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.840883  1984 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.845082  1927 tablet_bootstrap.cc:654] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.855633  1925 tablet_bootstrap.cc:492] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:32.856184  1925 ts_tablet_manager.cc:1403] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.030s	user 0.020s	sys 0.007s
I20260709 12:21:32.856529  1924 tablet_bootstrap.cc:492] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:32.857232  1924 ts_tablet_manager.cc:1403] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.043s	user 0.017s	sys 0.000s
I20260709 12:21:32.859112  1925 raft_consensus.cc:359] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.860003  1925 raft_consensus.cc:385] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.860386  1925 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.861207  1925 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.861917  1924 raft_consensus.cc:359] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.862950  1924 raft_consensus.cc:385] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.863301  1924 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.864634  1984 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.864420  1924 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.863221  1925 ts_tablet_manager.cc:1434] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:21:32.866268  1925 tablet_bootstrap.cc:492] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:32.866127  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:32.866750  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:32.867442  1730 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:32.868862  1924 ts_tablet_manager.cc:1434] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.011s	user 0.011s	sys 0.000s
I20260709 12:21:32.868791  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:32.869884  1984 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:32.870316  1984 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.870018  1924 tablet_bootstrap.cc:492] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
W20260709 12:21:32.870649  1771 leader_election.cc:343] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.870674  1984 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.873353  1925 tablet_bootstrap.cc:654] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.874030  2048 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:32.877696  1927 tablet_bootstrap.cc:492] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:32.878435  1924 tablet_bootstrap.cc:654] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.878230  1927 ts_tablet_manager.cc:1403] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.040s	user 0.008s	sys 0.012s
I20260709 12:21:32.907912  1994 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.908780  1994 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.915853  1730 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.920990  1879 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.926615  1994 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:32.929570  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:32.929692  1927 raft_consensus.cc:359] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.930897  1927 raft_consensus.cc:385] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.931298  1927 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
W20260709 12:21:32.932135  1771 leader_election.cc:343] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.932256  1927 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.934639  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:21:32.938287  1770 leader_election.cc:343] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.944098  1770 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:32.946318  2057 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:32.949340  1806 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:21:32.953047  2061 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:32.956043  1984 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.959870  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:32.960589  1732 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:32.970213  1925 tablet_bootstrap.cc:492] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:32.970865  1925 ts_tablet_manager.cc:1403] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.105s	user 0.037s	sys 0.036s
I20260709 12:21:32.974367  1925 raft_consensus.cc:359] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.975133  1925 raft_consensus.cc:385] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:32.975462  1925 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:32.976292  1925 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:32.978667  1925 ts_tablet_manager.cc:1434] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:21:32.980260  1925 tablet_bootstrap.cc:492] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:32.986336  1925 tablet_bootstrap.cc:654] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:32.991972  1732 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:32.996542  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:32.997421  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:21:32.998689  1771 leader_election.cc:343] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:32.998805  2057 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:32.999190  1927 ts_tablet_manager.cc:1434] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.120s	user 0.010s	sys 0.000s
I20260709 12:21:32.999385  2057 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:32.993979  1924 tablet_bootstrap.cc:492] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.000340  1927 tablet_bootstrap.cc:492] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:33.000897  1924 ts_tablet_manager.cc:1403] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.131s	user 0.048s	sys 0.072s
I20260709 12:21:33.002606  1951 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:33.003823  1924 raft_consensus.cc:359] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.004572  1924 raft_consensus.cc:385] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.004842  1924 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.000375  2057 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.005980  1924 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.009909  1924 ts_tablet_manager.cc:1434] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.001s
I20260709 12:21:33.011320  1924 tablet_bootstrap.cc:492] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:32.995872  1984 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.015751  1730 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:21:33.017554  1928 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:33.018749  1924 tablet_bootstrap.cc:654] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.019971  1925 tablet_bootstrap.cc:492] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.027559  1925 ts_tablet_manager.cc:1403] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.047s	user 0.015s	sys 0.002s
I20260709 12:21:33.033015  1925 raft_consensus.cc:359] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.033824  1925 raft_consensus.cc:385] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.034116  1925 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.034897  1925 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.038938  1925 ts_tablet_manager.cc:1434] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 12:21:33.040063  1925 tablet_bootstrap.cc:492] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.041934  1927 tablet_bootstrap.cc:654] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.044417  2048 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:33.055667  1984 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:33.056831  2048 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.057686  2048 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.066763  1730 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.070259  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:33.061591  2048 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.073055  1629 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:33.075266  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.078593  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:33.105098  1928 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.105669  1928 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.108125  1928 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:33.111999  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:33.115801  1984 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:33.123761  1881 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.133365  2062 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.132563  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.139608  1924 tablet_bootstrap.cc:492] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.139628  1994 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.140107  1924 ts_tablet_manager.cc:1403] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.129s	user 0.007s	sys 0.016s
I20260709 12:21:33.140441  1994 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.136117  1927 tablet_bootstrap.cc:492] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:33.142211  1927 ts_tablet_manager.cc:1403] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.142s	user 0.009s	sys 0.027s
I20260709 12:21:33.143436  1925 tablet_bootstrap.cc:654] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.146489  1924 raft_consensus.cc:359] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.147507  1924 raft_consensus.cc:385] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.147902  1924 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.136116  2062 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.154594  1994 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.155681  1983 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.160284  1983 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.162951  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:33.163646  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:33.163955  1732 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:33.164628  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.162635  1983 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.167230  1770 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:33.167266  1807 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:33.158074  1924 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.169390  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:33.174917  1951 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:33.168113  2062 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:33.179132  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:33.184502  2057 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:33.187388  1731 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:21:33.185562  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:21:33.189817  1696 leader_election.cc:343] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.191918  1924 ts_tablet_manager.cc:1434] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.051s	user 0.008s	sys 0.001s
I20260709 12:21:33.192941  1924 tablet_bootstrap.cc:492] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:33.179414  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:33.198742  1951 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.199437  1951 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.196293  1983 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:33.202874  1983 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.209815  1983 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.212560  1878 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.225945  1928 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.226531  1928 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.228802  1928 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:33.230026  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:33.232853  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.233186  1979 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.256042  2057 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:33.266254  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:21:33.268873  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:33.269974  1927 raft_consensus.cc:359] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.270835  1927 raft_consensus.cc:385] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.271163  1927 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.272966  1880 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.273887  1987 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:33.271867  1927 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.281237  1928 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.281342  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.281869  1928 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.283139  1845 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:33.283455  1987 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.283910  1987 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.286088  1987 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.287616  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.291913  1927 ts_tablet_manager.cc:1434] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.149s	user 0.005s	sys 0.004s
I20260709 12:21:33.293226  1927 tablet_bootstrap.cc:492] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:33.288582  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:33.291124  1696 leader_election.cc:343] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.293205  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:21:33.299817  1844 leader_election.cc:343] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:33.304355  1696 leader_election.cc:343] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.309645  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.310431  1807 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:33.305827  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:33.311906  1845 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:33.302636  1928 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:33.312870  2062 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:33.313241  2062 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.313441  1925 tablet_bootstrap.cc:492] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.313588  2062 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.314007  1925 ts_tablet_manager.cc:1403] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.274s	user 0.012s	sys 0.025s
I20260709 12:21:33.319447  2048 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.320422  1987 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.320946  1925 raft_consensus.cc:359] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.321705  1925 raft_consensus.cc:385] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.321982  1925 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
W20260709 12:21:33.302533  1771 leader_election.cc:343] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.322626  1925 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.329540  1983 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.331387  2061 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.336597  1983 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.337972  1925 ts_tablet_manager.cc:1434] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:21:33.339044  1925 tablet_bootstrap.cc:492] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.341115  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:33.342931  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:21:33.344635  1696 leader_election.cc:343] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:33.349501  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48278.8us
I20260709 12:21:33.349866  2062 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.326879  1951 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.351197  1984 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.351696  1984 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.352250  2062 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:33.353511  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:33.353775  1984 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.354171  1730 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.355122  1807 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 1: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:21:33.360941  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:33.361881  1805 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.378886  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:33.381948  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:33.362459  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:33.385435  1987 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.386428  1924 tablet_bootstrap.cc:654] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.392040  1951 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:21:33.385283  1771 leader_election.cc:343] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.403264  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:33.404457  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:33.405421  1731 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 1: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:21:33.408891  1771 leader_election.cc:343] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.414379  1979 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:21:33.416678  1770 leader_election.cc:343] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.443703  1730 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:33.432428  1770 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:33.445089  1984 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.445271  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:33.446460  1979 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:33.447134  1979 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:33.448047  1979 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.451994  1984 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:21:33.456167  1770 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:33.457458  1927 tablet_bootstrap.cc:654] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.460110  1924 tablet_bootstrap.cc:492] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.460677  1924 ts_tablet_manager.cc:1403] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.268s	user 0.019s	sys 0.018s
I20260709 12:21:33.447661  1805 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:33.462692  2104 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.471243  1924 raft_consensus.cc:359] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.472154  1924 raft_consensus.cc:385] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.472463  1924 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.473333  1924 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.476076  1924 ts_tablet_manager.cc:1434] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 12:21:33.478104  1924 tablet_bootstrap.cc:492] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:33.494962  1984 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:33.496284  1925 tablet_bootstrap.cc:654] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.518491  1924 tablet_bootstrap.cc:654] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.519729  1927 tablet_bootstrap.cc:492] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:33.520694  1927 ts_tablet_manager.cc:1403] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.228s	user 0.011s	sys 0.008s
I20260709 12:21:33.523335  1927 raft_consensus.cc:359] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.524202  1927 raft_consensus.cc:385] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.528525  1629 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:33.529711  1927 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.530987  1925 tablet_bootstrap.cc:492] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.530736  1927 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.531476  1925 ts_tablet_manager.cc:1403] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.193s	user 0.016s	sys 0.000s
I20260709 12:21:33.534034  1925 raft_consensus.cc:359] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.534677  1925 raft_consensus.cc:385] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.534961  1925 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.535604  1925 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.538496  1925 ts_tablet_manager.cc:1434] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:21:33.539497  1925 tablet_bootstrap.cc:492] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.539911  1924 tablet_bootstrap.cc:492] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.540483  1924 ts_tablet_manager.cc:1403] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.063s	user 0.013s	sys 0.001s
I20260709 12:21:33.543612  1924 raft_consensus.cc:359] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.544374  1924 raft_consensus.cc:385] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.544648  1924 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.545359  1924 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.547391  1925 tablet_bootstrap.cc:654] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.547777  1924 ts_tablet_manager.cc:1434] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:21:33.548825  1924 tablet_bootstrap.cc:492] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:33.551373  1927 ts_tablet_manager.cc:1434] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.030s	user 0.003s	sys 0.002s
W20260709 12:21:33.547648  1771 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:33.565335  1927 tablet_bootstrap.cc:492] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:33.565925  1924 tablet_bootstrap.cc:654] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.575165  1925 tablet_bootstrap.cc:492] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.575807  1925 ts_tablet_manager.cc:1403] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.037s	user 0.023s	sys 0.007s
I20260709 12:21:33.579589  1925 raft_consensus.cc:359] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.580543  1925 raft_consensus.cc:385] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.580883  1925 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.581138  2104 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.581809  2104 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.581774  1925 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.584405  2104 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.587623  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:33.585266  1925 ts_tablet_manager.cc:1434] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
W20260709 12:21:33.590332  1771 leader_election.cc:343] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.590466  1925 tablet_bootstrap.cc:492] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.599839  1925 tablet_bootstrap.cc:654] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.602360  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:21:33.603952  1770 leader_election.cc:343] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.604470  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:33.605697  2057 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.606588  1927 tablet_bootstrap.cc:654] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.613121  1925 tablet_bootstrap.cc:492] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.613947  1925 ts_tablet_manager.cc:1403] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.006s
I20260709 12:21:33.616894  1925 raft_consensus.cc:359] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.617553  1924 tablet_bootstrap.cc:492] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.617575  1925 raft_consensus.cc:385] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.618073  1924 ts_tablet_manager.cc:1403] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.069s	user 0.016s	sys 0.034s
I20260709 12:21:33.618001  1925 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.619267  1927 tablet_bootstrap.cc:492] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:33.618908  1925 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.619889  1927 ts_tablet_manager.cc:1403] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.055s	user 0.011s	sys 0.002s
I20260709 12:21:33.621218  1924 raft_consensus.cc:359] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.621906  1924 raft_consensus.cc:385] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.622195  1924 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.622717  1927 raft_consensus.cc:359] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.622988  1924 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.623469  1927 raft_consensus.cc:385] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.623950  1927 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.624644  1927 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.625279  1924 ts_tablet_manager.cc:1434] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:21:33.626178  1924 tablet_bootstrap.cc:492] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:33.627094  1927 ts_tablet_manager.cc:1434] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:21:33.628479  1927 tablet_bootstrap.cc:492] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:33.629324  1925 ts_tablet_manager.cc:1434] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.015s	user 0.006s	sys 0.000s
I20260709 12:21:33.630329  1925 tablet_bootstrap.cc:492] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.631254  2048 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.631880  2048 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.635136  1927 tablet_bootstrap.cc:654] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.635540  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:21:33.637923  1696 leader_election.cc:343] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.639485  2048 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.642556  1924 tablet_bootstrap.cc:654] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.640175  1925 tablet_bootstrap.cc:654] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.641443  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:33.648093  1696 leader_election.cc:343] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.648772  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:33.650725  2048 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.667469  1927 tablet_bootstrap.cc:492] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:33.668061  1927 ts_tablet_manager.cc:1403] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.040s	user 0.008s	sys 0.028s
I20260709 12:21:33.669425  2048 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.670199  2048 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.671392  1927 raft_consensus.cc:359] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.672178  1927 raft_consensus.cc:385] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.672492  1927 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.673280  2048 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.673296  1927 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.674866  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.675253  1924 tablet_bootstrap.cc:492] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.675756  1924 ts_tablet_manager.cc:1403] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.050s	user 0.015s	sys 0.002s
I20260709 12:21:33.675765  1927 ts_tablet_manager.cc:1434] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
W20260709 12:21:33.676190  1696 leader_election.cc:343] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.676290  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:33.676985  1927 tablet_bootstrap.cc:492] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
W20260709 12:21:33.677654  1696 leader_election.cc:343] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.678093  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:33.678769  1924 raft_consensus.cc:359] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.679212  2048 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.679534  1924 raft_consensus.cc:385] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.679963  1924 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.680740  1924 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.683125  1924 ts_tablet_manager.cc:1434] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:21:33.684095  1927 tablet_bootstrap.cc:654] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.684791  1924 tablet_bootstrap.cc:492] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:33.686036  1925 tablet_bootstrap.cc:492] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.686558  1925 ts_tablet_manager.cc:1403] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.056s	user 0.026s	sys 0.007s
I20260709 12:21:33.689756  1927 tablet_bootstrap.cc:492] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:33.690301  1927 ts_tablet_manager.cc:1403] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.014s	user 0.010s	sys 0.001s
I20260709 12:21:33.691339  2057 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.691829  2057 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.693303  1924 tablet_bootstrap.cc:654] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.693347  1927 raft_consensus.cc:359] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.693933  2057 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.694164  1927 raft_consensus.cc:385] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.694535  1927 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.695377  1927 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.689231  1925 raft_consensus.cc:359] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.696586  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:33.697579  1927 ts_tablet_manager.cc:1434] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.004s
I20260709 12:21:33.698272  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:33.698650  1927 tablet_bootstrap.cc:492] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
W20260709 12:21:33.698297  1770 leader_election.cc:343] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:33.700527  1771 leader_election.cc:343] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.700937  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:33.701795  2057 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:33.696254  1925 raft_consensus.cc:385] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.704869  1925 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.705638  1925 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.706161  1927 tablet_bootstrap.cc:654] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.709547  1925 ts_tablet_manager.cc:1434] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.023s	user 0.003s	sys 0.004s
I20260709 12:21:33.710531  1925 tablet_bootstrap.cc:492] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.716681  1925 tablet_bootstrap.cc:654] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.719103  1927 tablet_bootstrap.cc:492] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:33.719681  1927 ts_tablet_manager.cc:1403] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.003s
I20260709 12:21:33.720889  1924 tablet_bootstrap.cc:492] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.721378  1924 ts_tablet_manager.cc:1403] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.037s	user 0.006s	sys 0.012s
I20260709 12:21:33.724822  1927 raft_consensus.cc:359] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.725858  1927 raft_consensus.cc:385] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.726218  1927 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.727216  1927 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.728004  1924 raft_consensus.cc:359] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.728796  1924 raft_consensus.cc:385] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.729080  1924 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.729662  1927 ts_tablet_manager.cc:1434] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.002s
I20260709 12:21:33.729879  1924 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.731308  1927 tablet_bootstrap.cc:492] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:33.732257  1924 ts_tablet_manager.cc:1434] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:21:33.733297  1924 tablet_bootstrap.cc:492] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:33.738569  1927 tablet_bootstrap.cc:654] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.740315  1924 tablet_bootstrap.cc:654] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.749297  1925 tablet_bootstrap.cc:492] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.749850  1925 ts_tablet_manager.cc:1403] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.040s	user 0.009s	sys 0.000s
I20260709 12:21:33.752705  1925 raft_consensus.cc:359] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.753433  1925 raft_consensus.cc:385] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.753713  1925 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.754586  1925 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.756709  1925 ts_tablet_manager.cc:1434] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:21:33.757710  1925 tablet_bootstrap.cc:492] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.758903  1924 tablet_bootstrap.cc:492] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.759374  1924 ts_tablet_manager.cc:1403] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.026s	user 0.017s	sys 0.000s
I20260709 12:21:33.762429  1924 raft_consensus.cc:359] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.763327  1924 raft_consensus.cc:385] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.763664  1924 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.764400  1924 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.766968  1924 ts_tablet_manager.cc:1434] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:21:33.768265  1924 tablet_bootstrap.cc:492] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:33.770260  1925 tablet_bootstrap.cc:654] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.776681  1927 tablet_bootstrap.cc:492] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:33.777122  1927 ts_tablet_manager.cc:1403] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.046s	user 0.021s	sys 0.020s
I20260709 12:21:33.779210  1927 raft_consensus.cc:359] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.780270  1927 raft_consensus.cc:385] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.780563  1927 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.781334  1927 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.784263  1927 ts_tablet_manager.cc:1434] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:21:33.784703  1924 tablet_bootstrap.cc:654] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.785249  1927 tablet_bootstrap.cc:492] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:33.793895  1925 tablet_bootstrap.cc:492] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.794499  1925 ts_tablet_manager.cc:1403] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.037s	user 0.021s	sys 0.000s
I20260709 12:21:33.797387  1925 raft_consensus.cc:359] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.798123  1925 raft_consensus.cc:385] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.798393  1925 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.788444  1979 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.799220  1925 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.799825  1979 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.801225  1927 tablet_bootstrap.cc:654] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.801910  1925 ts_tablet_manager.cc:1434] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:21:33.803802  1979 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:33.804343  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:33.804816  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.805039  1730 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:33.805521  1805 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:33.806753  1925 tablet_bootstrap.cc:492] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.808732  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:33.809860  1979 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:33.810279  1979 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.817291  1925 tablet_bootstrap.cc:654] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.810642  1979 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.829598  1925 tablet_bootstrap.cc:492] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.830154  1925 ts_tablet_manager.cc:1403] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.003s
I20260709 12:21:33.832068  2057 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.832858  2057 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.836607  2057 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.833096  1925 raft_consensus.cc:359] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.838050  1925 raft_consensus.cc:385] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.837945  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:33.838491  1925 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.839227  1925 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.840090  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:33.840413  1979 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.840786  1878 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:33.842151  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:33.842973  1730 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:33.843400  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:33.843995  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:33.844756  1805 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.846240  1925 ts_tablet_manager.cc:1434] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:21:33.846774  2057 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:33.847378  1925 tablet_bootstrap.cc:492] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.847659  2057 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.848058  2057 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.844066  1728 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.850922  1928 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:33.853739  1925 tablet_bootstrap.cc:654] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.874428  1928 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:33.877113  1805 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:33.866218  1728 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:33.879051  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:33.880337  1928 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:33.881865  1928 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:33.883077  1928 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.885164  1979 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:33.885835  1924 tablet_bootstrap.cc:492] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.886407  1924 ts_tablet_manager.cc:1403] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.118s	user 0.026s	sys 0.031s
I20260709 12:21:33.890689  1924 raft_consensus.cc:359] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.891594  1924 raft_consensus.cc:385] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.892043  1924 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.892912  1924 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.896560  1924 ts_tablet_manager.cc:1434] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:21:33.897500  1924 tablet_bootstrap.cc:492] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:33.901448  1927 tablet_bootstrap.cc:492] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:33.902079  1927 ts_tablet_manager.cc:1403] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.117s	user 0.010s	sys 0.035s
I20260709 12:21:33.910447  2057 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.912890  1925 tablet_bootstrap.cc:492] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:33.913545  1925 ts_tablet_manager.cc:1403] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.066s	user 0.011s	sys 0.008s
I20260709 12:21:33.921597  2057 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.923234  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:33.923931  1925 raft_consensus.cc:359] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.924791  1925 raft_consensus.cc:385] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.925004  1925 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.925697  1925 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.927212  1878 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.925565  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:33.932102  1728 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.932860  1630 catalog_manager.cc:5697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:33.935539  1925 ts_tablet_manager.cc:1434] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.004s
I20260709 12:21:33.937889  1925 tablet_bootstrap.cc:492] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:33.943131  2061 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.943574  1924 tablet_bootstrap.cc:654] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.943743  2061 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.946318  2048 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.946982  2048 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.948109  2061 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:33.949033  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:33.951272  1927 raft_consensus.cc:359] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.952296  1927 raft_consensus.cc:385] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:33.952605  1927 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:33.956540  1927 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.949175  2048 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:33.949270  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.963459  1807 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:21:33.973284  1927 ts_tablet_manager.cc:1434] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.071s	user 0.008s	sys 0.001s
I20260709 12:21:33.966574  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:33.993870  1925 tablet_bootstrap.cc:654] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:33.991132  1987 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.000859  1878 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:34.007747  1924 tablet_bootstrap.cc:492] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:33.985302  1728 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:34.026362  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:34.027318  2057 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:33.975826  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:34.024763  1983 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.974880  1927 tablet_bootstrap.cc:492] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:34.028218  1983 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.028352  2057 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:34.027614  1924 ts_tablet_manager.cc:1403] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.130s	user 0.016s	sys 0.024s
I20260709 12:21:34.030347  1983 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.023798  1987 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:33.986008  2104 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.031567  2104 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.033515  2057 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.034488  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.010545  2048 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.035395  2048 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:34.035894  1770 leader_election.cc:343] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:33.976282  1729 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:21:34.037983  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:34.039244  2061 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:33.981547  1881 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.043378  2104 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.045050  1927 tablet_bootstrap.cc:654] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.046845  2061 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:34.047959  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:34.049131  2048 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.050384  1987 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.034111  1924 raft_consensus.cc:359] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.051204  1924 raft_consensus.cc:385] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.051614  1925 tablet_bootstrap.cc:492] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:34.052238  1925 ts_tablet_manager.cc:1403] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.115s	user 0.027s	sys 0.020s
I20260709 12:21:34.051630  1924 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.053746  1924 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.054267  1927 tablet_bootstrap.cc:492] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
W20260709 12:21:33.979296  1696 leader_election.cc:343] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.054864  1927 ts_tablet_manager.cc:1403] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.080s	user 0.012s	sys 0.003s
W20260709 12:21:34.055228  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 75985.7us
I20260709 12:21:34.054742  1878 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:34.055455  1925 raft_consensus.cc:359] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.056320  1925 raft_consensus.cc:385] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.056643  1925 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.057353  1925 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.059099  1927 raft_consensus.cc:359] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.059917  1927 raft_consensus.cc:385] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.060356  1927 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.060923  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:34.061190  1927 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.061797  2104 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.062300  2104 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.062667  2104 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.063560  1924 ts_tablet_manager.cc:1434] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.034s	user 0.004s	sys 0.002s
I20260709 12:21:34.063553  1927 ts_tablet_manager.cc:1434] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:21:34.064738  1927 tablet_bootstrap.cc:492] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:34.065068  1924 tablet_bootstrap.cc:492] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:34.071259  1927 tablet_bootstrap.cc:654] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.071126  2097 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.071774  1924 tablet_bootstrap.cc:654] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.071941  2097 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.059533  1925 ts_tablet_manager.cc:1434] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:21:34.075997  1925 tablet_bootstrap.cc:492] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:34.076586  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.077267  1729 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:34.078588  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:34.079638  1984 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.080024  1984 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.080370  1984 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.082428  1925 tablet_bootstrap.cc:654] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.084793  2097 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.085700  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:34.087252  1771 leader_election.cc:343] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.094175  1924 tablet_bootstrap.cc:492] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:34.094717  1924 ts_tablet_manager.cc:1403] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.030s	user 0.011s	sys 0.002s
I20260709 12:21:34.096448  1925 tablet_bootstrap.cc:492] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:34.096999  1925 ts_tablet_manager.cc:1403] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.006s
I20260709 12:21:34.097891  1924 raft_consensus.cc:359] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.098626  1924 raft_consensus.cc:385] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.099720  1924 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.100049  1925 raft_consensus.cc:359] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.100605  1924 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.100807  1925 raft_consensus.cc:385] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.102219  1925 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.103128  1925 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.105720  1925 ts_tablet_manager.cc:1434] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 12:21:34.106761  1925 tablet_bootstrap.cc:492] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:34.107863  1924 ts_tablet_manager.cc:1434] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:21:34.108908  1924 tablet_bootstrap.cc:492] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:34.110267  2097 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.110911  2097 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.113144  1927 tablet_bootstrap.cc:492] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:34.113710  1925 tablet_bootstrap.cc:654] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.113850  1927 ts_tablet_manager.cc:1403] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.049s	user 0.021s	sys 0.007s
I20260709 12:21:34.116873  1927 raft_consensus.cc:359] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.118183  1927 raft_consensus.cc:385] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.117710  1928 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.113230  2097 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.115769  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:34.119138  1928 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.119604  1877 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:34.121529  1928 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:34.121682  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:34.123317  2097 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.123700  2097 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.124003  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.124032  2097 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.126446  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.126654  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.127516  1807 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:34.127593  1927 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.128955  1924 tablet_bootstrap.cc:654] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.127481  1729 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:34.129716  1925 tablet_bootstrap.cc:492] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:34.128825  1927 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.130250  1925 ts_tablet_manager.cc:1403] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.024s	user 0.002s	sys 0.015s
I20260709 12:21:34.130434  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:34.131361  1928 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.131763  1928 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.132227  1928 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.133800  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:34.134074  1925 raft_consensus.cc:359] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.134712  1925 raft_consensus.cc:385] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.135042  1925 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.135882  1925 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.137293  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.137938  1807 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:34.138808  1925 ts_tablet_manager.cc:1434] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.002s
I20260709 12:21:34.139880  1925 tablet_bootstrap.cc:492] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:34.144001  1927 ts_tablet_manager.cc:1434] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.030s	user 0.005s	sys 0.002s
I20260709 12:21:34.145604  1927 tablet_bootstrap.cc:492] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:34.146363  1928 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.154472  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:34.155179  1984 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.155598  1729 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.156663  1805 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:21:34.157752  1984 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.159627  2097 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.161247  1928 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:34.161741  2097 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.162521  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:34.163992  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.172099  1805 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:21:34.172761  2097 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.173303  2097 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.164610  1925 tablet_bootstrap.cc:654] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:34.165906  1696 leader_election.cc:343] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.183642  1928 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.167560  2104 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.195415  1987 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.196154  1987 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.196703  1928 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.200575  1987 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.180943  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:34.203365  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.204412  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:34.204931  1928 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:34.205289  1879 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:21:34.207629  1844 leader_election.cc:343] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.206544  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.209420  1630 catalog_manager.cc:5697] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:34.191857  1696 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:34.213196  1805 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
W20260709 12:21:34.215476  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 74471.4us
I20260709 12:21:34.220660  1845 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:34.221442  1928 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.224345  1979 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.222278  1987 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:34.224870  1979 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.225672  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.226364  1728 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:34.227182  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:34.227852  1878 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:34.228715  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:34.214196  1729 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:34.228622  2104 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.221899  1928 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.229765  1877 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.236093  1928 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:34.235319  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69462.5us
W20260709 12:21:34.250316  1696 leader_election.cc:343] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.252343  1927 tablet_bootstrap.cc:654] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.230576  1979 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:34.230124  2097 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.256683  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:34.257139  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:34.259447  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:34.258440  1924 tablet_bootstrap.cc:492] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
W20260709 12:21:34.252406  1696 leader_election.cc:343] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.260810  1924 ts_tablet_manager.cc:1403] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.152s	user 0.041s	sys 0.048s
I20260709 12:21:34.257426  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:34.263792  1878 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.256784  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.264114  1730 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.264638  1729 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:34.257567  1728 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:21:34.266558  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:34.266826  1979 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:34.265808  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:34.277102  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.278100  1805 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:34.266966  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:34.285069  2062 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.285507  2062 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.285697  1984 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.285885  2062 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.288621  1928 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.287655  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:34.287669  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.282694  1979 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:34.292127  1807 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:34.291214  2061 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:21:34.294049  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:34.297180  2062 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.298111  1877 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:34.336584  1925 tablet_bootstrap.cc:492] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
W20260709 12:21:34.336465  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 52319.9us
I20260709 12:21:34.337072  1925 ts_tablet_manager.cc:1403] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.197s	user 0.019s	sys 0.008s
I20260709 12:21:34.337208  1730 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
W20260709 12:21:34.343267  1844 leader_election.cc:343] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.297958  1984 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.344115  1984 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.343961  1924 raft_consensus.cc:359] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.344820  1924 raft_consensus.cc:385] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.345129  1924 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.298962  1928 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:34.345789  1924 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.347599  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.333431  2048 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.350322  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:34.351073  1924 ts_tablet_manager.cc:1434] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.090s	user 0.005s	sys 0.003s
I20260709 12:21:34.350754  2048 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.351512  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:34.352027  1924 tablet_bootstrap.cc:492] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:34.352262  1807 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.352845  2048 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.297996  1878 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:34.350695  1730 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:34.345690  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:34.355233  1983 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.334234  2061 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.356904  1729 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.356981  1983 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.338533  1979 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.365751  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:34.370493  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:34.365772  1983 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.371141  1732 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.379000  2142 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:34.379539  2142 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:34.380352  2142 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.396236  1925 raft_consensus.cc:359] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.397090  1925 raft_consensus.cc:385] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.397344  1925 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.398005  1925 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.409051  1732 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:34.408175  2062 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:34.451169  1983 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.465654  1925 ts_tablet_manager.cc:1434] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.128s	user 0.010s	sys 0.005s
I20260709 12:21:34.467160  1925 tablet_bootstrap.cc:492] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:34.479328  1927 tablet_bootstrap.cc:492] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:34.479763  1927 ts_tablet_manager.cc:1403] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.334s	user 0.014s	sys 0.012s
W20260709 12:21:34.465191  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69642.5us
I20260709 12:21:34.463400  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:34.492449  1983 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.471213  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:34.507686  1987 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.508245  1987 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.508711  1987 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.509114  1984 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:34.511288  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59503.3us
I20260709 12:21:34.512136  2104 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:21:34.517709  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 103743us
I20260709 12:21:34.523326  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:34.524394  2104 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:34.524786  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:34.525398  2104 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.526526  1729 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:34.527292  2048 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:34.531065  2048 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.537206  2048 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.544459  1984 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.546507  2048 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.543608  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:34.563535  2148 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20260709 12:21:34.557623  1807 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:34.567134  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:34.567703  2061 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:34.568096  1805 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:21:34.568347  2061 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:34.569060  2061 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:34.569895  2061 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.572314  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.573979  1927 raft_consensus.cc:359] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.574951  1927 raft_consensus.cc:385] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.575322  1927 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.576176  1927 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.587459  1927 ts_tablet_manager.cc:1434] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.107s	user 0.008s	sys 0.000s
I20260709 12:21:34.588433  1927 tablet_bootstrap.cc:492] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:34.595290  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:34.597533  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:34.599916  1806 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.597645  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:34.601971  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.612984  2149 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:21:34.603696  2142 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
W20260709 12:21:34.624186  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:21:34.616245  1696 leader_election.cc:343] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.631441  1924 tablet_bootstrap.cc:654] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.626730  1627 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:34.618115  1845 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:34.620903  1984 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.629806  1925 tablet_bootstrap.cc:654] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.629940  1630 catalog_manager.cc:5697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:34.635761  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:34.641170  1728 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:21:34.638743  1984 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.643492  1984 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.643880  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:34.649204  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.630867  2048 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.617874  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:34.652844  2152 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.660444  2139 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:21:34.660804  2152 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.665141  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:34.667822  1696 leader_election.cc:343] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:34.668354  1771 leader_election.cc:343] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.668481  1806 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:34.671204  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:34.672041  1983 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:34.672376  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.673161  1728 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:34.654676  1987 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.675697  1987 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.676038  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:34.676992  1805 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.657514  2048 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.652524  2061 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:34.677198  2149 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.692451  1845 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:34.700250  2062 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:34.707756  1927 tablet_bootstrap.cc:654] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:34.703541  2061 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:34.704823  2152 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:34.711761  1983 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:34.722759  1924 tablet_bootstrap.cc:492] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:34.723412  1924 ts_tablet_manager.cc:1403] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.372s	user 0.014s	sys 0.014s
I20260709 12:21:34.725848  1924 raft_consensus.cc:359] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.726485  1924 raft_consensus.cc:385] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:34.726738  1924 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.703989  2149 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.727486  2061 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.703513  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:21:34.768280  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71878.1us
I20260709 12:21:34.788916  1925 tablet_bootstrap.cc:492] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:34.790289  1925 ts_tablet_manager.cc:1403] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.323s	user 0.009s	sys 0.017s
I20260709 12:21:34.792019  1924 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.801033  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:34.807893  1983 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.810086  1728 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:21:34.812278  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.814148  1924 ts_tablet_manager.cc:1434] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.090s	user 0.010s	sys 0.000s
I20260709 12:21:34.819733  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:34.822760  1877 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:34.771948  1696 leader_election.cc:343] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.823956  1924 tablet_bootstrap.cc:492] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:34.797428  2097 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:21:34.845515  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:34.852316  1928 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:34.822698  1630 catalog_manager.cc:5697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:34.729660  2149 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:34.836094  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:34.868567  2160 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:21:34.873162  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101601us
I20260709 12:21:34.877351  2057 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:34.877168  1925 raft_consensus.cc:359] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:34.855329  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84910.9us
I20260709 12:21:34.892685  1925 raft_consensus.cc:385] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:21:34.914041  1696 leader_election.cc:343] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:34.916668  1770 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:34.924716  2057 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:34.925323  2057 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.925751  2057 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.916867  1925 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:34.931355  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:34.933085  1804 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:34.932945  1925 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:34.935030  1925 ts_tablet_manager.cc:1434] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.142s	user 0.005s	sys 0.005s
I20260709 12:21:34.936574  1925 tablet_bootstrap.cc:492] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:34.954686  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:34.956254  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:34.957405  1807 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:21:34.951319  1984 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260709 12:21:34.978930  1927 tablet_bootstrap.cc:492] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:34.979921  1927 ts_tablet_manager.cc:1403] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.392s	user 0.015s	sys 0.003s
I20260709 12:21:34.998215  2160 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.000069  1927 raft_consensus.cc:359] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.001207  1927 raft_consensus.cc:385] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.001547  1927 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.003398  1927 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.003758  2145 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.008805  2145 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.006961  2160 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.016060  1927 ts_tablet_manager.cc:1434] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.036s	user 0.009s	sys 0.002s
I20260709 12:21:35.029722  1927 tablet_bootstrap.cc:492] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:35.033777  2145 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:35.034951  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:35.039309  2160 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.037989  1984 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:21:35.022437  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:35.041884  2145 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:35.045188  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:35.046397  1803 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:21:35.048050  1924 tablet_bootstrap.cc:654] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.025758  1877 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:35.051600  2062 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:35.053622  1845 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:35.055290  2061 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.059540  1804 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:35.060932  1629 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.073742  1925 tablet_bootstrap.cc:654] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.073411  2057 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:35.074213  1696 leader_election.cc:343] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.082020  2062 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:35.086417  2165 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.075314  1805 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:35.087384  2165 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.089764  2160 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.092165  2097 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:35.093420  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:35.098836  2160 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.099715  2165 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.103623  2061 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:35.104753  2160 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.105082  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:35.107788  1987 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:35.108837  1771 leader_election.cc:343] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.130483  1630 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.131733  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:35.136229  2160 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:35.136941  2160 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:35.140137  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:35.142026  1983 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:35.142591  1983 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:35.143950  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:35.144833  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:35.146147  1806 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:35.143980  1983 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.153600  2057 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.155170  1987 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:35.138135  2160 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.173355  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:21:35.187575  1770 leader_election.cc:343] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.188334  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:35.189152  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:35.191507  2165 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:35.192416  1627 catalog_manager.cc:5697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:35.195811  1771 leader_election.cc:343] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.205583  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:35.206658  1878 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:35.210253  2061 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:35.211539  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:35.212216  1729 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.220178  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:35.222165  1696 leader_election.cc:343] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.223577  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:35.228168  1924 tablet_bootstrap.cc:492] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:35.230013  1924 ts_tablet_manager.cc:1403] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.410s	user 0.021s	sys 0.006s
I20260709 12:21:35.200424  2165 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.029s
I20260709 12:21:35.249397  2139 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.235033  1983 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:35.251828  2139 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.255546  2139 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.255932  1927 tablet_bootstrap.cc:654] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.259859  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:35.261664  1732 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:35.257107  1983 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:35.266700  1696 leader_election.cc:343] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.273501  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:35.267751  1924 raft_consensus.cc:359] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.271587  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:35.272626  1983 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:35.275293  1924 raft_consensus.cc:385] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.275764  2048 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:35.276486  1924 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.277294  2142 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:35.282128  2142 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.282691  2142 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.278774  1925 tablet_bootstrap.cc:492] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:35.284658  1925 ts_tablet_manager.cc:1403] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.348s	user 0.027s	sys 0.048s
I20260709 12:21:35.288232  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:35.293125  1771 leader_election.cc:343] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.298357  1630 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.274230  1729 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
W20260709 12:21:35.303503  2174 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:21:35.303653  1770 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:35.309556  1924 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.310441  1630 catalog_manager.cc:5697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.312582  2057 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:35.315129  2057 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:35.316159  2057 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.319209  1925 raft_consensus.cc:359] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.320358  1925 raft_consensus.cc:385] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.322916  1924 ts_tablet_manager.cc:1434] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.093s	user 0.005s	sys 0.003s
I20260709 12:21:35.323612  1979 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.320632  1925 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.326292  1979 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.328816  1979 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:35.332525  1925 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.333191  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:35.333413  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:21:35.334954  1845 leader_election.cc:343] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.335243  1925 ts_tablet_manager.cc:1434] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.050s	user 0.009s	sys 0.000s
W20260709 12:21:35.343204  1844 leader_election.cc:343] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.345170  1925 tablet_bootstrap.cc:492] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:35.343889  1844 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:35.346095  1979 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:35.346544  1987 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:35.351784  2160 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.355118  2160 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.357328  2160 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.360296  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:35.362273  1983 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.363142  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:35.364450  1983 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.365257  1924 tablet_bootstrap.cc:492] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:35.368937  1979 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:35.365424  2142 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.374094  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:35.374832  1629 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.374771  1803 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:35.387308  1696 leader_election.cc:343] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:35.388216  1696 leader_election.cc:343] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.391093  1696 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:35.388252  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:35.392220  1987 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:35.392768  1729 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.391350  1979 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:35.399739  1925 tablet_bootstrap.cc:654] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.400462  1927 tablet_bootstrap.cc:492] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:35.407452  1927 ts_tablet_manager.cc:1403] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.378s	user 0.012s	sys 0.023s
I20260709 12:21:35.393600  2142 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.410315  1924 tablet_bootstrap.cc:654] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.409631  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:21:35.424678  1771 leader_election.cc:343] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.420895  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:35.437901  1878 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:35.436403  1803 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:21:35.441793  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:35.443162  1987 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:35.444058  1987 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:35.445016  1987 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.456950  2048 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:35.458250  2160 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.459113  2160 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.468356  2160 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.469137  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:35.470439  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:35.464732  1729 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
W20260709 12:21:35.481189  1696 leader_election.cc:343] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:35.482016  1696 leader_election.cc:343] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.482168  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:35.483337  1927 raft_consensus.cc:359] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.483933  1927 raft_consensus.cc:385] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.484159  1927 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.484656  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:35.485041  1927 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.486368  2160 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:35.487197  2057 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:35.488080  2057 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:35.489135  2057 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.493046  1927 ts_tablet_manager.cc:1434] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.085s	user 0.005s	sys 0.002s
I20260709 12:21:35.494515  1927 tablet_bootstrap.cc:492] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:35.496269  1925 tablet_bootstrap.cc:492] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:35.496850  1925 ts_tablet_manager.cc:1403] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.152s	user 0.026s	sys 0.029s
I20260709 12:21:35.501524  1925 raft_consensus.cc:359] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.502713  1925 raft_consensus.cc:385] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.503147  1925 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.502837  1878 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:21:35.505021  1925 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.510339  1925 ts_tablet_manager.cc:1434] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:21:35.511368  1925 tablet_bootstrap.cc:492] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:35.516832  1927 tablet_bootstrap.cc:654] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.523722  1630 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 2, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.540657  1924 tablet_bootstrap.cc:492] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:35.540730  1925 tablet_bootstrap.cc:654] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.541286  1924 ts_tablet_manager.cc:1403] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.187s	user 0.020s	sys 0.007s
I20260709 12:21:35.542588  2160 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.543414  2160 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.545894  2160 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.546509  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:35.547224  1803 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:35.553051  1924 raft_consensus.cc:359] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.553529  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:35.553906  1924 raft_consensus.cc:385] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.554226  1924 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.555084  1924 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.556716  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:35.557516  1924 ts_tablet_manager.cc:1434] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.002s
I20260709 12:21:35.558748  2160 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:35.559264  2160 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.559659  2160 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.562975  1924 tablet_bootstrap.cc:492] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
W20260709 12:21:35.567303  1696 leader_election.cc:343] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.570250  1924 tablet_bootstrap.cc:654] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.573647  2160 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.576560  1627 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.592423  1925 tablet_bootstrap.cc:492] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:35.592971  1925 ts_tablet_manager.cc:1403] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.082s	user 0.012s	sys 0.009s
I20260709 12:21:35.584407  1987 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:35.594537  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:35.595214  1803 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.595773  1925 raft_consensus.cc:359] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.586377  1927 tablet_bootstrap.cc:492] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:35.597024  1927 ts_tablet_manager.cc:1403] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.103s	user 0.027s	sys 0.016s
I20260709 12:21:35.596454  1925 raft_consensus.cc:385] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.603348  1925 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.607019  1927 raft_consensus.cc:359] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.608080  1927 raft_consensus.cc:385] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.608382  1927 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.609911  1925 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.610456  1927 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.610498  1987 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:35.613258  1927 ts_tablet_manager.cc:1434] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:21:35.615315  1803 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:35.616073  1927 tablet_bootstrap.cc:492] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:35.619891  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:35.620983  2152 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:35.621181  2160 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.621639  2152 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:35.622517  2152 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.623047  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:35.623211  1924 tablet_bootstrap.cc:492] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:35.624028  1924 ts_tablet_manager.cc:1403] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.061s	user 0.005s	sys 0.011s
I20260709 12:21:35.620942  1925 ts_tablet_manager.cc:1434] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.028s	user 0.003s	sys 0.006s
W20260709 12:21:35.624787  1696 leader_election.cc:343] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:35.638048  1924 raft_consensus.cc:359] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.638818  1924 raft_consensus.cc:385] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.639088  1924 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.639833  1924 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.641785  1925 tablet_bootstrap.cc:492] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:35.642233  1924 ts_tablet_manager.cc:1434] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:21:35.651084  1924 tablet_bootstrap.cc:492] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:35.678830  1925 tablet_bootstrap.cc:654] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.692278  1927 tablet_bootstrap.cc:654] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.696111  1629 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:35.697557  2048 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:35.706477  1924 tablet_bootstrap.cc:654] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.718207  1880 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:35.731993  1771 consensus_peers.cc:597] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:35.743835  1925 tablet_bootstrap.cc:492] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:35.744383  1925 ts_tablet_manager.cc:1403] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.110s	user 0.030s	sys 0.026s
I20260709 12:21:35.747639  1925 raft_consensus.cc:359] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.748384  1925 raft_consensus.cc:385] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.748715  1925 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.751221  1924 tablet_bootstrap.cc:492] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:35.751045  1925 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.751969  1924 ts_tablet_manager.cc:1403] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.105s	user 0.012s	sys 0.009s
I20260709 12:21:35.753456  1925 ts_tablet_manager.cc:1434] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.002s
I20260709 12:21:35.754737  1925 tablet_bootstrap.cc:492] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:35.763978  1925 tablet_bootstrap.cc:654] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.764581  1924 raft_consensus.cc:359] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.786950  1987 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:35.787451  1924 raft_consensus.cc:385] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.787865  1924 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.788856  1924 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.792222  2057 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.792793  2057 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.794850  2057 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.795536  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:35.796234  1729 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:35.797529  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:35.798466  2057 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:35.798861  2057 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.799201  2057 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.801329  1925 tablet_bootstrap.cc:492] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:35.801810  1925 ts_tablet_manager.cc:1403] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.047s	user 0.011s	sys 0.007s
I20260709 12:21:35.804385  1925 raft_consensus.cc:359] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.805052  1925 raft_consensus.cc:385] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.805306  1925 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.805962  1925 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.807401  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:35.808102  1879 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:35.809718  1925 ts_tablet_manager.cc:1434] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 12:21:35.811019  1925 tablet_bootstrap.cc:492] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:35.822482  1924 ts_tablet_manager.cc:1434] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.070s	user 0.007s	sys 0.002s
I20260709 12:21:35.825970  1924 tablet_bootstrap.cc:492] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:35.842566  2057 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.844967  2057 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.848577  1925 tablet_bootstrap.cc:654] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.855458  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:35.856325  1732 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.856941  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:35.872697  1924 tablet_bootstrap.cc:654] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:35.877655  1927 tablet_bootstrap.cc:492] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:35.878541  1927 ts_tablet_manager.cc:1403] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.263s	user 0.013s	sys 0.029s
I20260709 12:21:35.868105  1879 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.886611  1927 raft_consensus.cc:359] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.887786  1927 raft_consensus.cc:385] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.888123  1927 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:35.891978  2142 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.892645  2142 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.898574  1732 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:35.899489  1880 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:21:35.901491  1925 tablet_bootstrap.cc:492] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:35.902104  2152 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:35.902872  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:35.902076  1925 ts_tablet_manager.cc:1403] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.091s	user 0.017s	sys 0.013s
I20260709 12:21:35.904390  2097 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:35.905508  2142 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:35.906119  2167 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:35.909126  2142 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:35.912569  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:35.914592  1877 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:35.917327  1771 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:35.943202  2097 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:35.904243  1879 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:35.922909  2167 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:35.917824  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:35.961249  1732 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:35.960813  2097 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.962200  2167 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.963142  2167 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:35.948784  1924 tablet_bootstrap.cc:492] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:35.965811  1924 ts_tablet_manager.cc:1403] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.142s	user 0.016s	sys 0.005s
I20260709 12:21:35.941156  1925 raft_consensus.cc:359] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.966925  1925 raft_consensus.cc:385] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:35.967207  1925 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
W20260709 12:21:35.970386  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:35.921499  1927 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.976426  1925 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:35.982134  1927 ts_tablet_manager.cc:1434] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.103s	user 0.011s	sys 0.006s
I20260709 12:21:35.983633  1927 tablet_bootstrap.cc:492] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:35.992061  1925 ts_tablet_manager.cc:1434] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.087s	user 0.004s	sys 0.007s
I20260709 12:21:35.993371  1925 tablet_bootstrap.cc:492] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:36.002246  1805 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:21:36.005394  1987 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:36.015390  1924 raft_consensus.cc:359] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.016728  1924 raft_consensus.cc:385] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.017045  1924 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
W20260709 12:21:36.023388  1771 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:36.028227  1924 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.034919  2145 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.036064  2145 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.041531  2145 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:36.042693  1927 tablet_bootstrap.cc:654] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.043540  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:36.048130  1924 ts_tablet_manager.cc:1434] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.082s	user 0.007s	sys 0.002s
I20260709 12:21:36.048255  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:36.050798  1924 tablet_bootstrap.cc:492] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:36.050628  1729 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:36.057088  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:21:36.060683  1845 leader_election.cc:343] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.060889  1979 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:36.061388  1979 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.061712  1979 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.110375  1925 tablet_bootstrap.cc:654] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.114061  2167 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.115715  1979 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.127415  1979 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:36.128300  1627 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:36.128547  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:36.124212  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:36.136051  1729 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:36.139786  1696 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:36.161585  1924 tablet_bootstrap.cc:654] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.163897  1729 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:36.167273  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:36.167950  2167 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.168582  1979 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:36.169134  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:36.169606  1979 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:36.170127  1729 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.171931  1979 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.176532  2152 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:36.176807  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:36.177439  1879 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.179509  1927 tablet_bootstrap.cc:492] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:36.180104  1927 ts_tablet_manager.cc:1403] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.197s	user 0.016s	sys 0.010s
I20260709 12:21:36.166626  1880 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.182518  1729 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:36.184010  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:36.193908  2211 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:21:36.194190  1845 leader_election.cc:343] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.194572  2211 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:36.195868  2211 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.203809  1925 tablet_bootstrap.cc:492] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:36.204401  1925 ts_tablet_manager.cc:1403] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.211s	user 0.016s	sys 0.000s
I20260709 12:21:36.213865  1879 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:36.217144  1925 raft_consensus.cc:359] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.217980  1925 raft_consensus.cc:385] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.218487  1925 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.219732  1925 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.236627  2142 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.237440  2142 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.243479  2142 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.250936  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.254528  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:21:36.256100  1770 leader_election.cc:343] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:36.257306  1771 leader_election.cc:343] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.257884  1771 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:36.260511  1925 ts_tablet_manager.cc:1434] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.056s	user 0.009s	sys 0.000s
I20260709 12:21:36.263651  1627 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:36.259408  2221 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:36.277608  1924 tablet_bootstrap.cc:492] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:36.278169  1924 ts_tablet_manager.cc:1403] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.228s	user 0.005s	sys 0.009s
I20260709 12:21:36.283841  1927 raft_consensus.cc:359] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.284835  1927 raft_consensus.cc:385] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.286002  1927 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.290560  1927 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.294127  1927 ts_tablet_manager.cc:1434] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.114s	user 0.009s	sys 0.000s
I20260709 12:21:36.294909  1925 tablet_bootstrap.cc:492] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:36.297269  1927 tablet_bootstrap.cc:492] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:36.297925  1924 raft_consensus.cc:359] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.298507  1924 raft_consensus.cc:385] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.298731  1924 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.302910  1924 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.301205  1628 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:36.308383  2152 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.308976  2152 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.309952  1927 tablet_bootstrap.cc:654] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.311115  2152 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.312009  2152 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.312570  2152 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.314574  1925 tablet_bootstrap.cc:654] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.317598  2152 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.315685  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:36.316512  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.319583  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:36.315474  1924 ts_tablet_manager.cc:1434] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.037s	user 0.007s	sys 0.001s
I20260709 12:21:36.320611  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.336930  1925 tablet_bootstrap.cc:492] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:36.342389  1925 ts_tablet_manager.cc:1403] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.061s	user 0.005s	sys 0.015s
I20260709 12:21:36.342502  1924 tablet_bootstrap.cc:492] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:36.337857  1927 tablet_bootstrap.cc:492] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:36.344342  1927 ts_tablet_manager.cc:1403] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.049s	user 0.007s	sys 0.014s
I20260709 12:21:36.347190  1927 raft_consensus.cc:359] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.348188  1927 raft_consensus.cc:385] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.348536  1927 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.349532  1927 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:36.351507  1696 leader_election.cc:343] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:36.359728  1696 leader_election.cc:343] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.359959  1924 tablet_bootstrap.cc:654] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:36.360283  1696 leader_election.cc:343] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.360715  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:36.361436  2213 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:36.354662  1925 raft_consensus.cc:359] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.362748  1925 raft_consensus.cc:385] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.363036  1925 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.363755  1925 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.365656  1925 ts_tablet_manager.cc:1434] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.023s	user 0.004s	sys 0.000s
I20260709 12:21:36.352470  1927 ts_tablet_manager.cc:1434] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:21:36.366870  1927 tablet_bootstrap.cc:492] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:36.368834  1924 tablet_bootstrap.cc:492] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:36.369320  1924 ts_tablet_manager.cc:1403] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.000s
I20260709 12:21:36.372298  1925 tablet_bootstrap.cc:492] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:36.372670  1927 tablet_bootstrap.cc:654] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:36.375350  1696 leader_election.cc:343] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.375795  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:36.376670  2213 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:36.376555  1924 raft_consensus.cc:359] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.377503  1924 raft_consensus.cc:385] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.377789  1924 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.379125  1925 tablet_bootstrap.cc:654] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.389810  1924 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.394088  1924 ts_tablet_manager.cc:1434] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.000s
I20260709 12:21:36.395279  1927 tablet_bootstrap.cc:492] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:36.395752  1927 ts_tablet_manager.cc:1403] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.005s
I20260709 12:21:36.396698  2213 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.397334  2213 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.398440  1924 tablet_bootstrap.cc:492] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:36.399837  2213 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.404091  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:36.404785  1927 raft_consensus.cc:359] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.405040  1805 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:36.405596  1927 raft_consensus.cc:385] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.405923  1927 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.406096  1924 tablet_bootstrap.cc:654] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.404091  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.406556  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:36.407378  2213 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:36.407743  2213 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.408078  2213 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.409319  1925 tablet_bootstrap.cc:492] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:36.409837  1925 ts_tablet_manager.cc:1403] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.005s
I20260709 12:21:36.413089  1925 raft_consensus.cc:359] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.413978  1925 raft_consensus.cc:385] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.414323  1925 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.415091  1925 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:36.415361  1696 leader_election.cc:343] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.417629  1925 ts_tablet_manager.cc:1434] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 12:21:36.418926  1925 tablet_bootstrap.cc:492] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:36.420130  2213 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.424669  1927 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.426306  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:36.427181  1805 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.427858  1925 tablet_bootstrap.cc:654] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.427917  1927 ts_tablet_manager.cc:1434] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.032s	user 0.008s	sys 0.000s
I20260709 12:21:36.429136  1927 tablet_bootstrap.cc:492] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:36.431366  2213 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.437398  1927 tablet_bootstrap.cc:654] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.440529  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:21:36.443243  1696 leader_election.cc:343] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.444376  1925 tablet_bootstrap.cc:492] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:36.445341  1925 ts_tablet_manager.cc:1403] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.003s
I20260709 12:21:36.445484  1927 tablet_bootstrap.cc:492] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:36.446100  1927 ts_tablet_manager.cc:1403] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.000s
I20260709 12:21:36.449134  1927 raft_consensus.cc:359] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.449914  1927 raft_consensus.cc:385] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.450245  1927 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.451900  1927 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.453203  2221 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.453649  2221 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.455818  2221 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.457360  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:36.458052  1729 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:36.459437  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:36.453668  1925 raft_consensus.cc:359] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.460183  1925 raft_consensus.cc:385] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.460421  2167 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:36.461090  1924 tablet_bootstrap.cc:492] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:36.461194  2167 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.461640  1924 ts_tablet_manager.cc:1403] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.063s	user 0.017s	sys 0.032s
I20260709 12:21:36.461700  1927 ts_tablet_manager.cc:1434] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.015s	user 0.001s	sys 0.004s
I20260709 12:21:36.462178  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.462906  1927 tablet_bootstrap.cc:492] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:36.464393  1924 raft_consensus.cc:359] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.465023  1924 raft_consensus.cc:385] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:21:36.465181  1771 leader_election.cc:343] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.465265  1924 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.466234  1924 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.469159  1927 tablet_bootstrap.cc:654] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.470414  1924 ts_tablet_manager.cc:1434] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:21:36.461711  2167 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.460443  1925 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.471602  1924 tablet_bootstrap.cc:492] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:36.471696  1925 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.473277  1925 ts_tablet_manager.cc:1434] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.000s
I20260709 12:21:36.478730  1924 tablet_bootstrap.cc:654] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.481237  1925 tablet_bootstrap.cc:492] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:36.490114  1925 tablet_bootstrap.cc:654] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.493624  2211 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:36.496151  1927 tablet_bootstrap.cc:492] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:36.496853  1927 ts_tablet_manager.cc:1403] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.034s	user 0.012s	sys 0.003s
I20260709 12:21:36.504158  1927 raft_consensus.cc:359] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.504830  1927 raft_consensus.cc:385] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.505092  1927 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.505782  1927 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.513489  1927 ts_tablet_manager.cc:1434] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.002s
I20260709 12:21:36.514542  1927 tablet_bootstrap.cc:492] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:36.523200  2219 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:36.527700  1927 tablet_bootstrap.cc:654] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.529995  1879 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.535696  1925 tablet_bootstrap.cc:492] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:36.535892  1927 tablet_bootstrap.cc:492] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:36.536496  1925 ts_tablet_manager.cc:1403] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.055s	user 0.021s	sys 0.006s
I20260709 12:21:36.536602  1927 ts_tablet_manager.cc:1403] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.002s
I20260709 12:21:36.538139  1924 tablet_bootstrap.cc:492] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:36.538568  1924 ts_tablet_manager.cc:1403] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.067s	user 0.019s	sys 0.017s
I20260709 12:21:36.539305  1927 raft_consensus.cc:359] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.539980  1927 raft_consensus.cc:385] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.540246  1927 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.541966  1927 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.544059  1927 ts_tablet_manager.cc:1434] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:21:36.544988  1927 tablet_bootstrap.cc:492] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:36.544687  1924 raft_consensus.cc:359] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.545369  1924 raft_consensus.cc:385] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.545637  1924 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.546240  1924 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.548216  1924 ts_tablet_manager.cc:1434] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.004s
I20260709 12:21:36.549250  1924 tablet_bootstrap.cc:492] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:36.554860  1924 tablet_bootstrap.cc:654] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.557374  2213 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.557919  2213 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.557219  1925 raft_consensus.cc:359] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.564038  1925 raft_consensus.cc:385] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.564471  1925 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.565349  1925 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.568650  1925 ts_tablet_manager.cc:1434] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.000s
I20260709 12:21:36.568794  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:36.569657  1803 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:36.571113  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:36.572548  2152 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:36.573043  2152 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.573460  2152 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.575577  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.576380  1880 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:36.579603  1927 tablet_bootstrap.cc:654] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.580474  1987 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.580789  2213 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.581077  1987 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.585765  1987 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.588390  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.589519  2149 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.593915  2149 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.596405  2149 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:36.597173  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:36.598032  1727 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:36.598873  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:36.598215  1925 tablet_bootstrap.cc:492] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:36.599303  1844 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:36.599788  1806 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:36.600558  2145 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.598945  2152 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.601631  1979 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:36.602094  1979 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.602487  1979 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.605590  1927 tablet_bootstrap.cc:492] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:36.606230  1927 ts_tablet_manager.cc:1403] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.061s	user 0.009s	sys 0.009s
W20260709 12:21:36.608682  1696 leader_election.cc:343] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.610632  2152 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.592552  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:36.601187  2145 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:36.617619  1696 leader_election.cc:343] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.618203  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:36.619614  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:36.621961  1803 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.610970  2211 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.622663  2211 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.634029  2221 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:36.643273  1987 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:36.648774  1987 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.649260  1987 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.650657  1927 raft_consensus.cc:359] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.654076  1927 raft_consensus.cc:385] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.654556  1927 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.651392  1804 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.658521  1927 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.662153  1927 ts_tablet_manager.cc:1434] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.056s	user 0.016s	sys 0.003s
I20260709 12:21:36.664426  1927 tablet_bootstrap.cc:492] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:36.590147  1805 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:36.657434  1925 tablet_bootstrap.cc:654] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.667446  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:36.668155  1878 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.592154  2167 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.674378  2057 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.674952  2057 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.675684  2167 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.676913  2057 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.681035  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:36.681721  1732 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.659606  1979 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.686223  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:36.686848  1805 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.687211  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:36.687426  1927 tablet_bootstrap.cc:654] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.695927  2145 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:36.699775  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:36.700768  1806 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:36.702621  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:36.708647  1979 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:21:36.709455  1770 leader_election.cc:343] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.715008  1732 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:36.717171  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:36.709398  1979 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:36.725845  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:36.733644  1925 tablet_bootstrap.cc:492] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:36.733979  1987 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.734146  1925 ts_tablet_manager.cc:1403] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.136s	user 0.018s	sys 0.012s
I20260709 12:21:36.735651  1979 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:36.736094  1979 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.736436  1979 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.738423  2160 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:36.724834  2219 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:36.741446  2219 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:21:36.729635  2241 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.057s	user 0.000s	sys 0.004s
I20260709 12:21:36.743017  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:36.750340  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:36.751187  1728 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:36.752426  1925 raft_consensus.cc:359] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.753211  1925 raft_consensus.cc:385] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.754391  1925 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.739409  2211 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:36.760187  1927 tablet_bootstrap.cc:492] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
W20260709 12:21:36.759609  1696 leader_election.cc:343] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.764523  1927 ts_tablet_manager.cc:1403] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.100s	user 0.009s	sys 0.019s
I20260709 12:21:36.744550  2160 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:36.767654  2160 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.753845  2219 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.772558  1878 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:36.772851  1927 raft_consensus.cc:359] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.773653  1927 raft_consensus.cc:385] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.773953  1927 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.774608  1927 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.776950  1924 tablet_bootstrap.cc:492] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:36.777729  1924 ts_tablet_manager.cc:1403] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.229s	user 0.046s	sys 0.108s
I20260709 12:21:36.779779  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:36.784462  1987 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:36.784821  1927 ts_tablet_manager.cc:1434] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.001s
I20260709 12:21:36.785007  1987 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:36.783231  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:36.775368  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.786062  1987 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.800299  2097 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:21:36.816682  1696 leader_election.cc:343] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.819453  1696 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:36.821914  2160 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:36.829810  1979 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.808650  2097 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.827521  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.833238  2097 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.831010  1925 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.840953  1805 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:36.841248  1924 raft_consensus.cc:359] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.842109  1924 raft_consensus.cc:385] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:36.842388  1924 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:36.843089  1924 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.870208  1925 ts_tablet_manager.cc:1434] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.135s	user 0.009s	sys 0.003s
I20260709 12:21:36.870260  1924 ts_tablet_manager.cc:1434] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.092s	user 0.006s	sys 0.002s
I20260709 12:21:36.871569  1925 tablet_bootstrap.cc:492] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:36.872211  1924 tablet_bootstrap.cc:492] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:36.835065  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:36.877431  1803 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:36.888682  2149 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:21:36.889366  2149 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.893587  1979 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:36.901810  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:36.902374  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:36.902853  1730 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:36.903321  1880 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:21:36.907315  1927 tablet_bootstrap.cc:492] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:36.917155  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:36.918154  1727 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:36.920722  1730 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:36.964383  1924 tablet_bootstrap.cc:654] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:36.967187  2253 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.966133  2097 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:36.984100  2211 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.015882  2211 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.018179  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:37.018941  1730 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:37.023795  1771 leader_election.cc:343] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:36.992959  2253 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.005818  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:37.023970  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:37.031699  1979 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:37.024602  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:37.012293  2097 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.018132  2211 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.032298  1979 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:37.032672  2057 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:21:37.035254  1771 leader_election.cc:343] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:37.035573  1979 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.038648  2142 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260709 12:21:37.047536  1770 leader_election.cc:343] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:37.051342  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:37.052343  1804 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:21:37.060846  2149 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:37.063228  2219 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.063889  2219 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.066018  2219 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.071246  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:37.072062  1727 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:37.074625  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:37.075743  1879 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:21:37.077055  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:37.078172  1878 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.078601  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:37.078236  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:37.088668  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:37.084602  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:37.091343  1805 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:37.093178  1925 tablet_bootstrap.cc:654] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:37.095037  1696 leader_election.cc:343] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:37.096347  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80899.5us
I20260709 12:21:37.097106  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:21:37.095093  1771 leader_election.cc:343] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:37.098428  1771 leader_election.cc:343] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:37.098773  1771 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:37.100175  2048 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:37.100601  2048 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.100941  2048 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.101521  2149 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:37.107213  1927 tablet_bootstrap.cc:654] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.083607  2253 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.110956  2048 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.116621  1627 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:37.081465  1730 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:37.113893  2219 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:37.164646  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:37.168707  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:21:37.171437  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73545.4us
I20260709 12:21:37.183496  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:37.189031  2097 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:37.196197  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:37.196862  1805 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.197532  2218 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:37.198047  2218 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.198438  2218 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.198552  1924 tablet_bootstrap.cc:492] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:37.199213  1924 ts_tablet_manager.cc:1403] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.327s	user 0.036s	sys 0.037s
I20260709 12:21:37.185787  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:37.202611  1925 tablet_bootstrap.cc:492] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:37.203460  1925 ts_tablet_manager.cc:1403] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.332s	user 0.011s	sys 0.011s
I20260709 12:21:37.208689  2048 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.193555  2097 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:37.222991  2097 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.233304  1924 raft_consensus.cc:359] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.234393  1924 raft_consensus.cc:385] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.235298  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:37.234746  1924 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.227924  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:37.239403  1924 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.240384  2149 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:21:37.244153  1696 leader_election.cc:343] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:37.247539  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:37.256067  1927 tablet_bootstrap.cc:492] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:37.256676  1927 ts_tablet_manager.cc:1403] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.363s	user 0.016s	sys 0.011s
W20260709 12:21:37.258229  1771 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.251611  1845 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:37.264729  1924 ts_tablet_manager.cc:1434] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.065s	user 0.007s	sys 0.008s
I20260709 12:21:37.267891  2250 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:37.270093  1927 raft_consensus.cc:359] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.270830  1927 raft_consensus.cc:385] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.271090  1927 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.271353  1924 tablet_bootstrap.cc:492] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:37.271745  1927 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.271611  2253 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:21:37.274124  1805 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:37.275729  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:37.278635  1927 ts_tablet_manager.cc:1434] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260709 12:21:37.279810  1927 tablet_bootstrap.cc:492] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
W20260709 12:21:37.281540  1696 consensus_peers.cc:597] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.282187  2253 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:37.282703  2253 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:37.283910  2253 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.286290  1925 raft_consensus.cc:359] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.287060  1925 raft_consensus.cc:385] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.286903  2219 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:37.287432  1925 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.288122  1925 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.309481  1925 ts_tablet_manager.cc:1434] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.106s	user 0.007s	sys 0.000s
I20260709 12:21:37.280263  2218 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.358680  1979 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:21:37.359083  2218 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.359248  1979 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.365408  1925 tablet_bootstrap.cc:492] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:37.368158  2254 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:37.369706  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:37.370347  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:21:37.383127  1771 leader_election.cc:343] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:37.395323  1979 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:37.375216  1728 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.410208  1927 tablet_bootstrap.cc:654] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.412842  1924 tablet_bootstrap.cc:654] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.417392  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:37.431746  1728 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:37.437165  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:37.439602  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:37.450318  2253 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:37.451414  2218 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:37.451826  1845 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:37.452276  2218 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:37.453918  2267 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:37.454262  2218 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.495877  1927 tablet_bootstrap.cc:492] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:37.496589  1927 ts_tablet_manager.cc:1403] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.217s	user 0.012s	sys 0.018s
I20260709 12:21:37.505734  2149 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:37.509620  1927 raft_consensus.cc:359] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.510164  1927 raft_consensus.cc:385] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.510437  1927 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.512677  1927 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.516686  1927 ts_tablet_manager.cc:1434] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 12:21:37.520047  1927 tablet_bootstrap.cc:492] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:37.520916  1925 tablet_bootstrap.cc:654] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.529605  2267 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.530179  2267 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.537106  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:37.538069  2267 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:37.538239  1804 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:37.541342  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:37.544449  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:37.544487  1927 tablet_bootstrap.cc:654] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.544907  2250 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:37.545142  1727 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:37.545454  2250 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.546432  2250 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.548496  2149 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:37.549230  1924 tablet_bootstrap.cc:492] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:37.549721  1924 ts_tablet_manager.cc:1403] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.279s	user 0.010s	sys 0.010s
I20260709 12:21:37.573855  1924 raft_consensus.cc:359] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.574928  1924 raft_consensus.cc:385] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.575318  1924 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.576262  1924 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.580222  2250 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.583761  1924 ts_tablet_manager.cc:1434] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.034s	user 0.006s	sys 0.004s
I20260709 12:21:37.585055  1924 tablet_bootstrap.cc:492] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:37.588145  2250 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:37.589403  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:37.589713  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:37.590967  1803 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.591071  1730 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.592344  1927 tablet_bootstrap.cc:492] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:37.592412  1925 tablet_bootstrap.cc:492] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:37.593050  1927 ts_tablet_manager.cc:1403] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.074s	user 0.009s	sys 0.008s
I20260709 12:21:37.593330  1925 ts_tablet_manager.cc:1403] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.242s	user 0.027s	sys 0.024s
I20260709 12:21:37.595948  1927 raft_consensus.cc:359] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.596678  1924 tablet_bootstrap.cc:654] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.596684  1927 raft_consensus.cc:385] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.597421  1927 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.598450  1927 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.601553  1927 ts_tablet_manager.cc:1434] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:21:37.602636  1927 tablet_bootstrap.cc:492] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:37.604343  1925 raft_consensus.cc:359] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.605772  1730 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:37.608501  1844 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:37.609707  2250 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:37.605139  1925 raft_consensus.cc:385] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.610255  1925 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.610488  2250 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:37.610639  1803 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:37.611188  1925 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.612960  1927 tablet_bootstrap.cc:654] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.613443  1925 ts_tablet_manager.cc:1434] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.020s	user 0.000s	sys 0.004s
I20260709 12:21:37.614506  1925 tablet_bootstrap.cc:492] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:37.621120  1924 tablet_bootstrap.cc:492] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:37.620549  1925 tablet_bootstrap.cc:654] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.623886  1924 ts_tablet_manager.cc:1403] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.039s	user 0.004s	sys 0.008s
I20260709 12:21:37.622939  1927 tablet_bootstrap.cc:492] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:37.627626  1927 ts_tablet_manager.cc:1403] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.025s	user 0.016s	sys 0.000s
I20260709 12:21:37.627277  1924 raft_consensus.cc:359] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.628197  1924 raft_consensus.cc:385] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.628525  1924 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.629319  1924 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.616209  2250 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.632296  1927 raft_consensus.cc:359] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.677336  1927 raft_consensus.cc:385] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.677816  1927 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.678622  1927 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.680876  2149 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.681481  2149 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.683712  2149 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:37.683526  1630 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:37.685374  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:37.687186  1927 ts_tablet_manager.cc:1434] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.002s
I20260709 12:21:37.688437  1927 tablet_bootstrap.cc:492] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:37.689002  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:37.691387  1924 ts_tablet_manager.cc:1434] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.067s	user 0.009s	sys 0.001s
I20260709 12:21:37.692489  1924 tablet_bootstrap.cc:492] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:37.695171  2218 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.695473  1927 tablet_bootstrap.cc:654] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.695765  2218 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.698467  2218 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.700414  1628 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:37.723341  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:37.707115  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:37.734967  1925 tablet_bootstrap.cc:492] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:37.738869  1925 ts_tablet_manager.cc:1403] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.124s	user 0.020s	sys 0.039s
I20260709 12:21:37.711133  1845 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:21:37.736884  1696 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:21:37.740068  1771 leader_election.cc:343] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:37.711450  2152 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.745090  1924 tablet_bootstrap.cc:654] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.745473  1925 raft_consensus.cc:359] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.746440  2244 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:37.739734  1727 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:37.754346  1925 raft_consensus.cc:385] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.755347  1925 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.762703  1925 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.765539  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:37.767360  2219 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:37.770303  2219 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.772056  2219 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.775913  1925 ts_tablet_manager.cc:1434] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.037s	user 0.002s	sys 0.005s
I20260709 12:21:37.777202  1925 tablet_bootstrap.cc:492] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:37.777654  2213 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:37.791965  1925 tablet_bootstrap.cc:654] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:37.800405  2167 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:37.807657  1881 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.799551  1630 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:37.820751  2219 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.757758  2152 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.838547  1927 tablet_bootstrap.cc:492] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:37.839357  1927 ts_tablet_manager.cc:1403] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.151s	user 0.016s	sys 0.008s
I20260709 12:21:37.843225  2218 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.843832  2218 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.844095  1927 raft_consensus.cc:359] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.845163  1927 raft_consensus.cc:385] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.845510  1927 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.848945  2218 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.850894  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:37.851755  1730 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:37.853211  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:37.855207  2167 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:37.855626  2167 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.856002  2167 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.856030  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:37.857339  2152 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.858637  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:37.858920  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:37.860558  1696 leader_election.cc:343] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:37.861209  1696 leader_election.cc:343] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:37.861570  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:37.862535  2213 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:21:37.866093  1771 leader_election.cc:343] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:37.846393  1927 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.870891  2097 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:37.880332  1927 ts_tablet_manager.cc:1434] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.041s	user 0.009s	sys 0.000s
I20260709 12:21:37.881795  1927 tablet_bootstrap.cc:492] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:37.889202  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:21:37.890837  1771 leader_election.cc:343] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:37.894830  2219 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.896483  2218 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:37.900002  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:37.900876  1727 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.917109  2097 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.917667  2097 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.919843  2097 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.921298  1925 tablet_bootstrap.cc:492] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:37.921947  1925 ts_tablet_manager.cc:1403] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.145s	user 0.017s	sys 0.039s
I20260709 12:21:37.921734  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:37.922502  1880 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:37.923877  1771 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:37.924751  2097 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:37.925112  2097 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.925482  2097 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.932257  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:37.937067  1927 tablet_bootstrap.cc:654] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:37.945835  1770 leader_election.cc:343] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:37.950919  1771 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:37.963582  1927 tablet_bootstrap.cc:492] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:37.964198  1927 ts_tablet_manager.cc:1403] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.083s	user 0.012s	sys 0.001s
I20260709 12:21:37.956738  1925 raft_consensus.cc:359] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.966832  1925 raft_consensus.cc:385] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:37.932376  2097 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.967223  1925 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:37.894848  2167 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.969010  2097 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.954263  1628 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:21:37.995618  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:37.996518  1881 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:37.996063  1628 catalog_manager.cc:5697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:37.971738  1925 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:37.959163  1924 tablet_bootstrap.cc:492] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:37.999902  1924 ts_tablet_manager.cc:1403] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.308s	user 0.034s	sys 0.060s
I20260709 12:21:38.001336  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:38.004375  1925 ts_tablet_manager.cc:1434] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.082s	user 0.011s	sys 0.000s
I20260709 12:21:38.005774  1925 tablet_bootstrap.cc:492] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:37.973884  2167 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:37.958503  1727 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:37.998023  1628 catalog_manager.cc:5697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.019742  1925 tablet_bootstrap.cc:654] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.030915  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:38.031739  1726 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:38.039711  1771 leader_election.cc:343] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.041108  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.042078  2218 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:38.042608  2218 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:38.043486  2218 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.051350  2250 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:38.059325  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:38.064618  1927 raft_consensus.cc:359] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.065078  1924 raft_consensus.cc:359] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.065791  1924 raft_consensus.cc:385] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.068424  1924 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.069267  1924 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.073217  2253 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.065577  1927 raft_consensus.cc:385] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.075026  1927 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.076683  1881 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:38.076313  1630 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.078593  1927 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.083616  1927 ts_tablet_manager.cc:1434] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.119s	user 0.009s	sys 0.000s
I20260709 12:21:38.074345  2253 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.084919  1927 tablet_bootstrap.cc:492] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:38.085886  2253 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.088435  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:38.089265  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:38.090106  1804 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:38.073268  1924 ts_tablet_manager.cc:1434] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.073s	user 0.004s	sys 0.002s
I20260709 12:21:38.121721  1925 tablet_bootstrap.cc:492] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:38.122408  1925 ts_tablet_manager.cc:1403] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.117s	user 0.022s	sys 0.004s
I20260709 12:21:38.094911  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:38.103982  1771 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:21:38.124785  1696 leader_election.cc:343] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.130918  1924 tablet_bootstrap.cc:492] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:38.133431  2142 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:38.136066  2142 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:38.137192  2142 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.093719  1726 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:38.104671  2219 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.130301  2253 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:38.142900  1925 raft_consensus.cc:359] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.143862  1925 raft_consensus.cc:385] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.144196  1925 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
W20260709 12:21:38.142712  1770 leader_election.cc:343] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.149137  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.152770  1925 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.141314  2253 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.155193  2208 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:38.158901  2253 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.161512  1925 ts_tablet_manager.cc:1434] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.039s	user 0.007s	sys 0.000s
I20260709 12:21:38.162696  1925 tablet_bootstrap.cc:492] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:38.165391  2208 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:38.166750  2208 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.140833  2219 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.187570  2219 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.195339  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:21:38.197962  1771 leader_election.cc:343] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.201785  1924 tablet_bootstrap.cc:654] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.210240  1927 tablet_bootstrap.cc:654] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.209616  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:21:38.214123  1770 leader_election.cc:343] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.214854  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.216511  2302 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:38.221640  1628 catalog_manager.cc:5697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.220059  2254 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.223435  2254 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.232268  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:21:38.234123  1696 leader_election.cc:343] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.303408  2253 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.355752  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:38.355988  1925 tablet_bootstrap.cc:654] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.319034  2254 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.372191  1924 tablet_bootstrap.cc:492] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:38.372795  1924 ts_tablet_manager.cc:1403] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.242s	user 0.043s	sys 0.050s
W20260709 12:21:38.374565  1696 leader_election.cc:343] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.375207  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.374190  2244 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.383031  2244 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:21:38.384096  2303 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.184s	user 0.002s	sys 0.000s
I20260709 12:21:38.385679  2244 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:38.352831  2250 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.035s
I20260709 12:21:38.396457  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:38.380858  2152 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:38.394006  2213 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.401959  2213 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.406193  2253 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.407356  1924 raft_consensus.cc:359] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.408124  1924 raft_consensus.cc:385] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.408427  1924 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.414310  1924 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.420856  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:38.425022  1878 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.429939  1927 tablet_bootstrap.cc:492] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:38.430536  1927 ts_tablet_manager.cc:1403] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.346s	user 0.012s	sys 0.031s
I20260709 12:21:38.430414  2213 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.433938  1927 raft_consensus.cc:359] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.434609  1927 raft_consensus.cc:385] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.434913  1927 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.435487  1927 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.437289  1924 ts_tablet_manager.cc:1434] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.064s	user 0.007s	sys 0.000s
W20260709 12:21:38.444278  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:38.445272  1924 tablet_bootstrap.cc:492] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:38.437430  1927 ts_tablet_manager.cc:1434] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:21:38.459587  1927 tablet_bootstrap.cc:492] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:38.459348  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:38.475461  1925 tablet_bootstrap.cc:492] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:38.475996  1925 ts_tablet_manager.cc:1403] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.314s	user 0.013s	sys 0.011s
I20260709 12:21:38.478744  1924 tablet_bootstrap.cc:654] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.480756  2244 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:38.490000  1925 raft_consensus.cc:359] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.490902  1925 raft_consensus.cc:385] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.491276  1925 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.486474  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:38.493582  1803 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.493527  1925 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.495923  1925 ts_tablet_manager.cc:1434] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.000s
I20260709 12:21:38.497000  1925 tablet_bootstrap.cc:492] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:38.504122  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:38.509366  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:38.515090  2301 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:21:38.520844  1803 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
W20260709 12:21:38.513221  1696 leader_election.cc:343] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:38.523475  1696 leader_election.cc:343] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.537683  2253 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.538288  2253 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.540408  2253 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.542384  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.544731  2253 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:38.545235  2253 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:38.546113  2253 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.547286  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:38.557268  1627 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.566919  1627 catalog_manager.cc:5697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.572392  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:38.562573  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:38.573910  1880 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 1: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:21:38.574376  1803 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:38.601841  1924 tablet_bootstrap.cc:492] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:38.602653  1924 ts_tablet_manager.cc:1403] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.158s	user 0.044s	sys 0.022s
W20260709 12:21:38.618932  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 52877.3us
I20260709 12:21:38.638103  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.639014  2315 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.639444  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.640290  2048 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:38.640695  2048 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.641063  2048 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.644345  1927 tablet_bootstrap.cc:654] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:38.646843  1771 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:38.659761  1924 raft_consensus.cc:359] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.660652  1924 raft_consensus.cc:385] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.660938  1924 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.661688  1924 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.664620  1924 ts_tablet_manager.cc:1434] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.062s	user 0.006s	sys 0.000s
I20260709 12:21:38.668694  1924 tablet_bootstrap.cc:492] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:38.669734  1925 tablet_bootstrap.cc:654] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.692688  2048 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.699509  2048 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.701223  2149 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.701838  2149 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.713593  2149 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:38.715662  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:21:38.717567  1844 leader_election.cc:343] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.725026  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:38.727528  1803 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:38.729018  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:38.739063  1845 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:38.738715  1627 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:38.740392  1696 leader_election.cc:343] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.742204  2149 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:38.742666  2149 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:38.745517  1771 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:38.750298  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:38.751132  1807 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.754087  2208 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:38.754707  2315 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:38.756340  2149 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.766710  2218 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:38.759274  1807 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:38.770126  1925 tablet_bootstrap.cc:492] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:38.770735  1925 ts_tablet_manager.cc:1403] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.274s	user 0.015s	sys 0.009s
I20260709 12:21:38.771814  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.782438  1927 tablet_bootstrap.cc:492] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:38.783054  1927 ts_tablet_manager.cc:1403] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.324s	user 0.017s	sys 0.015s
I20260709 12:21:38.785929  2315 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:38.791018  1924 tablet_bootstrap.cc:654] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.791137  2315 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:38.792037  1925 raft_consensus.cc:359] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.792150  2315 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.795145  1925 raft_consensus.cc:385] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.797230  1925 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.798008  1925 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.799772  2301 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:38.805213  1925 ts_tablet_manager.cc:1434] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.034s	user 0.007s	sys 0.001s
I20260709 12:21:38.807279  1925 tablet_bootstrap.cc:492] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:38.803220  2149 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.812819  2149 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:38.815683  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:38.816541  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:38.817168  1807 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:38.817899  1844 leader_election.cc:343] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.829193  1807 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:38.828753  1927 raft_consensus.cc:359] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.829885  1927 raft_consensus.cc:385] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.830119  1927 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.830838  1845 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:21:38.831290  2317 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.832466  2317 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.830613  1927 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.834350  2149 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:38.839157  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:38.840760  1927 ts_tablet_manager.cc:1434] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.057s	user 0.009s	sys 0.000s
W20260709 12:21:38.842238  1696 leader_election.cc:343] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.843159  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:38.841626  2317 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:21:38.844350  1696 leader_election.cc:343] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.844725  1696 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.845037  1927 tablet_bootstrap.cc:492] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:38.846410  2048 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:38.840420  2149 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:38.848266  2149 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.860683  1925 tablet_bootstrap.cc:654] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.871434  1924 tablet_bootstrap.cc:492] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:38.872071  1924 ts_tablet_manager.cc:1403] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.204s	user 0.017s	sys 0.005s
I20260709 12:21:38.887737  1924 raft_consensus.cc:359] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.888536  1924 raft_consensus.cc:385] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.888808  1924 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.889525  1924 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.894168  1630 catalog_manager.cc:5697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.896751  1927 tablet_bootstrap.cc:654] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.901444  1925 tablet_bootstrap.cc:492] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:38.901981  1925 ts_tablet_manager.cc:1403] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.096s	user 0.010s	sys 0.005s
I20260709 12:21:38.902835  1924 ts_tablet_manager.cc:1434] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.000s
I20260709 12:21:38.905642  1925 raft_consensus.cc:359] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.906772  1925 raft_consensus.cc:385] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.907184  1925 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.908206  1925 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.908876  1924 tablet_bootstrap.cc:492] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:38.915920  1925 ts_tablet_manager.cc:1434] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.000s
I20260709 12:21:38.918119  1924 tablet_bootstrap.cc:654] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.917110  1925 tablet_bootstrap.cc:492] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Bootstrap starting.
I20260709 12:21:38.933960  1925 tablet_bootstrap.cc:654] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:38.936399  1627 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:38.941475  1924 tablet_bootstrap.cc:492] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:38.941831  2244 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.942059  1924 ts_tablet_manager.cc:1403] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.033s	user 0.009s	sys 0.013s
I20260709 12:21:38.942368  2244 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.944809  2244 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:38.945137  1924 raft_consensus.cc:359] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.945847  1924 raft_consensus.cc:385] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.945824  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:38.946406  1924 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.947368  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:38.947300  1924 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.941491  1925 tablet_bootstrap.cc:492] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:38.948987  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:38.949344  1925 ts_tablet_manager.cc:1403] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.032s	user 0.011s	sys 0.001s
I20260709 12:21:38.949734  1924 ts_tablet_manager.cc:1434] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:21:38.950533  2149 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:38.950832  1924 tablet_bootstrap.cc:492] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:38.956382  2315 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.956966  2315 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.957269  1925 raft_consensus.cc:359] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.958110  1925 raft_consensus.cc:385] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.958417  1925 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.959942  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:38.959398  1925 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.958983  1924 tablet_bootstrap.cc:654] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
W20260709 12:21:38.961416  1696 leader_election.cc:343] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.962498  2315 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.964174  2048 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.964776  2048 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.967080  2048 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.970151  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:38.970151  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:38.971042  1807 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
I20260709 12:21:38.972203  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:38.972654  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:38.974123  2253 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:38.974519  2253 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.974895  2253 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.975723  1925 ts_tablet_manager.cc:1434] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 12:21:38.985016  2253 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.987109  1925 tablet_bootstrap.cc:492] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Bootstrap starting.
W20260709 12:21:38.987146  1696 leader_election.cc:343] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:38.988195  1696 leader_election.cc:343] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:38.988577  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:21:38.990303  2218 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.991003  2218 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.991731  2315 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:38.985785  1927 tablet_bootstrap.cc:492] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:38.992933  1927 ts_tablet_manager.cc:1403] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.148s	user 0.008s	sys 0.008s
I20260709 12:21:38.993742  2218 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:38.994475  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:38.995100  1807 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:38.996210  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:38.996098  1927 raft_consensus.cc:359] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:38.996986  1927 raft_consensus.cc:385] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:38.997043  1724 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:38.997475  1927 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:38.998481  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:38.999310  2208 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:38.999747  2208 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.000098  2208 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.002772  1807 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:39.003574  2253 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:39.004459  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:39.005105  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:39.005738  2253 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:39.005465  1927 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.006546  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:39.006939  2253 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:39.008056  2253 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.012466  1927 ts_tablet_manager.cc:1434] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.001s
W20260709 12:21:39.013298  1696 leader_election.cc:343] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:21:39.013659  1771 leader_election.cc:343] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.014190  1927 tablet_bootstrap.cc:492] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:39.016080  1878 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.033090  1927 tablet_bootstrap.cc:654] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.041199  1925 tablet_bootstrap.cc:654] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.041754  2149 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.049651  2208 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.053761  1924 tablet_bootstrap.cc:492] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:39.054279  1924 ts_tablet_manager.cc:1403] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.104s	user 0.006s	sys 0.006s
I20260709 12:21:39.042426  2149 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.070330  2149 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:39.073132  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:39.073922  1724 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.076071  2208 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:39.088948  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:39.096596  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:39.098253  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:39.104238  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:21:39.106055  1696 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:39.107944  2149 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:39.108980  1924 raft_consensus.cc:359] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.109722  1924 raft_consensus.cc:385] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.109972  1924 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.110558  1924 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.111470  1724 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:39.115763  1924 ts_tablet_manager.cc:1434] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.061s	user 0.005s	sys 0.009s
I20260709 12:21:39.114916  1627 catalog_manager.cc:5697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:39.116212  1927 tablet_bootstrap.cc:492] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:39.116741  1927 ts_tablet_manager.cc:1403] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.103s	user 0.020s	sys 0.003s
I20260709 12:21:39.116866  1924 tablet_bootstrap.cc:492] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:39.119550  1878 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.119432  1927 raft_consensus.cc:359] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.120196  1927 raft_consensus.cc:385] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.121428  1927 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.122494  1879 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.127195  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:39.129010  2208 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:39.129647  2208 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:39.130508  2208 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.134754  1924 tablet_bootstrap.cc:654] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.136350  1925 tablet_bootstrap.cc:492] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: No bootstrap required, opened a new log
I20260709 12:21:39.136876  1925 ts_tablet_manager.cc:1403] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent bootstrapping tablet: real 0.150s	user 0.021s	sys 0.004s
I20260709 12:21:39.139024  2253 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:39.139351  1927 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.146551  1927 ts_tablet_manager.cc:1434] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.000s
I20260709 12:21:39.154620  1925 raft_consensus.cc:359] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.155507  1925 raft_consensus.cc:385] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.155817  1925 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.156517  1925 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.158682  1927 tablet_bootstrap.cc:492] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:39.166891  1925 ts_tablet_manager.cc:1434] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.000s
I20260709 12:21:39.166339  1627 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:39.172430  1927 tablet_bootstrap.cc:654] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.173679  2244 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.174252  2244 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.178561  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:39.179247  2244 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:39.181852  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:39.183974  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:39.184883  1927 tablet_bootstrap.cc:492] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:39.185221  2244 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:39.185421  1927 ts_tablet_manager.cc:1403] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.038s	user 0.013s	sys 0.004s
I20260709 12:21:39.189415  1927 raft_consensus.cc:359] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.190124  1927 raft_consensus.cc:385] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.190412  1927 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.191020  2253 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:39.191150  1927 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.194705  1927 ts_tablet_manager.cc:1434] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:21:39.196270  1927 tablet_bootstrap.cc:492] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Bootstrap starting.
I20260709 12:21:39.194491  1924 tablet_bootstrap.cc:492] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:39.197796  1924 ts_tablet_manager.cc:1403] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.081s	user 0.024s	sys 0.018s
I20260709 12:21:39.200718  1924 raft_consensus.cc:359] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.201479  1924 raft_consensus.cc:385] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.201803  1924 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.207892  1927 tablet_bootstrap.cc:654] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.214560  1924 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.219332  1924 ts_tablet_manager.cc:1434] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
W20260709 12:21:39.216024  1696 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:39.220791  1924 tablet_bootstrap.cc:492] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:39.223779  1878 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:39.228140  1924 tablet_bootstrap.cc:654] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.241936  1924 tablet_bootstrap.cc:492] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:39.242183  2301 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.242674  1924 ts_tablet_manager.cc:1403] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.022s	user 0.018s	sys 0.000s
I20260709 12:21:39.242950  2301 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.244133  2244 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.244684  2244 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.248720  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:39.249341  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:39.251811  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:39.252364  2301 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:39.252449  1730 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:39.252916  1845 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:39.254315  2149 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:39.254630  2244 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:39.258301  1924 raft_consensus.cc:359] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.258708  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:39.259114  1924 raft_consensus.cc:385] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.259382  1924 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
W20260709 12:21:39.260006  1771 leader_election.cc:343] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.260092  1924 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.261713  1770 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:39.247714  2253 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.263691  2253 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.265425  2301 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:39.265873  2301 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.266158  2253 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:39.270285  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:39.270751  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:39.271272  1807 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 0.
W20260709 12:21:39.272269  1696 leader_election.cc:343] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.273003  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:21:39.276531  1927 tablet_bootstrap.cc:492] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: No bootstrap required, opened a new log
I20260709 12:21:39.277012  1927 ts_tablet_manager.cc:1403] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent bootstrapping tablet: real 0.081s	user 0.022s	sys 0.023s
I20260709 12:21:39.266208  2301 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.279670  1927 raft_consensus.cc:359] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.280359  1927 raft_consensus.cc:385] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.280644  1927 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.281327  1927 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.282209  1924 ts_tablet_manager.cc:1434] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.039s	user 0.008s	sys 0.000s
I20260709 12:21:39.281549  2253 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:39.283540  2253 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.283960  2253 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.284428  1924 tablet_bootstrap.cc:492] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:39.285833  1927 ts_tablet_manager.cc:1434] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:21:39.291613  1924 tablet_bootstrap.cc:654] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.295634  2253 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.301282  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:39.309001  2301 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.312245  2301 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:39.312697  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:39.313405  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:39.313436  1724 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.315721  1806 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.322135  1924 tablet_bootstrap.cc:492] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:39.322273  1724 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:39.323864  1770 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:39.323063  1924 ts_tablet_manager.cc:1403] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.039s	user 0.020s	sys 0.005s
I20260709 12:21:39.326134  1878 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.326588  2208 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:39.327241  2208 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:39.328181  2208 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.330153  1924 raft_consensus.cc:359] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.331249  1924 raft_consensus.cc:385] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.331393  2219 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.331640  1924 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.332002  2219 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.332818  2149 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.333350  2149 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.335644  2149 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:39.337298  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:39.338011  1807 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:39.338636  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:39.342034  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:39.334527  2219 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:39.356243  2149 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:39.356691  2149 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.357071  2149 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.342182  2253 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:39.342360  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:39.340480  1924 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.343818  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:39.361972  1881 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:21:39.344013  1771 leader_election.cc:343] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:21:39.346443  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:39.339344  1726 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 0.
I20260709 12:21:39.363552  1924 ts_tablet_manager.cc:1434] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.000s
I20260709 12:21:39.365818  1924 tablet_bootstrap.cc:492] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:39.331665  1806 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:39.370455  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:39.371315  2253 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:39.368291  2149 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.375479  2253 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:21:39.375953  1879 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 1: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:21:39.376452  1924 tablet_bootstrap.cc:654] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.376495  2253 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.378340  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:21:39.369920  1881 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:21:39.379832  2219 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.390273  2149 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:39.392577  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:21:39.393029  1725 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.393620  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:39.394222  1726 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.402511  2219 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:39.403702  1726 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:39.404299  2244 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:39.406632  1844 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:21:39.407543  2244 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:39.408025  2244 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:21:39.408859  2244 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.409375  1807 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:21:39.417886  1627 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:39.431564  1924 tablet_bootstrap.cc:492] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:39.432314  1924 ts_tablet_manager.cc:1403] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.067s	user 0.024s	sys 0.009s
I20260709 12:21:39.435745  1924 raft_consensus.cc:359] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.437099  1924 raft_consensus.cc:385] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.437431  1924 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.440385  1924 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.443135  1924 ts_tablet_manager.cc:1434] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:21:39.443063  1628 catalog_manager.cc:5697] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 0 to 1, leader changed from <none> to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:39.444482  1924 tablet_bootstrap.cc:492] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:39.461071  1629 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 0 to 1, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:39.462129  1924 tablet_bootstrap.cc:654] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.473872  1924 tablet_bootstrap.cc:492] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:39.474509  1924 ts_tablet_manager.cc:1403] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.003s
I20260709 12:21:39.477928  1924 raft_consensus.cc:359] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.478649  1924 raft_consensus.cc:385] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.479048  1924 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.479481  2244 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:39.479839  1924 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.482348  1924 ts_tablet_manager.cc:1434] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 12:21:39.486356  1924 tablet_bootstrap.cc:492] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:39.494369  1924 tablet_bootstrap.cc:654] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.497404  1979 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.497924  1979 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.500578  1881 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.500806  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:39.501610  1807 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:21:39.502553  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:39.504251  1979 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:39.504451  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:39.516067  1924 tablet_bootstrap.cc:492] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:39.516526  1924 ts_tablet_manager.cc:1403] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.030s	user 0.013s	sys 0.000s
I20260709 12:21:39.519258  1924 raft_consensus.cc:359] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.519891  1924 raft_consensus.cc:385] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.520154  1924 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.520723  1924 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.521394  2149 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:39.522755  1924 ts_tablet_manager.cc:1434] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:21:39.524663  2253 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:39.527081  2308 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:39.527997  1924 tablet_bootstrap.cc:492] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
W20260709 12:21:39.535128  1678 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:21:39.543493  1924 tablet_bootstrap.cc:654] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.544183  2149 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.544663  2149 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.559396  2306 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:39.565235  1878 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.575680  2208 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:39.603111  2149 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:39.606281  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:39.607228  1730 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 1: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:21:39.610699  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:21:39.612704  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:21:39.613970  1979 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.627532  1979 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:21:39.635712  1924 tablet_bootstrap.cc:492] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:39.636184  1924 ts_tablet_manager.cc:1403] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.108s	user 0.030s	sys 0.018s
I20260709 12:21:39.639834  1924 raft_consensus.cc:359] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.640586  1924 raft_consensus.cc:385] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.640908  1924 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.641703  1924 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.643922  1924 ts_tablet_manager.cc:1434] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:21:39.647920  1924 tablet_bootstrap.cc:492] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Bootstrap starting.
I20260709 12:21:39.655313  1924 tablet_bootstrap.cc:654] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Neither blocks nor log segments found. Creating new log.
I20260709 12:21:39.659675  1924 tablet_bootstrap.cc:492] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: No bootstrap required, opened a new log
I20260709 12:21:39.660573  1924 ts_tablet_manager.cc:1403] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent bootstrapping tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:21:39.662863  1924 raft_consensus.cc:359] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.663321  1924 raft_consensus.cc:385] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:21:39.663558  1924 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Initialized, Role: FOLLOWER
I20260709 12:21:39.664196  1924 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.666029  1924 ts_tablet_manager.cc:1434] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20260709 12:21:39.718114  2302 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.718755  2302 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.720921  2302 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:39.721709  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:21:39.721997  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:21:39.722507  1726 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:39.722682  1881 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 0.
I20260709 12:21:39.723659  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:39.724431  2302 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:21:39.724766  2302 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.725013  2302 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.731422  2302 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.733114  2302 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:39.734194  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:21:39.734194  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:21:39.735208  1726 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.735165  1881 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.743530  1726 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:39.744766  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:21:39.745568  2302 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:21:39.746177  2302 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:21:39.747256  2302 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:21:39.748760  2219 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:39.750870  1881 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:21:39.762144  1629 catalog_manager.cc:5697] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:39.769114  2218 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:39.785980  1878 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.803767  1880 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:21:39.888734  2306 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:39.901391  2048 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:39.943086  2149 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:39.956076  2149 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:40.235565  2219 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:21:40.253959  2219 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:21:41.049698   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:21:41.089438  2372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:41.089540  2373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:21:41.091985  2375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:21:41.094889   561 server_base.cc:1061] running on GCE node
I20260709 12:21:41.096539   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:21:41.096956   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:21:41.097256   561 hybrid_clock.cc:648] HybridClock initialized: now 1783599701097236 us; error 0 us; skew 500 ppm
I20260709 12:21:41.098104   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:21:41.106119   561 webserver.cc:533] Webserver started at http://127.0.140.68:39969/ using document root <none> and password file <none>
I20260709 12:21:41.106925   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:21:41.107245   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:21:41.107673   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:21:41.109928   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-3-root/instance:
uuid: "123095127f334445889c21a42ba1d41e"
format_stamp: "Formatted at 2026-07-09 12:21:41 on dist-test-slave-7mcn"
I20260709 12:21:41.116734   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.002s
I20260709 12:21:41.121393  2381 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:41.122586   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:21:41.123032   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-3-root
uuid: "123095127f334445889c21a42ba1d41e"
format_stamp: "Formatted at 2026-07-09 12:21:41 on dist-test-slave-7mcn"
I20260709 12:21:41.123448   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:21:41.155156   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:21:41.157018   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:21:41.159201   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:21:41.162153   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:21:41.162360   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:21:41.162747   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:21:41.163021   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.001s
I20260709 12:21:41.238665   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.68:37665
I20260709 12:21:41.238869  2443 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.68:37665 every 8 connection(s)
I20260709 12:21:41.244875   561 auto_leader_rebalancer-test.cc:439] add a tserver: 123095127f334445889c21a42ba1d41e
I20260709 12:21:41.265276  2444 heartbeater.cc:344] Connected to a master server at 127.0.140.126:43371
I20260709 12:21:41.265769  2444 heartbeater.cc:461] Registering TS with master...
I20260709 12:21:41.266569  2444 heartbeater.cc:507] Master 127.0.140.126:43371 requested a full tablet report, sending...
I20260709 12:21:41.268882  1627 ts_manager.cc:194] Registered new tserver with Master: 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:21:41.270623  1627 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46106
I20260709 12:21:41.682224  1806 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:41.695960  1880 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:41.697973  2219 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:41.698007  1730 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:41.701364  2219 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:41.721402  2219 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.729776  1730 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.730125  1880 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:21:41.739573  1772 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84ce9896327e4c9d9888e80e8dbeeb0d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:41.743343  1803 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:41.743911  1627 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.770833  1880 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:41.772598  2377 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:41.774343  1772 consensus_peers.cc:597] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 62f288182b21420f8ef28e25fc10cd70. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:41.776429  1730 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:41.781019  2377 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.799599  2455 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.807539  1730 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.809927  1878 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.849838  1628 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:41.868183  1880 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:41.875921  1803 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:41.877537  2244 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.882689  1724 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:41.884405  1979 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.891304  2460 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:21:41.894114  1846 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36f1f37b7d544bb29cd9cace84f6e946. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:41.892668  1803 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.909727  1803 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:41.910863  1629 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.925640  1730 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:21:41.942745  1772 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82eacf0c2a8c4d8db4052327304a8793. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:41.942672  1880 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:41.944799  2219 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.945263  1726 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:41.946904  2377 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:41.957979  2219 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.962828  1880 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.982481  1726 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:41.987578  1629 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:42.186038  2474 ts_tablet_manager.cc:933] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:42.196287  2474 tablet_copy_client.cc:323] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:42.205899  2475 ts_tablet_manager.cc:933] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:42.220111  2475 tablet_copy_client.cc:323] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:42.237646  1817 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 84ce9896327e4c9d9888e80e8dbeeb0d from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:42.238397  1817 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 84ce9896327e4c9d9888e80e8dbeeb0d from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-84ce9896327e4c9d9888e80e8dbeeb0d
I20260709 12:21:42.238585  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 62f288182b21420f8ef28e25fc10cd70 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:42.242065  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 62f288182b21420f8ef28e25fc10cd70 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-62f288182b21420f8ef28e25fc10cd70
I20260709 12:21:42.251869  1816 tablet_copy_source_session.cc:215] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:42.251869  1817 tablet_copy_source_session.cc:215] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:42.257478  2475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f288182b21420f8ef28e25fc10cd70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:42.258838  2474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84ce9896327e4c9d9888e80e8dbeeb0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:42.273769  2475 tablet_copy_client.cc:806] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:42.273942  2444 heartbeater.cc:499] Master 127.0.140.126:43371 was elected leader, sending a full tablet report...
I20260709 12:21:42.274590  2475 tablet_copy_client.cc:670] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:42.281390  2475 tablet_copy_client.cc:538] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:42.290594  2474 tablet_copy_client.cc:806] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:42.291455  2474 tablet_copy_client.cc:670] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:42.295552  2475 tablet_bootstrap.cc:492] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:42.295907  2474 tablet_copy_client.cc:538] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:42.304154  2474 tablet_bootstrap.cc:492] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:42.337859  2475 tablet_bootstrap.cc:492] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:42.339416  2475 tablet_bootstrap.cc:492] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:42.340308  2475 ts_tablet_manager.cc:1403] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.045s	user 0.026s	sys 0.007s
I20260709 12:21:42.345455  2475 raft_consensus.cc:359] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:42.348541  2475 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:42.349339  2475 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:42.352120  2480 ts_tablet_manager.cc:933] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:42.366412  2475 ts_tablet_manager.cc:1434] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.004s
I20260709 12:21:42.369637  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-62f288182b21420f8ef28e25fc10cd70 received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:42.370147  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-62f288182b21420f8ef28e25fc10cd70 on tablet 62f288182b21420f8ef28e25fc10cd70 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:42.380317  2474 tablet_bootstrap.cc:492] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:42.381541  2474 tablet_bootstrap.cc:492] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:42.382391  2474 ts_tablet_manager.cc:1403] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.078s	user 0.027s	sys 0.007s
I20260709 12:21:42.385630  2474 raft_consensus.cc:359] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:42.386610  2474 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:42.387339  2474 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:42.390545  2474 ts_tablet_manager.cc:1434] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:21:42.394924  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-84ce9896327e4c9d9888e80e8dbeeb0d received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:42.395426  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-84ce9896327e4c9d9888e80e8dbeeb0d on tablet 84ce9896327e4c9d9888e80e8dbeeb0d with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:42.402161  2480 tablet_copy_client.cc:323] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:21:42.419829  1891 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 36f1f37b7d544bb29cd9cace84f6e946 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:21:42.420454  1891 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 36f1f37b7d544bb29cd9cace84f6e946 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-36f1f37b7d544bb29cd9cace84f6e946
I20260709 12:21:42.428212  1891 tablet_copy_source_session.cc:215] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:42.433692  2480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f1f37b7d544bb29cd9cace84f6e946. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:42.449944  2480 tablet_copy_client.cc:806] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:42.450588  2480 tablet_copy_client.cc:670] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:42.455076  2480 tablet_copy_client.cc:538] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:42.464505  2480 tablet_bootstrap.cc:492] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:42.484858  2480 tablet_bootstrap.cc:492] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:42.485737  2480 tablet_bootstrap.cc:492] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:42.486387  2480 ts_tablet_manager.cc:1403] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.005s
I20260709 12:21:42.489576  2480 raft_consensus.cc:359] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:42.490429  2480 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:42.491098  2480 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:42.493507  2480 ts_tablet_manager.cc:1434] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:21:42.495332  1891 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-36f1f37b7d544bb29cd9cace84f6e946 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:21:42.495877  1891 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-36f1f37b7d544bb29cd9cace84f6e946 on tablet 36f1f37b7d544bb29cd9cace84f6e946 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:42.513695  2480 ts_tablet_manager.cc:933] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:42.515857  2480 tablet_copy_client.cc:323] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:42.517645  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 82eacf0c2a8c4d8db4052327304a8793 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:42.518261  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 82eacf0c2a8c4d8db4052327304a8793 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-82eacf0c2a8c4d8db4052327304a8793
I20260709 12:21:42.524271  1816 tablet_copy_source_session.cc:215] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:42.527654  2480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82eacf0c2a8c4d8db4052327304a8793. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:42.542398  2480 tablet_copy_client.cc:806] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:42.543051  2480 tablet_copy_client.cc:670] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:42.547219  2480 tablet_copy_client.cc:538] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:42.558272  2480 tablet_bootstrap.cc:492] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:42.595156  2480 tablet_bootstrap.cc:492] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:42.596199  2480 tablet_bootstrap.cc:492] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:42.596917  2480 ts_tablet_manager.cc:1403] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.039s	user 0.013s	sys 0.019s
I20260709 12:21:42.599858  2480 raft_consensus.cc:359] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:42.601138  2480 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:42.602069  2480 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:42.604483  2480 ts_tablet_manager.cc:1434] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:21:42.608855  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-82eacf0c2a8c4d8db4052327304a8793 received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:42.609395  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-82eacf0c2a8c4d8db4052327304a8793 on tablet 82eacf0c2a8c4d8db4052327304a8793 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:42.676546  2419 raft_consensus.cc:1217] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:42.730872  2419 raft_consensus.cc:1217] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:42.790284  2419 raft_consensus.cc:1217] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:43.107605  2419 raft_consensus.cc:1217] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:43.164773  2219 raft_consensus.cc:1064] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:43.168083  2219 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:43.179683  2419 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.181063  1726 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.181267  1879 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.186681  2377 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:43.197682  2244 raft_consensus.cc:1064] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:43.189571  2218 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:43.202302  2490 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:43.204326  2244 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:21:43.242535  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58929.3us
I20260709 12:21:43.243671  2492 raft_consensus.cc:1064] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:43.246538  2492 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:43.263119  2377 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.267149  1806 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.268716  1979 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:43.271836  2418 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.277390  2460 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:43.288035  2419 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.295288  1726 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.298521  1979 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:43.299320  1724 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.320389  1879 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.322359  1628 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:43.317036  1979 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.330050  2417 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.335089  1730 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.354221  1806 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.374012  2417 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.376842  1726 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.382246  1630 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:43.384250  2492 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:43.392483  2218 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:43.405697  1880 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.408813  2219 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:43.413635  2492 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.415774  2419 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.452237  1726 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.472123  1629 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.481108  1880 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:43.518821  2419 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:43.520679  1979 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:43.523536  1730 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:43.526108  1979 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:43.528591  1880 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.544057  1806 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:43.546053  1726 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.547952  2244 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.558133  2417 raft_consensus.cc:2955] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.563689  2419 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:43.566919  2492 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:43.570327  1730 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:43.560897  1803 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:43.581707  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 36f1f37b7d544bb29cd9cace84f6e946 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:43.582567  2490 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:43.585724  2377 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.595531  1879 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:43.599287  2419 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:43.600385  2490 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:43.599436  2418 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.601243  2492 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:43.605594  1730 raft_consensus.cc:2955] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.617448  2492 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.621402  1627 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:43.627358  2419 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.637123  2219 raft_consensus.cc:1064] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:43.640759  2219 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:43.646425  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 84ce9896327e4c9d9888e80e8dbeeb0d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:43.645130  1881 raft_consensus.cc:2955] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.649392  1630 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.652415  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 62f288182b21420f8ef28e25fc10cd70 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:43.663033  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet 84ce9896327e4c9d9888e80e8dbeeb0d with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43056
I20260709 12:21:43.670989  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet 36f1f37b7d544bb29cd9cace84f6e946 with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44522
I20260709 12:21:43.677060  1630 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:43.687709  1730 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.689182  2419 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.689914  2219 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:43.690886  2491 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:43.691378  1879 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:43.704989  1714 tablet_service.cc:1558] Processing DeleteTablet for tablet 62f288182b21420f8ef28e25fc10cd70 with delete_type TABLET_DATA_TOMBSTONED (TS 075704438992440697732181e49ccd09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48982
I20260709 12:21:43.706386  2492 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:43.712690  2512 tablet_replica.cc:333] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:21:43.714047  2513 tablet_replica.cc:333] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:21:43.714157  2512 raft_consensus.cc:2243] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:43.720050  2513 raft_consensus.cc:2243] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:43.723795  2513 raft_consensus.cc:2272] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:43.726368  2512 raft_consensus.cc:2272] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:43.730293  2514 tablet_replica.cc:333] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:21:43.734012  2512 ts_tablet_manager.cc:1916] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:43.735539  2513 ts_tablet_manager.cc:1916] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:43.761618  2219 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.762923  2513 ts_tablet_manager.cc:1929] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:43.763617  2513 log.cc:1199] T 36f1f37b7d544bb29cd9cace84f6e946 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/36f1f37b7d544bb29cd9cace84f6e946
I20260709 12:21:43.765497  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet 36f1f37b7d544bb29cd9cace84f6e946 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:43.769032  1881 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.771392  2512 ts_tablet_manager.cc:1929] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:43.771785  2512 log.cc:1199] T 84ce9896327e4c9d9888e80e8dbeeb0d P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/84ce9896327e4c9d9888e80e8dbeeb0d
I20260709 12:21:43.773411  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet 84ce9896327e4c9d9888e80e8dbeeb0d (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:43.781564  2419 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.795876  1629 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.805533  1730 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.824973  2514 raft_consensus.cc:2243] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:43.825834  2514 raft_consensus.cc:2272] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:43.832285  2514 ts_tablet_manager.cc:1916] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:43.860951  1803 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:43.864543  2514 ts_tablet_manager.cc:1929] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:43.865069  2514 log.cc:1199] T 62f288182b21420f8ef28e25fc10cd70 P 075704438992440697732181e49ccd09: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/wals/62f288182b21420f8ef28e25fc10cd70
I20260709 12:21:43.867084  1614 catalog_manager.cc:5028] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): tablet 62f288182b21420f8ef28e25fc10cd70 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:43.876012  2419 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:43.878080  1730 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:43.882762  2219 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:43.884675  2491 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:43.893077  2492 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.895318  1724 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.902487  2419 raft_consensus.cc:2955] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.906837  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 82eacf0c2a8c4d8db4052327304a8793 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:43.916292  1630 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:43.930258  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet 82eacf0c2a8c4d8db4052327304a8793 with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43056
I20260709 12:21:43.931354  2512 tablet_replica.cc:333] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:21:43.932559  2512 raft_consensus.cc:2243] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:43.933282  2512 raft_consensus.cc:2272] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:43.937976  2512 ts_tablet_manager.cc:1916] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:43.955201  2512 ts_tablet_manager.cc:1929] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:43.955610  2512 log.cc:1199] T 82eacf0c2a8c4d8db4052327304a8793 P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/82eacf0c2a8c4d8db4052327304a8793
I20260709 12:21:43.957281  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet 82eacf0c2a8c4d8db4052327304a8793 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
W20260709 12:21:45.023994  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 95697.5us
I20260709 12:21:45.158301  1726 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:45.167866  1881 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:45.168967  1803 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:45.171345  2473 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:45.173010  2520 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:45.184604  2473 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:21:45.188113  1697 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23caea311a774ba3b29b7749bd3f1cd6. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:45.188544  1803 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.190176  1881 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.211817  1881 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:45.214986  1627 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:21:45.221356  1846 consensus_peers.cc:597] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bd4f4834ed2497290417394b678edc0. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:45.222348  1726 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:45.222534  1803 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:45.224241  2526 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:45.225523  2525 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:45.237764  2526 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.240235  1806 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.241840  1726 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.258880  1724 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:45.273176  1878 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:45.273550  1697 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b550688b5e84ddda5fcf049010ac1e8. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:45.275484  1630 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.278621  2317 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:45.284685  1806 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:45.287254  2317 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:45.294576  2473 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.296394  1878 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.302548  1806 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.318257  1627 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:45.319360  1881 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:21:45.332163  1846 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4f27ba9b299f46b7a89ef2d660780f8d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:45.333381  1726 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:45.333381  1806 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:45.335119  1979 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:45.336040  2526 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:45.349560  2525 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.357602  1724 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.384074  1629 catalog_manager.cc:5697] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.389384  1806 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:45.591032  2548 ts_tablet_manager.cc:933] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:21:45.605098  2548 tablet_copy_client.cc:323] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.65:35067
I20260709 12:21:45.640553  1742 tablet_copy_service.cc:140] P 075704438992440697732181e49ccd09: Received BeginTabletCopySession request for tablet 23caea311a774ba3b29b7749bd3f1cd6 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:48886)
I20260709 12:21:45.641376  1742 tablet_copy_service.cc:161] P 075704438992440697732181e49ccd09: Beginning new tablet copy session on tablet 23caea311a774ba3b29b7749bd3f1cd6 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:48886: session id = 123095127f334445889c21a42ba1d41e-23caea311a774ba3b29b7749bd3f1cd6
I20260709 12:21:45.651141  1742 tablet_copy_source_session.cc:215] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:45.654944  2548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23caea311a774ba3b29b7749bd3f1cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:45.669119  2548 tablet_copy_client.cc:806] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:45.669754  2548 tablet_copy_client.cc:670] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:45.673961  2548 tablet_copy_client.cc:538] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:45.681398  2548 tablet_bootstrap.cc:492] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:45.697917  2548 tablet_bootstrap.cc:492] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:45.698688  2548 tablet_bootstrap.cc:492] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:45.699352  2548 ts_tablet_manager.cc:1403] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.011s
I20260709 12:21:45.701792  2548 raft_consensus.cc:359] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:45.702455  2548 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:45.703060  2548 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:45.705776  2548 ts_tablet_manager.cc:1434] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:21:45.707701  1742 tablet_copy_service.cc:342] P 075704438992440697732181e49ccd09: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-23caea311a774ba3b29b7749bd3f1cd6 received from {username='slave'} at 127.0.0.1:48886
I20260709 12:21:45.708228  1742 tablet_copy_service.cc:434] P 075704438992440697732181e49ccd09: ending tablet copy session 123095127f334445889c21a42ba1d41e-23caea311a774ba3b29b7749bd3f1cd6 on tablet 23caea311a774ba3b29b7749bd3f1cd6 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:45.767030  2548 ts_tablet_manager.cc:933] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:45.768832  2548 tablet_copy_client.cc:323] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:21:45.771382  1891 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 4f27ba9b299f46b7a89ef2d660780f8d from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:21:45.772042  1891 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 4f27ba9b299f46b7a89ef2d660780f8d from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-4f27ba9b299f46b7a89ef2d660780f8d
I20260709 12:21:45.780728  1891 tablet_copy_source_session.cc:215] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:45.784715  2548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f27ba9b299f46b7a89ef2d660780f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:45.804086  2548 tablet_copy_client.cc:806] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:45.804979  2548 tablet_copy_client.cc:670] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:45.811612  2554 ts_tablet_manager.cc:933] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:21:45.811636  2548 tablet_copy_client.cc:538] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:45.813818  2554 tablet_copy_client.cc:323] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.65:35067
I20260709 12:21:45.815719  1742 tablet_copy_service.cc:140] P 075704438992440697732181e49ccd09: Received BeginTabletCopySession request for tablet 7b550688b5e84ddda5fcf049010ac1e8 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:48886)
I20260709 12:21:45.816273  1742 tablet_copy_service.cc:161] P 075704438992440697732181e49ccd09: Beginning new tablet copy session on tablet 7b550688b5e84ddda5fcf049010ac1e8 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:48886: session id = 123095127f334445889c21a42ba1d41e-7b550688b5e84ddda5fcf049010ac1e8
I20260709 12:21:45.823393  1742 tablet_copy_source_session.cc:215] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:45.823820  2548 tablet_bootstrap.cc:492] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:45.840197  2554 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b550688b5e84ddda5fcf049010ac1e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:45.895953  2557 ts_tablet_manager.cc:933] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:45.910835  2548 tablet_bootstrap.cc:492] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:45.912042  2548 tablet_bootstrap.cc:492] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:45.912812  2548 ts_tablet_manager.cc:1403] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.089s	user 0.016s	sys 0.017s
I20260709 12:21:45.924597  2548 raft_consensus.cc:359] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:45.925539  2548 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:45.926249  2548 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:45.928936  2548 ts_tablet_manager.cc:1434] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:21:45.930549  2557 tablet_copy_client.cc:323] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:21:45.934576  2554 tablet_copy_client.cc:806] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:45.935209  2554 tablet_copy_client.cc:670] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:45.940759  2554 tablet_copy_client.cc:538] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:45.944888  1891 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-4f27ba9b299f46b7a89ef2d660780f8d received from {username='slave'} at 127.0.0.1:58100
I20260709 12:21:45.945487  1891 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-4f27ba9b299f46b7a89ef2d660780f8d on tablet 4f27ba9b299f46b7a89ef2d660780f8d with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:45.949409  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 1bd4f4834ed2497290417394b678edc0 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:21:45.950068  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 1bd4f4834ed2497290417394b678edc0 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-1bd4f4834ed2497290417394b678edc0
I20260709 12:21:45.950299  2554 tablet_bootstrap.cc:492] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:45.958421  1890 tablet_copy_source_session.cc:215] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:45.986292  2557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd4f4834ed2497290417394b678edc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:45.986292  2554 tablet_bootstrap.cc:492] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:45.992033  2554 tablet_bootstrap.cc:492] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:45.993018  2554 ts_tablet_manager.cc:1403] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.043s	user 0.021s	sys 0.013s
I20260709 12:21:45.999488  2554 raft_consensus.cc:359] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:46.000515  2554 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:46.001258  2554 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:46.004067  2554 ts_tablet_manager.cc:1434] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.005s
I20260709 12:21:46.006037  1742 tablet_copy_service.cc:342] P 075704438992440697732181e49ccd09: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-7b550688b5e84ddda5fcf049010ac1e8 received from {username='slave'} at 127.0.0.1:48886
I20260709 12:21:46.006510  1742 tablet_copy_service.cc:434] P 075704438992440697732181e49ccd09: ending tablet copy session 123095127f334445889c21a42ba1d41e-7b550688b5e84ddda5fcf049010ac1e8 on tablet 7b550688b5e84ddda5fcf049010ac1e8 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:46.015661  2557 tablet_copy_client.cc:806] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:46.016649  2557 tablet_copy_client.cc:670] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:46.020568  2557 tablet_copy_client.cc:538] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:46.034224  2557 tablet_bootstrap.cc:492] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:46.059558  2557 tablet_bootstrap.cc:492] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:46.060817  2557 tablet_bootstrap.cc:492] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:46.061592  2557 ts_tablet_manager.cc:1403] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.016s
I20260709 12:21:46.065050  2557 raft_consensus.cc:359] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:46.066027  2557 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:46.066761  2557 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:46.069408  2557 ts_tablet_manager.cc:1434] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 12:21:46.071566  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-1bd4f4834ed2497290417394b678edc0 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:21:46.072036  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-1bd4f4834ed2497290417394b678edc0 on tablet 1bd4f4834ed2497290417394b678edc0 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:46.208190  2419 raft_consensus.cc:1217] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:46.220856  2419 raft_consensus.cc:1217] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:46.243538  2419 raft_consensus.cc:1217] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:46.373585  2419 raft_consensus.cc:1217] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:46.599546  2563 raft_consensus.cc:1064] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:46.601620  2563 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:46.608369  2419 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:46.608520  1881 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:46.609987  2317 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:46.610605  1806 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:46.611480  2563 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:46.612455  2521 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:46.624971  2525 raft_consensus.cc:1064] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:46.627756  2525 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:46.635088  2520 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.642899  1881 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.653287  1803 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.699748  1627 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.704736  2419 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:46.706678  2419 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.710211  1724 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:46.708249  2560 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260709 12:21:46.720799  1803 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:46.729857  2564 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:46.730839  2525 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:21:46.790169  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 77129.9us
I20260709 12:21:46.794965  2564 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.801892  2417 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.809980  1726 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.813199  2563 raft_consensus.cc:1064] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:46.815996  2563 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:46.815527  1806 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.851250  2417 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:46.851966  1879 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:46.853299  2521 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:46.856240  2317 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:46.861222  1806 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:46.862672  2563 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:46.887359  1629 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.894970  1726 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:46.894548  2521 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.897357  2417 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.908922  1878 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.913412  2419 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:46.915863  1807 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:46.916806  2317 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:46.918483  2317 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:46.913362  1806 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.914466  1880 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:46.927624  2317 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.933421  2418 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:46.932389  2419 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.936700  1614 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7b550688b5e84ddda5fcf049010ac1e8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:46.935276  2564 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:21:46.933605  1803 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:46.941742  2564 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:46.944494  2560 raft_consensus.cc:1064] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:46.946396  1979 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.955685  2416 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.966672  1803 raft_consensus.cc:2955] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:46.971660  1628 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:46.977857  2560 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:46.988720  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1bd4f4834ed2497290417394b678edc0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:46.977201  1807 raft_consensus.cc:2955] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.003553  2416 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:47.005378  1979 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:47.011946  1726 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:47.014290  2560 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:47.011961  1807 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:47.016767  1979 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:47.025736  1724 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:47.025874  1629 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.027886  1629 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.057814  1730 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.058460  1714 tablet_service.cc:1558] Processing DeleteTablet for tablet 1bd4f4834ed2497290417394b678edc0 with delete_type TABLET_DATA_TOMBSTONED (TS 075704438992440697732181e49ccd09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48982
I20260709 12:21:47.061252  2583 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.073559  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet 7b550688b5e84ddda5fcf049010ac1e8 with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43056
I20260709 12:21:47.075718  2419 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:47.077420  2317 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:47.077941  1878 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:47.080284  2520 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:47.100641  1807 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.104779  2588 tablet_replica.cc:333] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:21:47.112111  2317 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.119382  2419 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.121018  2416 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.141870  1629 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.143687  1614 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 23caea311a774ba3b29b7749bd3f1cd6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:47.147859  1627 catalog_manager.cc:5697] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.148741  1879 raft_consensus.cc:2955] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.156827  2589 tablet_replica.cc:333] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:21:47.157857  2588 raft_consensus.cc:2243] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:47.161796  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet 23caea311a774ba3b29b7749bd3f1cd6 with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44522
I20260709 12:21:47.164064  2589 raft_consensus.cc:2243] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:47.164638  2588 raft_consensus.cc:2272] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:47.164883  2589 raft_consensus.cc:2272] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:47.175498  2588 ts_tablet_manager.cc:1916] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:47.180238  2589 ts_tablet_manager.cc:1916] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:47.185084  2592 tablet_replica.cc:333] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:21:47.187970  2592 raft_consensus.cc:2243] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:47.188705  2592 raft_consensus.cc:2272] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:47.209545  1879 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:47.207325  2589 ts_tablet_manager.cc:1929] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:47.220614  2588 ts_tablet_manager.cc:1929] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:47.224433  1807 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:47.225368  2588 log.cc:1199] T 1bd4f4834ed2497290417394b678edc0 P 075704438992440697732181e49ccd09: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/wals/1bd4f4834ed2497290417394b678edc0
I20260709 12:21:47.217397  2592 ts_tablet_manager.cc:1916] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:47.226704  2583 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:47.223485  2416 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:47.228741  1614 catalog_manager.cc:5028] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): tablet 1bd4f4834ed2497290417394b678edc0 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:47.224524  2589 log.cc:1199] T 7b550688b5e84ddda5fcf049010ac1e8 P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/7b550688b5e84ddda5fcf049010ac1e8
I20260709 12:21:47.235224  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet 7b550688b5e84ddda5fcf049010ac1e8 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:47.235018  2525 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:47.236662  2564 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.245111  2416 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.252352  2592 ts_tablet_manager.cc:1929] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:47.252883  2592 log.cc:1199] T 23caea311a774ba3b29b7749bd3f1cd6 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/23caea311a774ba3b29b7749bd3f1cd6
I20260709 12:21:47.254990  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet 23caea311a774ba3b29b7749bd3f1cd6 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:47.245774  1807 raft_consensus.cc:2955] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.265717  1629 catalog_manager.cc:5697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:47.268716  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4f27ba9b299f46b7a89ef2d660780f8d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:47.281411  1714 tablet_service.cc:1558] Processing DeleteTablet for tablet 4f27ba9b299f46b7a89ef2d660780f8d with delete_type TABLET_DATA_TOMBSTONED (TS 075704438992440697732181e49ccd09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48982
I20260709 12:21:47.282611  2588 tablet_replica.cc:333] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:21:47.284049  2588 raft_consensus.cc:2243] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:47.284789  2588 raft_consensus.cc:2272] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:47.289258  2588 ts_tablet_manager.cc:1916] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:47.307395  2588 ts_tablet_manager.cc:1929] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:47.307756  2588 log.cc:1199] T 4f27ba9b299f46b7a89ef2d660780f8d P 075704438992440697732181e49ccd09: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/wals/4f27ba9b299f46b7a89ef2d660780f8d
I20260709 12:21:47.309239  1614 catalog_manager.cc:5028] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): tablet 4f27ba9b299f46b7a89ef2d660780f8d (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:48.356330  1726 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:48.364459  1879 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:48.364936  1697 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd91edf20ef54a30a2f5af0c0b682350. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:48.365022  1803 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:48.366210  2597 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:48.367447  2590 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:48.376420  2597 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.378224  1879 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.381491  1803 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.393692  1878 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:48.396622  1628 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:48.405014  1726 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:48.407455  1803 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:48.407898  2594 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:48.409530  2595 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:48.411216  1846 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b8cae5d30dd4a68a4fba5846e83873e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:48.440294  2595 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.448136  1803 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.471974  1730 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.497746  1730 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:48.502506  1629 catalog_manager.cc:5697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.523123  1879 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:48.525645  1803 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:48.528586  2590 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:48.529546  2317 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:21:48.532953  1697 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a2d4ac0ec9e4adcb3adb743647e0e78. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:48.543907  2611 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.544557  1803 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.548362  1879 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.569702  1879 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:48.570892  1628 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.579416  1803 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:48.580389  1730 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:48.581497  2595 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:48.582360  2594 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:48.584214  1846 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c072b820694347cabf7868a1b3805f76. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:48.604545  2594 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.611420  1726 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.617707  1803 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:48.635439  1628 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:48.765327  2615 ts_tablet_manager.cc:933] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:21:48.769372  2615 tablet_copy_client.cc:323] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.65:35067
I20260709 12:21:48.772447  1742 tablet_copy_service.cc:140] P 075704438992440697732181e49ccd09: Received BeginTabletCopySession request for tablet bd91edf20ef54a30a2f5af0c0b682350 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:48886)
I20260709 12:21:48.773090  1742 tablet_copy_service.cc:161] P 075704438992440697732181e49ccd09: Beginning new tablet copy session on tablet bd91edf20ef54a30a2f5af0c0b682350 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:48886: session id = 123095127f334445889c21a42ba1d41e-bd91edf20ef54a30a2f5af0c0b682350
I20260709 12:21:48.783587  1742 tablet_copy_source_session.cc:215] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:48.786763  2615 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd91edf20ef54a30a2f5af0c0b682350. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:48.802945  2615 tablet_copy_client.cc:806] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:48.803642  2615 tablet_copy_client.cc:670] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:48.808074  2615 tablet_copy_client.cc:538] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:48.818670  2615 tablet_bootstrap.cc:492] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:48.847499  2615 tablet_bootstrap.cc:492] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:48.848687  2615 tablet_bootstrap.cc:492] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:48.849385  2615 ts_tablet_manager.cc:1403] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.031s	user 0.027s	sys 0.000s
I20260709 12:21:48.852669  2615 raft_consensus.cc:359] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:48.853441  2615 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:48.854193  2615 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:48.861256  2615 ts_tablet_manager.cc:1434] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.012s	user 0.010s	sys 0.000s
I20260709 12:21:48.872344  2619 ts_tablet_manager.cc:933] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:48.872998  1742 tablet_copy_service.cc:342] P 075704438992440697732181e49ccd09: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-bd91edf20ef54a30a2f5af0c0b682350 received from {username='slave'} at 127.0.0.1:48886
I20260709 12:21:48.873572  1742 tablet_copy_service.cc:434] P 075704438992440697732181e49ccd09: ending tablet copy session 123095127f334445889c21a42ba1d41e-bd91edf20ef54a30a2f5af0c0b682350 on tablet bd91edf20ef54a30a2f5af0c0b682350 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:48.876341  2619 tablet_copy_client.cc:323] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:21:48.878307  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 9b8cae5d30dd4a68a4fba5846e83873e from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:21:48.878933  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 9b8cae5d30dd4a68a4fba5846e83873e from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-9b8cae5d30dd4a68a4fba5846e83873e
I20260709 12:21:48.887486  1890 tablet_copy_source_session.cc:215] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:48.891168  2619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b8cae5d30dd4a68a4fba5846e83873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:48.907672  2619 tablet_copy_client.cc:806] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:48.908377  2619 tablet_copy_client.cc:670] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:48.912865  2619 tablet_copy_client.cc:538] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:48.922576  2619 tablet_bootstrap.cc:492] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:48.944116  2619 tablet_bootstrap.cc:492] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:48.945051  2619 tablet_bootstrap.cc:492] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:48.945714  2619 ts_tablet_manager.cc:1403] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.008s
I20260709 12:21:48.948770  2619 raft_consensus.cc:359] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:48.949643  2619 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:48.950433  2619 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:48.953052  2619 ts_tablet_manager.cc:1434] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:21:48.955389  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-9b8cae5d30dd4a68a4fba5846e83873e received from {username='slave'} at 127.0.0.1:58100
I20260709 12:21:48.955904  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-9b8cae5d30dd4a68a4fba5846e83873e on tablet 9b8cae5d30dd4a68a4fba5846e83873e with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:49.017412  2619 ts_tablet_manager.cc:933] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:49.020316  2619 tablet_copy_client.cc:323] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:21:49.023792  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet c072b820694347cabf7868a1b3805f76 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:21:49.024428  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet c072b820694347cabf7868a1b3805f76 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-c072b820694347cabf7868a1b3805f76
I20260709 12:21:49.041270  1890 tablet_copy_source_session.cc:215] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:49.045547  2619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c072b820694347cabf7868a1b3805f76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:49.063295  2619 tablet_copy_client.cc:806] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:49.063942  2619 tablet_copy_client.cc:670] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:49.068096  2619 tablet_copy_client.cc:538] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:49.080647  2619 tablet_bootstrap.cc:492] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:49.101876  2619 tablet_bootstrap.cc:492] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:49.102965  2619 tablet_bootstrap.cc:492] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:49.103641  2619 ts_tablet_manager.cc:1403] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.010s
I20260709 12:21:49.106760  2619 raft_consensus.cc:359] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:49.107648  2619 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:49.108346  2619 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:49.110884  2619 ts_tablet_manager.cc:1434] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 12:21:49.113036  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-c072b820694347cabf7868a1b3805f76 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:21:49.113458  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-c072b820694347cabf7868a1b3805f76 on tablet c072b820694347cabf7868a1b3805f76 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:49.124333  2619 ts_tablet_manager.cc:933] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:21:49.131062  2619 tablet_copy_client.cc:323] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.65:35067
I20260709 12:21:49.133114  1742 tablet_copy_service.cc:140] P 075704438992440697732181e49ccd09: Received BeginTabletCopySession request for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:48886)
I20260709 12:21:49.133764  1742 tablet_copy_service.cc:161] P 075704438992440697732181e49ccd09: Beginning new tablet copy session on tablet 7a2d4ac0ec9e4adcb3adb743647e0e78 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:48886: session id = 123095127f334445889c21a42ba1d41e-7a2d4ac0ec9e4adcb3adb743647e0e78
I20260709 12:21:49.140431  1742 tablet_copy_source_session.cc:215] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:49.145634  2619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:49.164371  2619 tablet_copy_client.cc:806] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:49.164938  2619 tablet_copy_client.cc:670] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:49.179651  2619 tablet_copy_client.cc:538] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:49.186709  2619 tablet_bootstrap.cc:492] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:49.208323  2619 tablet_bootstrap.cc:492] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:49.209789  2619 tablet_bootstrap.cc:492] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:49.210613  2619 ts_tablet_manager.cc:1403] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.024s	user 0.020s	sys 0.003s
I20260709 12:21:49.215183  2619 raft_consensus.cc:359] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:49.216313  2619 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:49.217026  2619 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:49.219951  2619 ts_tablet_manager.cc:1434] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:21:49.222234  1742 tablet_copy_service.cc:342] P 075704438992440697732181e49ccd09: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-7a2d4ac0ec9e4adcb3adb743647e0e78 received from {username='slave'} at 127.0.0.1:48886
I20260709 12:21:49.225224  1742 tablet_copy_service.cc:434] P 075704438992440697732181e49ccd09: ending tablet copy session 123095127f334445889c21a42ba1d41e-7a2d4ac0ec9e4adcb3adb743647e0e78 on tablet 7a2d4ac0ec9e4adcb3adb743647e0e78 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:49.231869  2419 raft_consensus.cc:1217] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:49.333863  2419 raft_consensus.cc:1217] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:49.598469  2419 raft_consensus.cc:1217] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:49.736819  2627 raft_consensus.cc:1064] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:49.744524  2419 raft_consensus.cc:1217] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:49.747318  2627 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:49.762377  2419 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:49.764251  2628 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:49.765966  1803 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:49.766515  1879 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:49.767864  2628 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:49.768963  2627 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:49.781596  2628 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:49.807577  1878 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:49.800562  1803 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:49.833568  1629 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:49.857026  2419 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:49.899618  2594 raft_consensus.cc:1064] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:49.902951  2594 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:49.907235  1725 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:49.921231  2419 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:49.921886  1879 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:49.923161  2610 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:49.923671  2416 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:49.929270  2610 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:49.930166  2595 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:49.934845  2628 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:49.944267  1804 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:49.944293  1730 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:49.944659  1614 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet bd91edf20ef54a30a2f5af0c0b682350 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:49.946302  1979 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:49.946096  1878 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:49.950481  1979 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:49.961449  2419 raft_consensus.cc:2955] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:21:49.971127  2637 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:21:49.972689  1628 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:49.992059  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet bd91edf20ef54a30a2f5af0c0b682350 with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44522
I20260709 12:21:50.028191  1725 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.029246  2643 tablet_replica.cc:333] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:21:50.031180  2643 raft_consensus.cc:2243] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:50.032042  2643 raft_consensus.cc:2272] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:50.024159  2594 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.045979  2419 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.048323  2643 ts_tablet_manager.cc:1916] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:50.080780  1629 catalog_manager.cc:5697] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.093544  1804 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.106637  2643 ts_tablet_manager.cc:1929] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:50.107228  2643 log.cc:1199] T bd91edf20ef54a30a2f5af0c0b682350 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/bd91edf20ef54a30a2f5af0c0b682350
I20260709 12:21:50.109865  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet bd91edf20ef54a30a2f5af0c0b682350 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:50.149319  1878 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:50.156944  1725 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:50.159832  2419 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:50.162566  2644 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:50.172237  2644 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:50.177263  1979 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.183357  2419 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.190141  2645 raft_consensus.cc:1064] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:50.190443  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9b8cae5d30dd4a68a4fba5846e83873e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:50.193229  2645 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:50.193969  1725 raft_consensus.cc:2955] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.198611  1628 catalog_manager.cc:5697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:50.221179  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet 9b8cae5d30dd4a68a4fba5846e83873e with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44522
I20260709 12:21:50.222285  2643 tablet_replica.cc:333] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:21:50.223433  2643 raft_consensus.cc:2243] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:50.224460  2643 raft_consensus.cc:2272] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:50.231832  2419 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:50.233552  1979 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:50.241572  1803 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:50.242389  2643 ts_tablet_manager.cc:1916] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:50.245787  2594 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:50.247255  1725 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:50.248991  2644 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:50.281107  2594 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.285157  1807 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.288264  2643 ts_tablet_manager.cc:1929] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:50.287452  2419 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.288942  2643 log.cc:1199] T 9b8cae5d30dd4a68a4fba5846e83873e P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/9b8cae5d30dd4a68a4fba5846e83873e
I20260709 12:21:50.299798  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet 9b8cae5d30dd4a68a4fba5846e83873e (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:50.301964  2597 raft_consensus.cc:1064] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:50.307428  1627 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.307854  1725 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.329831  2597 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:50.334491  1880 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:50.341378  1878 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:50.343715  2610 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:50.344069  1807 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:50.345643  2610 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:50.357910  2416 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:50.358484  2419 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:50.361212  2644 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:50.361192  2597 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:21:50.366248  1807 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:50.367875  2644 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:50.380640  2594 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.381356  2416 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.392602  1807 raft_consensus.cc:2955] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.410125  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c072b820694347cabf7868a1b3805f76 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:50.421016  2627 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.422237  1628 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.445313  1714 tablet_service.cc:1558] Processing DeleteTablet for tablet c072b820694347cabf7868a1b3805f76 with delete_type TABLET_DATA_TOMBSTONED (TS 075704438992440697732181e49ccd09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48982
I20260709 12:21:50.452469  2654 tablet_replica.cc:333] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:21:50.432596  1804 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.455310  2654 raft_consensus.cc:2243] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:50.456110  2654 raft_consensus.cc:2272] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:50.459837  2654 ts_tablet_manager.cc:1916] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:50.460556  1878 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.476745  2654 ts_tablet_manager.cc:1929] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:50.477289  2654 log.cc:1199] T c072b820694347cabf7868a1b3805f76 P 075704438992440697732181e49ccd09: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/wals/c072b820694347cabf7868a1b3805f76
I20260709 12:21:50.479817  1614 catalog_manager.cc:5028] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): tablet c072b820694347cabf7868a1b3805f76 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:50.481689  2416 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.497741  1627 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.520156  1726 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:50.527406  2416 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:50.529410  1804 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:50.529317  2610 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:50.533700  2597 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:50.542253  2597 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.544809  1807 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.553221  2416 raft_consensus.cc:2955] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:50.555465  1614 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:50.565143  1628 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:50.582369  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a2d4ac0ec9e4adcb3adb743647e0e78 with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43056
I20260709 12:21:50.585366  2656 tablet_replica.cc:333] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:21:50.586259  2656 raft_consensus.cc:2243] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:50.587080  2656 raft_consensus.cc:2272] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:50.589763  2656 ts_tablet_manager.cc:1916] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:50.604362  2656 ts_tablet_manager.cc:1929] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:50.604817  2656 log.cc:1199] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/7a2d4ac0ec9e4adcb3adb743647e0e78
I20260709 12:21:50.606537  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet 7a2d4ac0ec9e4adcb3adb743647e0e78 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:51.669674  1804 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:51.678463  1726 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:51.678992  1772 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f285ce2457a47ebbd05920b42b35fba. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:51.678469  1878 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:51.680032  2657 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:51.681284  2642 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:51.690259  2657 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.692317  1726 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.692932  1878 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.708580  1725 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:51.719485  1629 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.735323  1879 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:51.736622  1804 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:51.737214  2655 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:51.742905  2655 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:21:51.747550  1697 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 513580259d0a4e6d96fb16084d5a90dc. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:51.766770  2667 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.775194  1879 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.784578  1807 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.799404  1804 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:21:51.807365  1772 consensus_peers.cc:597] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b9c190a9868442e4a7e6d84d752329f3. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:51.809798  1628 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:51.815071  1879 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:51.815349  1726 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:51.817000  2642 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:51.820585  2642 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:51.848142  2657 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.850847  1726 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.860116  1879 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.863689  1804 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:21:51.873977  1772 consensus_peers.cc:597] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c22ef2bc934c4b7ea57aad5853ab6122. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:51.874617  1879 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:51.875955  2642 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:51.877213  1725 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:51.878880  1629 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.881639  2657 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:51.890774  2657 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.893519  1881 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.903966  1726 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:51.929143  1628 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:52.177248  2676 ts_tablet_manager.cc:933] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:52.179227  2676 tablet_copy_client.cc:323] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:52.181257  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 8f285ce2457a47ebbd05920b42b35fba from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:52.181939  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 8f285ce2457a47ebbd05920b42b35fba from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-8f285ce2457a47ebbd05920b42b35fba
I20260709 12:21:52.189464  1816 tablet_copy_source_session.cc:215] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:52.193094  2676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f285ce2457a47ebbd05920b42b35fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:52.209692  2676 tablet_copy_client.cc:806] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:52.210479  2676 tablet_copy_client.cc:670] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:52.216140  2676 tablet_copy_client.cc:538] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:52.224478  2676 tablet_bootstrap.cc:492] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:52.250283  2676 tablet_bootstrap.cc:492] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:52.251307  2676 tablet_bootstrap.cc:492] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:52.251935  2676 ts_tablet_manager.cc:1403] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.028s	user 0.018s	sys 0.006s
I20260709 12:21:52.255012  2676 raft_consensus.cc:359] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:52.256438  2676 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:52.257138  2676 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:52.263829  2676 ts_tablet_manager.cc:1434] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:21:52.266774  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-8f285ce2457a47ebbd05920b42b35fba received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:52.267319  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-8f285ce2457a47ebbd05920b42b35fba on tablet 8f285ce2457a47ebbd05920b42b35fba with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:52.272272  2680 ts_tablet_manager.cc:933] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:52.298636  2680 tablet_copy_client.cc:323] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:52.300822  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet b9c190a9868442e4a7e6d84d752329f3 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:52.301455  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet b9c190a9868442e4a7e6d84d752329f3 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-b9c190a9868442e4a7e6d84d752329f3
I20260709 12:21:52.312186  1816 tablet_copy_source_session.cc:215] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:52.316991  2680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9c190a9868442e4a7e6d84d752329f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:52.327008  2676 ts_tablet_manager.cc:933] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:21:52.335271  2676 tablet_copy_client.cc:323] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.65:35067
I20260709 12:21:52.337148  1742 tablet_copy_service.cc:140] P 075704438992440697732181e49ccd09: Received BeginTabletCopySession request for tablet 513580259d0a4e6d96fb16084d5a90dc from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:48886)
I20260709 12:21:52.337795  1742 tablet_copy_service.cc:161] P 075704438992440697732181e49ccd09: Beginning new tablet copy session on tablet 513580259d0a4e6d96fb16084d5a90dc from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:48886: session id = 123095127f334445889c21a42ba1d41e-513580259d0a4e6d96fb16084d5a90dc
I20260709 12:21:52.338757  2680 tablet_copy_client.cc:806] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:52.339455  2680 tablet_copy_client.cc:670] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:52.344516  2680 tablet_copy_client.cc:538] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:52.346405  1742 tablet_copy_source_session.cc:215] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:52.350401  2676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 513580259d0a4e6d96fb16084d5a90dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:52.363216  2680 tablet_bootstrap.cc:492] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:52.365550  2676 tablet_copy_client.cc:806] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:52.366313  2676 tablet_copy_client.cc:670] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:52.371623  2676 tablet_copy_client.cc:538] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:52.381237  2676 tablet_bootstrap.cc:492] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:52.398696  2680 tablet_bootstrap.cc:492] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:52.399710  2680 tablet_bootstrap.cc:492] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:52.400406  2680 ts_tablet_manager.cc:1403] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.037s	user 0.021s	sys 0.009s
I20260709 12:21:52.409004  2680 raft_consensus.cc:359] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:52.409897  2680 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:52.411820  2680 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:52.415349  2680 ts_tablet_manager.cc:1434] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:21:52.422272  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-b9c190a9868442e4a7e6d84d752329f3 received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:52.422765  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-b9c190a9868442e4a7e6d84d752329f3 on tablet b9c190a9868442e4a7e6d84d752329f3 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:52.443029  2676 tablet_bootstrap.cc:492] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:52.444357  2676 tablet_bootstrap.cc:492] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:52.445102  2676 ts_tablet_manager.cc:1403] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.064s	user 0.021s	sys 0.013s
I20260709 12:21:52.445480  2680 ts_tablet_manager.cc:933] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:52.447460  2680 tablet_copy_client.cc:323] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:52.449513  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet c22ef2bc934c4b7ea57aad5853ab6122 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:52.450166  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet c22ef2bc934c4b7ea57aad5853ab6122 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-c22ef2bc934c4b7ea57aad5853ab6122
I20260709 12:21:52.448333  2676 raft_consensus.cc:359] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:52.451259  2676 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:52.452354  2676 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:52.454762  2676 ts_tablet_manager.cc:1434] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:21:52.458367  1742 tablet_copy_service.cc:342] P 075704438992440697732181e49ccd09: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-513580259d0a4e6d96fb16084d5a90dc received from {username='slave'} at 127.0.0.1:48886
I20260709 12:21:52.458884  1742 tablet_copy_service.cc:434] P 075704438992440697732181e49ccd09: ending tablet copy session 123095127f334445889c21a42ba1d41e-513580259d0a4e6d96fb16084d5a90dc on tablet 513580259d0a4e6d96fb16084d5a90dc with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:52.458890  1816 tablet_copy_source_session.cc:215] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:52.463693  2680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22ef2bc934c4b7ea57aad5853ab6122. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:52.489359  2680 tablet_copy_client.cc:806] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:52.490078  2680 tablet_copy_client.cc:670] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:52.494733  2680 tablet_copy_client.cc:538] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:52.504801  2680 tablet_bootstrap.cc:492] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:52.530589  2680 tablet_bootstrap.cc:492] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:52.531653  2680 tablet_bootstrap.cc:492] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:52.532408  2680 ts_tablet_manager.cc:1403] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.028s	user 0.021s	sys 0.004s
I20260709 12:21:52.535661  2680 raft_consensus.cc:359] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:52.536631  2680 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:52.537325  2680 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:52.540030  2680 ts_tablet_manager.cc:1434] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:21:52.542809  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-c22ef2bc934c4b7ea57aad5853ab6122 received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:52.543267  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-c22ef2bc934c4b7ea57aad5853ab6122 on tablet c22ef2bc934c4b7ea57aad5853ab6122 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:52.733881  2416 raft_consensus.cc:1217] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:52.821908  2416 raft_consensus.cc:1217] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:21:52.860394  2416 raft_consensus.cc:1217] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:52.994222  2416 raft_consensus.cc:1217] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:53.248929  2597 raft_consensus.cc:1064] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:53.251247  2597 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:53.259259  2416 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:53.259728  1881 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:53.261058  2687 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:53.261512  1804 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:21:53.262226  2691 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:53.267786  2687 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:53.284052  2690 raft_consensus.cc:1064] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:53.287860  2689 raft_consensus.cc:1064] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:53.287475  2690 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:53.290943  2689 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:53.290618  2687 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.305465  1881 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.307644  2419 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:53.309911  1879 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:53.310928  2642 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:53.312893  2419 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.316877  1804 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.320107  2642 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:53.321862  1627 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:53.324431  1726 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:53.326107  2689 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:53.345104  1725 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:53.348096  2416 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:53.349951  1878 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:53.353624  2657 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:53.355496  2661 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:53.356042  2657 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.359558  2642 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:53.365746  1879 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.376089  1726 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.400277  2419 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.469712  1628 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.476006  1725 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:53.477758  2642 raft_consensus.cc:1064] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:53.482815  2657 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.485399  2642 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:53.486485  1878 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:53.488163  2687 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:53.488663  2416 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:21:53.490494  2597 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:53.497412  2416 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.500485  2597 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.504468  1880 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.509965  1807 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:53.527861  2417 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:53.530071  2690 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:53.531951  2417 raft_consensus.cc:2955] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.542368  1629 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.542225  2690 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.547986  1725 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:53.550545  2419 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.557037  1730 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.570526  1614 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 513580259d0a4e6d96fb16084d5a90dc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:53.571718  2661 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:53.581699  1629 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:53.592913  2417 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:53.596240  2689 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:53.611761  1880 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.617026  1878 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:53.621259  1726 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:53.625104  1730 raft_consensus.cc:2955] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.641947  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet 513580259d0a4e6d96fb16084d5a90dc with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44522
I20260709 12:21:53.665661  1629 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.675401  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8f285ce2457a47ebbd05920b42b35fba with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:53.675704  2642 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:21:53.706218  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet 8f285ce2457a47ebbd05920b42b35fba with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43056
I20260709 12:21:53.719099  2717 tablet_replica.cc:333] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:21:53.720781  2718 tablet_replica.cc:333] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:21:53.730520  2718 raft_consensus.cc:2243] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:21:53.731900  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94156.4us
I20260709 12:21:53.749218  2657 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:53.739367  2717 raft_consensus.cc:2243] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:53.740803  1803 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:53.756301  2717 raft_consensus.cc:2272] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:53.760856  2717 ts_tablet_manager.cc:1916] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:53.762950  2419 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:53.748627  2718 raft_consensus.cc:2272] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:53.770438  2713 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:53.782016  2717 ts_tablet_manager.cc:1929] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:21:53.785473  1881 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:53.790411  2717 log.cc:1199] T 513580259d0a4e6d96fb16084d5a90dc P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/513580259d0a4e6d96fb16084d5a90dc
I20260709 12:21:53.792340  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet 513580259d0a4e6d96fb16084d5a90dc (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:53.793898  2714 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:53.800501  2718 ts_tablet_manager.cc:1916] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:53.800472  1730 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.798513  2657 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.806761  2690 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.816082  1880 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.820827  1879 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.822252  2718 ts_tablet_manager.cc:1929] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:53.822818  2718 log.cc:1199] T 8f285ce2457a47ebbd05920b42b35fba P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/8f285ce2457a47ebbd05920b42b35fba
I20260709 12:21:53.826893  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet 8f285ce2457a47ebbd05920b42b35fba (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:53.827090  2419 raft_consensus.cc:2955] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.831453  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b9c190a9868442e4a7e6d84d752329f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:53.837538  2417 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.837587  1627 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:53.849822  1629 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.865156  1714 tablet_service.cc:1558] Processing DeleteTablet for tablet b9c190a9868442e4a7e6d84d752329f3 with delete_type TABLET_DATA_TOMBSTONED (TS 075704438992440697732181e49ccd09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48982
I20260709 12:21:53.867445  2721 tablet_replica.cc:333] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:21:53.868672  2721 raft_consensus.cc:2243] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:53.869446  2721 raft_consensus.cc:2272] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:53.873188  2721 ts_tablet_manager.cc:1916] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:53.895628  1803 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:53.900574  2721 ts_tablet_manager.cc:1929] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:53.902320  2721 log.cc:1199] T b9c190a9868442e4a7e6d84d752329f3 P 075704438992440697732181e49ccd09: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/wals/b9c190a9868442e4a7e6d84d752329f3
I20260709 12:21:53.904928  1614 catalog_manager.cc:5028] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): tablet b9c190a9868442e4a7e6d84d752329f3 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:53.905151  1730 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:53.906451  2417 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:53.906555  2657 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:53.908174  2661 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:53.915731  2657 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.919708  2419 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.919754  1730 raft_consensus.cc:2955] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.925697  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c22ef2bc934c4b7ea57aad5853ab6122 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:53.938591  1628 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:53.957341  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet c22ef2bc934c4b7ea57aad5853ab6122 with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43056
I20260709 12:21:53.958612  2718 tablet_replica.cc:333] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:21:53.967377  2718 raft_consensus.cc:2243] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:53.968393  2718 raft_consensus.cc:2272] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:53.974687  2718 ts_tablet_manager.cc:1916] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:53.990929  2718 ts_tablet_manager.cc:1929] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:53.991299  2718 log.cc:1199] T c22ef2bc934c4b7ea57aad5853ab6122 P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/c22ef2bc934c4b7ea57aad5853ab6122
I20260709 12:21:53.992801  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet c22ef2bc934c4b7ea57aad5853ab6122 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
W20260709 12:21:55.162397  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53210.9us
I20260709 12:21:55.227934  1803 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:55.237756  1879 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:55.238240  1772 consensus_peers.cc:597] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c92163822f64868a647b688599f49bd. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:55.239624  1724 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:55.239332  2726 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:55.241393  2724 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:55.257876  2728 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.263226  1724 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.268136  1879 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.294098  1629 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.298707  1881 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:21:55.319494  1846 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b797b0728dc4e01976ab61ef164fa90. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:55.322608  1724 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:55.324381  1803 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:55.324771  2725 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:55.326586  2722 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:55.335603  2725 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.340092  1726 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.365221  1807 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:55.387349  1804 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:21:55.391027  1772 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4a57b66dec794db09a7f9048a3d0de10. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:55.394858  1880 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:55.394006  1726 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:55.392777  1628 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.396914  2724 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:55.398998  2726 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:55.422816  2724 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.433241  1724 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.435688  1879 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.444617  1804 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:55.453702  1629 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:55.485817  1772 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 034c167cd4e24138be60353036ce8092. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:55.488565  1879 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:55.490240  2713 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:55.493188  1724 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:55.495671  2713 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:55.500475  2724 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.502528  1879 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.511297  1724 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:55.521085  1629 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:55.804425  2745 ts_tablet_manager.cc:933] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:55.806653  2745 tablet_copy_client.cc:323] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:55.808939  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 8c92163822f64868a647b688599f49bd from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:55.809629  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 8c92163822f64868a647b688599f49bd from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-8c92163822f64868a647b688599f49bd
I20260709 12:21:55.818364  1816 tablet_copy_source_session.cc:215] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:55.822487  2745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c92163822f64868a647b688599f49bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:55.847285  2745 tablet_copy_client.cc:806] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:55.848417  2745 tablet_copy_client.cc:670] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:55.859220  2747 ts_tablet_manager.cc:933] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:55.862007  2745 tablet_copy_client.cc:538] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:55.862807  2747 tablet_copy_client.cc:323] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:55.867296  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 4a57b66dec794db09a7f9048a3d0de10 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:55.867967  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 4a57b66dec794db09a7f9048a3d0de10 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-4a57b66dec794db09a7f9048a3d0de10
I20260709 12:21:55.874449  2745 tablet_bootstrap.cc:492] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:55.877735  1816 tablet_copy_source_session.cc:215] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:55.884259  2747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a57b66dec794db09a7f9048a3d0de10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:55.914914  2749 ts_tablet_manager.cc:933] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:55.925372  2749 tablet_copy_client.cc:323] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:21:55.928627  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 2b797b0728dc4e01976ab61ef164fa90 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:21:55.929272  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 2b797b0728dc4e01976ab61ef164fa90 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-2b797b0728dc4e01976ab61ef164fa90
I20260709 12:21:55.938225  1890 tablet_copy_source_session.cc:215] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:55.942545  2747 tablet_copy_client.cc:806] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:55.943326  2747 tablet_copy_client.cc:670] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:55.944867  2745 tablet_bootstrap.cc:492] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:55.947275  2749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b797b0728dc4e01976ab61ef164fa90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:55.951767  2745 tablet_bootstrap.cc:492] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:55.952651  2745 ts_tablet_manager.cc:1403] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.079s	user 0.014s	sys 0.031s
I20260709 12:21:55.953406  2747 tablet_copy_client.cc:538] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:55.958639  2745 raft_consensus.cc:359] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:55.961022  2745 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:55.962339  2745 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:55.982295  2749 tablet_copy_client.cc:806] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:55.983247  2749 tablet_copy_client.cc:670] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:55.987178  2745 ts_tablet_manager.cc:1434] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.034s	user 0.009s	sys 0.009s
I20260709 12:21:55.990259  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-8c92163822f64868a647b688599f49bd received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:55.990873  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-8c92163822f64868a647b688599f49bd on tablet 8c92163822f64868a647b688599f49bd with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:55.993731  2749 tablet_copy_client.cc:538] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:55.996881  2745 ts_tablet_manager.cc:933] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:56.017942  2745 tablet_copy_client.cc:323] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:56.036659  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 034c167cd4e24138be60353036ce8092 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:56.037277  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 034c167cd4e24138be60353036ce8092 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-034c167cd4e24138be60353036ce8092
I20260709 12:21:56.041088  2749 tablet_bootstrap.cc:492] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:56.054304  1816 tablet_copy_source_session.cc:215] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:56.065488  2745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 034c167cd4e24138be60353036ce8092. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:56.086627  2749 tablet_bootstrap.cc:492] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:56.087680  2749 tablet_bootstrap.cc:492] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:56.087965  2747 tablet_bootstrap.cc:492] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:56.088366  2749 ts_tablet_manager.cc:1403] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.048s	user 0.026s	sys 0.011s
I20260709 12:21:56.091995  2745 tablet_copy_client.cc:806] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:56.091761  2749 raft_consensus.cc:359] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:56.092670  2749 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:56.092751  2745 tablet_copy_client.cc:670] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:56.093473  2749 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:56.096306  2749 ts_tablet_manager.cc:1434] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:21:56.097591  2745 tablet_copy_client.cc:538] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:56.100416  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-2b797b0728dc4e01976ab61ef164fa90 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:21:56.101015  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-2b797b0728dc4e01976ab61ef164fa90 on tablet 2b797b0728dc4e01976ab61ef164fa90 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:56.110446  2745 tablet_bootstrap.cc:492] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:56.127281  2747 tablet_bootstrap.cc:492] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:56.128367  2747 tablet_bootstrap.cc:492] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:56.129390  2747 ts_tablet_manager.cc:1403] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.042s	user 0.022s	sys 0.010s
I20260709 12:21:56.132838  2747 raft_consensus.cc:359] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:56.133713  2747 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:56.134371  2747 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:56.135131  2745 tablet_bootstrap.cc:492] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:56.136365  2745 tablet_bootstrap.cc:492] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:56.136962  2745 ts_tablet_manager.cc:1403] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.027s	user 0.022s	sys 0.004s
I20260709 12:21:56.140049  2747 ts_tablet_manager.cc:1434] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:21:56.140082  2745 raft_consensus.cc:359] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:56.141471  2745 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:56.142180  2745 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:56.144965  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-4a57b66dec794db09a7f9048a3d0de10 received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:56.145468  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-4a57b66dec794db09a7f9048a3d0de10 on tablet 4a57b66dec794db09a7f9048a3d0de10 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:56.147900  2745 ts_tablet_manager.cc:1434] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.001s
I20260709 12:21:56.150142  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-034c167cd4e24138be60353036ce8092 received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:56.150636  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-034c167cd4e24138be60353036ce8092 on tablet 034c167cd4e24138be60353036ce8092 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:56.227288  2416 raft_consensus.cc:1217] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:56.260324  2416 raft_consensus.cc:1217] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:56.317337  2416 raft_consensus.cc:1217] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:56.513342  2417 raft_consensus.cc:1217] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:56.778848  2724 raft_consensus.cc:1064] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:56.781626  2724 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:56.783074  2713 raft_consensus.cc:1064] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:56.789458  2713 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:56.808843  2417 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:56.811581  2759 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:56.818912  2417 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:56.822057  2759 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:56.822600  1880 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:56.831787  1880 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:56.840898  2726 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:56.843164  2726 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:56.879383  1726 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:56.880328  1724 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:56.881882  2763 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:56.883942  2759 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:21:56.904498  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 64287.2us
I20260709 12:21:56.914146  2713 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:56.923502  2416 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:56.939996  2725 raft_consensus.cc:1064] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:56.949735  2725 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:56.965221  1804 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:56.974625  2722 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 12:21:56.997350  2419 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:57.002439  1628 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:57.003273  1725 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.020761  2759 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.023138  1724 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.048465  2722 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260709 12:21:57.055218  1629 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.089462  2419 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.098464  1878 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.107348  1725 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:57.109658  2722 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:57.115366  1880 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.126300  2726 raft_consensus.cc:1064] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:21:57.131858  2726 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:57.144177  2722 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.149230  2416 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.174012  1804 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:57.178704  1725 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.184298  2419 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:57.185561  1724 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:57.194346  2726 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:57.196638  2726 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:57.178989  1803 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.216353  2416 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:57.217808  1724 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:57.220001  2713 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:57.230340  1629 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.232049  2713 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:57.222661  1880 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:21:57.239521  2765 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.253659  2761 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:57.259642  2763 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.265199  2416 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.272043  1726 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.277647  2417 raft_consensus.cc:2955] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.283025  1878 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:57.285825  1803 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:57.293923  1724 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:57.294718  2419 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:57.300465  2722 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:57.309414  2722 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:57.310653  1627 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.322773  2787 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.321643  1880 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.340631  2419 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:57.339792  2418 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.341763  1724 raft_consensus.cc:2955] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.348369  2713 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:57.363940  1881 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:57.366602  2761 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:57.368289  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2b797b0728dc4e01976ab61ef164fa90 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:57.375396  1629 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.380421  2713 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.391439  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b797b0728dc4e01976ab61ef164fa90 with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44522
I20260709 12:21:57.393162  1881 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.397488  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 8c92163822f64868a647b688599f49bd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:57.402420  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 4a57b66dec794db09a7f9048a3d0de10 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:57.402014  2417 raft_consensus.cc:2955] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.407801  1627 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.412606  1630 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:57.423571  2791 tablet_replica.cc:333] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:21:57.426522  2791 raft_consensus.cc:2243] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:57.428023  1724 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.429400  2791 raft_consensus.cc:2272] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:57.434583  2791 ts_tablet_manager.cc:1916] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:57.440282  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet 4a57b66dec794db09a7f9048a3d0de10 with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43056
I20260709 12:21:57.456950  1714 tablet_service.cc:1558] Processing DeleteTablet for tablet 8c92163822f64868a647b688599f49bd with delete_type TABLET_DATA_TOMBSTONED (TS 075704438992440697732181e49ccd09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48982
I20260709 12:21:57.485556  2793 tablet_replica.cc:333] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:21:57.487707  2793 raft_consensus.cc:2243] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:57.489674  2793 raft_consensus.cc:2272] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:57.487236  2792 tablet_replica.cc:333] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:21:57.491592  2792 raft_consensus.cc:2243] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:57.493311  2792 raft_consensus.cc:2272] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:57.497579  2793 ts_tablet_manager.cc:1916] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:57.505605  2791 ts_tablet_manager.cc:1929] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:57.506867  2791 log.cc:1199] T 2b797b0728dc4e01976ab61ef164fa90 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/2b797b0728dc4e01976ab61ef164fa90
I20260709 12:21:57.509018  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet 2b797b0728dc4e01976ab61ef164fa90 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:57.514312  1803 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:21:57.512554  2792 ts_tablet_manager.cc:1916] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:57.526008  2419 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:57.528432  2713 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:57.534345  2793 ts_tablet_manager.cc:1929] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:57.534932  2793 log.cc:1199] T 8c92163822f64868a647b688599f49bd P 075704438992440697732181e49ccd09: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/wals/8c92163822f64868a647b688599f49bd
I20260709 12:21:57.536929  1614 catalog_manager.cc:5028] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): tablet 8c92163822f64868a647b688599f49bd (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:57.536700  2763 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.542238  2792 ts_tablet_manager.cc:1929] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:57.542728  2792 log.cc:1199] T 4a57b66dec794db09a7f9048a3d0de10 P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/4a57b66dec794db09a7f9048a3d0de10
I20260709 12:21:57.540550  2419 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.544585  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet 4a57b66dec794db09a7f9048a3d0de10 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:57.548126  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 034c167cd4e24138be60353036ce8092 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:57.552937  1881 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:57.554690  2713 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:57.557437  1626 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:57.559379  1881 raft_consensus.cc:2955] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:21:57.578989  1714 tablet_service.cc:1558] Processing DeleteTablet for tablet 034c167cd4e24138be60353036ce8092 with delete_type TABLET_DATA_TOMBSTONED (TS 075704438992440697732181e49ccd09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48982
I20260709 12:21:57.579977  2793 tablet_replica.cc:333] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:21:57.581036  2793 raft_consensus.cc:2243] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:57.581851  2793 raft_consensus.cc:2272] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:21:57.587647  2793 ts_tablet_manager.cc:1916] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:57.608521  2793 ts_tablet_manager.cc:1929] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:57.608914  2793 log.cc:1199] T 034c167cd4e24138be60353036ce8092 P 075704438992440697732181e49ccd09: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/wals/034c167cd4e24138be60353036ce8092
I20260709 12:21:57.610316  1614 catalog_manager.cc:5028] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): tablet 034c167cd4e24138be60353036ce8092 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:21:58.653616  1724 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:58.661983  1881 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:21:58.662328  1697 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80ed6fcb806a4023b94ecd7b266b421e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:58.663498  1803 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:58.664567  2691 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:58.665481  2768 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:58.679663  2691 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.681692  1804 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.689332  1881 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.711433  1880 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:58.721036  1630 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:58.734278  1846 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9875eb35aaed48d7b2670b5ea57e3948. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:58.735247  1804 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:58.740182  2799 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:58.748037  1724 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:58.749969  2798 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:58.755977  2794 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.757731  1804 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.770709  1724 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.796259  1878 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:58.802650  1630 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:21:58.827230  1846 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 65799458c3fb44e7a162099338c0bb62. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:58.830161  1804 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:58.830644  1724 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:58.833030  2794 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:58.833925  2798 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:58.852422  2799 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.854079  1724 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.853984  1803 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.877619  1626 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.881580  1804 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:21:58.895633  1772 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79f5c05e64b945cbacc02914f31c5762. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:21:58.896142  1879 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:58.896142  1724 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:21:58.897691  2713 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:58.898505  2810 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:58.907143  2763 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.909413  1878 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.909507  1724 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:58.929433  1628 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:21:59.042758  2816 ts_tablet_manager.cc:933] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:21:59.044835  2816 tablet_copy_client.cc:323] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.65:35067
I20260709 12:21:59.046890  1742 tablet_copy_service.cc:140] P 075704438992440697732181e49ccd09: Received BeginTabletCopySession request for tablet 80ed6fcb806a4023b94ecd7b266b421e from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:48886)
I20260709 12:21:59.047580  1742 tablet_copy_service.cc:161] P 075704438992440697732181e49ccd09: Beginning new tablet copy session on tablet 80ed6fcb806a4023b94ecd7b266b421e from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:48886: session id = 123095127f334445889c21a42ba1d41e-80ed6fcb806a4023b94ecd7b266b421e
I20260709 12:21:59.056767  1742 tablet_copy_source_session.cc:215] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:59.060954  2816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80ed6fcb806a4023b94ecd7b266b421e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:59.079540  2816 tablet_copy_client.cc:806] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:59.080283  2816 tablet_copy_client.cc:670] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:59.084863  2816 tablet_copy_client.cc:538] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:59.095657  2816 tablet_bootstrap.cc:492] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:59.119828  2816 tablet_bootstrap.cc:492] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:59.121018  2816 tablet_bootstrap.cc:492] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:59.121832  2816 ts_tablet_manager.cc:1403] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.026s	user 0.016s	sys 0.008s
I20260709 12:21:59.125167  2816 raft_consensus.cc:359] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:59.126099  2816 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:59.126806  2816 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:59.140061  2816 ts_tablet_manager.cc:1434] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.004s
I20260709 12:21:59.147431  1742 tablet_copy_service.cc:342] P 075704438992440697732181e49ccd09: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-80ed6fcb806a4023b94ecd7b266b421e received from {username='slave'} at 127.0.0.1:48886
I20260709 12:21:59.148010  1742 tablet_copy_service.cc:434] P 075704438992440697732181e49ccd09: ending tablet copy session 123095127f334445889c21a42ba1d41e-80ed6fcb806a4023b94ecd7b266b421e on tablet 80ed6fcb806a4023b94ecd7b266b421e with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:59.304459  2816 ts_tablet_manager.cc:933] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:21:59.306248  2816 tablet_copy_client.cc:323] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:21:59.308097  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 79f5c05e64b945cbacc02914f31c5762 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:21:59.308661  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 79f5c05e64b945cbacc02914f31c5762 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-79f5c05e64b945cbacc02914f31c5762
I20260709 12:21:59.316298  1816 tablet_copy_source_session.cc:215] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:59.318553  2821 ts_tablet_manager.cc:933] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:59.319902  2816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79f5c05e64b945cbacc02914f31c5762. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:59.327152  2821 tablet_copy_client.cc:323] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:21:59.329083  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 9875eb35aaed48d7b2670b5ea57e3948 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:21:59.329814  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 9875eb35aaed48d7b2670b5ea57e3948 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-9875eb35aaed48d7b2670b5ea57e3948
I20260709 12:21:59.344506  1890 tablet_copy_source_session.cc:215] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:59.347512  2821 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9875eb35aaed48d7b2670b5ea57e3948. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:59.358855  2816 tablet_copy_client.cc:806] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:59.359495  2816 tablet_copy_client.cc:670] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:59.363552  2816 tablet_copy_client.cc:538] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:59.365811  2821 tablet_copy_client.cc:806] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:59.366520  2821 tablet_copy_client.cc:670] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:59.369810  2823 ts_tablet_manager.cc:933] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:21:59.372068  2821 tablet_copy_client.cc:538] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:59.375445  2816 tablet_bootstrap.cc:492] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:59.394268  2821 tablet_bootstrap.cc:492] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:59.435454  2823 tablet_copy_client.cc:323] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:21:59.437731  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 65799458c3fb44e7a162099338c0bb62 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:21:59.438369  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 65799458c3fb44e7a162099338c0bb62 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-65799458c3fb44e7a162099338c0bb62
I20260709 12:21:59.446941  1890 tablet_copy_source_session.cc:215] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:21:59.455571  2816 tablet_bootstrap.cc:492] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:59.456653  2816 tablet_bootstrap.cc:492] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:59.457062  2823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65799458c3fb44e7a162099338c0bb62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:21:59.457494  2816 ts_tablet_manager.cc:1403] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.082s	user 0.037s	sys 0.029s
I20260709 12:21:59.460752  2816 raft_consensus.cc:359] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:59.461649  2816 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:59.462363  2816 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:59.468852  2816 ts_tablet_manager.cc:1434] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:21:59.475109  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-79f5c05e64b945cbacc02914f31c5762 received from {username='slave'} at 127.0.0.1:42636
I20260709 12:21:59.475529  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-79f5c05e64b945cbacc02914f31c5762 on tablet 79f5c05e64b945cbacc02914f31c5762 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:59.492622  2823 tablet_copy_client.cc:806] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:21:59.493515  2823 tablet_copy_client.cc:670] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:21:59.500285  2823 tablet_copy_client.cc:538] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:21:59.509861  2823 tablet_bootstrap.cc:492] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:21:59.510030  2821 tablet_bootstrap.cc:492] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:59.511734  2821 tablet_bootstrap.cc:492] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:59.512499  2821 ts_tablet_manager.cc:1403] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.119s	user 0.029s	sys 0.013s
I20260709 12:21:59.516109  2821 raft_consensus.cc:359] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:59.517095  2821 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:59.518872  2821 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:59.525750  2821 ts_tablet_manager.cc:1434] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:21:59.527642  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-9875eb35aaed48d7b2670b5ea57e3948 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:21:59.528131  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-9875eb35aaed48d7b2670b5ea57e3948 on tablet 9875eb35aaed48d7b2670b5ea57e3948 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:59.545328  2823 tablet_bootstrap.cc:492] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:21:59.546294  2823 tablet_bootstrap.cc:492] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:21:59.547089  2823 ts_tablet_manager.cc:1403] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.037s	user 0.022s	sys 0.005s
I20260709 12:21:59.549894  2823 raft_consensus.cc:359] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:59.550632  2823 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:21:59.551326  2823 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:21:59.553579  2823 ts_tablet_manager.cc:1434] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:21:59.556003  2419 raft_consensus.cc:1217] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:59.556785  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-65799458c3fb44e7a162099338c0bb62 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:21:59.557217  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-65799458c3fb44e7a162099338c0bb62 on tablet 65799458c3fb44e7a162099338c0bb62 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:21:59.700662  2419 raft_consensus.cc:1217] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:21:59.747991  2419 raft_consensus.cc:1217] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:21:59.802248  2419 raft_consensus.cc:1217] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:22:00.003141  2691 raft_consensus.cc:1064] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:00.011358  2691 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:00.022400  1803 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:00.023443  2419 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:00.023463  1878 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:00.025094  2691 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:00.026129  2768 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:00.027526  2807 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:00.057667  2809 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.071954  1803 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.072855  2419 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.091538  1878 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.107513  1626 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.151139  1726 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:00.158224  2419 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:00.158224  1878 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:00.159965  2691 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:00.160889  2768 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:00.172669  2809 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.173408  1878 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.175565  2419 raft_consensus.cc:2955] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.193102  1627 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.208149  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet 80ed6fcb806a4023b94ecd7b266b421e with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44522
I20260709 12:22:00.211252  2839 tablet_replica.cc:333] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:22:00.212570  2839 raft_consensus.cc:2243] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.213510  2839 raft_consensus.cc:2272] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.219445  2839 ts_tablet_manager.cc:1916] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:22:00.219753  1614 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 80ed6fcb806a4023b94ecd7b266b421e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:22:00.228453  2810 raft_consensus.cc:1064] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:00.231001  2722 raft_consensus.cc:1064] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:00.231163  2810 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:00.233208  2722 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:00.251981  2839 ts_tablet_manager.cc:1929] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:22:00.252643  2839 log.cc:1199] T 80ed6fcb806a4023b94ecd7b266b421e P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/80ed6fcb806a4023b94ecd7b266b421e
I20260709 12:22:00.257182  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet 80ed6fcb806a4023b94ecd7b266b421e (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:22:00.259897  2419 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:00.260087  2417 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:22:00.261448  1879 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:00.262720  1724 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:00.262737  2826 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:00.264377  2713 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:00.267726  2830 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:00.274821  1803 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:22:00.276780  1726 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:22:00.277459  2713 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:00.278614  2722 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:00.281998  2829 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:00.330631  2826 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:22:00.352192  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52806.6us
I20260709 12:22:00.380663  2848 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.387151  1730 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.419335  1627 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:22:00.424759  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 125089us
I20260709 12:22:00.433663  2417 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.465037  1803 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:00.466745  2851 raft_consensus.cc:1064] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:00.469071  1878 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.469527  2851 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:00.472635  1730 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:00.482323  2713 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:00.489398  2830 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.500561  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 79f5c05e64b945cbacc02914f31c5762 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:22:00.503587  1730 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.507099  2418 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:00.512502  2761 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:00.518076  2418 raft_consensus.cc:2955] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.518729  1724 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.525302  1726 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:00.537248  2722 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:00.515173  2417 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:00.542274  2852 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:00.546109  1804 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.557802  2419 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.616168  1803 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:00.619426  1626 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:00.635550  2855 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.637737  1726 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.645629  2851 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:00.648674  1630 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.650072  1878 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:00.663880  2418 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:22:00.665184  1803 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:22:00.668167  2851 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:00.668151  2418 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.672808  2856 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:00.678130  2417 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.680254  2858 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.697358  1804 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.704046  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet 79f5c05e64b945cbacc02914f31c5762 with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43056
I20260709 12:22:00.706064  1626 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:00.739856  1879 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:00.741601  1630 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.745545  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9875eb35aaed48d7b2670b5ea57e3948 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:22:00.750605  1803 raft_consensus.cc:2955] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.753648  2865 tablet_replica.cc:333] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:22:00.768195  1726 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:00.768610  2417 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:00.770172  2858 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:00.770932  2851 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:00.771477  1714 tablet_service.cc:1558] Processing DeleteTablet for tablet 9875eb35aaed48d7b2670b5ea57e3948 with delete_type TABLET_DATA_TOMBSTONED (TS 075704438992440697732181e49ccd09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48982
I20260709 12:22:00.773265  2866 tablet_replica.cc:333] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:22:00.774369  2866 raft_consensus.cc:2243] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.775240  2866 raft_consensus.cc:2272] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.780373  2855 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.800360  2866 ts_tablet_manager.cc:1916] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:22:00.810748  2417 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.817680  2866 ts_tablet_manager.cc:1929] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:22:00.818264  2866 log.cc:1199] T 9875eb35aaed48d7b2670b5ea57e3948 P 075704438992440697732181e49ccd09: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/wals/9875eb35aaed48d7b2670b5ea57e3948
I20260709 12:22:00.820436  1614 catalog_manager.cc:5028] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): tablet 9875eb35aaed48d7b2670b5ea57e3948 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:22:00.833489  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 65799458c3fb44e7a162099338c0bb62 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:22:00.832131  1628 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.829834  1726 raft_consensus.cc:2955] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:00.847227  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet 65799458c3fb44e7a162099338c0bb62 with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44522
I20260709 12:22:00.853008  2865 raft_consensus.cc:2243] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.853912  2865 raft_consensus.cc:2272] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.857239  2865 ts_tablet_manager.cc:1916] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:22:00.863610  2867 tablet_replica.cc:333] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:22:00.865338  2867 raft_consensus.cc:2243] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:00.866264  2867 raft_consensus.cc:2272] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:00.873268  2867 ts_tablet_manager.cc:1916] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:22:00.880054  2865 ts_tablet_manager.cc:1929] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:22:00.881179  2865 log.cc:1199] T 79f5c05e64b945cbacc02914f31c5762 P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/79f5c05e64b945cbacc02914f31c5762
I20260709 12:22:00.883435  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet 79f5c05e64b945cbacc02914f31c5762 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:22:00.887624  2867 ts_tablet_manager.cc:1929] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:22:00.888113  2867 log.cc:1199] T 65799458c3fb44e7a162099338c0bb62 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/65799458c3fb44e7a162099338c0bb62
I20260709 12:22:00.889667  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet 65799458c3fb44e7a162099338c0bb62 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:22:01.246639   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:01.302076  1804 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:01.302901  1804 raft_consensus.cc:606] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:01.306160  1803 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c92163822f64868a647b688599f49bd"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:01.306859  1803 raft_consensus.cc:606] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:01.309688  1803 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:01.310153  1803 raft_consensus.cc:606] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:01.327196  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:01.327811  1730 raft_consensus.cc:606] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:01.330771  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:01.331492  1730 raft_consensus.cc:606] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:01.333007   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:01.334522   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:01.335348   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:01.397147  2850 raft_consensus.cc:993] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:01.397812  2849 raft_consensus.cc:1081] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:01.399703  1730 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:01.400432  1730 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:01.400688  1730 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.405233  1730 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:01.407317  1730 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:01.408198  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:01.408885  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:01.409052  1879 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.409345  1803 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:01.409592  1803 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:01.410248  1803 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:01.411228  1803 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.415695  1879 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:01.416141  1803 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:01.417580  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:01.418484  2768 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:01.419840  2768 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:01.420795  2768 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:01.438725  1627 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:01.519179  2691 raft_consensus.cc:993] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:01.519747  2768 raft_consensus.cc:1081] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:01.521268  1879 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:01.521777  1879 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:01.522023  1879 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.526659  1879 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:01.528129  1879 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:01.529230  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:01.529233  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:01.529995  1726 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:01.530009  1803 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.530539  1726 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:01.531615  1726 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:01.533156  1726 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.536545  1803 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:01.537719  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:01.538473  2851 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:01.539978  1726 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:01.539832  2851 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:01.541461  2851 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:01.553637  1627 catalog_manager.cc:5697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:01.588589  2850 raft_consensus.cc:993] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:01.589183  2850 raft_consensus.cc:1081] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:01.591010  2418 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8c92163822f64868a647b688599f49bd"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60928
I20260709 12:22:01.591673  2418 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:01.592087  2418 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.594506  2713 raft_consensus.cc:993] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:01.595155  2850 raft_consensus.cc:1081] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:01.596901  1726 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:01.597582  1726 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:01.597996  1726 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.601241  2418 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:01.604524  2418 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:01.605181  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:01.605584  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:01.605942  1803 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:01.606117  1879 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.606314  1803 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:01.606187  1726 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:01.609128  1726 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:01.610139  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:01.608397  1803 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:01.610909  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:01.610764  1804 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:01.611845  1803 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.611919  1804 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:01.613418  1879 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:01.613057  1804 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:01.611595  1878 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.614454  1804 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.621410  1803 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:01.622191  1804 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:01.623114  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:01.624974  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:01.625286  2879 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:01.625877  2872 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:01.623936  1878 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:01.626451  2872 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:01.627441  2872 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:01.627930  2879 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:01.629087  2879 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:01.656011  1627 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:01.663820  2881 raft_consensus.cc:993] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:01.664402  2881 raft_consensus.cc:1081] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:01.666098  1879 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:01.666755  1879 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:01.667224  1879 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.680932  1879 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:01.681157  1627 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:01.683225  1879 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:01.684388  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:01.684502  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:01.685226  1726 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:01.685586  1804 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.685590  1726 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:01.686578  1726 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:01.687934  1726 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:01.692963  1804 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:01.694489  1726 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:01.695796  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:01.697093  2877 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:01.697696  2877 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:01.698725  2877 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:01.715837  1627 catalog_manager.cc:5697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:01.860472  1880 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:01.862496  2872 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:01.930099  1804 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:02.730054  2889 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:22:02.785295  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.005s	sys 0.005s
W20260709 12:22:02.785696  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.005s	sys 0.005s
I20260709 12:22:02.824189  2901 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:02.825001  2901 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:02.827622  2901 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:02.844794  1807 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:02.852316  2768 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:02.853820  2768 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:02.854363  2768 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:02.856601  2768 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:02.869831  1730 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:02.873961  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:02.885782  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:02.889051  1879 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:02.905401   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:02.912576  2880 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:02.930536  2898 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:22:02.953330  2851 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:02.953938  2851 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:02.954654  2902 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:02.955175  2902 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:02.961163  2851 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:02.963982  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:02.964658  2880 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:02.965297  2880 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:02.966006  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:02.968616  2880 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:02.970395  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:02.970723  1732 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:02.972373  2897 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:02.972920  2897 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:02.978999  2713 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:02.980295  2892 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:02.980880  2892 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:02.979636  2713 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:02.984442  2713 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:02.985381  1661 debug-util.cc:398] Leaking SignalData structure 0x7b080008bde0 after lost signal to thread 564
W20260709 12:22:02.986655  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08000920a0 after lost signal to thread 1664
I20260709 12:22:02.987344  2849 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:02.988086  1661 debug-util.cc:398] Leaking SignalData structure 0x7b080033d6a0 after lost signal to thread 1756
I20260709 12:22:02.987910  2849 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:02.988440  1661 debug-util.cc:398] Leaking SignalData structure 0x7b080033b2e0 after lost signal to thread 1757
W20260709 12:22:02.989696  1661 debug-util.cc:398] Leaking SignalData structure 0x7b0800320560 after lost signal to thread 1831
I20260709 12:22:02.989971  2849 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:02.991104  1661 debug-util.cc:398] Leaking SignalData structure 0x7b080033ae60 after lost signal to thread 1905
W20260709 12:22:02.991518  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08002883e0 after lost signal to thread 1906
I20260709 12:22:02.993762  2892 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:02.994477  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:02.998708  1661 debug-util.cc:398] Leaking SignalData structure 0x7b0800321780 after lost signal to thread 2443
I20260709 12:22:03.002955  2871 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:03.003820  2897 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:03.003640  2871 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:03.005954  2871 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:03.006023  2858 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:03.008457  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.010023  2901 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:03.011045  2901 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.013204  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.013583  2901 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:03.014238  2901 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:03.014732  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.015733  1878 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:03.019503  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.032065  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.014757  2901 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.032644  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.050963  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.054497  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.055330  2768 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.089613  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.092941  2713 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:03.093657  2713 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:03.099228  2901 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:03.100049  2894 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.100266  2901 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:03.034888  2871 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260709 12:22:03.100847  2901 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.102531  2858 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:03.102967  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.104535  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.104377  2858 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:03.117362  2713 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:03.034627  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.115636  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.107398  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.090281  2902 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:03.116353  2880 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.182425  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:03.182878  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 57751us
I20260709 12:22:03.191716  2912 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.210534  2713 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:03.211957  2713 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.225301  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.226568  2713 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:03.213151  2902 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:03.253446  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.222770  2880 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:03.222973  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 115667us
I20260709 12:22:03.278954  1803 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:03.234201  2901 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:03.283893  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.246466  2849 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:03.284852  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.286823  2858 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:03.321448  1806 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:03.322413  1806 raft_consensus.cc:606] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:03.273402  2850 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:03.275797  2902 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:03.330205  2902 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:03.330988  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:03.324985  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.283355  2880 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.332116  2901 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:03.385353  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.292048  2849 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.400108  2880 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:03.352525  2912 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:03.371130  2896 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:03.423199  1881 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:22:03.426013  2911 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
W20260709 12:22:03.428311  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.442313  2896 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:03.413192  2902 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:03.451290  2922 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:03.451711  1801 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:03.332751  2850 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:03.410444  2912 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.391201  2901 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.461783  2850 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:03.457913  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58475us
I20260709 12:22:03.456063  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.420665  2880 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:03.461918  2912 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:03.476735  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.454545  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:03.475505  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.471138  2880 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.481209  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:03.472424  2912 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:03.459581  2902 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:03.490671  2913 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.235s	user 0.002s	sys 0.000s
I20260709 12:22:03.488663  1730 raft_consensus.cc:606] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:03.480036  2901 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:03.490823  2904 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
I20260709 12:22:03.506283  2912 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.545723  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.547111  2902 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:03.549469  2896 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:03.550520  2849 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:03.560561  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.565138  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.565673  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:03.593025  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 129508us
I20260709 12:22:03.567662  1803 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:03.568513  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.480777  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.636402  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.647006  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.651865  2713 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.651300  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.637190  2902 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:22:03.585716  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.656493  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.672225  2888 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.007s
I20260709 12:22:03.643762  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:03.683286  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 254835us
I20260709 12:22:03.683751  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.688766  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.690924  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.697207  2713 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.657943  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:03.471005  2922 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:03.725572  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.696609  2892 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:03.732234  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 320020us
I20260709 12:22:03.734318  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.737619  2890 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.729048  2917 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.745630  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.728016  1728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:03.749842  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.750629  2895 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.726769  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.753248  2901 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:03.759509  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.761600  1728 raft_consensus.cc:606] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.747035  2880 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:03.771582  2880 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.755995  2912 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:03.790971  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.791436  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.792904  2922 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:03.794667  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:03.810635  2906 raft_consensus.cc:993] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:03.810672  1801 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:03.811463  2906 raft_consensus.cc:1081] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:03.811650  1801 raft_consensus.cc:606] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:03.830456  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.833814  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.834899  2922 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.835358  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.836211  2922 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.852954  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.837179  2895 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.874032  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.878012  1723 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
W20260709 12:22:03.880342  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51847.9us
I20260709 12:22:03.878719  1723 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
W20260709 12:22:03.885258  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 234288us
I20260709 12:22:03.883745  1723 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:03.870512  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.894832  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.901031  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.900899  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.904711  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.909958  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.908936  1726 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:22:03.916064  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.903651  2922 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.921449  2919 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.913149  2877 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.897317  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.882932  2895 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:03.926162  2902 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.939133  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:03.948565  2895 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:03.949275  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.959107  2928 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:03.967914  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 77274.2us
I20260709 12:22:03.973572  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.973955  2920 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.976493  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:03.977059  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 115687us
W20260709 12:22:03.977911  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63813us
I20260709 12:22:03.978653  2924 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.979079  2912 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.983175  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:03.977979  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.987061  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:03.989780  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:03.982647  2920 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.993968  1801 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:22:03.995061  2895 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:03.997676  2930 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:22:04.015133  2901 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:04.009356  2850 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:04.042249  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 137727us
I20260709 12:22:04.038133  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:04.075044  2901 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:04.093971  2768 raft_consensus.cc:993] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:04.088657  2902 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
W20260709 12:22:04.088886  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50875.3us
I20260709 12:22:04.120831  2916 raft_consensus.cc:1081] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:04.085314  1723 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.124636  1723 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:04.127293  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:04.128669  1880 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.127194  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:04.158041  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 103560us
W20260709 12:22:04.159902  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77545.2us
W20260709 12:22:04.169852  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 131931us
W20260709 12:22:04.180990  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 98260.1us
W20260709 12:22:04.192109  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 109368us
I20260709 12:22:04.211688  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:04.214126  1804 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:04.218811  1804 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:04.219986  2412 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60950
I20260709 12:22:04.220816  2412 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:04.221267  2412 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.221496  1880 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:04.227837  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:04.229262  1846 consensus_peers.cc:597] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae23688e76624153b5a7089586669bb0. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:22:04.228423  1881 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:04.220002  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:04.231313  1804 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.233587  1804 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:04.264765  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63744.6us
I20260709 12:22:04.235011  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:04.370498  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 101862us
I20260709 12:22:04.373411  2912 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:04.379968  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 149952us
W20260709 12:22:04.407380  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.424908  2944 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:04.432420  1732 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:22:04.438923  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 158970us
I20260709 12:22:04.443207  2851 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:04.443975  2851 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.445236  2412 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:04.446486  2851 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:04.448637  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:04.448740  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:04.449450  1725 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:04.449759  1725 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:04.450557  1725 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:04.452054  1725 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.475437  1725 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:04.502674  1875 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:04.503515  1875 raft_consensus.cc:606] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:04.503934  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:04.505060  2912 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:04.505666  2912 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:04.506745  2912 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:04.477203  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.513980  2950 ts_tablet_manager.cc:933] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:04.522912  2924 raft_consensus.cc:993] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:04.524961  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:04.526141  2942 raft_consensus.cc:1081] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:04.532018  2412 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:04.533478  1804 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:04.536311  2944 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:58092 (request call id 8) took 1144 ms. Trace:
I20260709 12:22:04.541394  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:04.543013  2947 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:04.549574  1804 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:04.553644  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:04.561233  1800 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.563474  1628 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:04.555140  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:04.556957  1627 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:04.554282  2412 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60928
I20260709 12:22:04.579995  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:04.583992  2713 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:04.587003  2923 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:04.587618  2923 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:04.542321  2944 rpcz_store.cc:276] 0709 12:22:03.391739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.391944 (+   205us) service_pool.cc:224] Handling call
0709 12:22:03.446821 (+ 54877us) log.cc:844] Serialized 499 byte log entry
0709 12:22:04.536215 (+1089394us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":310,"raft.run_cpu_time_us":2322,"raft.run_wall_time_us":2321,"thread_start_us":321045,"threads_started":1,"wal-append.queue_time_us":316586}
I20260709 12:22:04.602106  1807 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:04.590281  2923 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.581153  2412 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:04.606284  2412 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.620293  1804 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:04.621044  1804 raft_consensus.cc:606] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:04.622292   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:22:04.624470   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:04.625296   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:04.614591  2950 tablet_copy_client.cc:323] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:22:04.629066  2412 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:04.633525  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:04.634225  1723 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.654831  2412 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:04.658088  1723 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:04.659009  1729 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:04.659463  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:04.660454  2912 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:04.661048  2912 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:04.662124  2912 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:04.671054  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:04.671875  1805 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:04.672333  1805 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:04.680691  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet ae23688e76624153b5a7089586669bb0 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:22:04.681314  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet ae23688e76624153b5a7089586669bb0 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-ae23688e76624153b5a7089586669bb0
I20260709 12:22:04.673545  1805 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:04.684870  1805 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.691391  1890 tablet_copy_source_session.cc:215] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:22:04.697319  2944 raft_consensus.cc:993] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:04.698092  2877 raft_consensus.cc:1081] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:04.692399  1805 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:04.704263  2950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae23688e76624153b5a7089586669bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:22:04.708827  1723 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57228
I20260709 12:22:04.715916  1800 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:04.729046  2894 raft_consensus.cc:993] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:04.729579  2894 raft_consensus.cc:1081] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:04.732698  1873 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:22:04.734453  2927 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:04.746258  1729 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:22:04.748665  1772 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66d3405528d44edb8c0be3b296ef72be. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:22:04.727869  1723 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:04.751562  1723 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:04.758956  2890 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:04.766742  1878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:04.773998  1878 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:04.774454  1878 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.806034  1878 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.831980  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:04.832880  1732 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:04.833319  1732 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:04.835388  1878 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:04.836333  1732 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.837774  1732 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.802001  1723 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.847985  1872 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:04.852715  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:04.853260  1732 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:04.853576  1800 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.855822  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:04.856774  2877 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:04.857386  2877 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:04.861125  2892 raft_consensus.cc:993] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:04.861706  2892 raft_consensus.cc:1081] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:04.865283  1725 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:04.866096  1725 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:04.866600  1725 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.873143  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:04.873905  1798 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:04.858462  2877 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.881239  1725 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.886770  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:04.887475  1805 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:04.887776  1805 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:04.888495  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:04.888552  1805 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.889906  1725 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:04.889499  1875 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.884575  2713 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:04.907603  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:04.908346  1878 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:04.908675  1878 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:04.909286  1805 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:04.909487  1878 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.911705  1878 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:04.912626  1723 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:04.912036  1873 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:04.926376  1878 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:04.929154  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:04.934195  2918 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:04.935525  2918 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:04.936321  2918 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.945063  1805 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:04.930141  1798 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:04.946545  1807 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:04.946535  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:04.948015  2916 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:04.948575  2916 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:04.948122  1729 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:04.949445  2916 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:04.952529  2950 tablet_copy_client.cc:806] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:22:04.953338  2950 tablet_copy_client.cc:670] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:22:04.952843  1723 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:04.964809  1800 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:04.995168  1805 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:04.981455  1875 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:05.024315  2950 tablet_copy_client.cc:538] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:22:05.054134  1732 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:05.056027  2912 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:05.062526  1626 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.064851  1627 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.068397  2912 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:05.067504  2950 tablet_bootstrap.cc:492] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:22:05.077325  1630 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:05.134351  1725 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:05.137236  2912 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:22:05.186061  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 75380.8us
W20260709 12:22:05.313483  2956 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:22:05.331830  2950 tablet_bootstrap.cc:700] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent opening tablet: real 0.146s	user 0.004s	sys 0.001s
W20260709 12:22:05.338553  2970 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:22:05.347643  2977 ts_tablet_manager.cc:933] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:05.350052  2977 tablet_copy_client.cc:323] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
W20260709 12:22:05.376748  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 238943us
I20260709 12:22:05.380149  1627 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:05.383612  1627 catalog_manager.cc:5697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 3, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 3 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:05.389433  1697 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6388b1a5243d463589750794e8ac5511. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:22:05.386053  1627 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:05.439069  1877 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:05.444319  1872 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:22:05.501093  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 110556us
I20260709 12:22:05.509840  2975 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:05.546721  1800 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:22:05.577478  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 77447.3us
I20260709 12:22:05.616343  2985 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:22:05.619841  1804 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:05.628599  1807 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:05.636547  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 66d3405528d44edb8c0be3b296ef72be from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:22:05.637471  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 66d3405528d44edb8c0be3b296ef72be from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-66d3405528d44edb8c0be3b296ef72be
I20260709 12:22:05.658922  1799 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:22:05.660127  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45470.2us
I20260709 12:22:05.666749  1729 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:05.707259  2994 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:22:05.718146  1873 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:05.740509  1804 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:22:05.743579  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58633.5us
I20260709 12:22:05.747196  2858 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:05.749464  2912 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:05.734644  2950 tablet_bootstrap.cc:492] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:22:05.784813  1816 tablet_copy_source_session.cc:215] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:22:05.800995  2950 tablet_bootstrap.cc:492] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:22:05.802273  2950 ts_tablet_manager.cc:1403] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.735s	user 0.021s	sys 0.023s
I20260709 12:22:05.816977   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:05.818490  1880 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:05.823133  1804 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:05.870311  2977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d3405528d44edb8c0be3b296ef72be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:22:05.865106  2889 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 12:22:05.863376  2985 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:22:05.989693  3002 ts_tablet_manager.cc:933] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:22:06.016438  2889 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260709 12:22:06.043790  2889 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:22:06.084239  2950 raft_consensus.cc:359] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:06.085322  2950 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:22:06.093464  3002 tablet_copy_client.cc:323] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.65:35067
I20260709 12:22:06.086077  2950 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:06.109695  2989 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:06.129582  1742 tablet_copy_service.cc:140] P 075704438992440697732181e49ccd09: Received BeginTabletCopySession request for tablet 6388b1a5243d463589750794e8ac5511 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:48886)
I20260709 12:22:06.130343  1742 tablet_copy_service.cc:161] P 075704438992440697732181e49ccd09: Beginning new tablet copy session on tablet 6388b1a5243d463589750794e8ac5511 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:48886: session id = 123095127f334445889c21a42ba1d41e-6388b1a5243d463589750794e8ac5511
I20260709 12:22:06.187935  2985 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:06.189793  2950 ts_tablet_manager.cc:1434] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.387s	user 0.004s	sys 0.009s
I20260709 12:22:06.197508  1742 tablet_copy_source_session.cc:215] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:22:06.252825  2973 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:22:06.255573  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68874.7us
I20260709 12:22:06.276659  3002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6388b1a5243d463589750794e8ac5511. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:22:06.288616  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-ae23688e76624153b5a7089586669bb0 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:22:06.289376  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-ae23688e76624153b5a7089586669bb0 on tablet ae23688e76624153b5a7089586669bb0 with peer 123095127f334445889c21a42ba1d41e
W20260709 12:22:06.297169  3006 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.107s	user 0.001s	sys 0.001s
I20260709 12:22:06.334760  2977 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:60928 (request call id 319) took 1015 ms. Trace:
I20260709 12:22:06.380990  2977 rpcz_store.cc:276] 0709 12:22:05.318979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.319584 (+   605us) service_pool.cc:224] Handling call
0709 12:22:05.348162 (+ 28578us) ts_tablet_manager.cc:934] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
0709 12:22:05.868397 (+520235us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:06.220078 (+351681us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:06.228552 (+  8474us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:06.333867 (+105315us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:06.334182 (+   315us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:06.334513 (+   331us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:06.334627 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":297,"tablet-copy.queue_time_us":20914,"thread_start_us":19828,"threads_started":2}
I20260709 12:22:06.382673  2977 tablet_copy_client.cc:806] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:22:06.383332  2977 tablet_copy_client.cc:670] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:22:06.408324  2977 tablet_copy_client.cc:538] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:22:06.432659  1880 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:06.447306  2973 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:48870 (request call id 6) took 1503 ms. Trace:
I20260709 12:22:06.448484  2973 rpcz_store.cc:276] 0709 12:22:04.943386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.945542 (+  2156us) service_pool.cc:224] Handling call
0709 12:22:04.988178 (+ 42636us) log.cc:844] Serialized 499 byte log entry
0709 12:22:06.447133 (+1458955us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26123,"raft.queue_time_us":42917,"raft.run_cpu_time_us":1579,"raft.run_wall_time_us":1580,"thread_start_us":269028,"threads_started":2,"wal-append.queue_time_us":202325,"wal-append.run_cpu_time_us":2537,"wal-append.run_wall_time_us":1148752}
I20260709 12:22:06.449939  2977 tablet_bootstrap.cc:492] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:22:06.463289  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:06.464190  1729 raft_consensus.cc:606] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:06.472368  1799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:06.473145  1799 raft_consensus.cc:606] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:06.479161  3002 tablet_copy_client.cc:806] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:22:06.490186  2973 raft_consensus.cc:993] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:06.490734  2973 raft_consensus.cc:1081] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:06.497711  1630 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:06.506568  3002 tablet_copy_client.cc:670] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:22:06.480471  1723 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:06.522500  3002 tablet_copy_client.cc:538] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:22:06.518354  1799 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:06.526340  1798 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:06.527146  1798 raft_consensus.cc:606] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:06.583449  1880 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:06.584131  1880 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:06.584511  1880 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:06.588874  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:06.589634  1729 raft_consensus.cc:606] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:06.592729  1872 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:06.593510  1872 raft_consensus.cc:606] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:06.597443  1725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:06.598310  1725 raft_consensus.cc:606] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:06.601711   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:22:06.603556   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:06.604897   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:06.607563  1880 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.609761  1880 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:06.612648  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:06.613517  1807 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:06.612577  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:06.617027  1725 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:06.617355  1725 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:06.618098  1725 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.625779  1807 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:06.627393  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:06.628863  1725 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:06.629294  3021 raft_consensus.cc:993] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:06.629909  3021 raft_consensus.cc:1081] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
W20260709 12:22:06.630272  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.630290  2944 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:06.631278  2944 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:06.631788  2892 raft_consensus.cc:993] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:06.632395  2892 raft_consensus.cc:1081] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:06.632272  2944 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.646448  1725 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:06.650396  1630 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:06.652940  1697 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a4c45f93428e4cc68f85f17ea3ec3ad0. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:22:06.659063  2417 raft_consensus.cc:1217] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:22:06.651189  1725 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:06.663275  1872 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:06.663497  1725 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:06.667378  1725 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:06.665681  1872 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:06.670288  1872 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:06.673250  3002 tablet_bootstrap.cc:492] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:22:06.695641  1873 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:22:06.695722  1725 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.698067  1725 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:06.702204  1799 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:22:06.719039  3021 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:06.736125  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:06.736948  1799 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:06.737323  1799 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:06.744750  3002 tablet_bootstrap.cc:492] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:22:06.745298  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:06.746086  1873 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:06.747570  3002 tablet_bootstrap.cc:492] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:22:06.748421  3002 ts_tablet_manager.cc:1403] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.075s	user 0.021s	sys 0.010s
I20260709 12:22:06.759018  3002 raft_consensus.cc:359] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.762250  2989 raft_consensus.cc:993] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:06.762476  3002 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:22:06.762890  2989 raft_consensus.cc:1081] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:06.744261  1799 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.763368  3002 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:06.767777  1872 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.769078  1880 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:06.767884  2417 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60950
I20260709 12:22:06.769626  2417 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:06.769974  1799 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:06.770018  2417 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:06.773873  3002 ts_tablet_manager.cc:1434] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.003s
I20260709 12:22:06.774435  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:06.775094  1725 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:06.775393  1725 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:06.776253  1725 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.776907  2977 tablet_bootstrap.cc:492] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:22:06.777351  1725 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:06.781204  1742 tablet_copy_service.cc:342] P 075704438992440697732181e49ccd09: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-6388b1a5243d463589750794e8ac5511 received from {username='slave'} at 127.0.0.1:48886
I20260709 12:22:06.771553  2988 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:06.781734  1742 tablet_copy_service.cc:434] P 075704438992440697732181e49ccd09: ending tablet copy session 123095127f334445889c21a42ba1d41e-6388b1a5243d463589750794e8ac5511 on tablet 6388b1a5243d463589750794e8ac5511 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:22:06.767444  1873 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:06.784389  2417 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:06.787598  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:06.787948  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:06.788273  1723 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:06.788548  1873 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:06.788642  1723 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:06.789722  1723 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:06.790678  1723 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:06.799751  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:06.800746  2989 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:06.805003  2989 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:06.806291  2989 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.808904  1723 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:06.811434  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:06.813162  3028 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:06.813810  3028 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:06.814888  3028 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:06.816138  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:06.816848  1798 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:06.820698  2417 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:06.829757  1873 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:06.832305  1799 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:06.835261  1875 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:06.777993  2977 tablet_bootstrap.cc:492] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:22:06.839663  1872 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:06.846957  2977 ts_tablet_manager.cc:1403] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.397s	user 0.024s	sys 0.009s
I20260709 12:22:06.853426  2977 raft_consensus.cc:359] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:06.854486  2977 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:22:06.860800  1627 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:06.866183  1630 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:06.874418  3021 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:06.872113  1798 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:06.873365  1725 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:06.882571  1878 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:06.884763  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:06.885972  2920 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:06.886674  2920 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:06.887699  2920 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:06.855310  2977 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:06.909498  2977 ts_tablet_manager.cc:1434] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.062s	user 0.007s	sys 0.000s
I20260709 12:22:06.912664  2415 raft_consensus.cc:1217] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:22:06.917011  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-66d3405528d44edb8c0be3b296ef72be received from {username='slave'} at 127.0.0.1:42636
I20260709 12:22:06.917539  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-66d3405528d44edb8c0be3b296ef72be on tablet 66d3405528d44edb8c0be3b296ef72be with peer 123095127f334445889c21a42ba1d41e
I20260709 12:22:06.934010  1807 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:06.957530  1630 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:06.999306  1627 catalog_manager.cc:5697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 3 to 4, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:07.035897  2986 raft_consensus.cc:993] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:07.036468  2986 raft_consensus.cc:1081] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:07.038543  1732 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:07.039228  1732 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:07.039616  1732 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:07.041673  3037 raft_consensus.cc:993] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:07.042289  3037 raft_consensus.cc:1081] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:07.044576  1723 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57228
I20260709 12:22:07.045282  1723 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:07.045670  1723 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:07.049810  2415 raft_consensus.cc:1217] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:22:07.056977  1723 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:07.051098  2415 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:07.061830  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:07.062474  1807 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:07.070318  1807 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:07.071836  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:07.071749  1732 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:07.074427  1723 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:07.076717  3013 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:07.085076  3013 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:07.086046  1732 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:07.086161  3013 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:07.095676  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:07.096397  1878 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:07.106011  1878 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:07.108899  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:07.082512  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:07.110127  1872 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:07.110363  2985 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:07.110651  1872 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:07.111770  1872 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:07.111068  2985 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:07.115880  1872 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:07.115548  2985 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:07.122450  1725 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:07.128773  2943 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:07.131747  2977 ts_tablet_manager.cc:933] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:22:07.154233  1799 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:07.156142  3037 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:07.186502  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:07.187122  1807 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:07.187412  1807 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:07.188199  1807 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:07.189391  1807 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:07.205250  1798 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:07.209393  3013 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:07.211500  2977 tablet_copy_client.cc:323] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.65:35067
I20260709 12:22:07.213661  1742 tablet_copy_service.cc:140] P 075704438992440697732181e49ccd09: Received BeginTabletCopySession request for tablet a4c45f93428e4cc68f85f17ea3ec3ad0 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:48886)
I20260709 12:22:07.214380  1742 tablet_copy_service.cc:161] P 075704438992440697732181e49ccd09: Beginning new tablet copy session on tablet a4c45f93428e4cc68f85f17ea3ec3ad0 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:48886: session id = 123095127f334445889c21a42ba1d41e-a4c45f93428e4cc68f85f17ea3ec3ad0
I20260709 12:22:07.220877  3037 raft_consensus.cc:1064] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:07.226262  1872 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:07.230235  3037 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:07.234191  1872 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:07.237947  3028 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:07.238165  1628 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:07.230271  1807 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:07.269295  1878 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:07.271204  3021 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:07.274200  2415 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:07.276022  2920 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:07.287724  1627 catalog_manager.cc:5697] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:07.291729  1742 tablet_copy_source_session.cc:215] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:22:07.291630  1798 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:07.293927  2943 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:07.301119  2977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4c45f93428e4cc68f85f17ea3ec3ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:22:07.348227  1725 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:07.350445  2920 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:07.396450  2977 tablet_copy_client.cc:806] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:22:07.397166  2977 tablet_copy_client.cc:670] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:22:07.399013  2943 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:07.404194  1723 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:07.416348  1725 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:07.422946  2977 tablet_copy_client.cc:538] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:22:07.433115  3028 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:07.474025  2982 raft_consensus.cc:1064] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:07.490868  1807 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:07.456691  2415 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:07.501608  2944 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 12:22:07.510344  1628 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:07.511606  1726 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:07.513084  2920 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 12:22:07.445792  1798 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:07.529621  1878 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:07.533527  3021 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:07.542073  1872 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:07.599488  1872 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:07.599851  2982 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:07.601279  2988 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260709 12:22:07.621418  2411 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:07.625168  2944 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:07.651780  1804 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:07.659838  2977 tablet_bootstrap.cc:492] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:22:07.669906  1804 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:07.671350  2920 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:07.661214  3021 raft_consensus.cc:1064] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:07.678283  2973 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:07.682538  2943 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:07.716318  1807 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:07.723868  2411 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:07.694293  3021 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:07.735473  3013 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:07.819717   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:07.844345  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ae23688e76624153b5a7089586669bb0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:22:07.877800  1807 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:07.880833  1873 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:07.885399  2988 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:07.888305  2988 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:07.909138  2411 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:07.913498  1726 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:07.923724  1799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:07.936890  1799 raft_consensus.cc:606] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:07.951664  3067 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:07.968919  2977 tablet_bootstrap.cc:492] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:22:07.946024  2988 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 12:22:07.980813  1732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:07.981796  1732 raft_consensus.cc:606] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:07.988179  2980 raft_consensus.cc:993] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:07.988787  2980 raft_consensus.cc:1081] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:07.989689  1878 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:07.993525  1732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:07.994355  1732 raft_consensus.cc:606] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:08.001597  1804 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:08.002461  1804 raft_consensus.cc:606] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:08.015276  2981 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:08.024338  2977 tablet_bootstrap.cc:492] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
W20260709 12:22:08.053738  3066 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
W20260709 12:22:08.058755  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86059.2us
W20260709 12:22:08.084818  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 143456us
I20260709 12:22:08.087249  3072 raft_consensus.cc:993] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:08.089354  3074 raft_consensus.cc:1081] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:08.103612  1723 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:08.104374  1723 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:08.104794  1723 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.036037  2977 ts_tablet_manager.cc:1403] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.377s	user 0.021s	sys 0.008s
I20260709 12:22:08.117463  2977 raft_consensus.cc:359] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:08.123219  2977 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:22:08.124223  2977 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:08.129565  2977 ts_tablet_manager.cc:1434] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.001s
I20260709 12:22:08.130461  1627 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 1 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.132511  1723 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:08.150182  1723 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:08.150830  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:08.168514  1798 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:08.168983  1798 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:08.170367  1798 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:08.172155  1798 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.202773  1798 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:08.205374  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:08.208472  1873 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae23688e76624153b5a7089586669bb0"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:08.209306  1873 raft_consensus.cc:606] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:08.210630  3013 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:08.211318  3013 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:08.212267  3013 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:08.211936  1798 raft_consensus.cc:2955] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.223395  1723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:08.224473  1723 raft_consensus.cc:606] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:08.226015   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:22:08.227919   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:08.228832   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:08.258652  3021 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.264190  2411 raft_consensus.cc:1217] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:22:08.267408  2410 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:08.270231  2418 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60950
I20260709 12:22:08.270915  2418 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:08.271302  2418 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:08.287110  2973 raft_consensus.cc:993] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:08.287735  2973 raft_consensus.cc:1081] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:08.298759  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:08.299465  1881 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.306881  1874 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.330868  1742 tablet_copy_service.cc:342] P 075704438992440697732181e49ccd09: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-a4c45f93428e4cc68f85f17ea3ec3ad0 received from {username='slave'} at 127.0.0.1:48886
I20260709 12:22:08.331460  1742 tablet_copy_service.cc:434] P 075704438992440697732181e49ccd09: ending tablet copy session 123095127f334445889c21a42ba1d41e-a4c45f93428e4cc68f85f17ea3ec3ad0 on tablet a4c45f93428e4cc68f85f17ea3ec3ad0 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:22:08.348460  2988 raft_consensus.cc:993] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:08.357285  2988 raft_consensus.cc:1081] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:08.354830  2982 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:08.359020  3069 raft_consensus.cc:993] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:08.365451  3069 raft_consensus.cc:1081] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:08.368162  2417 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60950
I20260709 12:22:08.374194  2417 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:08.376785  1878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:08.383286  1878 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:08.383747  1878 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.379863  3076 raft_consensus.cc:993] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:08.370422  2418 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.386583  3061 raft_consensus.cc:1081] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:08.388491  2418 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:08.389539  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:08.389819  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:08.390292  1723 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:08.390697  1723 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:08.390626  1873 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:08.391772  1723 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.393090  1723 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:08.396862  1799 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.400794  1881 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:08.406699  1878 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:08.410519  1877 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.436215  1873 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:08.435667  2414 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.429018  2410 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ae23688e76624153b5a7089586669bb0"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60934
I20260709 12:22:08.439589  2410 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:08.439996  2410 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.434929  1880 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:08.444577  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:08.445227  1725 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:08.445559  1725 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:08.444789  1629 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:08.446144  1878 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:08.446257  1725 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:08.447803  1725 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.460156  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:08.461282  3029 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:08.461769  3029 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:08.462867  2418 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.463596  3029 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.484143  2411 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60928
I20260709 12:22:08.485198  2411 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:08.485913  2411 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.499079  1629 catalog_manager.cc:5697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:08.403319  2417 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.386103  3069 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.510193  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:08.510990  1799 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.515067  1725 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:08.517117  1723 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:08.516649  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:08.457407  2410 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.519644  3061 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:08.520269  3061 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:08.521567  2410 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:08.523556  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:08.531029  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:08.531801  1798 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.525318  1873 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:08.532598  1873 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:08.533519  1873 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.540637  1873 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.552294  1799 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:08.565007  2417 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.569587  2417 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:08.572232  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:08.573042  1799 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.572664  1723 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:08.572503  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:08.577456  1725 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:08.577684  1725 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:08.579201  1873 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:08.585783  1725 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.594408  1725 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.565971  1798 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:08.608541  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:08.609999  3053 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:08.610553  3053 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:08.611428  3053 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.622198  1725 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:08.623595  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:08.624760  3029 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:08.568477  3061 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:08.626248  2988 raft_consensus.cc:1064] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:08.625330  3029 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:08.628382  3029 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.635090  2411 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.629519  2988 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.646831  1799 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:08.648207  2411 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:08.648190  1714 tablet_service.cc:1558] Processing DeleteTablet for tablet ae23688e76624153b5a7089586669bb0 with delete_type TABLET_DATA_TOMBSTONED (TS 075704438992440697732181e49ccd09 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:48982
I20260709 12:22:08.649502  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:08.650374  1798 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:08.650807  1798 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:08.651916  1798 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.653383  1798 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.657732  1723 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:08.672739  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:08.678431  1730 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:08.684314  2411 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:08.685969  1873 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:08.686169  3021 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:08.687685  2973 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:08.949934  1730 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
W20260709 12:22:09.016626  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 283420us
I20260709 12:22:09.022980  3090 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:09.051892  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:22:09.094532  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 55385.4us
W20260709 12:22:09.096024  1828 debug-util.cc:398] Leaking SignalData structure 0x7b0800168200 after lost signal to thread 564
I20260709 12:22:09.097533  3029 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:22:09.098275  1828 debug-util.cc:398] Leaking SignalData structure 0x7b0800162920 after lost signal to thread 1664
I20260709 12:22:09.044363  3090 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:09.102743  1828 debug-util.cc:398] Leaking SignalData structure 0x7b08003217a0 after lost signal to thread 1756
I20260709 12:22:09.115631  3090 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:09.098333  3029 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:09.120736  3029 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:09.124598  1798 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
W20260709 12:22:09.121158  1828 debug-util.cc:398] Leaking SignalData structure 0x7b08002479c0 after lost signal to thread 1831
W20260709 12:22:09.129142  1828 debug-util.cc:398] Leaking SignalData structure 0x7b080030a2a0 after lost signal to thread 1905
I20260709 12:22:09.141522  1878 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:22:09.140888  1828 debug-util.cc:398] Leaking SignalData structure 0x7b0800336c80 after lost signal to thread 2443
I20260709 12:22:09.187440  3088 tablet_replica.cc:333] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:22:09.189085  3088 raft_consensus.cc:2243] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:22:09.182667  1828 debug-util.cc:398] Leaking SignalData structure 0x7b080025cc00 after lost signal to thread 3026
I20260709 12:22:09.231153  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:09.193638  2988 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 12:22:09.232390  3088 raft_consensus.cc:2272] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:22:09.303819  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57294.8us
I20260709 12:22:09.315583  2414 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:09.329679  1628 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:22:09.343252  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61351.5us
I20260709 12:22:09.354820  3088 ts_tablet_manager.cc:1916] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:22:09.360591  3013 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:09.378360  1881 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:09.383911  2410 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:09.399250  1800 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:09.377074  3074 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:22:09.411350  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 67031.8us
I20260709 12:22:09.411582  1804 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:09.423138  1732 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:09.424067  1725 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:09.423651  1875 raft_consensus.cc:2955] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:09.426131  1874 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:09.427969  1731 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:09.436344  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:09.436985  1880 raft_consensus.cc:1813] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Handling vote request from an unknown peer 075704438992440697732181e49ccd09
I20260709 12:22:09.437463  1880 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 2: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:22:09.516049  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76925.3us
W20260709 12:22:09.691057  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 251796us
W20260709 12:22:09.563002  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149888us
I20260709 12:22:09.713589  1696 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:22:09.747226  1696 raft_consensus.cc:2686] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:22:09.747462  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:09.748364  1806 raft_consensus.cc:1813] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Handling vote request from an unknown peer 075704438992440697732181e49ccd09
I20260709 12:22:09.748893  1806 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 2: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:09.705425  3029 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:22:09.751874   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:09.754577  3025 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 12:22:09.761950  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64777.3us
I20260709 12:22:09.754621  3109 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
I20260709 12:22:09.776008  3037 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:09.754805  3102 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 12:22:09.801939  3070 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:09.842375  1804 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:22:09.820029  3070 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:09.879652  3121 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:22:09.880247  1801 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:09.887604  3106 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:09.888343  3070 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:09.889379  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 57956.8us
W20260709 12:22:09.856560  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 90565.4us
I20260709 12:22:09.919924  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:09.920938  3093 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:09.952473  1730 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:09.960697  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:09.921510  3093 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:09.959837  3053 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:22:09.987564  3037 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:09.987866  3102 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:09.987999  3097 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 12:22:09.992594  1800 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:09.994254  1801 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:10.001273  1873 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:09.971302  3074 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:48982 (request call id 74) took 1297 ms. Trace:
I20260709 12:22:10.030452  3013 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:10.047274  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:10.062260  3131 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:22:10.074954  3114 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.036851  3037 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:10.063797  3070 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:10.031782  3074 rpcz_store.cc:276] 0709 12:22:08.653072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.654927 (+  1855us) service_pool.cc:224] Handling call
0709 12:22:08.673979 (+ 19052us) log.cc:844] Serialized 494 byte log entry
0709 12:22:09.936023 (+1262044us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":568,"raft.run_cpu_time_us":1153,"raft.run_wall_time_us":1152}
I20260709 12:22:10.050499  3097 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:22:10.087476  3102 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:22:10.087455  3124 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260709 12:22:10.063494  3090 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260709 12:22:10.102536  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:10.075814  3114 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:10.124541  1614 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6388b1a5243d463589750794e8ac5511 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:22:10.113569  2982 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:10.167096  3114 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:10.172924  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:10.170507  2982 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:22:10.199834  3128 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.099s	user 0.000s	sys 0.001s
I20260709 12:22:10.208778  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:10.211906  3093 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:10.222249  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:10.226148  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:10.253078  1628 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 223) took 1035 ms. Trace:
I20260709 12:22:10.272912  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:10.255318  1629 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:10.293509  1772 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:10.322773  2892 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:10.295861  2982 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:10.322857  3088 ts_tablet_manager.cc:1929] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:22:10.343178  3088 log.cc:1199] T ae23688e76624153b5a7089586669bb0 P 075704438992440697732181e49ccd09: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-0-root/wals/ae23688e76624153b5a7089586669bb0
I20260709 12:22:10.351931  3104 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.352555  3104 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:10.355487  3088 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48982 (request call id 73) took 1707 ms. Trace:
I20260709 12:22:10.356022  3088 rpcz_store.cc:276] 0709 12:22:08.647713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.647865 (+   152us) service_pool.cc:224] Handling call
0709 12:22:09.014475 (+366610us) ts_tablet_manager.cc:1151] Deleting tablet ae23688e76624153b5a7089586669bb0
0709 12:22:09.023685 (+  9210us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:22:09.187172 (+163487us) spinlock_profiling.cc:243] Waited 89.5 ms on lock 0x7b540030b954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abde0ff9 00007f16b0dee482 00007f16b0e1f2d0 00007f16b0e0583d 00007f16b0e057e9 00007f16b0e05779 00007f16b0e05741 00007f16b0e0496d 0000000000388634 0000000000388479
0709 12:22:09.945141 (+757969us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:10.320876 (+375735us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:10.354277 (+ 33401us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16204,"spinlock_wait_cycles":205932928,"tablet-delete.queue_time_us":331917,"thread_start_us":206153,"threads_started":1}
I20260709 12:22:10.357482  3104 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:10.358065  1614 catalog_manager.cc:5028] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): tablet ae23688e76624153b5a7089586669bb0 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:22:10.333709  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:10.253594  1628 rpcz_store.cc:276] 0709 12:22:09.217244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.217414 (+   170us) service_pool.cc:224] Handling call
0709 12:22:10.252992 (+1035578us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:09.380097 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:09.382864 (+  2767us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:09.382906 (+    42us) write_op.cc:435] Acquired schema lock
0709 12:22:09.382935 (+    29us) tablet.cc:662] Decoding operations
0709 12:22:09.383456 (+   521us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:09.383552 (+    96us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:09.383595 (+    43us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:22:09.383865 (+   270us) tablet.cc:701] Row locks acquired
0709 12:22:09.383893 (+    28us) write_op.cc:260] PREPARE: finished
0709 12:22:09.384094 (+   201us) write_op.cc:270] Start()
0709 12:22:09.384234 (+   140us) write_op.cc:276] Timestamp: P: 1783599729384030 usec, L: 0
0709 12:22:09.384267 (+    33us) op_driver.cc:348] REPLICATION: starting
0709 12:22:09.516284 (+132017us) log.cc:844] Serialized 595 byte log entry
0709 12:22:09.920401 (+404117us) op_driver.cc:464] REPLICATION: finished
0709 12:22:10.112902 (+192501us) write_op.cc:301] APPLY: starting
0709 12:22:10.115845 (+  2943us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:10.120845 (+  5000us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:10.131964 (+ 11119us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:10.168280 (+ 36316us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:10.189024 (+ 20744us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:22:10.195268 (+  6244us) write_op.cc:312] APPLY: finished
0709 12:22:10.225030 (+ 29762us) log.cc:844] Serialized 23 byte log entry
0709 12:22:10.225262 (+   232us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:10.225364 (+   102us) write_op.cc:454] Released schema lock
0709 12:22:10.225406 (+    42us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:10.225854 (+   448us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:10.225884 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":165504,"child_traces":[["op",{"apply.queue_time_us":176175,"apply.run_cpu_time_us":2814,"apply.run_wall_time_us":120520,"num_ops":1,"prepare.queue_time_us":47328,"prepare.run_cpu_time_us":6475,"prepare.run_wall_time_us":299095,"replication_time_us":535974,"thread_start_us":164189,"threads_started":2}]]}
I20260709 12:22:10.375564  3037 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:10.375941  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:10.383739  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:10.397310  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:10.398020  1804 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:10.404345  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:10.497181  1845 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:22:10.509696  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 376873us
W20260709 12:22:10.511649  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 60541us
I20260709 12:22:10.520759  3112 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:10.521519  3112 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:10.522078  3112 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:10.687697  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.019s	sys 0.045s
W20260709 12:22:10.670874  3013 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.013s	sys 0.032s
W20260709 12:22:10.688422  1880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.023s	sys 0.042s
W20260709 12:22:10.693890  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.020s	sys 0.045s
W20260709 12:22:10.698246  1880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.023s	sys 0.042s
W20260709 12:22:10.698064  3013 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.013s	sys 0.032s
W20260709 12:22:10.767604  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 248900us
I20260709 12:22:10.754849  3108 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:10.773383  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 347969us
I20260709 12:22:10.779965  1771 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:10.797290  3140 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:22:10.778321  3108 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:10.807826  3138 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
W20260709 12:22:10.815207  3141 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:22:10.823969  3108 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:10.840345  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:10.814620  3112 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:10.859694  2981 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:10.870770  2981 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:10.873431  3104 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:10.877660  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:10.894857  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:10.910665  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:10.874774  2981 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:10.911034  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:10.914072  1729 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:10.904268  2986 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:10.928344  3107 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:10.955379  3107 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:10.977684  3107 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:10.978410  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:22:10.981592  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 72684us
W20260709 12:22:10.985479  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54002.2us
W20260709 12:22:10.989908  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 326646us
I20260709 12:22:10.996410  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:10.981878  2981 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:10.989830  3117 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:11.019119  3117 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:11.019575  3117 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:11.030243  3161 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:22:11.043542  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:11.049278  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:11.049002  2985 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:11.067565  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:11.069031  3112 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:11.073030  3111 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:11.103940  3090 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:11.107911  3090 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.073577  1804 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:11.110867  1874 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 1. Current term is 2. Ops: []
W20260709 12:22:11.113251  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 121174us
I20260709 12:22:11.118908  3090 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:11.126170  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:11.140552  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:11.141427  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:11.154444  1629 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 241) took 1986 ms. Trace:
I20260709 12:22:11.165839  1626 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.171463  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:11.175366  1845 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:11.176838  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:11.177354  1626 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.185290  1626 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 3 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.182861  3168 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:11.178851  3111 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:11.199196  3111 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 12:22:11.204805  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:11.228849  3117 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.240512  3117 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:11.241420  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:11.242084  1725 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:11.243217  1626 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 144) took 2256 ms. Trace:
I20260709 12:22:11.243569  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:11.154978  1629 rpcz_store.cc:276] 0709 12:22:09.197486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.212367 (+ 14881us) service_pool.cc:224] Handling call
0709 12:22:11.154336 (+1941969us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:10.759962 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:10.798962 (+ 39000us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:10.799014 (+    52us) write_op.cc:435] Acquired schema lock
0709 12:22:10.799037 (+    23us) tablet.cc:662] Decoding operations
0709 12:22:10.799412 (+   375us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:10.799477 (+    65us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:10.799508 (+    31us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:22:10.799727 (+   219us) tablet.cc:701] Row locks acquired
0709 12:22:10.799748 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:22:10.799953 (+   205us) write_op.cc:270] Start()
0709 12:22:10.800096 (+   143us) write_op.cc:276] Timestamp: P: 1783599730799889 usec, L: 0
0709 12:22:10.800125 (+    29us) op_driver.cc:348] REPLICATION: starting
0709 12:22:10.800861 (+   736us) log.cc:844] Serialized 538 byte log entry
0709 12:22:10.977837 (+176976us) op_driver.cc:464] REPLICATION: finished
0709 12:22:11.059946 (+ 82109us) write_op.cc:301] APPLY: starting
0709 12:22:11.060051 (+   105us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:11.060165 (+   114us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:11.061501 (+  1336us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:11.061789 (+   288us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:11.061954 (+   165us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:22:11.061998 (+    44us) write_op.cc:312] APPLY: finished
0709 12:22:11.062227 (+   229us) log.cc:844] Serialized 23 byte log entry
0709 12:22:11.063200 (+   973us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:11.063295 (+    95us) write_op.cc:454] Released schema lock
0709 12:22:11.063337 (+    42us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:11.063661 (+   324us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:11.063685 (+    24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4255,"reported_tablets":3,"spinlock_wait_cycles":21632,"child_traces":[["op",{"apply.queue_time_us":80040,"apply.run_cpu_time_us":1988,"apply.run_wall_time_us":3724,"num_ops":1,"prepare.queue_time_us":445337,"prepare.run_cpu_time_us":3381,"prepare.run_wall_time_us":49769,"replication_time_us":177564,"thread_start_us":467269,"threads_started":2}]]}
I20260709 12:22:11.258046  1798 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:11.261052  3095 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:11.261508  3095 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:11.260852  3102 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:11.264925  1725 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:11.268380  3102 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:11.268826  3102 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:11.288314  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:11.294760  3117 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:11.295321  3117 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:11.294127  1725 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:11.243701  1626 rpcz_store.cc:276] 0709 12:22:08.986807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.987781 (+   974us) service_pool.cc:224] Handling call
0709 12:22:11.243100 (+2255319us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:11.189617 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:11.190432 (+   815us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:11.190484 (+    52us) write_op.cc:435] Acquired schema lock
0709 12:22:11.190511 (+    27us) tablet.cc:662] Decoding operations
0709 12:22:11.195874 (+  5363us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:11.195943 (+    69us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:11.195994 (+    51us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:22:11.196390 (+   396us) tablet.cc:701] Row locks acquired
0709 12:22:11.196413 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:22:11.196599 (+   186us) write_op.cc:270] Start()
0709 12:22:11.196722 (+   123us) write_op.cc:276] Timestamp: P: 1783599731196541 usec, L: 0
0709 12:22:11.196753 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 12:22:11.215931 (+ 19178us) log.cc:844] Serialized 1293 byte log entry
0709 12:22:11.229178 (+ 13247us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b1800192ac0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16af80f41d 00007f16af81171a 00007f16af8116a8 00007f16af811628 00007f16af8115e3 00007f16af8107fc 00007f16b1c432f6 00007f16b1c3daa8 00007f16abde4310
0709 12:22:11.229239 (+    61us) op_driver.cc:464] REPLICATION: finished
0709 12:22:11.230539 (+  1300us) write_op.cc:301] APPLY: starting
0709 12:22:11.230638 (+    99us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:11.230797 (+   159us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:11.230818 (+    21us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:11.231335 (+   517us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:11.231521 (+   186us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 12:22:11.231585 (+    64us) write_op.cc:312] APPLY: finished
0709 12:22:11.231833 (+   248us) log.cc:844] Serialized 35 byte log entry
0709 12:22:11.232023 (+   190us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:11.232122 (+    99us) write_op.cc:454] Released schema lock
0709 12:22:11.232173 (+    51us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:11.233247 (+  1074us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:11.234198 (+   951us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":75524,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":1070,"apply.run_cpu_time_us":1923,"apply.run_wall_time_us":1923,"num_ops":3,"prepare.queue_time_us":153,"prepare.run_cpu_time_us":4371,"prepare.run_wall_time_us":29811,"replication_time_us":25918,"spinlock_wait_cycles":5781248}]]}
I20260709 12:22:11.309537  1732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:11.310186  1732 raft_consensus.cc:606] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:11.322474  3117 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.304930  2411 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:11.272621  3102 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.271756  3095 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.337728  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:11.337772  3029 consensus_queue.cc:1175] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "075704438992440697732181e49ccd09" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:11.340958  3168 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:11.337442  3102 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:11.392105  1628 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.402523  1627 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 222) took 1994 ms. Trace:
I20260709 12:22:11.407039  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:11.407749  1730 raft_consensus.cc:606] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:11.403029  1627 rpcz_store.cc:276] 0709 12:22:09.408233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.408448 (+   215us) service_pool.cc:224] Handling call
0709 12:22:09.728611 (+320163us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd42090 00007f16abde0f3a 00007f16b1b96e9c 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:22:11.402457 (+1673846us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45215,"reported_tablets":6,"spinlock_wait_cycles":618845568}
I20260709 12:22:11.403601  1628 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 3 to 4, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.421375  3095 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.420385  1628 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:11.461642  3111 raft_consensus.cc:993] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:11.462198  3111 raft_consensus.cc:1081] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:11.450729  1731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:11.466259  1731 raft_consensus.cc:606] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:11.497135  1876 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:11.500159  1878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:11.509116  1878 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:11.509622  1878 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:11.509040  1876 raft_consensus.cc:606] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:11.502997  3109 raft_consensus.cc:993] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:11.515055  3109 raft_consensus.cc:1081] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:11.492100  1801 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.520498  1877 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:11.521327  1877 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:11.521775  1877 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.526396  1732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:11.527194  1732 raft_consensus.cc:606] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:11.527717  1878 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:11.534478  1798 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:11.534586  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:11.535347  1732 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:11.535143  1798 raft_consensus.cc:606] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:11.535749  1732 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:11.536598  1732 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:11.537927  1732 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:11.537297  1878 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:11.543398   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:22:11.545406   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:11.547447  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:11.548187  1800 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:11.548792  1732 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:11.555603  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:11.560459  3098 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:11.561618  3098 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:11.564546   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:11.585561  1629 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 2 to 3, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:11.586620  1877 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:11.589843  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:11.590463  1731 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:11.590809  1731 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:11.591447  3098 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:11.591631  1731 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:11.593204  1731 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.622013  3109 raft_consensus.cc:993] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:11.622678  3109 raft_consensus.cc:1081] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:11.624392  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet 6388b1a5243d463589750794e8ac5511 with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44522
I20260709 12:22:11.632609  2418 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60950
I20260709 12:22:11.633414  2418 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:11.633963  2418 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.643157  1800 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:11.649652  2418 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.669579  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:11.670480  1730 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:11.671016  1730 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:11.671828  1730 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.676440  2418 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:11.681968  1730 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.692636  1730 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:11.693611  1732 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:11.694294  2418 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:22:11.695294  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:11.700928  3176 tablet_replica.cc:333] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:22:11.702314  3176 raft_consensus.cc:2243] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:11.703228  3176 raft_consensus.cc:2272] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:11.706151  3102 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:11.706215  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:11.709998  2973 raft_consensus.cc:993] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:11.711104  3111 raft_consensus.cc:1081] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:11.707080  1876 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.718525  2411 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60950
I20260709 12:22:11.703616  3067 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260709 12:22:11.719362  2411 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:11.719812  2411 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.721009  2986 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:22:11.696965  1614 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a4c45f93428e4cc68f85f17ea3ec3ad0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 075704438992440697732181e49ccd09 (127.0.140.65:35067) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 075704438992440697732181e49ccd09 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:22:11.745306  1614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63333.9us
I20260709 12:22:11.725906  3102 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:11.750386  3102 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.752141  1876 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:11.708073  1731 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:11.767035  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:11.767819  1801 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.774268  2986 raft_consensus.cc:993] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:11.775045  2986 raft_consensus.cc:1081] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:11.787089  1731 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:11.790954  1731 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:11.791414  1731 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.804674  1628 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:11.747545  1877 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:11.830986  3176 ts_tablet_manager.cc:1916] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:22:11.837914  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:11.837261  2411 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.866130  2411 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:11.871335  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:11.872082  1877 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.880297  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:11.881384  1725 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:11.881778  1725 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:11.884927  1725 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:11.854833  3119 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:11.899216  1725 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:11.917637  3119 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:11.918944  3119 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:11.927394  1729 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:11.910993  1801 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:11.934752  3091 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:11.944628  2410 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:11.949785  1725 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:11.951227  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:11.987546  2418 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:11.961275  3102 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:11.992655  1731 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:11.996645  1731 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:11.953557  3092 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 12:22:11.999604  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:12.000306  1798 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:12.000607  1798 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:11.999346  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:12.001346  1798 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:12.001909  1880 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:12.020148  3180 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:22:12.020872  1798 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:12.026325  1880 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:12.029829  3176 ts_tablet_manager.cc:1929] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:22:12.031266  1798 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:12.031502  1877 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:11.945602  2981 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:12.064508  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:12.040373  3176 log.cc:1199] T 6388b1a5243d463589750794e8ac5511 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/6388b1a5243d463589750794e8ac5511
I20260709 12:22:12.074942  1732 raft_consensus.cc:2955] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:12.077111  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet 6388b1a5243d463589750794e8ac5511 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:22:12.087066  3021 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:12.087742  3021 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:12.089094  3021 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:12.098851  3102 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:12.125106  1626 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:12.146538  3102 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:12.148929  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 66d3405528d44edb8c0be3b296ef72be with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:22:12.195084  1730 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:22:12.195298  3181 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.149s	user 0.003s	sys 0.002s
I20260709 12:22:12.205271  3144 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:12.221294  1798 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:12.229305  1726 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:12.240360  3168 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:12.245114  3155 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:12.251418  3168 raft_consensus.cc:993] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:12.252126  3168 raft_consensus.cc:1081] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:12.259573  1725 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57228
I20260709 12:22:12.260437  1725 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:12.260944  1725 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:12.298569  1725 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:12.303413  1725 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:12.304930  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:12.305208  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:12.305645  1804 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:12.333673  1804 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:12.335318  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:12.337985  1628 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 3 to 4, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:12.338692  1627 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:12.374975  1880 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:12.339175  3186 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:12.376560  1880 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:12.386359  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59292.6us
I20260709 12:22:12.388273  1880 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:12.390774  3186 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:12.403759  1880 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:12.404505  1878 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:12.408870  3029 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:12.426762  1880 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:12.403293  3186 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:12.445734  1728 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:12.464769  3098 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:22:12.504969  3191 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.134s	user 0.005s	sys 0.000s
I20260709 12:22:12.558562  3188 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:12.661404  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 89303.4us
I20260709 12:22:12.603173  3188 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:12.693995  1629 catalog_manager.cc:5697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:12.752614  1614 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet a4c45f93428e4cc68f85f17ea3ec3ad0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 075704438992440697732181e49ccd09 (127.0.140.65:35067) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 075704438992440697732181e49ccd09 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:12.704425  3198 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:12.793421  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 85120.8us
I20260709 12:22:12.814599  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet 66d3405528d44edb8c0be3b296ef72be with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43056
I20260709 12:22:12.818812  3029 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:12.819530  3029 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:12.811247  3198 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:12.822727  3190 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.069s	user 0.000s	sys 0.002s
I20260709 12:22:12.822474  3029 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:12.833078  3188 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:12.843869  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:12.846570  1726 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:12.846727  3062 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:12.847043  3168 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.847564  3062 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:12.847702  3168 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:12.851238  3062 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:12.851581  1730 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:12.855086  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:12.856050  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:12.856680  1725 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:12.863363  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:12.862695  3144 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:22:12.866112  3198 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:12.880280  3168 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:12.882155  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 204686us
I20260709 12:22:12.911363  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:12.912070  2410 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:12.916275  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:12.917109  1800 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:12.917382  1876 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 2: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:12.916641  1626 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:12.921588  1876 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:12.924458  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:12.919828  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:12.925465  2415 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:12.925874  1799 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:12.920549  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:12.927045  1881 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:12.932571  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:12.933444  2415 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:22:12.934237  3107 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:12.941949  1846 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:22:12.948287  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 301378us
I20260709 12:22:12.948563   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:12.959450  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:12.960309  1799 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 1.
I20260709 12:22:12.962714  3111 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:12.975970  3219 tablet_replica.cc:333] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:22:12.975989  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:12.981966  3167 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:12.991990  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:13.008682  1881 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 2: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:12.993019  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 51170.6us
I20260709 12:22:13.004303  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:13.006968  1726 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:13.009961  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:13.033197  1798 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:13.036372  1881 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:13.038687  1770 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:13.023415  2973 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:13.049069  2973 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:13.049485  2973 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:13.049805  3219 raft_consensus.cc:2243] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:13.070734  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:13.071426  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:13.072130  1806 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:13.073235  1877 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:13.075317  1801 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:13.075366  3219 raft_consensus.cc:2272] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:13.075085  1627 catalog_manager.cc:5697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:13.079202  3219 ts_tablet_manager.cc:1916] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:22:13.086977  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48423us
I20260709 12:22:13.070871  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:13.101898  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57056us
I20260709 12:22:13.095611  3167 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:13.125347  3167 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:13.126066  3167 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:13.137332  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68649.7us
I20260709 12:22:13.103617  2982 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:13.108001  3216 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:13.159026  3216 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:13.159549  3216 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.144273  3188 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:13.141294  2982 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:13.164597  2982 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.107643  2973 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.167605  3107 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:22:13.179639  3183 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:13.172385  3218 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:13.195680  1876 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:13.196465  1876 raft_consensus.cc:606] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:13.200342  2973 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:13.204381  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:13.211670  3112 raft_consensus.cc:993] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:13.212817  3112 raft_consensus.cc:1081] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:13.216238  1729 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57228
I20260709 12:22:13.217015  1729 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:13.217381  1729 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.230499  1729 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.226142  2982 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.273777  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:13.285550  1799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:13.291638  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:13.307760  1799 raft_consensus.cc:606] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:13.293744  3167 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.211756  3219 ts_tablet_manager.cc:1929] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:22:13.211738  3216 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.312372  1807 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:13.333904  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:13.334590  2418 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.335999  1729 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:13.336319  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:13.336885  1877 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:13.337224  1877 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:13.338002  1877 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.343466  1877 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.347571  1807 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:13.350340  1805 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:13.351722  1805 raft_consensus.cc:606] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:13.323554  3219 log.cc:1199] T 66d3405528d44edb8c0be3b296ef72be P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/66d3405528d44edb8c0be3b296ef72be
I20260709 12:22:13.337858  2415 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:13.369339  3216 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:13.371374  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:13.373879  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:13.379971  1877 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:13.374768  1732 raft_consensus.cc:2368] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3.
I20260709 12:22:13.382684  2418 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
W20260709 12:22:13.360677  3227 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:22:13.400732  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:13.406452  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:13.403347  1876 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:13.409014  2382 leader_election.cc:400] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3.
I20260709 12:22:13.426067  2382 leader_election.cc:403] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:22:13.428916  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:13.410897  3167 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:13.412155  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:13.412611  3111 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:13.431915  3111 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:13.439406  1845 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:13.415416  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet 66d3405528d44edb8c0be3b296ef72be (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:22:13.414891  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:13.447829  1732 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:13.448305  3110 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:13.417466  3091 raft_consensus.cc:993] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:13.417794  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:13.450218  1726 raft_consensus.cc:2368] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 2. Current term is 3.
I20260709 12:22:13.418529  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:13.424978  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:13.432914  3111 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.452673  1772 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:13.453584  3091 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:13.415256  2415 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:13.454948  3144 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3.
I20260709 12:22:13.455843  1770 leader_election.cc:400] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 2. Current term is 3.
I20260709 12:22:13.417165  2982 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:13.456801  3212 raft_consensus.cc:1081] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:22:13.459666  1877 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c072b820694347cabf7868a1b3805f76"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:13.460212  2414 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60928
I20260709 12:22:13.461251  2414 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:13.461934  2414 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:13.462733  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61269.9us
I20260709 12:22:13.460309  1877 raft_consensus.cc:606] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:13.481668  1805 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:13.484004  1626 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:13.484851  1805 raft_consensus.cc:606] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:13.486491   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:13.488412   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:13.489295   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:13.499280  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:13.500710  3109 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:13.501225  3109 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:13.502156  3109 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.505442  2414 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:13.580474  3228 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:22:13.584237  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:13.586140  1805 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:13.586467  1805 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:13.587219  1805 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.587071  1628 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:13.588373  1805 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.603456  2414 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:13.604622  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:13.605394  1880 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.583750  3212 raft_consensus.cc:993] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:13.621029  1805 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:13.626969  3170 raft_consensus.cc:1081] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:13.634462  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:13.635569  3183 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:13.636118  3183 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:13.637251  3183 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:13.664521  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53397.1us
I20260709 12:22:13.681494  1732 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:13.682253  1732 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:13.682639  1732 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.702629  1880 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:13.703194  1628 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:13.739210  3110 raft_consensus.cc:993] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: : Instructing follower 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
I20260709 12:22:13.743197  3110 raft_consensus.cc:1081] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Signalling peer 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
I20260709 12:22:13.746644  1804 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c072b820694347cabf7868a1b3805f76"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:33790
I20260709 12:22:13.747283  1804 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:13.747650  1804 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.753985  1732 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.759299  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:13.760011  1807 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:13.760321  1807 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:13.760484  1804 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.761184  1807 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.762471  1807 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.764297  1732 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:13.764640  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:13.765249  2414 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.770886  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:13.771586  1880 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:13.771637  1804 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:13.771946  1880 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:13.774626  1880 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.775882  1880 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.776677  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:13.777307  2415 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.789230  2414 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:13.791100  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:13.792163  2973 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:13.792822  2973 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:13.793790  2973 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.795524  1880 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:13.811117  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:13.812456  3170 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:13.813122  3170 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:13.814424  3170 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.825979  1626 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:13.828243  1807 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:13.854655  3212 raft_consensus.cc:993] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:13.855265  3212 raft_consensus.cc:1081] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:13.859000  1730 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:13.814386  2415 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:13.861136  1880 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:13.859656  1730 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:13.863960  1730 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.865149  2973 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260709 12:22:13.881407  1730 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.887293  1730 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:13.898612  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:13.899510  2415 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.908874  1804 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:22:13.901764  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:13.910485  1807 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:13.910842  1807 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:13.911638  1807 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:13.913120  1807 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:13.922889  2411 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:13.926898  1626 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 1 to 2, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:13.932174  1807 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:13.939504  3107 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:13.941318  3107 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:13.951826  2415 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:13.959311  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:13.971065  3107 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:13.971833  3107 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:13.972934  3107 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:14.022675  1628 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:14.043067  2415 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:22:14.046830  3111 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:14.092039  1880 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:22:14.094827  3111 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:14.105607  1876 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:14.108947  1880 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:14.116938  3183 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:14.205673  1798 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:14.211388  1798 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:14.213317  3144 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:14.216466  3111 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:14.275990  2415 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:14.278234  3111 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:14.294639  2411 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:14.297504  3170 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260709 12:22:14.337626  1881 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:14.346738  2982 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:14.358466  1732 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:14.373799  2415 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:14.374476  1880 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:14.375361  3111 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:14.376364  2973 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:14.400257  2411 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:14.400542  3107 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:14.412885  3111 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:14.422391  1614 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a4c45f93428e4cc68f85f17ea3ec3ad0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 8)
I20260709 12:22:14.424026  1880 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:14.436856  2411 raft_consensus.cc:2955] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:22:14.442916  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48020.7us
W20260709 12:22:14.448544  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50195.9us
I20260709 12:22:14.479486  1629 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 3 to 5, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 3 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:14.513113  1805 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:14.515887  3186 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:14.537408  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet a4c45f93428e4cc68f85f17ea3ec3ad0 with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44522
I20260709 12:22:14.573789  3255 tablet_replica.cc:333] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:22:14.575408  3255 raft_consensus.cc:2243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:14.576597  3255 raft_consensus.cc:2272] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:14.611572  3255 ts_tablet_manager.cc:1916] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:22:14.658114  3255 ts_tablet_manager.cc:1929] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260709 12:22:14.658586  3255 log.cc:1199] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/a4c45f93428e4cc68f85f17ea3ec3ad0
I20260709 12:22:14.665225  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet a4c45f93428e4cc68f85f17ea3ec3ad0 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:22:14.709461   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:14.836207  1807 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:14.836937  1807 raft_consensus.cc:606] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:14.855954  1881 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:14.856639  1881 raft_consensus.cc:606] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:14.861958  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:14.862722  1730 raft_consensus.cc:606] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:14.864763   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:22:14.866700   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:14.867580   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:15.167667  3258 raft_consensus.cc:993] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:15.168211  3258 raft_consensus.cc:1081] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:15.168931  3256 raft_consensus.cc:993] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: : Instructing follower 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
I20260709 12:22:15.169448  3256 raft_consensus.cc:1081] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Signalling peer 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
I20260709 12:22:15.170671  1730 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:15.171233  1807 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:33790
I20260709 12:22:15.171311  1730 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:15.171851  1807 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:15.171945  1730 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:15.172247  1807 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:15.180830  1730 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:15.181504  1807 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:15.184305  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:15.185019  1805 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:15.185151  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:15.185700  1805 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:15.185878  2415 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:15.186569  1805 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:15.187736  1805 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:15.197109  2415 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:15.201828  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:15.185014  1730 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:15.212337  1805 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:15.220136  1807 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:15.237231  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:15.237836  1881 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:15.238125  1881 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:15.238824  1881 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:15.240101  1881 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:15.243366  3223 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:15.243909  3223 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:15.244791  3223 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:15.248486  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 46761.6us
I20260709 12:22:15.250645  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:15.251189  1730 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:15.260560  3107 raft_consensus.cc:993] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:15.261225  3107 raft_consensus.cc:1081] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:15.268038  1876 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:15.268748  1876 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:15.269126  1876 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:15.285295  1876 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:15.289438  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:15.290169  1807 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:15.299073  1807 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:15.300549  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:15.301455  3108 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:15.301991  3108 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:15.303326  3108 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:15.308574  1876 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:15.313565  1626 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:15.318449  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 117659us
I20260709 12:22:15.331857  1730 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:15.348465  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:15.358148  1732 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:15.367722  1732 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:15.368705  1732 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:15.370954  1732 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:15.361503  1881 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:15.362564  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:15.393368  1732 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:15.402311  3257 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:22:15.421923  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69488.3us
W20260709 12:22:15.427358  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49075.1us
I20260709 12:22:15.428778  3257 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:15.434424  3257 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:15.463897  1628 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:15.509871  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 127315us
W20260709 12:22:15.519086  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 118055us
W20260709 12:22:15.531862  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 144923us
I20260709 12:22:15.548229  1626 catalog_manager.cc:5697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 2 to 3, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:15.621340  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 55147.4us
I20260709 12:22:15.822892  2415 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:15.833314  1807 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:15.845451  3260 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:15.895349  1729 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:15.903359  3282 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:22:15.910408  1805 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:15.919139  3223 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:15.931792   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:15.957201  1730 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:15.959851  2982 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:16.004163  3275 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:16.031869  1877 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:16.036818  3259 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
W20260709 12:22:16.239223  3295 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.065s	user 0.000s	sys 0.002s
I20260709 12:22:16.272702  1880 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:16.283623  1876 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:16.284576  1876 raft_consensus.cc:606] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:16.292374  1798 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:16.293048  1798 raft_consensus.cc:606] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:16.297744  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:16.298420  1730 raft_consensus.cc:606] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 1 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:16.301507   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:22:16.303365   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:16.304590   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:16.315548  3107 raft_consensus.cc:993] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: : Instructing follower a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:16.316226  3107 raft_consensus.cc:1081] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 1 LEADER]: Signalling peer a20e99e125b84513a44e88860667ed61 to start an election
I20260709 12:22:16.324553  1877 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:55690
I20260709 12:22:16.325338  1877 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:16.325768  1877 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:16.337071  1798 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:22:16.339044  3256 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:22:16.343585  1846 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bb813e4fab14a61b810a24b56899586. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:22:16.349179  1730 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:22:16.355571  3267 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:16.371968  1877 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:16.374073  1877 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:16.376000  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:16.376830  1726 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:16.377146  1726 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:16.378093  1726 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:16.379915  1726 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:16.401262  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:16.401933  1805 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:16.412421  1805 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:16.415747  1726 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:16.416021  3266 raft_consensus.cc:2955] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.427148  1798 raft_consensus.cc:2955] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.433791  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:16.435003  3267 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:16.435563  3267 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:16.436604  3267 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:16.443507  3256 raft_consensus.cc:993] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: : Instructing follower 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
I20260709 12:22:16.444222  3112 raft_consensus.cc:1081] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Signalling peer 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
I20260709 12:22:16.446472  1804 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:33790
I20260709 12:22:16.447103  1804 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:16.447484  1804 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:16.455801  1804 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:16.459700  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:16.460398  1730 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:16.459141  1627 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 3, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.463871  1729 raft_consensus.cc:2955] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.490723  1726 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:16.497269  1628 catalog_manager.cc:5697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:16.499017  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:16.499621  1880 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:16.499977  1880 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:16.500932  1880 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:16.502471  1880 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:16.508663  1804 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:16.499161  1730 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:16.517055  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:16.524633  3212 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:16.527173  1880 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:16.528401  3212 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:16.532466  3212 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:16.536572  1881 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:22:16.539161  3260 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:22:16.540596  1697 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d891ed30ca204d09b7b083088037b63d. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:22:16.545956  3283 raft_consensus.cc:993] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:16.546473  3283 raft_consensus.cc:1081] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:16.547408  1804 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:22:16.548730  1726 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57234
I20260709 12:22:16.549295  1726 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:16.549553  1726 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:16.564401  1726 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:16.559176  3260 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260709 12:22:16.580147  1726 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:16.581519  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:16.582134  1877 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:16.587781  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:16.588472  1798 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:16.588816  1798 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:16.589670  1798 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:16.608520  3260 raft_consensus.cc:2955] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.615880  1798 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:16.616261  1880 raft_consensus.cc:2955] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.630330  1798 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:16.634595  1877 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:16.643553  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:16.644876  3223 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:16.645427  3223 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:16.646371  3223 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:16.654011  1877 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:16.629899  1626 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 3 to 4, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 4 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:16.673166  1627 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:16.706637  1804 raft_consensus.cc:2955] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.731999  1732 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 12:22:16.735296  1846 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ea5d8f652fc448d6af846ff6850004b8. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:22:16.736423  3108 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260709 12:22:16.763546  1798 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:22:16.766607  3267 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:16.787230  3108 raft_consensus.cc:2955] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.791795  1805 raft_consensus.cc:2955] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.792878  1729 raft_consensus.cc:2955] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.829337  1798 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:16.850376  1627 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 3, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.851760  3313 ts_tablet_manager.cc:933] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:16.876685  1626 catalog_manager.cc:5697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:16.913707  1772 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: def0036446324543ba07e132e885133a. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:22:16.913909  1881 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:22:16.915810  3212 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:16.922927  1726 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:16.925328  3266 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:16.929979  1726 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:16.932638  3257 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:16.940910  1730 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:22:16.951303  3257 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:16.958630  3283 raft_consensus.cc:2955] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:16.972946  1881 raft_consensus.cc:2955] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:17.006716  3313 tablet_copy_client.cc:323] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:22:17.047487  1877 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:17.049371  1798 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:17.085388  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 8bb813e4fab14a61b810a24b56899586 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:22:17.086105  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 8bb813e4fab14a61b810a24b56899586 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-8bb813e4fab14a61b810a24b56899586
I20260709 12:22:17.085187  3112 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:22:17.119419  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59827.8us
I20260709 12:22:17.243676  1890 tablet_copy_source_session.cc:215] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:22:17.244048  1627 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 1 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:22:17.247334  3306 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:22:17.247982  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 145717us
W20260709 12:22:17.308208  3323 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:22:17.310253  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 88278.4us
I20260709 12:22:17.322211  3324 ts_tablet_manager.cc:933] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
W20260709 12:22:17.357331  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47511.6us
I20260709 12:22:17.391968  3327 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:22:17.399962  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 244722us
I20260709 12:22:17.425516   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:17.541764  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46290.2us
W20260709 12:22:17.551565  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49500.8us
I20260709 12:22:17.549264  3324 tablet_copy_client.cc:323] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.65:35067
W20260709 12:22:17.584384  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58821.7us
I20260709 12:22:17.647789  1742 tablet_copy_service.cc:140] P 075704438992440697732181e49ccd09: Received BeginTabletCopySession request for tablet d891ed30ca204d09b7b083088037b63d from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:48886)
I20260709 12:22:17.648712  1742 tablet_copy_service.cc:161] P 075704438992440697732181e49ccd09: Beginning new tablet copy session on tablet d891ed30ca204d09b7b083088037b63d from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:48886: session id = 123095127f334445889c21a42ba1d41e-d891ed30ca204d09b7b083088037b63d
W20260709 12:22:17.693570  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54674.2us
I20260709 12:22:17.737668  3313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bb813e4fab14a61b810a24b56899586. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:22:17.739383  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84409.5us
W20260709 12:22:18.053997  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 312399us
W20260709 12:22:18.108419  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53391.3us
W20260709 12:22:18.123435  3256 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.844s	user 0.013s	sys 0.015s
W20260709 12:22:18.170308  3256 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.884s	user 0.014s	sys 0.015s
W20260709 12:22:18.400167  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 290444us
I20260709 12:22:18.380942  3325 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.380079  3328 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:18.367916  3344 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.398311  3337 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.461813  3355 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.526569  3344 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.525841  3328 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.529564  3344 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.529865  3328 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.519186  3334 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:18.537938  3364 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:18.538641  3364 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.530835  3358 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:18.539449  3365 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:18.523739  3337 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.549403  3337 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.542609  3358 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.552285  3358 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.552863  3358 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.553371  3358 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.547005  3365 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.557415  3365 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.523026  3325 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.560827  3325 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.537230  3334 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.562963  3334 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.563457  3265 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:18.563862  3265 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.567662  3364 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.535282  3355 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.573130  3334 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.574054  3334 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.597186  3369 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:18.575645  3358 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.598351  3369 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.600646  3265 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.625824  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:18.634730  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 232170us
I20260709 12:22:18.599380  3325 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:18.620294  3328 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.634362  3265 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:18.646677  3367 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.665072  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:18.665604  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:18.651027  3344 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:18.655643  3334 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.669348  3328 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.668429  3367 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.677348  2418 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:18.640012  3325 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.685218  3334 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.685796  3334 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.688256  3334 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.689038  3334 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:18.689774  3334 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.692833  3334 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.693590  3334 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.670707  3259 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.655505  3265 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.696748  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:18.702909  3375 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.703887  3375 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.705407  3379 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:18.680370  3256 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:18.722981  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:18.694308  3334 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:18.706115  3379 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.681304  3344 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.718730  3355 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.743505  3375 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.745810  3379 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.748200  3328 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.770125  3325 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.772619  3373 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.773326  3381 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.775115  3344 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.740679  3369 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:18.779176  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 59560.3us
I20260709 12:22:18.719595  3256 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.709709  3259 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:18.798245  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 124944us
I20260709 12:22:18.804654  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.805605  1804 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 1.
I20260709 12:22:18.807925  3379 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.809018  3379 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.814262  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:18.815248  2417 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:22:18.794979  3381 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:22:18.821723  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 94333.2us
I20260709 12:22:18.822252  3381 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:22:18.821209  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.818395  3385 ts_tablet_manager.cc:933] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.827102  3334 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.832882  3267 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.837598  3267 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:18.840551  3267 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.842957  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.812814  3376 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:18.841208  3267 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.798240  3373 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.856820  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.859398  3334 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.863988  3334 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.872956  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:18.881335  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 80307.4us
I20260709 12:22:18.876624  1627 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 268) took 1760 ms. Trace:
I20260709 12:22:18.883929  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.884186  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:18.859930  3376 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.861650  3267 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.880141  3325 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:18.891963  3376 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.892423  3387 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.896144  3373 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.893069  3267 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.881672  3265 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.893430  3328 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.894872  3379 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.894807  3377 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.896056  3387 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.917737  3388 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.893075  3359 ts_tablet_manager.cc:933] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.908244  3259 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:18.894114  3325 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:18.909423  3328 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.910548  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:18.921241  3379 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.921553  3387 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.918443  3388 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:18.929780  3328 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.926087  3325 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:18.930949  3379 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.933177  3379 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.911240  3377 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.936489  3267 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.937614  3267 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.939472  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:18.940711  3267 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.942096  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:18.942662  1881 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 1.
I20260709 12:22:18.944691  3394 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.945243  3394 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.948009  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:18.883294  1627 rpcz_store.cc:276] 0709 12:22:17.110484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.120547 (+ 10063us) service_pool.cc:224] Handling call
0709 12:22:18.870326 (+1749779us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:17.289836 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:17.551387 (+261551us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:17.560927 (+  9540us) write_op.cc:435] Acquired schema lock
0709 12:22:17.564391 (+  3464us) tablet.cc:662] Decoding operations
0709 12:22:17.577528 (+ 13137us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:17.577595 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:17.577625 (+    30us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:22:17.577931 (+   306us) tablet.cc:701] Row locks acquired
0709 12:22:17.577963 (+    32us) write_op.cc:260] PREPARE: finished
0709 12:22:17.578145 (+   182us) write_op.cc:270] Start()
0709 12:22:17.578263 (+   118us) write_op.cc:276] Timestamp: P: 1783599737578086 usec, L: 0
0709 12:22:17.578290 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:22:17.607746 (+ 29456us) log.cc:844] Serialized 595 byte log entry
0709 12:22:17.694875 (+ 87129us) op_driver.cc:464] REPLICATION: finished
0709 12:22:18.524675 (+829800us) write_op.cc:301] APPLY: starting
0709 12:22:18.524785 (+   110us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:18.524902 (+   117us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:18.529026 (+  4124us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:18.529332 (+   306us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:18.529511 (+   179us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:22:18.529564 (+    53us) write_op.cc:312] APPLY: finished
0709 12:22:18.529797 (+   233us) log.cc:844] Serialized 23 byte log entry
0709 12:22:18.530002 (+   205us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:18.530427 (+   425us) write_op.cc:454] Released schema lock
0709 12:22:18.530480 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:18.537777 (+  7297us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:18.537814 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":808487,"apply.run_cpu_time_us":2760,"apply.run_wall_time_us":7569,"num_ops":1,"prepare.queue_time_us":5245,"prepare.run_cpu_time_us":22324,"prepare.run_wall_time_us":336935,"replication_time_us":116381,"thread_start_us":9435,"threads_started":1}]]}
I20260709 12:22:18.944339  3328 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:18.949880  3328 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:18.948673  3393 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:18.953431  3391 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:22:18.954092  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.756s	user 0.021s	sys 0.030s
I20260709 12:22:18.953015  3328 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.953581  3393 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.956148  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.956918  1807 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:18.946228  3387 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.957768  3387 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.954031  3391 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:18.961859  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.764s	user 0.022s	sys 0.031s
W20260709 12:22:18.963325  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 1.76506e+06us
I20260709 12:22:18.969563  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.979314  3385 tablet_copy_client.cc:323] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:22:18.981585  3398 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:18.982467  3398 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.982913  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet def0036446324543ba07e132e885133a from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:22:18.983978  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet def0036446324543ba07e132e885133a from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-def0036446324543ba07e132e885133a
I20260709 12:22:18.985450  3259 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:18.986177  3259 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:18.990242  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.990994  1799 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:18.992431  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.992518  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:18.993218  1802 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:18.993362  1801 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:18.997402  3377 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:18.998749  3394 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:18.999403  3394 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:19.000190  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:19.002503  1807 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:19.004422  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:19.005059  1807 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:19.006579  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:18.959681  3376 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.009029  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.010043  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:18.962949  3391 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:19.012339  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.015015  1877 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:19.000839  3365 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:18.985666  3398 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:19.026069  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.030366  3393 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:19.031436  3259 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:19.005362  3379 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:19.027012  1877 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:19.010848  3394 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:19.037372  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:19.016614  3370 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:19.059950  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.058399  3370 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.068751  1877 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:19.074280  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.068347  3370 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:19.017612  3376 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:19.085512  3399 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:19.086004  1816 tablet_copy_source_session.cc:215] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:22:19.088346  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.089054  1880 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:19.117681  3403 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:19.118672  3403 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.033741  3365 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:19.122326  1877 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:19.056430  3379 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:19.126487  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.082527  3265 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:19.138370  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 157227us
I20260709 12:22:19.086114  3399 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.139312  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:19.139436  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 127317us
I20260709 12:22:19.074524  3325 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.149567  3399 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:19.163641  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.164532  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.165323  1876 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:19.127930  3359 tablet_copy_client.cc:323] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:22:19.167424  3404 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:19.168404  3404 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.169361  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.170068  1873 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:19.171738  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.172932  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.173629  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.171728  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.175758  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:19.175951  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:19.177387  3377 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:19.182447  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.183259  3390 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:19.183974  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.184295  3390 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:19.185317  3401 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.185482  3405 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:19.185979  3401 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.187004  3328 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:19.187422  3328 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.187801  3328 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.183526  3377 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:19.172678  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.188513  3401 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:19.188588  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.189153  1875 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:22:19.186190  3405 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.201514  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.148800  3398 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:19.216116  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.203125  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:19.238037  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 92850.3us
I20260709 12:22:19.200250  3265 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.204735  3387 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:19.237617  3393 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:19.204229  3369 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:19.204474  3337 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:19.202827  3377 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:19.268370  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 73037us
I20260709 12:22:19.270699  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.300982  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.246987  3342 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:19.248909  3394 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:19.248005  3325 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:19.266578  3377 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.266578  3369 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.266824  3337 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.249747  3404 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:19.246438  3409 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:19.305028  3373 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:19.335400  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.337060  2982 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:19.337854  2982 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.338385  2982 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.298592  3265 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.328986  3369 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.332260  3388 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:19.328356  3342 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.327124  3337 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.268481  3398 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.343549  3402 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:19.315320  3393 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:19.335250  3373 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.356230  3414 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:19.357093  3414 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.350162  3342 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.353482  1802 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:19.361742  3373 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.389942  3375 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:19.390259  3386 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:19.379832  2982 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:19.394760  2982 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:19.361757  3403 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:19.399667  1803 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:19.400264  3398 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:19.402891  3389 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:19.403561  3389 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.404065  3389 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:19.370859  3376 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:19.372143  3313 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:60934 (request call id 437) took 2544 ms. Trace:
I20260709 12:22:19.376242  3348 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:19.373217  3402 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.391067  3386 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.397176  3392 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:19.353713  3409 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:19.376114  3406 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.412752  3405 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:19.423689  3348 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.450752  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.463380  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:19.452899  3348 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:19.455513  3265 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:19.453377  3406 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.456362  3392 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:19.878944  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:19.591899  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 96594.2us
I20260709 12:22:19.863474  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:19.468640  3344 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:19.890439  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:19.891458  3415 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:19.891538  1725 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:19.890722  3402 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:19.899907  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:19.900875  1879 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:19.813429  3370 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.729s	user 0.006s	sys 0.009s
I20260709 12:22:19.907269  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:19.907905  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:19.908908  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:19.911819  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:19.911990  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:19.909318  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:19.894099  3344 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:19.892112  3415 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.484704  3386 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:19.554610  3393 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:19.499972  3398 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:19.904110  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:19.786382  3409 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:19.928620  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:19.932717  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:19.861925  3411 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:19.935086  1847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.000s	sys 0.021s
I20260709 12:22:19.937201  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:19.938123  2418 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:19.477099  3391 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:19.946192  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:19.948109  3392 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:19.925277  3344 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:19.968813  1847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.000s	sys 0.022s
I20260709 12:22:19.968483  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:19.969465  1880 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:22:19.913933  3402 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:19.411051  3313 rpcz_store.cc:276] 0709 12:22:16.809380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.809530 (+   150us) service_pool.cc:224] Handling call
0709 12:22:16.852279 (+ 42749us) ts_tablet_manager.cc:934] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
0709 12:22:17.736016 (+883737us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:18.832018 (+1096002us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:18.832189 (+   171us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:19.300119 (+467930us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:19.336477 (+ 36358us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:19.347463 (+ 10986us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:19.353811 (+  6348us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":4288,"tablet-copy.queue_time_us":41460,"thread_start_us":191568,"threads_started":2}
W20260709 12:22:19.943912  3389 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.539s	user 0.008s	sys 0.007s
I20260709 12:22:19.974705  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:19.992177  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84094.4us
W20260709 12:22:19.961444  3399 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.811s	user 0.005s	sys 0.009s
I20260709 12:22:19.992583  1879 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:19.924579  3370 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:19.975313  3334 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:19.936292  3398 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:19.974136  3419 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:20.028995  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:20.031307  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.020s	sys 0.042s
W20260709 12:22:20.041008  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.020s	sys 0.043s
W20260709 12:22:20.041616  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 835426us
I20260709 12:22:20.043817  1876 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 2: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:20.050562  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:19.988786  3402 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:20.060732  3422 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:20.024652  3403 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:20.015038  3389 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:20.061380  3427 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:20.041505  3313 tablet_copy_client.cc:806] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
W20260709 12:22:20.066430  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45163.1us
I20260709 12:22:20.040373  3334 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:20.025686  3417 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:20.055627  3369 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.702s	user 0.014s	sys 0.000s
I20260709 12:22:19.991386  3411 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:20.018953  3399 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:20.088384  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 92679us
I20260709 12:22:20.062069  3404 raft_consensus.cc:2764] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:22:20.095173  3415 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:20.050052  3419 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:20.167857  3344 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:20.167346  3417 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:20.074024  3393 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:20.253829  1728 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:20.255016  3370 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:20.257184  3402 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:20.259374  3406 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:20.261610  3373 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.855s	user 0.007s	sys 0.005s
I20260709 12:22:20.263552  3393 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:20.162549  3337 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.792s	user 0.007s	sys 0.002s
I20260709 12:22:20.081853  3334 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:20.238864  3348 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.680s	user 0.011s	sys 0.001s
I20260709 12:22:20.086836  3424 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:20.075194  3403 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:20.262240  3373 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:20.098476  3422 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:20.271756  3313 tablet_copy_client.cc:670] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:22:20.268357  3391 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:20.266733  3393 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:20.099390  3369 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:20.385809  3420 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:20.702080  3334 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:20.877080  3419 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:20.719501  3348 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:20.776175  3337 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:20.957837  3424 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:21.018728  3342 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.638s	user 0.011s	sys 0.003s
I20260709 12:22:21.556584  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:21.556836  3342 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.102128  3389 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:22:21.103163  3417 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:21.106159  3377 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.771s	user 0.009s	sys 0.004s
I20260709 12:22:21.566361  3377 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.569105  3377 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.108161  3403 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.178285  3369 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.571332  3369 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:21.571844  3369 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:21.572249  3369 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:21.196985  3348 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.197672  3373 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.015357  3423 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.576426  3423 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.039218  3344 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.204412  3398 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.020960  3420 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.207103  3399 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.213956  3424 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.338032  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.349638  3432 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.581344  3420 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.581512  1723 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:21.358736  1742 tablet_copy_source_session.cc:215] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Tablet Copy: opened 0 blocks and 1 log segments
W20260709 12:22:21.582129  1723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2060) took 1681 ms (client timeout 1729 ms). Trace:
I20260709 12:22:21.581724  3432 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:21.582564  1723 rpcz_store.cc:269] 0709 12:22:19.901083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.337481 (+1436398us) service_pool.cc:224] Handling call
0709 12:22:21.582030 (+244549us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.583288  1742 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:48886 (request call id 166) took 3942 ms. Trace:
I20260709 12:22:21.583676  1742 rpcz_store.cc:276] 0709 12:22:17.647035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.647387 (+   352us) service_pool.cc:224] Handling call
0709 12:22:21.583172 (+3935785us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.348160  3426 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.584434  3432 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.038269  3422 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.097469  3411 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:21.097801  3427 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.022s	user 0.009s	sys 0.009s
I20260709 12:22:21.365458  3434 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.028283  3419 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.586895  3434 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.364353  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2063) took 1462 ms. Trace:
I20260709 12:22:21.588097  1728 rpcz_store.cc:276] 0709 12:22:19.902022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.909042 (+  7020us) service_pool.cc:224] Handling call
0709 12:22:21.363962 (+1454920us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b54000d16d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:21.364193 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":717484928}
I20260709 12:22:21.393718  3256 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.437008  3437 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.404043  3435 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.591005  3437 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.591627  3435 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.592993  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2045) took 2247 ms. Trace:
I20260709 12:22:21.593713  3437 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.594383  3423 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.484028  3400 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.595320  3400 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.596717  3342 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.490347  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:21.598658  2412 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:21.468379  2411 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:21.600337  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 333) took 4098 ms. Trace:
I20260709 12:22:21.485579  3367 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.538048  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:21.602253  2415 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:21.603152  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:21.605105  3423 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.552302  3340 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.605762  3423 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:21.586486  3427 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.510s	user 0.009s	sys 0.009s
I20260709 12:22:21.606292  3432 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.550348  2982 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.605926  3437 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.553081  3421 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:21.599969  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:21.411132  1799 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:21.600924  2411 rpcz_store.cc:276] 0709 12:22:17.502056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.502339 (+   283us) service_pool.cc:224] Handling call
0709 12:22:19.816228 (+2313889us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b5800003f98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:19.816527 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.599409 (+1782882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.600021 (+   612us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b54005609d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:21.600222 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1078659328}
I20260709 12:22:21.593411  1730 rpcz_store.cc:276] 0709 12:22:19.345775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.508625 (+2162850us) service_pool.cc:224] Handling call
0709 12:22:21.509054 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.589380 (+ 80326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.589417 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:21.589461 (+    44us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:21.589495 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.592687 (+  3192us) spinlock_profiling.cc:243] Waited 80 ms on lock 0x7b5400436cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:21.592741 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.592925 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183961984}
I20260709 12:22:21.552521  3382 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:21.488127  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.396s	user 0.031s	sys 0.029s
I20260709 12:22:21.617141  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1289) took 2362 ms. Trace:
I20260709 12:22:21.597863  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:21.586978  3419 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:21.628055  1799 rpcz_store.cc:276] 0709 12:22:19.254376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.255097 (+   721us) service_pool.cc:224] Handling call
0709 12:22:21.410632 (+2155535us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:21.410906 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.616882 (+205976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.617084 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5564160}
I20260709 12:22:21.611588  2982 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:21.552387  1772 connection.cc:256] RPC call timeout handler was delayed by -1.3201205020000089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20260709 12:22:21.501492  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 161) took 2985 ms. Trace:
I20260709 12:22:21.489928  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:21.496356  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.610244  3437 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:21.553541  3431 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.623445  3403 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.438097  3414 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.470669  3380 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.624033  3424 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:21.677944  2982 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:21.656987  1697 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.703s (SENT) [suppressed 3 similar messages]
I20260709 12:22:21.597836  3400 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.448196  3438 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.172s
I20260709 12:22:21.616819  3432 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:21.608562  3427 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:21.482239  3390 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.508347  3375 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.719080  3375 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:21.474280  3391 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.413442  3186 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.719187  3439 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.572280  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:21.627956  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.536s	user 0.032s	sys 0.029s
W20260709 12:22:21.724151  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.63222e+06us
I20260709 12:22:21.659696  2412 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:21.637238  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.636579  1805 rpcz_store.cc:276] 0709 12:22:18.515834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.515945 (+   111us) service_pool.cc:224] Handling call
0709 12:22:18.516194 (+   249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:18.528283 (+ 12089us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:18.528318 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.528379 (+    61us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:18.528414 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.501122 (+2972708us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b54001676d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:21.501188 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.501381 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27358592}
I20260709 12:22:21.627671  3417 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.627442  3382 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.667376  3405 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.667618  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.618204  3434 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.677711  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.668771  3431 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.660179  3411 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.594441  3435 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.607137  3340 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.684162  3424 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.699738  3403 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.625628  3423 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.677353  2410 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
W20260709 12:22:21.720592  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 164119us
I20260709 12:22:21.669397  3380 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:21.722673  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46112.6us
I20260709 12:22:21.585106  3426 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.723630  3186 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.726220  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:21.737243  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.732242  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:21.722186  3420 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.743953  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.743798  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.744171  3419 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.747467  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.622442  3421 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:21.739754  1727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2062) took 1838 ms (client timeout 2085 ms). Trace:
I20260709 12:22:21.761082  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.761272  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.762290  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:21.733314  3391 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:21.764230  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 107288us
I20260709 12:22:21.765214  1725 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:21.766333  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.767261  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:21.767869  1798 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 2: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:21.770361  2982 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:21.740485  3405 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.772552  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:21.774235  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.781121  1732 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:21.781605  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:21.786515  2982 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:21.742712  3417 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.786976  2982 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:21.748265  3411 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.788894  2982 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:21.788210  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.769394  3422 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:21.726379  3439 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.787963  3186 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.764390  3344 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:21.791764  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:21.792635  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:21.731565  3427 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.797122  3405 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.771811  3420 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.792210  3422 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:21.784257  3377 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:21.793749  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:21.802834  1770 connection.cc:256] RPC call timeout handler was delayed by -1.5438828859996647s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
I20260709 12:22:21.803031  1723 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
W20260709 12:22:21.803383  1770 connection.cc:256] RPC call timeout handler was delayed by -1.1019828859998597s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
I20260709 12:22:21.772979  1802 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:21.785989  3397 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:21.805291  1724 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:21.793335  3421 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.807659  3437 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.808457  3391 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.809255  3403 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.811686  3424 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.800467  3344 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:21.822958  3411 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.803401  3373 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:21.821275  3382 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:21.764822  1727 rpcz_store.cc:269] 0709 12:22:19.901733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.495868 (+1594135us) service_pool.cc:224] Handling call
0709 12:22:21.738745 (+242877us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.824122  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.803782  3422 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:21.807693  3432 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.831571  1723 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:21.833997  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.835181  3426 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.829377  3373 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.836083  3373 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:21.815240  3397 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.858744  2982 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.861687  3431 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.862071  2982 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.828436  3344 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:21.863050  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.864498  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.840049  3417 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.827273  3340 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.862011  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.873625  3375 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:21.877043  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 1942) took 2796 ms (client timeout 2098 ms). Trace:
W20260709 12:22:21.878156  1877 rpcz_store.cc:269] 0709 12:22:19.073765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.073913 (+   148us) service_pool.cc:224] Handling call
0709 12:22:21.870033 (+2796120us) spinlock_profiling.cc:243] Waited 46.9 ms on lock 0x7b5400117e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:21.870171 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":107873024}
I20260709 12:22:21.880980  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.883397  3375 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.883704  3420 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.884083  1799 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:21.887758  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1369) took 1860 ms. Trace:
I20260709 12:22:21.888351  1877 rpcz_store.cc:276] 0709 12:22:20.027254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.886628 (+1859374us) service_pool.cc:224] Handling call
0709 12:22:21.887073 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.887310 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.887351 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:21.887409 (+    58us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:21.887454 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.887528 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.887681 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.884387  3380 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.874567  3344 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.841173  3391 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:21.856506  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:21.923388  1802 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:21.898800  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.924837  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.924965  3344 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.925540  1723 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:21.900833  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.902717  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.910863  3337 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:21.926538  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:21.903600  1771 connection.cc:256] RPC call timeout handler was delayed by -1.1344144049999159s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2098s for a response
W20260709 12:22:21.928334  1771 connection.cc:256] RPC call timeout handler was delayed by -1.0867144049998387s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21509999999999999s for a response
I20260709 12:22:21.928562  3439 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:21.905115  3382 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:21.929693  3382 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:21.931447  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:21.917721  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.896206  1877 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:21.927287  1726 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:21.938819  1730 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:21.938836  1805 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:21.939941  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.901741  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.941229  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:21.941790  1805 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:21.943076  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.944026  3369 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:21.945415  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 1947) took 2778 ms (client timeout 2151 ms). Trace:
I20260709 12:22:21.932047  3379 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.932439  3382 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:21.945909  1876 rpcz_store.cc:269] 0709 12:22:19.167407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.172476 (+  5069us) service_pool.cc:224] Handling call
0709 12:22:21.945188 (+2772712us) spinlock_profiling.cc:243] Waited 870 ms on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:21.945333 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2000123776}
I20260709 12:22:21.932973  3391 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.947806  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.947947  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:21.949081  3391 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.949272  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:21.949920  1876 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:21.936019  3422 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:21.896816  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 130805us
I20260709 12:22:21.961272  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1370) took 1907 ms. Trace:
W20260709 12:22:21.946153  3334 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.918s	user 0.013s	sys 0.004s
I20260709 12:22:21.946120  3369 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.950737  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:21.962314  1805 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
W20260709 12:22:21.180217  3328 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.992s	user 0.012s	sys 0.004s
I20260709 12:22:21.962502  1877 rpcz_store.cc:276] 0709 12:22:20.054280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.889669 (+1835389us) service_pool.cc:224] Handling call
0709 12:22:21.890061 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.960749 (+ 70688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.961177 (+   428us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.968814  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:21.952478  3365 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:21.970705  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:21.971345  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.039592  3373 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.044785  1877 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
W20260709 12:22:22.046322  1846 connection.cc:256] RPC call timeout handler was delayed by -1.8180373469999722s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
W20260709 12:22:22.047947  1846 connection.cc:256] RPC call timeout handler was delayed by -1.6669654929996796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1557s for a response
I20260709 12:22:22.048465  3388 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:22.050096  3388 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:22.033631  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:21.951582  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.051286  1880 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:22.052392  3388 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:22.046723  3373 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:22.053393  1802 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:22.026062  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.049511  3363 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:22.035193  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.053985  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1366) took 2068 ms. Trace:
I20260709 12:22:22.046247  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.056304  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:22.057067  1846 connection.cc:256] RPC call timeout handler was delayed by -1.5237373469999511s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20260709 12:22:22.068378  1846 connection.cc:256] RPC call timeout handler was delayed by -1.4976373469999089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1938s for a response
W20260709 12:22:22.069056  1846 connection.cc:256] RPC call timeout handler was delayed by -1.4814373469998827s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
W20260709 12:22:22.069684  1846 connection.cc:256] RPC call timeout handler was delayed by -1.3312654929995915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20260709 12:22:22.070077  1846 connection.cc:256] RPC call timeout handler was delayed by -1.2494396580000284s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260709 12:22:22.070443  1846 connection.cc:256] RPC call timeout handler was delayed by -1.0028396580000845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1774s for a response
I20260709 12:22:22.066282  1880 rpcz_store.cc:276] 0709 12:22:19.985717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.985886 (+   169us) service_pool.cc:224] Handling call
0709 12:22:19.986268 (+   382us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.051833 (+2065565us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.053547 (+  1714us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b5400052154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:22.053861 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4748452096}
I20260709 12:22:22.080549  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.082008  3441 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:22.082392  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.083493  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.074223  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:22:22.084991  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 116216us
I20260709 12:22:22.055528  3363 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:21.986423  3328 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.088424  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.092289  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.092922  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.094854  3422 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:22.065907  3397 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:22.088090  3379 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:21.965390  3334 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.134075  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1291) took 2832 ms. Trace:
I20260709 12:22:22.141076  1803 rpcz_store.cc:276] 0709 12:22:19.302085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.332637 (+ 30552us) service_pool.cc:224] Handling call
0709 12:22:19.349985 (+ 17348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:19.400331 (+ 50346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.133626 (+2733295us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b540026f054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:22.133973 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97535872}
I20260709 12:22:22.141881  3381 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:22.142319  3381 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:22.142706  1804 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:22.143749  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69694.4us
I20260709 12:22:22.144196  3328 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:22.144338  3363 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:22.144891  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1286) took 2892 ms. Trace:
I20260709 12:22:22.105525  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.145329  1804 rpcz_store.cc:276] 0709 12:22:19.252953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.265871 (+ 12918us) service_pool.cc:224] Handling call
0709 12:22:19.316668 (+ 50797us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.143451 (+2826783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.144654 (+  1203us) spinlock_profiling.cc:243] Waited 2.83 s on lock 0x7b54001eb4d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:22.144838 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6498730752}
I20260709 12:22:22.146059  2410 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 2: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:22.146167  3379 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:22.096199  1880 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:22.142834  3381 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.146629  3379 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.158516  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:22.096910  1844 connection.cc:256] RPC call timeout handler was delayed by -1.735232447999806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18560000000000001s for a response
W20260709 12:22:22.159644  1844 connection.cc:256] RPC call timeout handler was delayed by -1.6875324479997289s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19090000000000001s for a response
W20260709 12:22:22.159958  1844 connection.cc:256] RPC call timeout handler was delayed by -1.677345604999573s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
W20260709 12:22:22.160230  1844 connection.cc:256] RPC call timeout handler was delayed by -1.6452324479996605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
W20260709 12:22:22.160514  1844 connection.cc:256] RPC call timeout handler was delayed by -1.5731281399998807s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1647s for a response
W20260709 12:22:22.160822  1844 connection.cc:256] RPC call timeout handler was delayed by -1.5327324479999334s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2081s for a response
W20260709 12:22:22.161092  1844 connection.cc:256] RPC call timeout handler was delayed by -1.500045604999741s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
W20260709 12:22:22.161371  1844 connection.cc:256] RPC call timeout handler was delayed by -1.4667456049996872s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1816s for a response
W20260709 12:22:22.161630  1844 connection.cc:256] RPC call timeout handler was delayed by -1.460445604999677s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18230000000000002s for a response
W20260709 12:22:22.161908  1844 connection.cc:256] RPC call timeout handler was delayed by -1.3641456049995213s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
I20260709 12:22:22.161806  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.137825  3324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d891ed30ca204d09b7b083088037b63d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:22:22.163372  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.163543  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.155984  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 1935) took 3154 ms. Trace:
I20260709 12:22:22.147287  1800 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.165031  1881 rpcz_store.cc:276] 0709 12:22:19.001980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.002120 (+   140us) service_pool.cc:224] Handling call
0709 12:22:19.002395 (+   275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:19.071445 (+ 69050us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.071493 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:19.071551 (+    58us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:19.071595 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.155646 (+3084051us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b5400118354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:22.155721 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.155903 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":158195072}
I20260709 12:22:22.115149  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.107095  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.151557  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 1959) took 2695 ms. Trace:
I20260709 12:22:22.162151  3334 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:22.169391  1873 rpcz_store.cc:276] 0709 12:22:19.456579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.464283 (+  7704us) service_pool.cc:224] Handling call
0709 12:22:22.122041 (+2657758us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:22.122313 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.122531 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:22.122572 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.122632 (+    60us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:22.122686 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.122756 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.151465 (+ 28709us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60218240}
I20260709 12:22:22.170856  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.102351  3441 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:22.174911  3441 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:22.162760  1875 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.141288  2419 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:22.192641  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1287) took 2939 ms. Trace:
W20260709 12:22:22.096596  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 132899us
W20260709 12:22:22.193531  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.121s	user 0.002s	sys 0.015s
I20260709 12:22:22.183615  3381 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:22.194308  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.128s	user 0.003s	sys 0.016s
I20260709 12:22:22.195583  1730 raft_consensus.cc:2955] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:22:22.198035  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2059) took 2297 ms (client timeout 1594 ms). Trace:
I20260709 12:22:22.198956  3422 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:22.199623  3422 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:22.202085  3422 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:22.203500  1799 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:22.214934  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.212455  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:22.198624  1731 rpcz_store.cc:269] 0709 12:22:19.900777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.197649 (+2296872us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:22.197785 (+   136us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:22.214355  1802 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.227207  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.147862  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:22.203977  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.229274  1804 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:22.219645  1803 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:22.230144  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.193338  1801 rpcz_store.cc:276] 0709 12:22:19.253605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.259766 (+  6161us) service_pool.cc:224] Handling call
0709 12:22:22.191267 (+2931501us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:22.191540 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.191779 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:22.191820 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.192098 (+   278us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:22.192144 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.192218 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.192566 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5228416}
I20260709 12:22:22.232925  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.233819  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.234647  1873 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:22.235229  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.235567  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.236194  1873 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:22.237257  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.237836  1873 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:22.238857  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.239480  1873 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.239179  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 1956) took 2787 ms. Trace:
I20260709 12:22:22.239993  1872 rpcz_store.cc:276] 0709 12:22:19.451328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.455811 (+  4483us) service_pool.cc:224] Handling call
0709 12:22:22.179117 (+2723306us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:22.238242 (+ 59125us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3e9c 00007f16a8cb3a5b 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:22.238450 (+   208us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.238668 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:22.238698 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.238871 (+   173us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:22.238922 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.238995 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.239115 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":136443904}
I20260709 12:22:22.241765  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.202736  3422 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:22.242645  1872 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:22.227254  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.243264  1805 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
W20260709 12:22:22.245095  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1292) took 2921 ms (client timeout 2133 ms). Trace:
I20260709 12:22:22.245596  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:22.245543  1807 rpcz_store.cc:269] 0709 12:22:19.323548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.244861 (+2921313us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:22.244999 (+   138us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:22.246430  3422 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:22.247475  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.248756  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.249354  1872 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:22.248704  3422 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:22.255728  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:22.256536  2419 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 2: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:22.257031  3379 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.257607  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.250485  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.261404  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.263291  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.265633  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.266641  3379 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:22.271124  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.271533  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:22.271916  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:22.272652  2412 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.242470  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.277073  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 1961) took 2805 ms. Trace:
I20260709 12:22:22.277436  1873 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:22.278498  3385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet def0036446324543ba07e132e885133a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:22:22.277488  1878 rpcz_store.cc:276] 0709 12:22:19.472022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.484110 (+ 12088us) service_pool.cc:224] Handling call
0709 12:22:22.276342 (+2792232us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:22.276572 (+   230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.276759 (+   187us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:22.276786 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.276830 (+    44us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:22.276864 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.276919 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.277020 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55604992}
I20260709 12:22:22.278898  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.279431  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.280318  1878 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:22.280655  1753 debug-util.cc:398] Leaking SignalData structure 0x7b0800135ce0 after lost signal to thread 564
I20260709 12:22:22.281687  3436 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:22.282878  3186 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:22.283311  3186 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:22.283567  3436 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:22.284202  3186 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.284790  3433 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:22.285243  3433 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:22.285600  3186 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.286095  3433 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.287467  1626 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 252) took 4802 ms. Trace:
I20260709 12:22:22.287549  3433 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:22.281976  1753 debug-util.cc:398] Leaking SignalData structure 0x7b08001893e0 after lost signal to thread 1664
I20260709 12:22:22.277050  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.293052  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.291116  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.295310  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.287955  1626 rpcz_store.cc:276] 0709 12:22:17.485211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.485377 (+   166us) service_pool.cc:224] Handling call
0709 12:22:22.287350 (+4801973us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":942329,"reported_tablets":1}
I20260709 12:22:22.291409  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.293164  1876 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.298046  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.192749  1875 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:22.298957  1805 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:22.299110  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.299919  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.300623  1732 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:22.300964  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.302138  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:22.302433  1753 debug-util.cc:398] Leaking SignalData structure 0x7b08001d7c40 after lost signal to thread 1756
W20260709 12:22:22.302765  1753 debug-util.cc:398] Leaking SignalData structure 0x7b0800353ba0 after lost signal to thread 1757
I20260709 12:22:22.303973  3402 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:22.304617  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.305855  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.296926  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.299932  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.308676  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.308843  1805 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:22.309407  1877 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:22.302685  1806 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 1.
I20260709 12:22:22.311049  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.305171  3438 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:22:22.312966  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 154544us
I20260709 12:22:22.311124  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:22.162508  1844 connection.cc:256] RPC call timeout handler was delayed by -1.2957456049994107s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2006s for a response
W20260709 12:22:22.314826  1844 connection.cc:256] RPC call timeout handler was delayed by -1.2155580680000639s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1774s for a response
W20260709 12:22:22.315228  1844 connection.cc:256] RPC call timeout handler was delayed by -1.1382456049996108s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
W20260709 12:22:22.315526  1844 connection.cc:256] RPC call timeout handler was delayed by -1.1328456049996021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
W20260709 12:22:22.315802  1844 connection.cc:256] RPC call timeout handler was delayed by -1.1146468679999089s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21230000000000004s for a response
W20260709 12:22:22.316087  1844 connection.cc:256] RPC call timeout handler was delayed by -1.0571580679998078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.195s for a response
I20260709 12:22:22.319304  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 1957) took 2864 ms. Trace:
I20260709 12:22:22.282003  1802 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:22.323697  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:22.324525  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 176718us
I20260709 12:22:22.314531  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:22.321471  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:22.325268  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:22.319819  1874 rpcz_store.cc:276] 0709 12:22:19.454913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.455675 (+   762us) service_pool.cc:224] Handling call
0709 12:22:22.318427 (+2862752us) spinlock_profiling.cc:243] Waited 26.7 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:22.318688 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.318916 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:22.318947 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.319001 (+    54us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:22.319035 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.319103 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.319232 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61301504}
I20260709 12:22:22.327138  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.328004  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1290) took 3068 ms. Trace:
I20260709 12:22:22.328929  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.329478  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:22.332333  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.332890  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.333033  1725 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:22.334107  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.335821  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.337483  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:22.338658  1798 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 2: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:22.340386  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.342382  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.342823  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.343127  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.343853  1725 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:22.334251  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.348462  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.349303  1879 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.351522  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1368) took 2334 ms. Trace:
I20260709 12:22:22.351717  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.352263  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.352989  1880 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 2: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:22.310410  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.313297  3438 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:22.353709  1879 rpcz_store.cc:276] 0709 12:22:20.016903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.096906 (+ 80003us) service_pool.cc:224] Handling call
0709 12:22:22.348901 (+2251995us) spinlock_profiling.cc:243] Waited 34.6 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:22.349096 (+   195us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.351243 (+  2147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.351459 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79551872}
I20260709 12:22:22.356539  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:22.353165  3108 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:22.364698  3381 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:22.364876  3425 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:22.365413  3425 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:22.348019  3280 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:22.367079  3280 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:22.367152  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.343875  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.368580  3429 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:22.368685  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.368934  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.369555  1727 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:22.370195  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.313265  3402 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:22:22.371565  3402 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:22.372102  3397 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:22.367519  3280 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:22.376415  1807 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.377660  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.378314  1881 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:22.357856  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.379087  1723 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:22.380568  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:22.375102  3429 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:22:22.373248  1753 debug-util.cc:398] Leaking SignalData structure 0x7b08002aa060 after lost signal to thread 1831
I20260709 12:22:22.383342  3429 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.323503  3322 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:22.349994  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.371816  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:22.384289  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:22.344400  3424 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:22.389142  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.389681  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.390328  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.385370  1876 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:22.392840  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.345860  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.397131  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:22.388360  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 73943.7us
I20260709 12:22:22.386143  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.392824  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:22.393638  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.372539  3397 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:22.383540  1753 debug-util.cc:398] Leaking SignalData structure 0x7b0800335ec0 after lost signal to thread 1832
I20260709 12:22:22.404656  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:22.407624  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.408442  1873 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:22.392381  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.408644  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.409698  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.410637  1873 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:22.397743  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.367523  3425 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:22.413934  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.382630  2412 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:22.416379  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.386584  3402 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:22.417050  1879 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:22.418107  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.417579  1770 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:22.420712  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.386982  3322 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:22.421541  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.405615  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.396425  3424 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:22.396600  3350 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:22.403467  3328 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:22.427083  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1371) took 2363 ms. Trace:
I20260709 12:22:22.406421  3321 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:22.396799  3108 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:22.428539  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:22.537432  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78878us
W20260709 12:22:22.470640  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53130.9us
I20260709 12:22:22.538966  1881 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.539144  1799 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.459286  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.459637  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:22.519131  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49023.3us
I20260709 12:22:22.468852  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.403965  3407 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:22.383628  3438 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.542363  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.403209  1802 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:22.473253  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.548477  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.471838  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:22.548923  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.514072  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.328434  1800 rpcz_store.cc:276] 0709 12:22:19.259424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.266391 (+  6967us) service_pool.cc:224] Handling call
0709 12:22:19.334610 (+ 68219us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.165684 (+2831074us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.327719 (+162035us) spinlock_profiling.cc:243] Waited 2.81 s on lock 0x7b5400132754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:22.327937 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6467756288}
I20260709 12:22:22.578800  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.540220  1876 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:22.467197  3322 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:22.472594  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.543040  1805 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:22.419118  3420 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:22.579139  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.543835  1879 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.582536  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1372) took 2493 ms. Trace:
I20260709 12:22:22.582962  1806 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:22.583055  1879 rpcz_store.cc:276] 0709 12:22:20.089721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.471099 (+2381378us) service_pool.cc:224] Handling call
0709 12:22:22.543684 (+ 72585us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.582243 (+ 38559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.582467 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:22.583918  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.584502  3436 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:22.585067  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.586047  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.579588  3397 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:22.587348  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.588125  3397 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:22.588805  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.589082  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.426164  3425 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:22.469750  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.579218  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.464435  3402 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:22.541651  1873 rpcz_store.cc:276] 0709 12:22:20.064199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.426110 (+2361911us) service_pool.cc:224] Handling call
0709 12:22:22.426480 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.426684 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:22.426711 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.426757 (+    46us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:22.426810 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.426867 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.427025 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:22.586711  1806 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:22.592252  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.614423  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:22.615490  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.425344  3398 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.164s
I20260709 12:22:22.597002  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.606866  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.616945  1798 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:22.537190  3321 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:22.578528  3381 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:22.609741  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.620443  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.625949  1806 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.622090  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.609694  3357 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:22.540181  1877 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:22.548872  1753 debug-util.cc:398] Leaking SignalData structure 0x7b0800380640 after lost signal to thread 1905
I20260709 12:22:22.513608  3350 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:22.467187  3424 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:22.516829  3373 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:22.536614  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.699048  3373 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:22.699618  3373 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.699729  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.700183  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.701706  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.702952  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.704095  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.616940  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:22.608217  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 136495us
I20260709 12:22:22.618137  1876 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.708276  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.515493  3328 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:22:22.619649  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 197048us
I20260709 12:22:22.709228  3328 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.709611  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:22.550887  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.542411  3407 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:22.711349  3407 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:22.613854  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.424541  3379 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.125s
I20260709 12:22:22.717546  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:22.598915  3420 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:22.643287  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.718559  3420 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.718933  1803 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:22.643362  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.719882  1731 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:22.614112  3324 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:60950 (request call id 332) took 5543 ms. Trace:
I20260709 12:22:22.624291  3321 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:22.729511  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.730875  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:22.731191  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:22.676191  1804 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:22:22.666673  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.678938  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.679904  3108 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:22.752214  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.753021  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:22.681032  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet ea5d8f652fc448d6af846ff6850004b8 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:22:22.681315  2382 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:22.756000  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74750.8us
I20260709 12:22:22.675622  3350 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.681623  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.623674  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.691749  1806 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.696945  1799 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.727025  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:22.771324  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:22.663383  3433 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:22.772652  3358 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:22.778699  3433 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:22.779119  3358 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:22.779280  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet ea5d8f652fc448d6af846ff6850004b8 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-ea5d8f652fc448d6af846ff6850004b8
I20260709 12:22:22.779899  3433 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.781426  3433 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:22.667052  3438 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.783293  1872 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:22.763643  1875 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:22.735918  3398 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:22.743530  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.749536  1628 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 249) took 4095 ms. Trace:
I20260709 12:22:22.753703  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:22.787794  1628 rpcz_store.cc:276] 0709 12:22:18.640566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.640804 (+   238us) service_pool.cc:224] Handling call
0709 12:22:22.735526 (+4094722us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3636409,"reported_tablets":3}
I20260709 12:22:22.779611  3358 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:22.790563  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:22.772815  1726 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:22.766310  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:22.792034  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:22.793380  1803 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:22.787992  3429 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:22.796767  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:22.818615  1804 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:22.792443  3401 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:22.821658  3401 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:22.822100  3401 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:22.825098  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:22.853513  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 56828.1us
W20260709 12:22:22.799062  1753 debug-util.cc:398] Leaking SignalData structure 0x7b080031ef40 after lost signal to thread 2443
I20260709 12:22:22.741820  3186 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260709 12:22:22.774398  3366 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:22.858747  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:22.796002  3351 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:22.805907  3339 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:22.805689  3355 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:22.880712  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:22.729564  3407 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.892690  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:22.853807  3369 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:22.741281  3402 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:22.870275  3351 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:22.923704  3351 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:22.933980  1725 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:22.725700  3324 rpcz_store.cc:276] 0709 12:22:17.046112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.046267 (+   155us) service_pool.cc:224] Handling call
0709 12:22:17.322706 (+276439us) ts_tablet_manager.cc:934] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 075704438992440697732181e49ccd09 (127.0.140.65:35067)
0709 12:22:22.136068 (+4813362us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:22.172187 (+ 36119us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:22.172368 (+   181us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:22.387804 (+215436us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:22.420808 (+ 33004us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:22.548422 (+127614us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:22.589563 (+ 41141us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":44535,"tablet-copy.queue_time_us":210605,"thread_start_us":240854,"threads_started":2}
I20260709 12:22:22.939828  3324 tablet_copy_client.cc:806] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:22:22.940609  3324 tablet_copy_client.cc:670] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:22:22.903941  3369 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:22.878237  1753 debug-util.cc:398] Leaking SignalData structure 0x7b08002c1320 after lost signal to thread 3266
I20260709 12:22:22.827946  3385 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:60928 (request call id 517) took 4119 ms. Trace:
I20260709 12:22:22.899708  1730 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.865814  3392 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:22:22.967088  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 240104us
I20260709 12:22:22.980315  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:22.980949  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 269857us
W20260709 12:22:22.958689  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 166731us
I20260709 12:22:22.961946  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:22.969515  1724 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:22.983800  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:22.984462  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 335070us
I20260709 12:22:22.984783  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:22.975808  3369 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:22.987805  3407 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:22.989061  3363 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:22.989555  3363 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:22.989939  3363 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:22.986250  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:22.995659  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:22.998374  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:22.983536  1753 debug-util.cc:398] Leaking SignalData structure 0x7b080030bfa0 after lost signal to thread 3315
I20260709 12:22:22.980710  2410 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:22.981528  3402 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:23.011977  3322 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:23.014441  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.017112  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.018757  1844 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:22.981619  3392 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:23.027694  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:23.028245  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:22.993043  2413 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:22.994747  3379 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.045437  3357 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:23.045945  3357 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:23.046041  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:23.047200  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:23.039042  1881 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:23.047040  3357 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.007355  3364 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.038589  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:23.052538  3457 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.059156  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:23.055418  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:22:23.022382  1753 debug-util.cc:398] Leaking SignalData structure 0x7b0800319de0 after lost signal to thread 3329
I20260709 12:22:22.999123  3451 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:22:23.051929  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 183776us
I20260709 12:22:23.085273  3429 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:23.087276  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:23.088009  1876 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:23.084578  3455 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:22:23.095860  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54394.9us
I20260709 12:22:23.103998  1876 raft_consensus.cc:606] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:23.104557  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.039099  3439 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:23.030821  3392 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:23.042768  3354 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.088565  3339 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.118850  3322 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:23.060685  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:23.030436  3352 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.100522  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:22:23.101022  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62556.4us
I20260709 12:22:23.106909  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:23.122413  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 61933.6us
I20260709 12:22:23.123780  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:23.006524  3355 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
W20260709 12:22:23.126446  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 71119.8us
I20260709 12:22:23.059566  3449 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:23.079651  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:23.125499  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:23.109656  1727 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.740s	user 0.011s	sys 0.007s
W20260709 12:22:23.130344  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 168447us
I20260709 12:22:23.196106  1880 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:23.112900  3368 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:23.132992  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49744us
I20260709 12:22:23.098278  3451 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:23.131574  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:23.195473  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:23.151144  1731 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:22:23.198968  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 117498us
I20260709 12:22:23.117336  3407 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:23.212042  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1384) took 1276 ms. Trace:
I20260709 12:22:23.122852  3357 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:23.125725  3280 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.743s	user 0.007s	sys 0.011s
W20260709 12:22:23.215708  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 92008.9us
I20260709 12:22:23.273249  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:22:23.216234  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 90883.2us
I20260709 12:22:23.112301  3436 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:23.268375  1880 rpcz_store.cc:276] 0709 12:22:21.935970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.583563 (+647593us) service_pool.cc:224] Handling call
0709 12:22:22.583843 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.197589 (+613746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.211710 (+ 14121us) spinlock_profiling.cc:243] Waited 612 ms on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:23.211969 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1407544192}
I20260709 12:22:23.274889  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:23.107746  3420 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:23.279382  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 147941us
I20260709 12:22:23.282464  3407 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:23.288416  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:23.284368  3358 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.128464  3387 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:23.200356  3429 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.201363  3108 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.212924  3451 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:23.318025  1729 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:23.319463  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 219075us
I20260709 12:22:23.319804  3339 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:23.320349  3339 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:23.320902  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.321998  3186 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.321408  3339 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.325760  3452 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:23.327368  3186 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:23.327837  3186 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:23.328910  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:23.332645  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:23.334641  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:23.341241  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.979396  3385 rpcz_store.cc:276] 0709 12:22:18.708392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.708554 (+   162us) service_pool.cc:224] Handling call
0709 12:22:18.841059 (+132505us) ts_tablet_manager.cc:934] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
0709 12:22:22.277133 (+3436074us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:22.438223 (+161090us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:22.438445 (+   222us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:22.791695 (+353250us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:22.792093 (+   398us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:22.806216 (+ 14123us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:22.816176 (+  9960us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":2762,"tablet-copy.queue_time_us":78589,"thread_start_us":110637,"threads_started":2}
I20260709 12:22:23.154146  3383 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:23.195854  3439 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.151501  3457 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.202001  1727 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:23.264858  3322 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.279006  3324 tablet_copy_client.cc:538] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:22:23.298120  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:23.277071  3436 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:23.289744  3450 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.300529  3407 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:23.297545  3352 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.293395  3379 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:23.352733  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 61108.4us
I20260709 12:22:23.293774  3461 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:23.354486  1732 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:23.293484  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:23.318571  3373 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.618s	user 0.013s	sys 0.010s
W20260709 12:22:23.372730  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 173921us
I20260709 12:22:23.391829  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.359732  3385 tablet_copy_client.cc:806] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
W20260709 12:22:23.376431  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 78447.7us
I20260709 12:22:23.345610  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:23.378888  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86986.9us
I20260709 12:22:23.351928  3422 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:22:23.357426  3328 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.648s	user 0.008s	sys 0.008s
I20260709 12:22:23.402493  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:23.403551  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:23.385962  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.405335  1730 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:23.387104  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.289407  3280 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.354642  3339 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:23.351934  3383 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:23.361863  3321 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.628s	user 0.007s	sys 0.013s
W20260709 12:22:23.390503  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.374090  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2095) took 1005 ms. Trace:
I20260709 12:22:23.402822  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.348266  3108 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:23.392256  3457 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:22:23.426666  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 274144us
I20260709 12:22:23.328927  3186 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.434376  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:23.437755  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1192) took 1102 ms. Trace:
I20260709 12:22:23.438270  1732 rpcz_store.cc:276] 0709 12:22:22.335246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.619524 (+284278us) service_pool.cc:224] Handling call
0709 12:22:23.354325 (+734801us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.389671 (+ 35346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.437545 (+ 47874us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b54000d16d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:23.437693 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1587907328}
I20260709 12:22:23.396494  3373 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:23.462958  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 356083us
I20260709 12:22:23.463888  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.408411  3383 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:23.412537  3385 tablet_copy_client.cc:670] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:22:23.424414  3422 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:23.408054  3458 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.473847  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.431289  3425 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:23.485008  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:23.496343  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.423818  3328 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.461808  3466 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:23.423847  1727 rpcz_store.cc:276] 0709 12:22:22.356867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.368404 (+ 11537us) service_pool.cc:224] Handling call
0709 12:22:23.361999 (+993595us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.470652  3358 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:23.516335  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:23.473225  3401 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.651s	user 0.009s	sys 0.016s
W20260709 12:22:23.520629  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 60158.3us
I20260709 12:22:23.424376  3321 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.508247  3472 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:23.394635  3461 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:23.528599  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 126270us
I20260709 12:22:23.472573  3334 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:23.530081  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.555850  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:23.532011  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 82164.9us
I20260709 12:22:23.557111  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:23.547938  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.551486  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:23.494243  3186 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:23.554737  1807 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:23.512121  3439 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.511413  3373 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:23.523272  3112 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.601631  1799 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:22:23.603170  1726 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:23.603050  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:23.565698  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.600289  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.516760  3448 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.606416  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.553683  3429 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:23.553597  3420 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:23.449046  3436 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:23.610386  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.614765  1730 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:23.553825  3401 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.601500  3470 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.562816  3379 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.627425  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.618544  3472 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.617040  3429 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:23.620450  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:23.652837  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 440803us
I20260709 12:22:23.599135  3441 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.054s
I20260709 12:22:23.622826  3334 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:23.598922  3454 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260709 12:22:23.661405  2414 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:23.606675  3452 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:22:23.634217  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:23.671655  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.676394  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.672407  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.673703  1798 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:23.665334  3438 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.882s	user 0.006s	sys 0.006s
I20260709 12:22:23.654049  3436 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:23.658083  3429 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:23.723297  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1326) took 1329 ms. Trace:
I20260709 12:22:23.729281  1807 rpcz_store.cc:276] 0709 12:22:22.344295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.514759 (+170464us) service_pool.cc:224] Handling call
0709 12:22:23.673272 (+1158513us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b54001e4454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:23.673428 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2236019456}
I20260709 12:22:23.725793  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.621724  3422 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:23.735481  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.735903  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:23.736653  1724 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:23.726822  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72961.7us
I20260709 12:22:23.738965  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.680267  3458 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:23.634008  3476 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
W20260709 12:22:23.752775  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.731739  3472 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:23.731941  3108 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:23.732460  3280 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:23.744643  3366 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:23.756655  3472 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:23.780988  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 337857us
I20260709 12:22:23.781837  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.765161  3328 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:23.766268  3334 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:23.774708  1890 tablet_copy_source_session.cc:215] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:22:23.786240  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:23.787774  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.789145  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:23.791551  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.794612  1890 rpcz_store.cc:275] Call kudu.tserver.TabletCopyService.BeginTabletCopySession from 127.0.0.1:58100 (request call id 156) took 1159 ms. Trace:
I20260709 12:22:23.795065  1890 rpcz_store.cc:276] 0709 12:22:22.635608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.643867 (+  8259us) service_pool.cc:224] Handling call
0709 12:22:23.794509 (+1150642us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:23.795982  3464 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:23.784598  3321 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:23.880409  1728 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:23.760141  3458 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:23.799281  3466 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:22:23.894937  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 293173us
I20260709 12:22:23.899999  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:23.896687  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 105248us
I20260709 12:22:23.900885  3321 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:23.904323  3369 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.916s	user 0.017s	sys 0.017s
I20260709 12:22:23.880843  3464 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:23.904994  3476 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:23.886075  3328 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:23.896265  3351 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.964s	user 0.016s	sys 0.002s
I20260709 12:22:23.910077  3429 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:23.938041  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:23.939725  3429 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:23.938091  3351 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.943039  3351 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:23.944808  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:23.946105  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:23.908834  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.948959  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:23.928537  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.949360  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:23.927158  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:23.915570  3369 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:23.934056  3476 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:23.937486  3455 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:23.947045  3354 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:23.881409  3480 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:22:23.962184  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 218279us
I20260709 12:22:23.975050  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:23.979081  3451 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.624s	user 0.007s	sys 0.006s
I20260709 12:22:23.969852  3429 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:23.980219  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.922928  3461 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.008953  1724 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:23.987336  3452 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.015075  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.015327  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:23.984460  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:23.900048  3466 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:23.997624  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 479266us
I20260709 12:22:23.952020  3469 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.004863  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:23.955744  3464 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:24.002863  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:23.980566  3407 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.583s	user 0.009s	sys 0.003s
I20260709 12:22:23.991523  3455 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:23.987021  3425 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:23.992566  3351 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.017942  1729 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
W20260709 12:22:24.014806  3433 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.233s	user 0.012s	sys 0.014s
I20260709 12:22:24.010641  3321 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:23.892297  3472 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:24.036939  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.018581  3398 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:22:24.037559  1876 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:24.037863  3481 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.022621  3439 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:24.038574  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.038913  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:24.039050  3449 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.774s	user 0.013s	sys 0.000s
I20260709 12:22:24.040074  1875 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:24.040288  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.036966  3410 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:24.042080  3350 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.243s	user 0.015s	sys 0.000s
W20260709 12:22:24.079483  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 95149.2us
W20260709 12:22:24.042908  3397 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.454s	user 0.010s	sys 0.019s
I20260709 12:22:24.043972  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.046707  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.048852  3458 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:24.023514  3392 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.904s	user 0.012s	sys 0.000s
I20260709 12:22:24.043731  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:24.094007  3392 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.027822  3429 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:22:24.096230  3429 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.096907  3429 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.097997  3429 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:24.096916  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53040.5us
I20260709 12:22:24.103232  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:24.053462  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.055465  1806 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:23.989876  3438 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260709 12:22:24.014142  3451 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.064296  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.067546  3478 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:22:24.123340  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.068349  3369 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:24.062368  3321 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:24.092128  3424 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.361s	user 0.008s	sys 0.008s
I20260709 12:22:24.065850  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.076323  3449 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.142108  3454 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.138293  3321 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.077310  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:24.042316  3401 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:24.081327  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.079113  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.061059  3484 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 12:22:24.093304  3486 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.150110  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:24.150947  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69704.1us
I20260709 12:22:24.151748  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.152452  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:24.153275  1730 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
W20260709 12:22:24.139909  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74120.9us
I20260709 12:22:24.154141  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:24.029454  3407 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.162442  3324 tablet_bootstrap.cc:492] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:22:24.162886  3407 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.100692  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.107367  3482 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.111693  3464 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:24.116923  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52729.7us
I20260709 12:22:24.185464  3481 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.186182  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.187654  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.117772  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.112900  3394 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:24.120236  3313 tablet_copy_client.cc:538] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:22:24.142675  3454 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.120613  3461 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.163364  3328 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:24.114076  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.122097  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:22:24.202385  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.122256  3339 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.691s	user 0.006s	sys 0.006s
W20260709 12:22:24.217686  1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1204) took 1572 ms (client timeout 1994 ms). Trace:
W20260709 12:22:24.218202  1728 rpcz_store.cc:269] 0709 12:22:22.644984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.703846 (+ 58862us) service_pool.cc:224] Handling call
0709 12:22:24.217490 (+1513644us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54002b75d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:24.217618 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2361529600}
I20260709 12:22:24.219511  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:24.195469  3466 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:24.220513  1846 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:22:24.220983  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 57770.6us
I20260709 12:22:24.221449  1728 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:24.221675  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:24.222550  3112 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:24.227448  2412 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:24.228731  3469 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.229424  3394 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:24.231016  3394 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:24.124460  1723 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:24.069489  3398 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.125434  3439 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:24.231819  3398 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.231976  3439 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:24.232496  3439 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.129319  3448 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.232671  3472 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.133009  3366 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.096347  3392 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:24.135350  3357 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.858s	user 0.013s	sys 0.000s
I20260709 12:22:24.237218  3366 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:22:24.240511  1723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1202) took 1692 ms (client timeout 1921 ms). Trace:
I20260709 12:22:24.240737  3366 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.242007  3366 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.242097  1729 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:24.122560  3383 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.637s	user 0.010s	sys 0.011s
I20260709 12:22:24.079105  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.051493  3450 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:24.091135  3350 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.170219  3438 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.172468  3410 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.173341  3452 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:22:24.182291  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 81739.5us
I20260709 12:22:24.191028  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.250690  3452 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.251122  3452 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:24.251343  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 62546.1us
I20260709 12:22:24.251500  3450 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:24.191272  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.252104  3450 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:24.252676  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63233.9us
I20260709 12:22:24.252859  3441 raft_consensus.cc:2764] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:22:24.236101  3448 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.253397  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:24.253268  3450 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.254168  3448 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:24.255203  3357 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:24.255512  3448 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.255606  3357 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.256026  3357 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:24.163578  3429 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.164840  3351 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.194901  3433 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:24.271101  3433 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:24.271508  3433 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.271561  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:24.197655  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.193028  3482 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 12:22:24.192916  3449 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.194904  3486 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.205798  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 175) took 1919 ms. Trace:
I20260709 12:22:24.274690  1798 rpcz_store.cc:276] 0709 12:22:22.285930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.665371 (+379441us) service_pool.cc:224] Handling call
0709 12:22:23.673538 (+1008167us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.677679 (+  4141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.205528 (+527849us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001e4450. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:24.205687 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2317752576}
I20260709 12:22:24.274734  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.207845  3451 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:24.209192  3476 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.209563  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.210016  3454 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:24.195359  3402 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.183s	user 0.004s	sys 0.012s
I20260709 12:22:24.212193  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:24.153277  3424 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.214403  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 190) took 1120 ms. Trace:
W20260709 12:22:24.216532  3460 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.903s	user 0.010s	sys 0.001s
I20260709 12:22:24.230744  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:24.297341  3410 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:24.297760  3410 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:24.298707  3410 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.300696  3410 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.129779  3455 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.231477  3394 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:24.238162  2415 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:24.232898  3398 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.310058  3394 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.311235  3398 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.311594  3394 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:24.203549  3363 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.213s	user 0.011s	sys 0.015s
W20260709 12:22:24.138548  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 226763us
I20260709 12:22:24.249987  3438 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:24.191865  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 74190.4us
I20260709 12:22:24.120517  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:24.252104  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:24.258387  3392 raft_consensus.cc:993] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: : Instructing follower 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:24.267681  3350 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:24.269004  1879 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:24.206944  3379 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:24.284525  3313 tablet_bootstrap.cc:492] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:22:24.265118  3487 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:24.285938  1807 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.337451  3487 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:24.337909  3487 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:24.246604  3328 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:24.191133  3425 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.288944  1725 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:24.259549  3450 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.288919  1726 rpcz_store.cc:276] 0709 12:22:23.093807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.096427 (+  2620us) service_pool.cc:224] Handling call
0709 12:22:23.096736 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:23.603650 (+506914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.214085 (+610435us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b54003ccdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:24.214296 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164165888}
I20260709 12:22:24.292587  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.293627  3441 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:24.300513  3321 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:24.325577  3454 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.301878  3322 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:24.304646  1730 raft_consensus.cc:606] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.355086  1730 raft_consensus.cc:614] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:22:24.096370  3186 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.531s	user 0.010s	sys 0.000s
I20260709 12:22:24.308269  3455 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.308779  3469 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:22:24.240932  1723 rpcz_store.cc:269] 0709 12:22:22.547737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.550509 (+  2772us) service_pool.cc:224] Handling call
0709 12:22:24.240326 (+1689817us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540035d754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:24.240456 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3250018560}
I20260709 12:22:24.251967  3452 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.313572  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:24.345356  3471 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:24.318918  3363 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:24.327608  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75608us
I20260709 12:22:24.327790  1731 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:24.372602  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.373883  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.329515  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.330415  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.331079  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:24.331468  3359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea5d8f652fc448d6af846ff6850004b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:22:24.374385  1800 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:24.333024  3438 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.333469  3392 raft_consensus.cc:1081] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Signalling peer 075704438992440697732181e49ccd09 to start an election
I20260709 12:22:24.327119  3379 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:24.330332  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.328696  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:24.335121  1798 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:24.335376  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 154) took 1709 ms (client timeout 1608 ms). Trace:
I20260709 12:22:24.335559  1880 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:24.389441  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.247177  3383 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.344063  3476 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:22:24.348428  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55883.4us
I20260709 12:22:24.393275  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.300957  3357 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.351914  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:24.373258  3186 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:24.400492  1726 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 2: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:24.352136  3441 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.353037  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:24.356897  3322 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:24.403083  3379 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.403839  3322 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.404347  3379 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:24.352216  3385 tablet_copy_client.cc:538] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:22:24.406934  3441 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:24.409353  3441 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.352723  3401 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260709 12:22:24.352176  3339 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20260709 12:22:24.350595  3112 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.411988  3476 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.412423  3476 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:24.342720  3448 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:24.363247  1723 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:24.366449  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52972.5us
I20260709 12:22:24.326050  1806 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:24.367084  1732 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:24.368834  3452 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.369715  3351 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:22:24.380190  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50763.9us
I20260709 12:22:24.376176  1803 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.382493  3402 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:24.434276  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:24.434422  3402 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:24.434828  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:24.435125  3402 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.435623  1877 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 2: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:24.338521  3433 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:24.387187  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2105) took 1052 ms. Trace:
I20260709 12:22:24.340019  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:24.439281  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:22:24.439711  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 99786.1us
W20260709 12:22:24.389747  1879 rpcz_store.cc:269] 0709 12:22:22.625751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:22.631074 (+  5323us) service_pool.cc:224] Handling call
0709 12:22:22.708070 (+ 76996us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:22:24.335134 (+1627064us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b5400223b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:24.335284 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3615504000}
I20260709 12:22:24.390733  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.383936  1800 raft_consensus.cc:606] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:24.452291  3339 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.452816  3339 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:24.453053  3401 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:24.453557  3401 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:24.453740  3339 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.396533  3474 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.438405  1731 rpcz_store.cc:276] 0709 12:22:23.335179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.336955 (+  1776us) service_pool.cc:224] Handling call
0709 12:22:23.337388 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.372388 (+1035000us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.386951 (+ 14563us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54004362d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:24.387127 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2277230336}
I20260709 12:22:24.397112  3449 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:24.457247  3449 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:24.397382  3387 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.460812  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.378975  3397 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.401356  1730 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62"
dest_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:57228
I20260709 12:22:24.402536  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:24.443070  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52431us
I20260709 12:22:24.449580  3458 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.450474  3438 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:24.450750  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.451236  3451 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.469754  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.451133  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:24.470700  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.471208  1723 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:24.471426  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:24.449626  3495 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:22:24.398830  3363 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:24.462453  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:24.467986   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:22:24.482066   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:24.482946   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:24.468901  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.484687  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.475143  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.475286  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.476681  3483 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.413287  3441 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.415824  3392 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:24.461364  3449 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.458441  3482 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:24.487792  3407 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:24.487888  3392 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.490137  3482 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.490738  3387 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.480959  3458 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.492357  3392 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:24.473681  3486 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.455331  3474 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:24.469655  3438 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:24.493443  3438 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.492542  3482 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:24.416769  3448 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:22:24.486810  3468 consensus_queue.cc:1175] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "075704438992440697732181e49ccd09" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20260709 12:22:24.433327  3351 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:24.439468  3424 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.488963  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.489238  1730 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:24.503515  1730 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.506349  1807 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:24.496073  3433 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.477593  3466 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.494445  3355 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:24.524724  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49517.5us
I20260709 12:22:24.498208  3351 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.458760  3186 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.511133  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:24.516774  3357 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:24.479507  3339 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.525524  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.531101  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.531278  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.474144  3461 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:22:24.541342  1879 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:24.541429  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.532202  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.542982  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.499754  3448 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.535888  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.536402  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:24.544545  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.454509  3401 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.545092  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.545246  1878 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.547487  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:24.416455  3328 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.552697  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:24.553356  1731 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:24.553290  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:24.375433  3471 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.561151  1807 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:24.561291  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.523698  3402 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.525200  3407 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.532483  3451 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.543509  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:24.532243  3387 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.566243  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:24.572850  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:24.586966  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.588145  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.588691  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.560127  3424 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.595723  1723 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:24.591511  1873 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:24.593784  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:24.591583  1799 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:24.563455  3463 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:24.538340  3452 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:22:24.560949  3352 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:24.482560  3476 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.562592  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.606276  1730 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.608193  1732 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:24.615015  1879 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:24.619411  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.585255  3321 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:24.623853  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.622133  1881 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:24.598850  3429 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.577979  3357 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 12:22:24.600497  3461 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.636128  1879 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:22:24.605867  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.638743  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.596355  1874 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:24.604768  3352 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:24.765300  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 182587us
I20260709 12:22:24.686318  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:24.733812  1803 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:24.722012  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.704026  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.642486  1879 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:24.690353  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:24.768949  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.623116  3488 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.772512  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.623009  3463 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.723218  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:24.780401  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76523us
I20260709 12:22:24.788887  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:22:24.756342  3439 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.523s	user 0.004s	sys 0.008s
I20260709 12:22:24.638182  3407 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:24.758159  3476 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.664307  3471 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:24.815225  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.642992  3363 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.748431  3355 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.783455  3364 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:24.721647  3383 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:24.653785  3402 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.641659  3186 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.837548  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:24.838913  3455 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:24.839499  3455 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:24.663796  3449 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.840786  3455 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.798437  3499 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.811926  3407 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.843065  3449 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.724814  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.777647  3352 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.843894  3407 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.798559  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:24.724948  3483 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.647738  3433 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:24.829450  3369 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.847536  3433 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.847833  3369 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.771617  3392 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:24.814268  1878 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:24.743641  3438 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:24.850235  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51785.1us
I20260709 12:22:24.647068  3491 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.361s
I20260709 12:22:24.857049  3449 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:24.857532  3449 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:24.857883  3449 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.863106  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.845947  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:24.834227  1880 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:24.851625  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:24.792622  3439 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.727994  3461 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.879355  3454 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:24.876209  3366 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.634s	user 0.006s	sys 0.004s
I20260709 12:22:24.876617  2410 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:24.848502  3433 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.823009  3471 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.822896  3394 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.878387  3363 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:24.878234  3402 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:24.886653  3439 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.889084  1729 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
W20260709 12:22:24.880725  3472 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.637s	user 0.009s	sys 0.002s
I20260709 12:22:24.895584  2413 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
W20260709 12:22:24.888767  3487 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.550s	user 0.009s	sys 0.003s
I20260709 12:22:24.898875  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.849503  3334 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260709 12:22:24.904531  1878 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.904325  3487 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.904978  3334 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:24.905380  3334 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:24.906590  3487 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.906194  3334 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.907482  3334 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:24.907636  1725 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260709 12:22:24.908944  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.910343  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.912918  1874 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:24.913162  1878 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:24.916424  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:24.917500  3487 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:24.918270  3487 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.918447  3439 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:24.920127  3487 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.920755  3321 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:24.921988  3321 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.923171  3321 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.924620  3321 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:24.897919  3463 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:24.946466  1879 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:24.896006  3369 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:24.894013  3397 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:24.945323  3410 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.612s	user 0.010s	sys 0.000s
I20260709 12:22:24.891747  3454 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.943583  3402 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:24.951695  3433 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:24.957010  3424 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:24.956347  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:24.977423  3438 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:24.978257  1801 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:24.982430  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.984704  1730 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:24.985193  3352 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:24.986212  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:24.987200  3497 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.165s	user 0.001s	sys 0.001s
I20260709 12:22:24.988831  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.989756  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:24.959420  3397 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:24.993285  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.996042  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.997649  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:24.998224  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:24.999100  1879 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:24.999507  1879 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:24.999964  3454 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:25.000372  1879 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.001143  3424 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:24.973806  1800 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:25.001631  1879 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:25.002959  3385 tablet_bootstrap.cc:492] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:22:25.004632  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:25.008122  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:25.008644  1877 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:25.009640  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:25.010543  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 412546us
I20260709 12:22:25.011663  3461 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.012602  3410 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.013098  3410 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.015731  1728 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:25.019552  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.020388  1728 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:25.023941  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:22:25.032927  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.992085  3394 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.016009  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:25.035312  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:25.032090  3461 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:25.063688  1726 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:25.060003  3461 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.065675  3461 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.030616  3441 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:25.015612  3402 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:25.030578  3454 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:22:25.093268  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.106930  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.108081  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.108623  1798 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:25.108702  1804 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:25.109807  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:25.096174  1875 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:25.064973  2414 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:25.114079  1879 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:25.127314  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:25.113114  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:25.130491  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.093685  3438 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.092231  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.098240  3441 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.152840  3397 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:25.159842  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.162813  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55448.7us
I20260709 12:22:25.061142  3394 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.098858  3457 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:25.145179  3505 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.121s	user 0.004s	sys 0.001s
I20260709 12:22:25.111584  3359 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:60934 (request call id 445) took 7422 ms. Trace:
I20260709 12:22:25.068598  3448 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.179762  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:25.114359  3410 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.131848  3424 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:25.126499  3483 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:22:25.168363  3441 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:25.167054  3352 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:25.175045  3359 rpcz_store.cc:276] 0709 12:22:17.689616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.689758 (+   142us) service_pool.cc:224] Handling call
0709 12:22:18.892557 (+1202799us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b5800003f98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb41a2 00007f16b1bf5889 00007f16b0d26d88 00007f16b0deaa07 00007f16b0e1cf4e 00007f16b0e01fdd 00007f16b0e01f89 00007f16b0e01f19 00007f16b0e01ee1
0709 12:22:18.925481 (+ 32924us) ts_tablet_manager.cc:934] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
0709 12:22:24.326962 (+5401481us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:24.749725 (+422763us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:24.749843 (+   118us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:25.104624 (+354781us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:25.104924 (+   300us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:25.105222 (+   298us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:25.111513 (+  6291us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":86837,"spinlock_wait_cycles":43268352,"tablet-copy.queue_time_us":4904,"thread_start_us":104239,"threads_started":2}
I20260709 12:22:25.204897  3359 tablet_copy_client.cc:806] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:22:25.205533  3359 tablet_copy_client.cc:670] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:22:25.207567  1723 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:25.210767  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:25.216150  1877 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:25.218631  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.223308  1726 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:25.221452  3487 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:25.226125  3508 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:22:25.160945  3438 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.228835  3438 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:25.208447  3424 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:25.230432  3424 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.189570  3363 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:25.235656  3363 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.236018  3363 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:25.240008  1726 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:25.240090  3448 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:25.244670  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:25.244848  1725 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:25.245384  1879 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:25.247009  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.249495  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:25.252575  1724 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:25.255648  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:25.251807  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:25.260286  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:25.234231  3441 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.264586  1724 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 5 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:25.180413  3449 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:25.269243  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 100136us
I20260709 12:22:25.187281  3457 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:25.271674  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 161938us
I20260709 12:22:25.277724  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:25.252205  3322 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:25.262137  3422 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:25.262217  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:25.267216  3438 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:25.270036  3487 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:25.294183  3498 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 12:22:25.304260  3498 consensus_queue.cc:1243] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 5.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:22:25.262727  3458 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
W20260709 12:22:25.308301  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.283630  3469 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.322916  3469 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:25.325655  1806 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:25.291500  3322 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.283562  3352 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:22:25.319553  3350 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.307495  3500 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:25.283252  3482 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 12:22:25.320510  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:25.321533  3359 tablet_copy_client.cc:538] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:22:25.311017  3438 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:25.326274  3469 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.297549  3351 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:25.307986  3487 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.272145  3456 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:22:25.327623  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.315223  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.272003  3499 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:22:25.343710  1877 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:25.358888  1728 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:25.361804  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.364745  1724 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:25.366269  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:25.366664  1728 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:25.367961  1728 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:25.368621  1730 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 2 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:25.370033  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.370814  1724 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:25.372144  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:25.374933  1727 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:25.356076  3454 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:25.377782  1804 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:25.380162  1804 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:25.360447  2414 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:25.383168  3512 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:25.383831  3512 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.384302  3512 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.376189  3474 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 12:22:25.388049  3322 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:25.392760  3498 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:25.394018  3498 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:25.378463  3186 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 12:22:25.395555  3498 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:25.395907  3498 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:25.383836  3387 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:25.395970  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.396811  3498 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.354240  3461 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.399720  3457 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:25.400602  2415 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:25.400991  3461 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.401305  3394 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.401952  3401 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.346182  3350 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.402384  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.360818  3487 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.406474  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:25.407301  1732 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.408946  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:25.415956  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 43920.8us
I20260709 12:22:25.373410  3469 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:25.422930  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.418208  1804 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.425383  1728 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:25.396703  3387 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:25.351704  3500 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.428599  1804 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:25.430044  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.429883  2410 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:25.427702  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.422434  3448 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.427976  3464 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:22:25.428309  3321 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.503s	user 0.014s	sys 0.000s
I20260709 12:22:25.429926  3387 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:25.435071  3424 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.452046  1804 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:25.432497  3469 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:25.440327  3464 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:25.436501  3448 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:25.438232  3438 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.472014  3387 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:25.476150  3449 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:25.474129  3392 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:25.479817  3392 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:25.566450  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.573794  3441 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:25.574919  3448 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:25.575816  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.576587  1730 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:25.578052  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:25.580073  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:25.580340  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.573900  3422 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.580801  1804 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:25.580992  1806 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:25.583876  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1332) took 1049 ms. Trace:
I20260709 12:22:25.473843  3464 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.470805  3321 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.139s
W20260709 12:22:25.585444  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.585457  3469 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.588375  3381 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:25.457571  3424 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.603358  3422 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.604899  3424 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:25.591374  1799 rpcz_store.cc:276] 0709 12:22:24.534538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.536243 (+  1705us) service_pool.cc:224] Handling call
0709 12:22:25.583529 (+1047286us) spinlock_profiling.cc:243] Waited 46.6 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:25.583726 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":107245440}
I20260709 12:22:25.605684  3422 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:25.600142  3464 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:25.568333  3454 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.607920  3454 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.610054  3454 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:25.581050  3502 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.611310  3502 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.478032  3350 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.476707  3500 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:25.600087  3450 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:25.598403  1804 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:25.621766   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:22:25.590623  3433 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.000s
I20260709 12:22:25.633342  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 170) took 1216 ms. Trace:
I20260709 12:22:25.633845  1880 rpcz_store.cc:276] 0709 12:22:24.417346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.417475 (+   129us) service_pool.cc:224] Handling call
0709 12:22:25.633123 (+1215648us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b540021c5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:25.633266 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":810422656}
I20260709 12:22:25.616434  3364 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:25.635923  3364 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.636358  3364 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.640918  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:25.643463  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:25.644266  2418 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:25.645577  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.646530  1730 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:25.648662  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:25.649770  3280 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:25.649959  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.650290  3280 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:25.650606  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.650815  3280 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:25.651748  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.652917  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:25.654661  3489 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:25.655884  3489 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:25.656251  3489 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:25.644361  3350 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:25.659312  3502 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.657160  3489 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.660831  3489 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:25.642657  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:25.665729  2419 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:25.671511  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:25.620554  3468 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:25.663028  3381 consensus_queue.cc:1243] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 2.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:22:25.627753  3450 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:25.678964  3433 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:25.687635  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1327) took 1531 ms (client timeout 1502 ms). Trace:
I20260709 12:22:25.677106  3350 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:25.680734  3491 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:25.676549  3351 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.698011  3351 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.700500  3351 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.702327  1800 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:42892
I20260709 12:22:25.707267  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:25.708107  1730 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
W20260709 12:22:25.688172  1801 rpcz_store.cc:269] 0709 12:22:24.156597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.165349 (+  8752us) service_pool.cc:224] Handling call
0709 12:22:25.687409 (+1522060us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b54001e3a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:25.687534 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1366196096}
I20260709 12:22:25.697376  3381 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.709677  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:25.714399  3512 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:25.715142  3441 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:25.686502  3438 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.717834  1732 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:25.687798  3433 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.681581  3468 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:25.736757  1726 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:22:25.738485  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:25.739413  3502 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:25.740157  3502 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:25.742455  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 1993) took 1170 ms. Trace:
I20260709 12:22:25.741503  3433 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:25.760314  3487 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.764279  3512 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.741106  3502 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.765789  1872 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:25.736320  3438 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.694186  3491 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.773833  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:25.719669  3441 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.712311  3350 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:25.777954  3186 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:25.773993  3450 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:25.806146  3521 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:22:25.710974  3363 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.720844  3381 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.806950  3468 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.776993  3280 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.810282  3324 tablet_bootstrap.cc:492] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:22:25.815258  3366 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.815269  3313 tablet_bootstrap.cc:492] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:22:25.817449  3455 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.819442  3359 tablet_bootstrap.cc:492] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:22:25.742960  1875 rpcz_store.cc:276] 0709 12:22:24.571563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.575387 (+  3824us) service_pool.cc:224] Handling call
0709 12:22:25.742243 (+1166856us) spinlock_profiling.cc:243] Waited 319 ms on lock 0x7b5400026f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:25.742384 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":732667776}
I20260709 12:22:25.825254  3487 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:25.806051  3364 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:25.830353  2415 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:25.824026  3491 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.826969  3441 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.852469  3455 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.859812  3350 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.860595  3280 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.861308  3499 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:25.861946  3455 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:25.862689  3498 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:25.863008  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2114) took 1682 ms. Trace:
I20260709 12:22:25.867385  1729 rpcz_store.cc:276] 0709 12:22:24.180389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.255289 (+ 74900us) service_pool.cc:224] Handling call
0709 12:22:24.255680 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.903449 (+647769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.862677 (+959228us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b54002b7fd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:25.862935 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1456224128}
I20260709 12:22:25.861752  3499 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:25.845855  3450 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:25.851377  3351 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.854216  3468 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:25.888103  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.869031  3381 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:25.905366  1805 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:25.888513  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.889621  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:25.890478  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:25.880466  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:25.907276  1807 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:25.899161  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:25.903801  1873 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:25.900360  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:25.909475  1879 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:25.896708  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:25.906697  1806 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:25.905421  3313 tablet_bootstrap.cc:492] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:22:25.914407  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:25.917425  1873 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:22:25.835108  3186 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:25.932803  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:25.933192  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:25.932689  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:25.936582  1878 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:25.903589  3385 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:22:25.940442  3385 tablet_bootstrap.cc:492] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:22:25.941231  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:25.852138  3366 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:25.943013  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.943256  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:25.948865  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 196) took 1047 ms. Trace:
I20260709 12:22:25.949349  1723 rpcz_store.cc:276] 0709 12:22:24.901786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.902001 (+   215us) service_pool.cc:224] Handling call
0709 12:22:24.902400 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.218901 (+316501us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.948621 (+729720us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b54000d2ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:25.948793 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":701319552}
I20260709 12:22:25.906036  1874 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:22:25.949487  3397 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:25.916627  3487 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:25.961356  1801 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:25.933818  3424 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:25.918203  3469 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:25.949148  3324 tablet_bootstrap.cc:492] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:22:26.013950  1807 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
W20260709 12:22:26.028249  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 74957.9us
W20260709 12:22:26.016498  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 66956.4us
I20260709 12:22:26.032585  1878 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:25.954197  3381 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:25.957324  3449 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:26.053387  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:25.917708  3351 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.036829  3487 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.037581  3532 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.037601  3422 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:26.041006  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:26.032115  3352 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.042968  3425 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:25.916807  3454 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260709 12:22:26.081370  1881 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:25.963217  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:25.941215  3321 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20260709 12:22:26.058223  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:26.061962  3280 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:26.055713  3449 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.067919  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:26.050598  3313 ts_tablet_manager.cc:1403] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 1.766s	user 0.036s	sys 0.024s
W20260709 12:22:26.053315  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.042167  3385 tablet_bootstrap.cc:492] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:22:26.057965  3474 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.950431  3499 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:26.085089  1879 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:26.130468  3321 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 12:22:26.139422  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:26.140630  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:26.141419  3186 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.141723  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:26.126473  3407 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:26.142589  3407 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.090688  3334 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:26.127324  3543 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.144208  3334 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:26.144619  3543 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.081625  3352 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.153429  3352 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.090210  1878 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:26.155565  1881 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:26.092445  1874 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:26.089136  1723 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:26.170363  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.072630  3487 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.098026  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:26.185958  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 87895.1us
I20260709 12:22:26.187901  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:26.117332  1876 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:26.058656  3381 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.134141  3397 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:26.134935  3424 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:26.075047  3532 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.085537  3438 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:26.076309  3324 ts_tablet_manager.cc:1403] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 1.938s	user 0.026s	sys 0.023s
I20260709 12:22:26.142910  3512 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:26.143494  3366 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.145166  3334 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.077909  3357 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.183367  3369 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:26.160357  1879 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:26.174309  2419 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:26.034276  3469 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.102072  3449 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:26.210942  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 126604us
I20260709 12:22:26.144670  3407 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:26.135972  3363 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.219851  1878 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:26.130478  3280 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:26.173789  3394 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.204707  3352 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.094877  3482 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.184007  3535 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:26.204191  3186 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:26.243152  3543 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:26.222539  3385 ts_tablet_manager.cc:1403] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 1.220s	user 0.028s	sys 0.016s
I20260709 12:22:26.209057  3539 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:26.244097  3369 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.172571  3474 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.214627  3456 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:22:26.213901  3321 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
W20260709 12:22:26.253400  3387 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.774s	user 0.007s	sys 0.009s
I20260709 12:22:26.261729  3363 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.239924  3438 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:22:26.245644  3512 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.250499  3357 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.274362  3381 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.266673  3482 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.316578  3369 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.317385  3466 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:26.317713  3482 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.313864  3363 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.318861  3280 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.313381  3424 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.262234  3350 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.302347  3539 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:26.261030  3538 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260709 12:22:26.265517  3394 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.324123  3466 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:26.338092  3466 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:26.339834  3534 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.278s	user 0.003s	sys 0.000s
I20260709 12:22:26.343418  3487 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:26.344547  3532 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:26.346074  3466 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:26.359297  3464 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:26.359853  3464 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:26.361318  3464 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:26.361865  3536 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.294s	user 0.003s	sys 0.000s
I20260709 12:22:26.362812  3464 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:26.276633  3535 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.356210  3474 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:26.367509  3491 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.504s	user 0.012s	sys 0.004s
I20260709 12:22:26.368608  3539 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.161953  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.420646  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.422981  1730 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:26.087941  1727 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:26.425127  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.444139  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2120) took 1234 ms. Trace:
I20260709 12:22:26.445259  1730 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:26.445076  1725 rpcz_store.cc:276] 0709 12:22:25.209258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.210277 (+  1019us) service_pool.cc:224] Handling call
0709 12:22:25.210616 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.245378 (+ 34762us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.443306 (+1197928us) spinlock_profiling.cc:243] Waited 33.9 ms on lock 0x7b54000d25d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:26.443874 (+   568us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":77891456}
I20260709 12:22:26.449640  1723 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:26.451987  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2136) took 1041 ms. Trace:
I20260709 12:22:26.452529  1726 rpcz_store.cc:276] 0709 12:22:25.410647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.410849 (+   202us) service_pool.cc:224] Handling call
0709 12:22:25.736501 (+325652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:25.737303 (+   802us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.438418 (+701115us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b540044a550. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:26.451766 (+ 13348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":712542976}
I20260709 12:22:26.158763  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:26.476032  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.477046  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.476799  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 317966us
W20260709 12:22:26.225536  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.472075  1876 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:26.472903  1800 raft_consensus.cc:606] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:26.487939  1800 raft_consensus.cc:614] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:26.488741  1729 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:26.497282  1878 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:26.499423  1874 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:26.500662  1878 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:22:26.507155  3321 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.508208  3321 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.515870  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:26.517354  3350 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.521520  3350 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.522297  3350 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.264691  3352 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.529412  3357 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.531736  3407 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.532804  3407 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.535612  3401 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.536571  3401 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.538365  1729 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 5 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:26.540746  3313 raft_consensus.cc:359] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:26.542106  3313 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:22:26.303159  3397 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.545979  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:26.546955  3476 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:26.547828  3351 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.548030  3476 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.547117  3466 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:26.548451  3474 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:26.549266  3466 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.549911  3488 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:26.493605  3324 raft_consensus.cc:359] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:26.551368  3500 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:26.551867  3324 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
W20260709 12:22:26.562266  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 79663.8us
I20260709 12:22:26.563284  3433 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:26.563722  3433 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:26.564078  3433 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:26.566231  3321 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.550005  3489 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:22:26.570637  3476 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:26.569744  3186 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:26.571767  3313 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:26.573843  3350 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.577143  3350 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.578656  3352 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.581043  3466 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:26.587006  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:26.552856  3324 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:26.611076  3313 ts_tablet_manager.cc:1434] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.477s	user 0.009s	sys 0.004s
W20260709 12:22:26.611752  3313 ts_tablet_manager.cc:1467] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Tablet startup took 501ms
I20260709 12:22:26.612278  3469 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.618325  3364 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.619549  3476 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.619814  3468 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:26.620289  3476 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:26.620687  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 61702.9us
I20260709 12:22:26.620824  3468 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.624701  3324 ts_tablet_manager.cc:1434] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.392s	user 0.006s	sys 0.005s
W20260709 12:22:26.624969  1697 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:22:26.626358  3532 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:26.627221  3532 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.628584  3450 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:26.634915  3474 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.635444  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.636484  2413 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:26.636711  3483 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:26.637601  3483 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.638042  3540 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.010s
I20260709 12:22:26.368527  3491 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.648450  3540 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:26.648902  3540 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:26.651180  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.651361  3547 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:26.651994  1730 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:26.652137  3350 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.652987  3350 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.653373  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.654448  1730 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 2: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:26.640241  2415 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:26.641037  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.656631  2412 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:26.658277  3500 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:26.659500  3500 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:26.660688  3500 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:26.661410  3469 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:26.661767  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55700.2us
I20260709 12:22:26.661967  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.662083  3469 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.663301  3422 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:26.664364  3422 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:26.664556  1726 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:26.664764  3422 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:26.665019  3469 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.665705  3381 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:26.665805  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.666245  3381 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:26.667351  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 991600us
I20260709 12:22:26.650004  3540 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.668764  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.670312  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:26.647279  3551 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.273s	user 0.002s	sys 0.000s
I20260709 12:22:26.650452  3491 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.724607  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.721498  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:26.794896  3186 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:26.795888  3186 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.796353  3186 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:26.684150  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:26.689499  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.688618  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.683629  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1268) took 1259 ms. Trace:
I20260709 12:22:26.825930  2418 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:26.690377  3498 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:26.803814  1726 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:26.800633  1725 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:22:26.830849  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 109436us
I20260709 12:22:26.684063  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:26.687348  3512 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.633890  3425 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:26.713647  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.683014  3466 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:26.827929  2417 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:26.684732  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.683590  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.836673  2411 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:22:26.821087  1875 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.004s	sys 0.011s
I20260709 12:22:26.685325  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.724850  3468 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:26.639770  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1415) took 2109 ms (client timeout 1638 ms). Trace:
I20260709 12:22:26.672204  3488 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.162s
I20260709 12:22:26.698258  3449 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.699779  3364 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.671834  3401 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:26.690800  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.820816  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.828145  1728 rpcz_store.cc:276] 0709 12:22:25.423919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.426101 (+  2182us) service_pool.cc:224] Handling call
0709 12:22:25.426636 (+   535us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:25.574733 (+148097us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:25.574767 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:25.575275 (+   508us) log.cc:844] Serialized 26 byte log entry
0709 12:22:25.679008 (+103733us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:25.689691 (+ 10683us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.689914 (+   223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.350689 (+660775us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b18003b4ac0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:22:26.465470 (+114781us) spinlock_profiling.cc:243] Waited 4.83 ms on lock 0x7b18003b4ac0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419
0709 12:22:26.638403 (+172933us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b54002a9254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:26.638459 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:26.638526 (+    67us) raft_consensus.cc:1685] finished
0709 12:22:26.638539 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:26.683431 (+ 44892us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":209528,"prepare.queue_time_us":10519,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":750,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":2755,"spinlock_wait_cycles":355688192,"thread_start_us":102946,"threads_started":1,"wal-append.queue_time_us":97496}
I20260709 12:22:26.671293  3407 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.734324  3540 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:26.652011  3547 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:26.686305  3381 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.683450  2412 raft_consensus.cc:1217] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3]   Dedup: 2.3->[]
W20260709 12:22:26.668545  3392 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.154s	user 0.012s	sys 0.001s
W20260709 12:22:26.645948  3441 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.779s	user 0.014s	sys 0.013s
I20260709 12:22:26.695015  3394 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.709537  3350 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.698452  2413 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:26.637620  3474 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:26.892298  1875 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.835s	user 0.004s	sys 0.013s
I20260709 12:22:26.703631  1727 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:26.946287  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1278) took 1010 ms. Trace:
I20260709 12:22:26.946854  1727 rpcz_store.cc:276] 0709 12:22:25.935753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.941447 (+  5694us) service_pool.cc:224] Handling call
0709 12:22:26.946164 (+1004717us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:26.947319  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.723362  3450 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.210s
I20260709 12:22:26.948405  3450 consensus_queue.cc:1243] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:22:26.949177  2410 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:26.950955  3468 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.951370  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.951591  3468 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.689563  3363 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:26.642616  3535 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:26.952744  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:26.953672  2419 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:26.953835  3441 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.956306  3392 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.959168  1880 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:26.904464  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.950732  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.963279  3401 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:26.964404  3401 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.721078  3433 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:26.897764  2414 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:26.850912  3466 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:26.897462  1872 rpcz_store.cc:269] 0709 12:22:24.529893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.532613 (+  2720us) service_pool.cc:224] Handling call
0709 12:22:26.639272 (+2106659us) spinlock_profiling.cc:243] Waited 988 ms on lock 0x7b5400117454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:26.639482 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2271772032}
I20260709 12:22:26.908761  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.838819  3498 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:26.677711  3499 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:26.914600  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:26.918264  2418 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
W20260709 12:22:26.924830  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 393680us
I20260709 12:22:26.924566  3364 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.925346  3556 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:26.924688  1732 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:22:26.984094  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 294551us
I20260709 12:22:26.931140  3488 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.947837  3449 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:26.612183  3313 ts_tablet_manager.cc:1469] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Trace:
0709 12:22:16.809380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.809530 (+   150us) service_pool.cc:224] Handling call
0709 12:22:16.852279 (+ 42749us) ts_tablet_manager.cc:934] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
0709 12:22:17.736016 (+883737us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:18.832018 (+1096002us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:18.832189 (+   171us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:19.300119 (+467930us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:19.336477 (+ 36358us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:19.347463 (+ 10986us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:19.353811 (+  6348us) inbound_call.cc:177] Queueing success response
0709 12:22:24.284088 (+4930277us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:22:26.160710 (+1876622us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:22:26.491485 (+330775us) tablet_replica.cc:261] Starting instrumentation
0709 12:22:26.493032 (+  1547us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":4288,"mutex_wait_us":38193,"raft.queue_time_us":45667,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":193,"tablet-copy.queue_time_us":41460,"tablet-download-pool.queue_time_us":46332,"tablet-download-pool.run_cpu_time_us":1972,"tablet-download-pool.run_wall_time_us":3109981,"thread_start_us":191568,"threads_started":2}
I20260709 12:22:26.936520  1730 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:26.937098  3456 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:26.937666  3547 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:26.931140  3422 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.017377  3422 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:26.842254  3425 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:26.948356  1726 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:26.974444  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:26.967761  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.644027  3385 raft_consensus.cc:359] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:26.928269  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:26.956036  3532 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:26.931933  3352 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:27.009349  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:26.982285  2410 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:26.939471  3489 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
W20260709 12:22:27.033911  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 112519us
I20260709 12:22:27.036682  1729 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:27.041126  1877 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:27.025998  3385 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:22:27.048712  3498 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.049883  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:27.050774  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 64696.7us
I20260709 12:22:27.049297  3364 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:27.057891  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 400025us
I20260709 12:22:27.058642  3561 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:27.059151  3424 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.059090  1628 catalog_manager.cc:5697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:22:27.061869  3350 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:27.065554  3186 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.069897  1880 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:27.039373  3474 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:22:27.051220  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.029659  3352 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.051126  3535 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:27.091523  3464 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.728s	user 0.005s	sys 0.012s
I20260709 12:22:27.091898  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.094281  1873 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:27.071409  3350 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.098996  3350 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.099758  3563 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.100649  3563 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.083626  3481 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:27.112465  1872 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:27.113772  3352 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.102468  3535 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:27.067801  3561 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.069552  3392 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:27.119184  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:27.120512  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.077162  3441 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:27.103271  3498 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:27.115358  1879 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:27.114454  3424 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:27.052561  3488 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.115156  3535 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.108778  3456 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.047035  3556 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.119475  3561 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.122622  1628 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:27.083220  3425 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:27.133083  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1431) took 1232 ms. Trace:
I20260709 12:22:27.137499  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.138347  3401 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:27.120545  3385 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:27.141166  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.157660  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.164729  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.173264  1628 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 3, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 3 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:27.131372  3481 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.176263  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.155385  3456 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.178687  3394 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:27.203413  1879 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 2: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:27.196702  3560 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
W20260709 12:22:27.203240  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0640653239997846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
I20260709 12:22:27.207631  1628 catalog_manager.cc:5697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:27.208839  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 88759.2us
I20260709 12:22:27.196684  3394 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.189070  3564 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.083s
W20260709 12:22:27.222936  3565 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:22:27.224509  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:27.225816  3392 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:27.226336  3392 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:27.226755  3392 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:27.227648  3556 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.227790  3488 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.227996  3441 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:27.213343  3563 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.228886  3425 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.215380  3469 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:27.229673  3469 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.230172  3469 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.231936  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.232795  1878 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:27.233897  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.235998  3456 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.212806  3394 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:27.240146  2410 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Advancing to term 2
I20260709 12:22:27.240767  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:27.241628  2413 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:27.242707  3352 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.243326  1800 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:22:27.243476  3352 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.244057  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:27.245736  3352 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:27.247503  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:27.248198  1724 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.249073  3352 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.249737  3352 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.250823  3466 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.252174  3466 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:27.252453  3352 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:27.253279  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.254354  3512 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:27.254758  3512 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:27.255167  3512 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:27.256544  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.257458  1729 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:27.258652  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.259287  1731 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:27.259443  3424 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.264365  3468 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.264531  3512 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.266863  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.221786  3363 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.267570  1726 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:27.267830  3363 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.268589  1628 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 5, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:27.269878  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.155046  1875 rpcz_store.cc:276] 0709 12:22:25.896094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.903215 (+  7121us) service_pool.cc:224] Handling call
0709 12:22:25.958317 (+ 55102us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:26.522146 (+563829us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16a8cce0ac 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060
0709 12:22:26.944304 (+422158us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:26.944343 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:26.945196 (+   853us) log.cc:844] Serialized 26 byte log entry
0709 12:22:27.040860 (+ 95664us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:27.044663 (+  3803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.069721 (+ 25058us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:27.114904 (+ 45183us) raft_consensus.cc:1685] finished
0709 12:22:27.114936 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:27.128469 (+ 13533us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3427,"raft.queue_time_us":9222,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":11324800,"thread_start_us":976491,"threads_started":2,"wal-append.queue_time_us":149889}
I20260709 12:22:27.271664  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.272264  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:27.272970  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.274154  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.275315  3385 ts_tablet_manager.cc:1434] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 1.008s	user 0.008s	sys 0.013s
W20260709 12:22:27.275660  3385 ts_tablet_manager.cc:1467] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Tablet startup took 1009ms
I20260709 12:22:27.276002  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:27.278532  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:27.226964  3476 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.278668  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.279700  1725 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:27.282441  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.283878  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:27.284476  3407 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.284935  3186 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.285661  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.287914  2416 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:27.297863  1724 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:27.296177  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.284937  3449 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.299577  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.270431  1628 catalog_manager.cc:5697] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:27.302035  3488 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:27.302734  3488 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.303282  3481 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.304836  3424 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:27.305609  3424 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:27.300289  3569 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.301725  1628 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:27.304982  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:27.276772  3512 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.314133  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:27.318684  3438 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:27.319584  1879 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:27.321393  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:27.322404  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1436) took 1424 ms. Trace:
I20260709 12:22:27.305042  3488 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.323316  1881 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:27.310194  3407 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.324074  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.270452  3363 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.328225  1732 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:27.327320  3569 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.348006  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:27.349153  1730 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:27.333361  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:27.335773  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:27.338199  1877 rpcz_store.cc:276] 0709 12:22:25.897558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.900000 (+  2442us) service_pool.cc:224] Handling call
0709 12:22:27.310834 (+1410834us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400026054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:27.322296 (+ 11462us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2439954688}
I20260709 12:22:27.350203  2419 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.314308  3556 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:27.299532  3449 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.306857  3424 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:27.358225  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.356313  1628 catalog_manager.cc:5697] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:27.369647  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.372830  3350 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.375556  1804 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:22:27.375787  3350 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.381173  3407 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:27.384532  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63342.5us
I20260709 12:22:27.348461  3552 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:27.372375  1742 tablet_copy_service.cc:342] P 075704438992440697732181e49ccd09: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-d891ed30ca204d09b7b083088037b63d received from {username='slave'} at 127.0.0.1:48886
I20260709 12:22:27.409715  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.419128  1742 tablet_copy_service.cc:434] P 075704438992440697732181e49ccd09: ending tablet copy session 123095127f334445889c21a42ba1d41e-d891ed30ca204d09b7b083088037b63d on tablet d891ed30ca204d09b7b083088037b63d with peer 123095127f334445889c21a42ba1d41e
I20260709 12:22:27.421516  3407 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:27.422236  3407 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.423101  3394 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.426090  3394 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:27.409586  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 76039.6us
I20260709 12:22:27.391011  3540 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:27.436791  2411 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:27.403318  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.407501  1628 catalog_manager.cc:5697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:22:27.435074  1879 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:27.365423  3556 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:27.437862  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.391786  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.440385  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.403823  3483 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.422689  3407 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.444197  3569 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.437091  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.447935  1726 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:27.441419  1800 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:27.451345  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.461457  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.455895  1724 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:27.454767  3468 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:27.457089  3397 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260709 12:22:27.457635  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.457850  3539 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:27.457652  3476 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:27.458178  3538 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:27.459800  3568 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.464819  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:27.452767  1628 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } }
I20260709 12:22:27.467675  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.475864  1723 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:27.461052  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.476476  1731 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:22:27.469650  3392 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.477900  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.479704  1725 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:27.479977  3392 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.480911  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.481849  1801 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:27.482319  3466 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.483178  3488 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:27.483282  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.483980  3499 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.484213  1801 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:27.484659  3569 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.485069  3499 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.485447  3569 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:27.487465  1799 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:27.487526  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.488533  1806 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:27.488997  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.489358  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.489991  1806 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:27.490657  3571 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.483031  3466 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.490800  3570 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.491596  3394 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:27.491427  3571 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.491742  3570 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:27.492399  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.493072  3499 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.493252  1798 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:27.494426  1878 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:27.490938  3489 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.495311  3489 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:27.499014  3568 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.499754  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.500944  3563 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.501308  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:27.275954  3385 ts_tablet_manager.cc:1469] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Trace:
0709 12:22:18.708392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.708554 (+   162us) service_pool.cc:224] Handling call
0709 12:22:18.841059 (+132505us) ts_tablet_manager.cc:934] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
0709 12:22:22.277133 (+3436074us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:22.438223 (+161090us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:22.438445 (+   222us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:22.791695 (+353250us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:22.792093 (+   398us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:22.806216 (+ 14123us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:22.816176 (+  9960us) inbound_call.cc:177] Queueing success response
0709 12:22:25.002562 (+2186386us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:22:26.273112 (+1270550us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:22:26.528039 (+254927us) tablet_replica.cc:261] Starting instrumentation
0709 12:22:26.529030 (+   991us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":2762,"mutex_wait_us":56591,"raft.queue_time_us":16654,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":7984,"tablet-copy.queue_time_us":78589,"tablet-download-pool.queue_time_us":20000,"tablet-download-pool.run_cpu_time_us":1792,"tablet-download-pool.run_wall_time_us":843481,"thread_start_us":110637,"threads_started":2}
I20260709 12:22:27.501624  3563 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.503883  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:27.505189  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:27.473212  3538 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.507246  3538 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.451176  3540 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.509454  3540 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.511533  3556 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.511960  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:27.451205  3561 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:27.512975  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67360.3us
I20260709 12:22:27.513836  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:27.499468  3469 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.521039  1798 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:27.522367  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.485970  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.522555  1799 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:27.527495  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.545691  1877 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:27.536111  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:27.537720  2419 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:27.540205  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 93412.7us
I20260709 12:22:27.540745  3570 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:27.533733  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.551762  1807 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:27.540509  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.543118  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.491452  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:27.542508  1628 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:22:27.544179  2417 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
W20260709 12:22:27.527704  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1433) took 1631 ms (client timeout 2080 ms). Trace:
I20260709 12:22:27.541383  3582 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:27.554716  1806 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:27.518494  3500 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.557425  1805 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:27.555186  1880 rpcz_store.cc:269] 0709 12:22:25.896688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.898908 (+  2220us) service_pool.cc:224] Handling call
0709 12:22:27.516845 (+1617937us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:27.527623 (+ 10778us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2416695552}
W20260709 12:22:27.558655  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.542104  3450 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:27.544844  3540 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:27.542831  3186 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:27.564492  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.527712  3574 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.572930  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:27.533667  3394 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:27.518956  3476 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.597196  1880 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:27.560251  3483 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:27.484092  3488 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.603066  3574 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.611997  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.612334  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.613188  1732 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:27.613456  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:27.613745  3488 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.614182  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.614380  3186 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.615229  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.616077  1727 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:27.616430  3186 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.617300  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-8bb813e4fab14a61b810a24b56899586 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:22:27.617890  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-8bb813e4fab14a61b810a24b56899586 on tablet 8bb813e4fab14a61b810a24b56899586 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:22:27.617506  3394 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.618395  3381 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.615329  3449 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.619405  3467 raft_consensus.cc:2764] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:22:27.534188  3468 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.578213  3392 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:27.560032  3582 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.633028  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.580078  3359 tablet_bootstrap.cc:492] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:22:27.643603  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:27.557356  1801 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:27.646682  1725 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:27.603710  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:27.635375  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.635459  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:27.636099  3186 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:27.564394  3450 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:27.638046  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:27.645203  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.622917  3363 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:27.574868  3540 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.624976  1800 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:27.621151  3538 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:27.649873  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 135095us
I20260709 12:22:27.654444  3469 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.645579  3449 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.612133  3574 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.659053  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:27.659953  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.687477  1872 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:27.660558  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1352) took 1771 ms. Trace:
I20260709 12:22:27.689849  1804 rpcz_store.cc:276] 0709 12:22:25.889090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.891047 (+  1957us) service_pool.cc:224] Handling call
0709 12:22:25.891416 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.375986 (+1484570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.660252 (+284266us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54003b7ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:27.660459 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3380095872}
I20260709 12:22:27.692132  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2008) took 1605 ms. Trace:
I20260709 12:22:27.667035  3540 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.692620  1881 rpcz_store.cc:276] 0709 12:22:26.087092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.156396 (+ 69304us) service_pool.cc:224] Handling call
0709 12:22:27.322952 (+1166556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.354736 (+ 31784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.691877 (+337141us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400026050. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:27.692062 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2681905408}
W20260709 12:22:27.695777  3535 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.553s	user 0.010s	sys 0.002s
I20260709 12:22:27.696337  3535 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.661279  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.632400  3401 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:27.636832  3422 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.666639  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.574982  3500 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.667661  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.668983  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:27.701180  3401 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.703830  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.604537  3350 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:22:27.707255  1805 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.642845  3392 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.643115  3556 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.709579  3422 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.710274  3489 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.710707  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.711977  3538 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.712932  3476 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.700856  3468 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:27.713171  3450 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.714566  3450 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.715364  1874 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:27.716729  3535 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.715863  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.717996  1802 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 2: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:27.718827  3407 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.719218  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.720041  1802 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:27.720520  1728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:22:27.721131  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.673035  3569 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.675496  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:27.676898  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:27.721282  1728 raft_consensus.cc:606] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:22:27.674711  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:27.682734  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.683106  3575 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.662245  3488 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:27.684541  3587 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:27.684654  3561 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.686013  1873 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:27.691627  3449 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.652019  3363 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.695832  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.661178  3359 tablet_bootstrap.cc:492] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:22:27.644824  3582 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.700471  1806 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:27.704726  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.706025  1880 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:27.703365  3401 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.651400  3576 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:27.664247  3352 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:27.625425  3547 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.721315  3407 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.680056  3539 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:27.712387  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 117181us
I20260709 12:22:27.727250  1728 raft_consensus.cc:614] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:27.728516  1807 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:22:27.733419  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 58845.9us
W20260709 12:22:27.732239  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 56898.1us
I20260709 12:22:27.734000  1805 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:27.722625  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.736634  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:27.745232  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:27.747872  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:27.710376  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.726186  3392 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.711974  3422 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:27.756057  1800 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:27.731848  3488 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.735229  3552 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:27.762765  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.739856  3468 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:27.748342  3363 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.810057  1726 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.811592  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:27.810070  1725 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:27.832093  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:27.740778  3588 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:27.842173  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:27.846599  2419 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:27.839466  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.847697  1799 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:27.845178  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:27.858919  3547 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:27.841708  1806 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:27.848869  1729 raft_consensus.cc:2368] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:27.860376  3449 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.861375  3547 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:27.861683  1729 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:27.808617  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.863322  3359 ts_tablet_manager.cc:1403] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 2.044s	user 0.043s	sys 0.041s
I20260709 12:22:27.845822  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.863359  1874 raft_consensus.cc:2368] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5.
I20260709 12:22:27.759346  3352 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.854499  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.863402  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.864671  2411 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:22:27.876822  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.736124  3556 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.880802  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.882650  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.883289  3556 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:27.883695  1872 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 2: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:27.839216  3466 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.885044  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.886607  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.887498  1880 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:27.888361  3589 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:27.888969  3589 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:27.809312  3576 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.892028  1727 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:27.893865  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:27.877622  1877 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:27.889981  3589 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:27.846863  3553 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
W20260709 12:22:27.794193  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.841s (SENT) [suppressed 28 similar messages]
I20260709 12:22:27.874260  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:27.878237  1806 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:27.879128  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:27.891930  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:27.897899  1729 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
W20260709 12:22:27.864718  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53313.2us
I20260709 12:22:27.904263  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:27.915261  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:27.741034  3587 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:27.904633  1725 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:27.909161  2419 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:27.845968  3488 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:27.913825  3449 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:27.840883  3468 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.905646  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:27.757045  3476 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.116s
I20260709 12:22:27.915953  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:27.931717  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:27.740645  3575 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.938632  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.941615  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:27.919154  1802 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:27.946594  1697 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:27.895120  3464 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:27.876600  3552 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:27.894733  3576 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:27.941406  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:27.951436  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45918us
I20260709 12:22:27.955128  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:27.957600  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:27.955113  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:22:27.959828  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 165691us
I20260709 12:22:27.967267  1799 raft_consensus.cc:2368] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3.
I20260709 12:22:27.929574  3381 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:27.973409  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52828.3us
I20260709 12:22:27.966113  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:27.974905  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:22:27.945446  3364 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.782s	user 0.006s	sys 0.012s
I20260709 12:22:27.973397  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:27.980729  1806 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:27.975979  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:27.928615  3540 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:27.954738  3464 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.967100  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:27.896983  3397 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260709 12:22:27.953994  2411 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:28.028615  3394 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:28.037091  3186 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:27.937847  3466 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 12:22:27.978051  1801 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:28.030721  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 89196us
I20260709 12:22:28.061810  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:28.062824  1798 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:27.941111  3467 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.090s
I20260709 12:22:27.895758  3392 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:27.944978  3468 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:28.044704  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 60837.3us
I20260709 12:22:28.044822  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:28.046623  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:27.973978  3381 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:27.987694  3364 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.006s	sys 0.012s
I20260709 12:22:27.975543  3538 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:28.058162  3186 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.051625  3569 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:28.068507  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:28.107967  3186 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:28.108421  1804 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:28.068641  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:28.068850  3589 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:28.069518  3552 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:28.073449  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:28.028977  3424 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:28.031375  3591 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:28.106521  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:28.074864  3392 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:28.078742  1727 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:28.032024  3407 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.073s
I20260709 12:22:28.156183  1799 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:28.110913  3433 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:28.049615  3464 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:28.099421  2413 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 1 LEARNER]: Advancing to term 2
I20260709 12:22:28.173357  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:28.224237  2417 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:28.180320  3422 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.181748  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:28.182379  3498 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:28.237308  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.054075  3576 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.157s
I20260709 12:22:28.102216  3587 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:28.216058  3476 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:28.105959  3381 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:28.220762  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 183658us
I20260709 12:22:28.146562  3589 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.179657  3334 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.224745  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:28.107658  3538 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:27.981284  3539 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:28.332998  1877 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:28.238749  3186 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:28.239333  1726 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:28.204526  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:28.339619  1804 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:28.341517  1804 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:22:28.260866  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 154422us
I20260709 12:22:28.187753  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:28.230383  3586 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.518s	user 0.003s	sys 0.000s
I20260709 12:22:28.204371  3449 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:28.343443  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:22:28.342916  3401 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.531s	user 0.015s	sys 0.000s
I20260709 12:22:28.352454  3401 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:28.301563  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:28.347080  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:28.255209  3394 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:28.347894  3593 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:28.296275  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:28.345118  3449 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:28.312477  3450 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.590s	user 0.012s	sys 0.004s
I20260709 12:22:28.230535  3363 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:28.303100  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:28.297430  3476 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:28.254937  3606 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:28.297392  3397 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:28.296207  3489 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.368387  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 508) took 1129 ms. Trace:
I20260709 12:22:28.369477  3363 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:28.318526  3589 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:28.371518  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:28.371807  3186 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:28.337929  3538 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:28.376587  3604 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:22:28.359738  3593 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:28.359791  3591 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:28.305678  3422 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:28.379930  3450 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:28.403211  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47847.3us
I20260709 12:22:28.377295  3606 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.357156  3334 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:28.414235  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:28.421355  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:28.419173  3595 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.153s	user 0.006s	sys 0.000s
W20260709 12:22:28.433164  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 467152us
W20260709 12:22:28.422719  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 51374.2us
I20260709 12:22:28.429073  1877 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:28.431051  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 475995us
I20260709 12:22:28.435348  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:28.395776  3489 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:22:28.424214  3601 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 12:22:28.436281  2417 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:28.412043  3593 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:28.436746  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:28.436389  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:28.414688  1874 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:28.415138  3424 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.412982  3575 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:28.458998  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 153980us
W20260709 12:22:28.465760  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 284103us
I20260709 12:22:28.467096  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:28.427166  3540 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:28.479817  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:28.495707  1876 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:28.429581  3606 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:28.457582  3401 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:28.415303  3397 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:28.493029  3363 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:28.536335  2417 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:28.513280  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78966.1us
I20260709 12:22:28.514887  3425 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.437296  3476 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:28.529488  3433 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:28.491238  3424 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:28.549676  3602 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 12:22:28.537827  3613 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.640376  1804 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:22:28.631328  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 164354us
I20260709 12:22:28.548043  3609 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
W20260709 12:22:28.629093  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 165105us
I20260709 12:22:28.553619  3450 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.503621  3359 raft_consensus.cc:359] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:22:28.647888  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 211215us
I20260709 12:22:28.552865  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:28.651859  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:28.642741  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:28.669723  1806 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.632s	user 0.006s	sys 0.002s
I20260709 12:22:28.671684  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1253) took 1013 ms. Trace:
W20260709 12:22:28.664132  1798 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.000s
I20260709 12:22:28.676266  1798 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:28.673521  1806 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:28.675362  1800 rpcz_store.cc:276] 0709 12:22:27.658292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.183645 (+525353us) service_pool.cc:224] Handling call
0709 12:22:28.670067 (+486422us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:28.670347 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:28.670639 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.670669 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:28.670713 (+    44us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:28.670746 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.671429 (+   683us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.671596 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17279872}
I20260709 12:22:28.677523  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1256) took 1018 ms. Trace:
I20260709 12:22:28.680574  1798 rpcz_store.cc:276] 0709 12:22:27.659385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.061394 (+402009us) service_pool.cc:224] Handling call
0709 12:22:28.677453 (+616059us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:28.684576  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 250521us
I20260709 12:22:28.685768  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:28.686098  1873 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:22:28.665340  3599 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:22:28.688776  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:28.651269  3359 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:22:28.695684  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:28.696497  3456 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.696369  3359 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:28.612708  3425 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:28.697297  3456 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:28.699208  3425 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:28.699932  3425 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.700407  3425 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:28.671557  3582 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.863s	user 0.005s	sys 0.010s
I20260709 12:22:28.706238  3582 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:28.712710  3561 raft_consensus.cc:2764] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:22:28.620828  3611 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260709 12:22:28.658973  3613 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:28.666409  3568 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:28.639936  3433 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:28.560082  3464 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:22:28.665226  3608 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:22:28.695732  1807 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:28.689987  3438 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:28.696394  1800 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:28.737998  1730 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:28.740847  1877 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:28.736670  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:28.744855  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 195777us
I20260709 12:22:28.681058  3618 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:28.739703  3468 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.634s	user 0.008s	sys 0.006s
I20260709 12:22:28.657089  3450 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:28.750526  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 297479us
I20260709 12:22:28.750438  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:28.761719  3186 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:28.806257  1798 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:28.806555  3186 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:28.748247  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1252) took 1090 ms. Trace:
I20260709 12:22:28.799813  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:28.778479  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:28.811753  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:28.807866  1807 rpcz_store.cc:276] 0709 12:22:27.657966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.180514 (+522548us) service_pool.cc:224] Handling call
0709 12:22:28.671164 (+490650us) spinlock_profiling.cc:243] Waited 7.23 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:28.674713 (+  3549us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:28.744859 (+ 70146us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.748153 (+  3294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16618752}
W20260709 12:22:28.822731  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 147516us
I20260709 12:22:28.824302  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:28.798454  3441 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:28.835026  3186 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:28.836551  3561 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:28.837173  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:28.778692  3422 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:28.792045  3334 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:28.803002  3618 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.843613  1804 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:28.836621  3456 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:22:28.844776  3615 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260709 12:22:28.780042  3609 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260709 12:22:28.770828  3438 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:28.840058  3561 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:28.871685  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 213102us
I20260709 12:22:28.842248  3425 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:28.876075  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 129486us
I20260709 12:22:28.813427  3468 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:28.838130  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1450) took 1557 ms. Trace:
I20260709 12:22:28.883878  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:28.887315  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:28.888597  3613 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:28.884781  1804 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:28.863595  3618 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:28.862803  3582 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:28.840509  3441 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:28.872572  3456 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:28.878753  1874 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:22:28.931259  3629 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:22:28.932642  3468 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:28.933549  3613 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:28.938014  3397 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:28.879961  3424 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:28.939721  3626 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:22:28.954914  3359 ts_tablet_manager.cc:1434] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 1.091s	user 0.004s	sys 0.011s
W20260709 12:22:28.955346  3359 ts_tablet_manager.cc:1467] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Tablet startup took 1091ms
I20260709 12:22:28.939860  3450 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:28.940825  3614 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:22:28.926173  3623 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.876839  3561 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:28.942817  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 291000us
I20260709 12:22:28.943826  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:28.950281  3464 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:28.938434  3438 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:28.950915  3613 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.952548  3610 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:28.958058  1872 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 4
W20260709 12:22:28.969827  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86204.6us
W20260709 12:22:28.971740  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84524.9us
I20260709 12:22:28.941362  3401 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:29.013844  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:28.974428  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:28.883304  3616 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:29.023500  3575 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:29.024124  3450 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:29.024441  1807 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:29.024595  3450 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.025035  3450 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:28.973665  3433 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:28.986140  3488 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.027s	user 0.009s	sys 0.003s
I20260709 12:22:28.988659  3571 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.019966  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-def0036446324543ba07e132e885133a received from {username='slave'} at 127.0.0.1:42636
I20260709 12:22:28.995332  3489 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.020181  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:28.986480  3483 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:29.021056  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 77330.9us
I20260709 12:22:29.015293  3613 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:28.883325  1875 rpcz_store.cc:276] 0709 12:22:27.281052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.551904 (+270852us) service_pool.cc:224] Handling call
0709 12:22:27.640657 (+ 88753us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.700578 (+ 59921us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:27.700617 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:27.703059 (+  2442us) log.cc:844] Serialized 26 byte log entry
0709 12:22:27.948896 (+245837us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:27.951045 (+  2149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.972871 (+ 21826us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:28.816458 (+843587us) raft_consensus.cc:1685] finished
0709 12:22:28.818905 (+  2447us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.833912 (+ 15007us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4257,"raft.queue_time_us":931,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":29606,"spinlock_wait_cycles":963328,"thread_start_us":205706,"threads_started":1,"wal-append.queue_time_us":736035}
W20260709 12:22:28.973994  3381 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.669s	user 0.011s	sys 0.003s
I20260709 12:22:28.949196  3456 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:28.973397  3394 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.611s	user 0.012s	sys 0.001s
I20260709 12:22:28.968504  3606 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:28.991430  3627 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:22:29.045305  3483 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:29.063745  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.065248  3394 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:29.036705  1872 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:29.048187  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:28.991686  3500 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:29.038688  3593 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.587s	user 0.013s	sys 0.000s
W20260709 12:22:29.044538  2413 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.802s	user 0.014s	sys 0.000s
I20260709 12:22:29.031092  3466 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.026472  3623 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.037856  3488 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.062041  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-def0036446324543ba07e132e885133a on tablet def0036446324543ba07e132e885133a with peer 123095127f334445889c21a42ba1d41e
I20260709 12:22:29.069864  1879 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:22:29.065059  3589 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.651s	user 0.011s	sys 0.004s
I20260709 12:22:29.083653  3456 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.086128  3488 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.093009  3456 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:29.086915  1799 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:29.090472  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.083472  3466 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.097003  3466 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:29.102636  1804 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:29.111649  3441 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:29.086941  3633 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:29.119884  3352 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:29.125087  3623 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.134529  3363 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:29.135282  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 135068us
I20260709 12:22:29.139183  3610 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:29.135388  3363 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:29.142951  3449 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.150379  3468 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:29.151341  3468 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.152875  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.153879  1803 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:29.156075  3610 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:29.157326  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.156958  3610 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.161615  3610 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:29.162768  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:29.168612  2411 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:22:29.164657  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:29.171624  2412 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:29.173595  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:28.392119  2410 rpcz_store.cc:276] 0709 12:22:27.239345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.239509 (+   164us) service_pool.cc:224] Handling call
0709 12:22:27.239949 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.547075 (+307126us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:27.550012 (+  2937us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:27.634010 (+ 83998us) log.cc:844] Serialized 26 byte log entry
0709 12:22:27.850595 (+216585us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:28.363581 (+512986us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.367990 (+  4409us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.368080 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:28.368128 (+    48us) raft_consensus.cc:1685] finished
0709 12:22:28.368145 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.368297 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":612,"prepare.queue_time_us":541106,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":2979,"raft.queue_time_us":179330,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":9810,"thread_start_us":563849,"threads_started":2,"wal-append.queue_time_us":122720}
I20260709 12:22:29.075875  3394 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:29.059492  3381 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:29.180723  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2173) took 1567 ms (client timeout 1573 ms). Trace:
I20260709 12:22:29.181123  1802 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:29.182139  3381 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:29.181154  1730 rpcz_store.cc:269] 0709 12:22:27.612894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.613211 (+   317us) service_pool.cc:224] Handling call
0709 12:22:29.180464 (+1567253us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540044af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:29.180620 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2517108096}
I20260709 12:22:29.185189  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.186007  1730 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:29.154413  3468 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.165411  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:29.201752  2417 raft_consensus.cc:2368] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
W20260709 12:22:29.201807  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 516151us
I20260709 12:22:29.186757  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.202718  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:29.202916  1798 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
I20260709 12:22:29.203882  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.203994  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:29.204612  1798 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:29.205668  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:29.158403  1805 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:29.192202  2411 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:29.154528  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1276) took 1320 ms. Trace:
I20260709 12:22:29.162508  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1303) took 1498 ms. Trace:
I20260709 12:22:29.200002  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.200919  1845 leader_election.cc:400] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3.
I20260709 12:22:29.214035  1845 leader_election.cc:403] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:22:29.215164  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1270) took 1452 ms. Trace:
I20260709 12:22:29.215431  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:29.215986  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:29.215682  1805 rpcz_store.cc:276] 0709 12:22:27.755100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.844328 (+ 89228us) service_pool.cc:224] Handling call
0709 12:22:29.157091 (+1312763us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:22:29.207012 (+ 49921us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3543424}
I20260709 12:22:29.217605  1805 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:22:29.130347  3540 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:29.218146  1805 pending_rounds.cc:85] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 2
I20260709 12:22:29.218500  3540 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:29.219084  3540 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:29.218705  1805 pending_rounds.cc:107] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:22:29.219762  1805 raft_consensus.cc:2889] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:29.220597  3538 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.797s	user 0.006s	sys 0.007s
I20260709 12:22:29.221097  3538 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.080668  3593 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.227253  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:29.211122  1803 rpcz_store.cc:276] 0709 12:22:27.833653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.168210 (+334557us) service_pool.cc:224] Handling call
0709 12:22:29.152622 (+984412us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:22:29.154447 (+  1825us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44382464}
I20260709 12:22:29.219996  3610 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:29.091543  3588 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.191s	user 0.012s	sys 0.002s
I20260709 12:22:29.232622  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.233039  3588 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:29.233498  1730 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 1.
I20260709 12:22:29.233930  3468 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:29.233726  3363 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.234400  3468 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:29.235283  3468 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:29.236272  3364 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:29.236594  2412 raft_consensus.cc:2368] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:29.234535  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.237634  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:29.238725  2418 raft_consensus.cc:2368] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:29.242331  1728 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:29.238233  1730 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:29.242985  2411 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:29.246160  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:29.247123  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1448) took 2031 ms (client timeout 1973 ms). Trace:
I20260709 12:22:29.250384  1844 leader_election.cc:400] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:29.217442  3476 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:29.232662  3593 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:29.156536  3563 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.247141  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1310) took 1538 ms. Trace:
W20260709 12:22:29.247634  1878 rpcz_store.cc:269] 0709 12:22:27.215520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.233581 (+ 18061us) service_pool.cc:224] Handling call
0709 12:22:29.246906 (+2013325us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b5400051754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:29.247013 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":597718400}
W20260709 12:22:29.252724  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 178818us
I20260709 12:22:29.259104  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:29.212316  1731 rpcz_store.cc:276] 0709 12:22:27.663840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.663985 (+   145us) service_pool.cc:224] Handling call
0709 12:22:28.419626 (+755641us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:28.419874 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:28.552284 (+132410us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:28.638108 (+ 85824us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.804652 (+166544us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:28.804731 (+    79us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:28.804805 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.162114 (+357309us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54002b75d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:29.162188 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.162408 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":86326,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":124103,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":9075,"spinlock_wait_cycles":317681792,"thread_start_us":166095,"threads_started":1,"wal-append.queue_time_us":284982}
I20260709 12:22:29.256261  1732 rpcz_store.cc:276] 0709 12:22:27.708272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.739069 (+ 30797us) service_pool.cc:224] Handling call
0709 12:22:29.246931 (+1507862us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54001905d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:29.247067 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2866129152}
I20260709 12:22:29.256417  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:29.257161  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 478720us
I20260709 12:22:29.265740  2383 leader_election.cc:400] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5.
I20260709 12:22:29.266268  2383 leader_election.cc:403] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:29.265749  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.266942  1732 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:29.250885  1844 leader_election.cc:403] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:22:29.270145  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:29.272390  1800 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:22:29.273811  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.282334  3621 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5.
I20260709 12:22:29.284789  3394 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:29.289986  3575 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:29.298251  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1254) took 1618 ms. Trace:
I20260709 12:22:29.299214  1801 rpcz_store.cc:276] 0709 12:22:27.658626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.210134 (+551508us) service_pool.cc:224] Handling call
0709 12:22:28.356109 (+145975us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.171098 (+814989us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.171133 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.171183 (+    50us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:29.171220 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.257279 (+ 86059us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x7b5400166cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:29.261689 (+  4410us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.277221 (+ 15532us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1866447616}
I20260709 12:22:29.301223  3618 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.302137  1801 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:29.255813  3449 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:29.303558  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1255) took 1644 ms. Trace:
I20260709 12:22:29.252390  3616 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.304468  3616 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.303107  3449 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.304045  1799 rpcz_store.cc:276] 0709 12:22:27.659040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.211409 (+552369us) service_pool.cc:224] Handling call
0709 12:22:28.231402 (+ 19993us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.093968 (+862566us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.303252 (+209284us) spinlock_profiling.cc:243] Waited 855 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:29.303480 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1966710016}
I20260709 12:22:29.311848  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.312409  3593 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:29.187764  3589 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.315019  3538 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.273296  1800 pending_rounds.cc:85] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 5
I20260709 12:22:29.325685  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.334833  1800 pending_rounds.cc:107] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:22:29.335309  1800 raft_consensus.cc:2889] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:29.335217  1697 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:29.313179  1729 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:29.349846  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:29.353272  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83195.2us
I20260709 12:22:29.357812  1732 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:29.262042  3606 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.362381  3186 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:29.255990  3364 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3.
I20260709 12:22:29.270118  3401 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:29.362746  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:29.361472  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:29.351824  3539 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.067s	sys 0.069s
I20260709 12:22:29.316704  3589 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.365938  3624 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.369426  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:29.362865  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.371905  3186 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:29.386765  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:29.388497  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:29.395632  1730 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:29.364255  3538 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.368997  3407 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:29.418800  3186 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.439930  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.440191  3186 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:29.266862  3622 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.119s
I20260709 12:22:29.440759  1730 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:29.441840  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 106730us
I20260709 12:22:29.443061  3441 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:29.418727  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49396.7us
I20260709 12:22:29.375809  3401 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:29.476575  3642 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:29.483786  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:29.391721  2410 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:29.366603  3575 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260709 12:22:29.495981  2418 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:22:29.491940  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49155.3us
W20260709 12:22:29.500016  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54941.5us
I20260709 12:22:29.417181  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:29.447083  3609 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:29.387431  3561 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.540838  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:29.541944  3186 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:29.542949  3628 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:22:29.484367  3407 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.564515  3407 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:29.566614  3642 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:29.561053  3609 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.567857  3642 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.501766  3441 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:29.554037  3616 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:28.955749  3359 ts_tablet_manager.cc:1469] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Trace:
0709 12:22:17.689616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.689758 (+   142us) service_pool.cc:224] Handling call
0709 12:22:18.892557 (+1202799us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b5800003f98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb41a2 00007f16b1bf5889 00007f16b0d26d88 00007f16b0deaa07 00007f16b0e1cf4e 00007f16b0e01fdd 00007f16b0e01f89 00007f16b0e01f19 00007f16b0e01ee1
0709 12:22:18.925481 (+ 32924us) ts_tablet_manager.cc:934] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
0709 12:22:24.326962 (+5401481us) tablet_copy_client.cc:368] Tablet copy session begun
0709 12:22:24.749725 (+422763us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:24.749843 (+   118us) tablet_copy_client.cc:497] Wrote new tablet metadata
0709 12:22:25.104624 (+354781us) tablet_copy_client.cc:418] Wrote new consensus metadata
0709 12:22:25.104924 (+   300us) tablet_replica.cc:206] Creating consensus instance
0709 12:22:25.105222 (+   298us) ts_tablet_manager.cc:989] Replica registered.
0709 12:22:25.111513 (+  6291us) inbound_call.cc:177] Queueing success response
0709 12:22:25.819038 (+707525us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0709 12:22:27.863975 (+2044937us) ts_tablet_manager.cc:1435] Starting tablet replica
0709 12:22:28.031270 (+167295us) tablet_replica.cc:261] Starting instrumentation
0709 12:22:28.074667 (+ 43397us) tablet_replica.cc:280] Starting consensus
Metrics: {"dns_us":86837,"mutex_wait_us":284773,"raft.queue_time_us":257623,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":114,"spinlock_wait_cycles":43268352,"tablet-copy.queue_time_us":4904,"tablet-download-pool.queue_time_us":518,"tablet-download-pool.run_cpu_time_us":1844,"tablet-download-pool.run_wall_time_us":114711,"thread_start_us":104239,"threads_started":2}
I20260709 12:22:29.441143  3538 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.560261  1730 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
W20260709 12:22:29.557690  3639 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:22:29.596935  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.566900  3424 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:29.485967  3401 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:29.559129  2411 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:29.396312  3636 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.168s
I20260709 12:22:29.482398  3364 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:29.569362  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1464) took 1014 ms. Trace:
I20260709 12:22:29.587411  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:29.602914  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:29.560747  3483 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.594688  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:29.482007  3624 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:29.563886  3613 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.507s	user 0.004s	sys 0.007s
I20260709 12:22:29.608058  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:29.597954  1874 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.013s	sys 0.018s
I20260709 12:22:29.602176  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.602617  1879 rpcz_store.cc:276] 0709 12:22:28.555368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.555612 (+   244us) service_pool.cc:224] Handling call
0709 12:22:28.616854 (+ 61242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.083212 (+466358us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.568971 (+485759us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b5400118354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:29.569262 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1032634112}
I20260709 12:22:29.608688  1806 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:29.605854  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.561133  3186 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.610064  1730 raft_consensus.cc:2368] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5.
I20260709 12:22:29.608879  1804 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:29.611195  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.611761  1800 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:29.612430  1874 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.013s	sys 0.018s
I20260709 12:22:29.634835  2416 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:29.636934  1876 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:29.672405  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.674319  1804 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:29.675091  1800 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:29.635401  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.676288  1730 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 1.
W20260709 12:22:29.607473  3576 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.264s	user 0.005s	sys 0.028s
I20260709 12:22:29.681104  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:29.612715  3618 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.611384  3364 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:29.689847  1879 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:29.632426  3397 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.701699  3630 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:29.702526  3630 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:29.703192  3441 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.677719  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:29.707044  3468 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.708894  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:29.710189  3468 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.612367  3424 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.713191  3424 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:29.720968  3363 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:29.725261  3450 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.700s	user 0.011s	sys 0.001s
I20260709 12:22:29.726125  3450 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.729257  3624 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:29.732769  3561 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:29.632704  3590 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:29.737754  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.739053  1806 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 12:22:29.687461  3576 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.347s	user 0.006s	sys 0.028s
W20260709 12:22:29.687673  1727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.052s	sys 0.079s
I20260709 12:22:29.638346  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:29.753110  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:29.753880  1805 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.014s	sys 0.036s
I20260709 12:22:29.749317  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:29.747165  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 553143us
I20260709 12:22:29.748659  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:29.758109  1804 raft_consensus.cc:2368] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5.
I20260709 12:22:29.623972  3613 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:29.756628  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:29.734870  3552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.989s	user 0.006s	sys 0.045s
I20260709 12:22:29.763244  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:29.695716  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:29.764817  1805 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.014s	sys 0.036s
I20260709 12:22:29.766760  3612 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:29.769961  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 74394.4us
I20260709 12:22:29.771061  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:29.751499  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:29.781905  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:29.784209  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.796388  1730 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:29.789363  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1455) took 1575 ms. Trace:
I20260709 12:22:29.795857  3364 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.800098  1880 rpcz_store.cc:276] 0709 12:22:28.213499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.213764 (+   265us) service_pool.cc:224] Handling call
0709 12:22:28.296266 (+ 82502us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.343502 (+ 47236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.344085 (+   583us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.351705 (+  7620us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.132959 (+781254us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x7b180001ccc0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:22:29.594892 (+461933us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:29.594942 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.595027 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.788987 (+193960us) raft_consensus.cc:1685] finished
0709 12:22:29.789016 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.789285 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12335,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":5625,"spinlock_wait_cycles":16775424,"thread_start_us":461637,"threads_started":1,"wal-append.queue_time_us":1435104}
I20260709 12:22:29.791762  1876 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:29.809063  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:29.771034  3552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.027s	user 0.007s	sys 0.045s
I20260709 12:22:29.817921  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:29.819257  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:29.820151  1799 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:29.747499  3590 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:29.747417  3397 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.817536  1801 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:29.837205  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.795791  3538 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.821749  3624 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:29.832600  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:29.806304  3630 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.820688  3561 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:29.847383  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 452031us
W20260709 12:22:29.847096  3540 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.002s
I20260709 12:22:29.816380  3612 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:29.820132  3609 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:29.820746  3588 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:29.822398  3618 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:29.779699  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1469) took 1014 ms. Trace:
W20260709 12:22:29.822834  3476 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.572s	user 0.008s	sys 0.004s
I20260709 12:22:29.913551  3483 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:29.915257  3450 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:29.915401  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.916297  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1459) took 1521 ms. Trace:
I20260709 12:22:29.850912  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1268) took 2124 ms. Trace:
I20260709 12:22:29.917548  1802 rpcz_store.cc:276] 0709 12:22:27.726039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.053751 (+327712us) service_pool.cc:224] Handling call
0709 12:22:29.841743 (+1787992us) spinlock_profiling.cc:243] Waited 837 ms on lock 0x7b54001d4a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:29.847150 (+  5407us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1924651904}
I20260709 12:22:29.821979  3539 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:22:29.824043  3438 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
W20260709 12:22:29.840126  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.011s	sys 0.008s
W20260709 12:22:29.921211  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.011s	sys 0.008s
W20260709 12:22:29.921893  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 662858us
I20260709 12:22:29.922995  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.923357  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:29.924536  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:29.924885  3613 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:29.925484  1723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2164) took 2315 ms (client timeout 1682 ms). Trace:
W20260709 12:22:29.866261  3394 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.581s	user 0.013s	sys 0.000s
I20260709 12:22:29.869040  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.901129  1730 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:29.848186  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.852396  3561 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.902864  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.938584  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.911885  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.910327  1874 rpcz_store.cc:276] 0709 12:22:28.765205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.032153 (+266948us) service_pool.cc:224] Handling call
0709 12:22:29.065453 (+ 33300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.065687 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:29.066044 (+   357us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.694341 (+628297us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.697441 (+  3100us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:29.745822 (+ 48381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.752256 (+  6434us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.779612 (+ 27356us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":214,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":37846,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":574524,"threads_started":1,"wal-append.queue_time_us":700415}
I20260709 12:22:29.935658  3394 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:29.916945  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:29.845870  3624 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.941478  3394 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:29.961798  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.962579  1724 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 5
W20260709 12:22:29.925915  1723 rpcz_store.cc:269] 0709 12:22:27.609783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.632552 (+ 22769us) service_pool.cc:224] Handling call
0709 12:22:29.924233 (+2291681us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b5400360ea0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abddbe18 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:29.925240 (+  1007us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54002a9c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:29.925376 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4215638144}
I20260709 12:22:29.934947  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.939935  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:29.916709  1881 rpcz_store.cc:276] 0709 12:22:28.394917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.395132 (+   215us) service_pool.cc:224] Handling call
0709 12:22:28.416609 (+ 21477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:28.425650 (+  9041us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:28.425688 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.432160 (+  6472us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.135951 (+703791us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b180001cae0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:22:29.616861 (+480910us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:29.636833 (+ 19972us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.637161 (+   328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.637248 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.909270 (+272022us) raft_consensus.cc:1685] finished
0709 12:22:29.916046 (+  6776us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.916210 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3465,"prepare.queue_time_us":19460,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":3835,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":48266,"spinlock_wait_cycles":20512896,"thread_start_us":472942,"threads_started":1,"wal-append.queue_time_us":1369003}
I20260709 12:22:29.939390  1731 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:29.967777  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:29.969129  2417 raft_consensus.cc:2410] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:29.909507  3618 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.916311  1728 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:29.846576  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:29.936427  3363 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:29.866945  3540 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:29.935979  3352 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:22:29.979216  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 136401us
I20260709 12:22:29.985992  1876 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:29.988217  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:29.995424  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:29.996234  1730 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:29.991942  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:29.983608  3363 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.999974  1723 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:29.942281  3561 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:29.996546  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:30.014132  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:29.941944  3624 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:29.987433  3612 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:30.011140  1724 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:30.096589  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:30.097219  1724 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:30.019277  1696 leader_election.cc:400] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5.
W20260709 12:22:30.027168  3652 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 12:22:30.030100  3556 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:30.032367  1723 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:30.012554  3438 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:30.063776  3397 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:30.064987  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:30.066358  3590 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:30.001803  3352 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:30.162772  3352 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:30.069682  3540 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:30.071844  3394 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:30.013918  3363 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:30.080552  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:30.081784  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1376) took 1276 ms. Trace:
I20260709 12:22:30.179221  1807 rpcz_store.cc:276] 0709 12:22:28.809488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.818947 (+  9459us) service_pool.cc:224] Handling call
0709 12:22:30.081483 (+1262536us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001eb4d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:30.081688 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423194880}
I20260709 12:22:30.084473  3575 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:30.094578  2414 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:30.099138  1696 leader_election.cc:403] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:22:30.188328  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 169727us
I20260709 12:22:30.108729  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:30.110646  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:22:30.190972  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 80477.9us
I20260709 12:22:30.191897  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:30.112246  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:30.113636  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1375) took 1321 ms. Trace:
I20260709 12:22:30.129701  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:30.135452  1730 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:22:30.138092  1872 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:30.142130  3468 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:22:30.201508  3449 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:30.192552  1727 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:30.144177  2382 leader_election.cc:400] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5.
W20260709 12:22:30.206743  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 109166us
I20260709 12:22:30.207623  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:29.986545  3618 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:30.126953  3476 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:30.127403  3483 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.142542  3438 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:30.218487  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:30.159868  1731 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:30.174388  3450 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:30.219740  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:30.128326  3538 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:30.194800  1798 rpcz_store.cc:276] 0709 12:22:28.792579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.246348 (+453769us) service_pool.cc:224] Handling call
0709 12:22:29.246763 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.253342 (+  6579us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.253392 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.253996 (+   604us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.773733 (+519737us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:29.984032 (+210299us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.986055 (+  2023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.986313 (+   258us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.113309 (+126996us) raft_consensus.cc:1685] finished
0709 12:22:30.113343 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.113557 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6091,"raft.queue_time_us":7252,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":23641,"thread_start_us":672703,"threads_started":2,"wal-append.queue_time_us":719550}
I20260709 12:22:30.206386  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:30.222077  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2192) took 1026 ms. Trace:
I20260709 12:22:30.223908  1731 rpcz_store.cc:276] 0709 12:22:29.195743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.911521 (+715778us) service_pool.cc:224] Handling call
0709 12:22:30.221995 (+310474us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.195427  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:30.096028  3612 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:30.146013  3630 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260709 12:22:30.148631  3556 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.236404  3540 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:30.247133  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:30.193176  3576 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:30.251408  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:30.255685  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:30.246579  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:30.253834  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:30.251073  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:30.205332  3616 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:30.195904  3590 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5.
I20260709 12:22:30.270880  3540 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.239874  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:30.275054  1804 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:30.266273  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1374) took 1661 ms. Trace:
I20260709 12:22:30.282902  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:30.252594  3401 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.645s	user 0.012s	sys 0.000s
I20260709 12:22:30.195808  3394 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:30.294981  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.311946  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:30.232401  3538 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:30.330034  2418 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:30.267542  1802 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:30.235616  3467 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:30.231827  3397 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.098s
I20260709 12:22:30.283694  2419 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:30.285054  1725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2178) took 2383 ms (client timeout 2250 ms). Trace:
W20260709 12:22:30.375941  1725 rpcz_store.cc:269] 0709 12:22:27.901197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.938295 (+ 37098us) service_pool.cc:224] Handling call
0709 12:22:30.284774 (+2346479us) spinlock_profiling.cc:243] Waited 1.88 s on lock 0x7b54002d14d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:30.284947 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4313421568}
I20260709 12:22:30.258980  3450 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.335501  3552 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:22:30.377908  3645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.002s	sys 0.028s
W20260709 12:22:30.263677  3407 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.699s	user 0.003s	sys 0.010s
I20260709 12:22:30.247946  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:30.259889  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 395) took 2313 ms. Trace:
W20260709 12:22:30.333134  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50357us
I20260709 12:22:30.334059  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:30.312304  1798 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:22:30.231884  3438 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:30.247474  3649 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.236588  3612 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:30.311111  1803 rpcz_store.cc:276] 0709 12:22:28.605105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.230141 (+625036us) service_pool.cc:224] Handling call
0709 12:22:29.230545 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.231039 (+   494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.231082 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.264510 (+ 33428us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.856733 (+592223us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:29.856775 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.856841 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.265986 (+409145us) raft_consensus.cc:1685] finished
0709 12:22:30.266014 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.266192 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7211,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":8322,"thread_start_us":492617,"threads_started":1,"wal-append.queue_time_us":939295}
I20260709 12:22:30.216688  3468 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:30.325523  3469 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.003s	sys 0.021s
I20260709 12:22:30.333357  3538 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:30.355230  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:30.236028  3483 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.313818  3582 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:30.356362  3352 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.329344  3576 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:30.496031  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:30.361382  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:30.365561  3561 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:30.255949  3449 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.081s
I20260709 12:22:30.387751  1697 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:22:30.323539  3186 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.004s	sys 0.016s
W20260709 12:22:30.535293  3186 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.004s	sys 0.017s
W20260709 12:22:30.324546  1805 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.013s	sys 0.011s
W20260709 12:22:30.536100  1805 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.014s	sys 0.011s
I20260709 12:22:30.517905  1798 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:30.325057  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.070s	user 0.011s	sys 0.014s
I20260709 12:22:30.353332  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 528) took 1119 ms. Trace:
I20260709 12:22:30.438234  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:30.330471  3658 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:30.441326  1800 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.007s	sys 0.003s
I20260709 12:22:30.490633  3613 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:30.547299  3561 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:22:30.547452  3613 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:30.548444  3561 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.548998  3561 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.548673  3613 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.549971  3618 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:30.494237  3640 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.763s	user 0.000s	sys 0.005s
I20260709 12:22:30.373603  3467 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:30.551767  3467 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:30.314355  3540 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.385720  3569 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:30.332391  3401 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.555116  2417 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:30.439772  3645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.002s	sys 0.029s
I20260709 12:22:30.457193  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:30.449947  3407 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.563509  1725 raft_consensus.cc:2368] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3.
I20260709 12:22:30.564203  3483 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:30.565348  3561 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:30.458565  3441 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.011s	sys 0.015s
I20260709 12:22:30.351164  3616 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:30.538379  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.284s	user 0.011s	sys 0.014s
I20260709 12:22:30.545647  3539 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:30.541594  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 153992us
I20260709 12:22:30.566252  3616 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:30.566603  3539 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:30.566965  3616 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:30.567142  3539 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:30.553866  1730 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:30.360406  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:30.553872  1800 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.008s	sys 0.003s
I20260709 12:22:30.449586  1731 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:30.556802  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 99708.5us
I20260709 12:22:30.542590  2414 rpcz_store.cc:276] 0709 12:22:29.233743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.242139 (+  8396us) service_pool.cc:224] Handling call
0709 12:22:29.315985 (+ 73846us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.202886 (+886901us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.312596 (+109710us) spinlock_profiling.cc:243] Waited 778 ms on lock 0x7b5400561b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:30.335583 (+ 22987us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1789825920}
I20260709 12:22:30.564298  3407 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:30.550966  3658 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.550422  3613 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:30.573187  3352 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:30.577912  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:30.576156  3186 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:30.579336  3658 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:30.566321  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 1.36715e+06us
I20260709 12:22:30.583246  3540 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:30.582994  3587 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:30.583875  3576 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:30.584123  3587 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:30.585819  3628 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:30.585386  3576 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.555512  3569 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:30.586294  3628 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:30.477067  3469 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.914s	user 0.003s	sys 0.022s
I20260709 12:22:30.587896  3401 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:30.594391  3628 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.595678  3450 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:30.596088  3628 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:30.596539  3450 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.597114  3450 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.542068  3556 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:30.602003  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.026s	sys 0.037s
W20260709 12:22:30.603762  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.026s	sys 0.038s
W20260709 12:22:30.605875  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 611074us
I20260709 12:22:30.599623  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:22:30.566056  3441 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.012s	sys 0.016s
I20260709 12:22:30.611229  1846 leader_election.cc:400] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:30.611763  1846 leader_election.cc:403] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:22:30.609555  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:30.587944  3569 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.458276  3649 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.489711  3659 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.124s
I20260709 12:22:30.616598  1844 leader_election.cc:400] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3.
I20260709 12:22:30.644953  1844 leader_election.cc:403] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:22:30.643776  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:30.619184  1846 leader_election.cc:400] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:30.645589  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:30.562397  3538 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.645795  1846 leader_election.cc:403] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:30.618543  2417 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:30.476830  3468 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:30.642923  3364 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.038s	sys 0.062s
W20260709 12:22:30.671398  3364 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.767s	user 0.038s	sys 0.062s
I20260709 12:22:30.654716  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:30.672101  3364 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:30.673142  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:30.673583  3649 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:30.675082  3397 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:30.634688  3450 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:30.448495  2413 rpcz_store.cc:276] 0709 12:22:27.946724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.948640 (+  1916us) service_pool.cc:224] Handling call
0709 12:22:28.054605 (+105965us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.235805 (+1181200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.235970 (+   165us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.236528 (+   558us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.696356 (+459828us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:29.736737 (+ 40381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.747730 (+ 10993us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.259487 (+511757us) raft_consensus.cc:1685] finished
0709 12:22:30.259540 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.259754 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13945,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"thread_start_us":436994,"threads_started":1,"wal-append.queue_time_us":607056}
I20260709 12:22:30.616218  3588 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:22:30.679491  3449 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:30.680294  3449 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.647572  3561 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:30.653079  3352 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:30.662519  3612 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:30.635197  1802 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:30.629590  3618 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:30.663434  3441 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:30.711072  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.019s	sys 0.035s
I20260709 12:22:30.664116  3659 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:30.662719  3539 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:30.664783  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48395.7us
I20260709 12:22:30.665261  3467 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.717715  3467 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:30.665967  1729 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:30.666999  3569 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:30.667444  3538 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:30.675640  3671 raft_consensus.cc:1064] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:30.674561  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:30.642022  3658 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:30.476336  3394 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.647012  3582 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:30.679553  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 60574us
W20260709 12:22:30.707870  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.635711  3624 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.682994  3630 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.633011  3438 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:30.687651  3642 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.957s	user 0.035s	sys 0.060s
I20260709 12:22:30.728371  3364 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3.
I20260709 12:22:30.641783  3556 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260709 12:22:30.636734  3363 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.753252  3363 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:30.723726  3659 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:30.733736  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.000s	sys 0.030s
W20260709 12:22:30.762033  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.000s	sys 0.030s
I20260709 12:22:30.763892  3612 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:30.764777  3449 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:30.733983  3433 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.815s	user 0.002s	sys 0.029s
W20260709 12:22:30.765781  3433 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.847s	user 0.002s	sys 0.029s
I20260709 12:22:30.758095  3630 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.583801  3616 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.688335  3468 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:30.767359  3468 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.767959  3468 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.769832  3467 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:22:30.736663  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:30.737164  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62683.5us
I20260709 12:22:30.771490  3624 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:30.756453  3642 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.026s	user 0.036s	sys 0.061s
I20260709 12:22:30.773182  3642 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:22:30.745431  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.019s	sys 0.035s
I20260709 12:22:30.750581  1846 leader_election.cc:400] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:30.739792  3582 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:30.690765  3561 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:30.779153  1728 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:30.782192  3540 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:30.749872  3538 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:30.784626  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:30.820987  1876 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:22:30.733390  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.009s	sys 0.021s
I20260709 12:22:30.770269  3468 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:30.822813  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.009s	sys 0.021s
W20260709 12:22:30.823638  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 617340us
I20260709 12:22:30.799459  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:30.819054  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:30.825031  1807 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:30.826375  1877 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 2: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:30.827677  1879 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:30.778147  3363 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:30.834539  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:30.835865  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:30.836630  1881 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:30.838668  1878 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:30.831773  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:30.839941  1873 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:30.833282  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:30.840816  1804 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:30.768723  3630 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:30.867753  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:30.869678  1799 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.001s	sys 0.030s
I20260709 12:22:30.790892  3582 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:30.935242  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57481.1us
I20260709 12:22:30.889601  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:30.935534  1727 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:30.759299  3352 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:22:30.894694  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:30.910089  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:30.829159  3476 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.001s	sys 0.029s
W20260709 12:22:30.912889  1799 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.001s	sys 0.031s
I20260709 12:22:30.834134  3363 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:30.907626  3538 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:30.972064  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62084.6us
I20260709 12:22:30.936597  1880 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:30.974140  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:30.968237  1726 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.013s	sys 0.017s
W20260709 12:22:30.977278  1726 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.014s	sys 0.017s
W20260709 12:22:30.980692  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.001s	sys 0.031s
W20260709 12:22:30.981159  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.001s	sys 0.031s
W20260709 12:22:30.869307  2412 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.013s	sys 0.019s
I20260709 12:22:30.817126  3556 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.126s
I20260709 12:22:30.907791  3649 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:30.981518  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 786112us
I20260709 12:22:30.896625  3642 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.827167  3441 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.136s
I20260709 12:22:30.828685  3540 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:30.934204  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:30.980840  3476 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.002s	sys 0.029s
I20260709 12:22:30.942423  3624 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.936874  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:30.944206  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:30.783075  3671 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:31.002687  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:31.003005  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:30.995791  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1373) took 2391 ms. Trace:
I20260709 12:22:31.018584  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:31.027606  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:30.996706  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:31.036616  1880 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:22:31.036017  1724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.841s	user 0.011s	sys 0.026s
W20260709 12:22:31.037334  1724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.011s	sys 0.026s
I20260709 12:22:31.003333  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:31.036039  1730 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:31.021481  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:31.040964  1798 raft_consensus.cc:2368] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5.
I20260709 12:22:31.035703  1805 rpcz_store.cc:276] 0709 12:22:28.604837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.221330 (+616493us) service_pool.cc:224] Handling call
0709 12:22:29.221699 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.222597 (+   898us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.222631 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.223391 (+   760us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.765337 (+541946us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:30.536532 (+771195us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.995391 (+458859us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.995500 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.995545 (+    45us) raft_consensus.cc:1685] finished
0709 12:22:30.995558 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.995725 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":413872,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":65,"raft.queue_time_us":56946,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":6624,"thread_start_us":1770369,"threads_started":3,"wal-append.queue_time_us":921828}
I20260709 12:22:31.035020  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:31.047533  1879 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:31.048282  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:31.049494  1881 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
W20260709 12:22:31.067205  1803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.027s	sys 0.072s
W20260709 12:22:31.067844  1803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.027s	sys 0.072s
W20260709 12:22:30.991351  2412 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.015s	sys 0.020s
W20260709 12:22:30.984853  1801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.000s	sys 0.033s
I20260709 12:22:31.033512  3539 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:30.986021  3630 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.985834  3469 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:31.019547  3476 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:22:31.091562  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.044097  3397 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:31.027861  3624 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:31.012864  3642 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:31.033617  3363 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:31.018010  3659 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:31.157367  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:31.419029  1847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.009s	sys 0.021s
W20260709 12:22:31.487865  1847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.041s	user 0.011s	sys 0.026s
W20260709 12:22:31.194720  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.000s	sys 0.024s
I20260709 12:22:31.255632  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:31.489161  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.004s	sys 0.040s
I20260709 12:22:31.489517  1879 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:22:31.490003  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 880474us
W20260709 12:22:31.490758  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.491110  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:31.488785  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 219) took 2099 ms. Trace:
I20260709 12:22:31.492726  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:31.054713  3609 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:31.497498  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:22:31.186580  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 389996us
W20260709 12:22:31.103632  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 322280us
W20260709 12:22:31.384749  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 407886us
I20260709 12:22:31.138585  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:31.582384  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 89749.6us
I20260709 12:22:31.012624  3675 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:31.492431  1724 rpcz_store.cc:276] 0709 12:22:29.389470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.098018 (+708548us) service_pool.cc:224] Handling call
0709 12:22:30.098350 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.147778 (+ 49428us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.147813 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.148359 (+   546us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.038103 (+889744us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:31.038170 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.038260 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.424914 (+386654us) raft_consensus.cc:1685] finished
0709 12:22:31.424951 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.488708 (+ 63757us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61928,"raft.queue_time_us":49881,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":5224,"thread_start_us":842974,"threads_started":1,"wal-append.queue_time_us":980865}
W20260709 12:22:31.607486  2383 leader_election.cc:336] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.605s (SENT)
I20260709 12:22:31.046555  3556 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:22:31.089813  1801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.000s	sys 0.033s
I20260709 12:22:31.591025  1881 raft_consensus.cc:2410] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:22:31.105100  1874 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:31.578845  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.007s	sys 0.003s
I20260709 12:22:31.590749  2417 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:31.629438  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.009s	sys 0.003s
W20260709 12:22:31.623697  1807 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.796s	user 0.012s	sys 0.000s
W20260709 12:22:31.500914  1804 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.623s	user 0.011s	sys 0.001s
I20260709 12:22:31.488245  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1333) took 1165 ms. Trace:
I20260709 12:22:31.623607  3675 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.630750  1804 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:31.631181  1726 rpcz_store.cc:276] 0709 12:22:30.323002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.391912 (+ 68910us) service_pool.cc:224] Handling call
0709 12:22:30.392345 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.392610 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:30.977898 (+585288us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.487806 (+509908us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.487873 (+    67us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:31.487949 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.488036 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.488184 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":630535,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":57812,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":28139,"thread_start_us":942264,"threads_started":2}
I20260709 12:22:31.632978  1807 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:31.631552  3675 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:31.591493  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:31.104146  2413 raft_consensus.cc:2410] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:31.640075  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:31.563087  3649 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.027s	sys 0.065s
W20260709 12:22:31.351310  3433 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.015s	sys 0.021s
W20260709 12:22:31.330744  3679 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:22:31.439663  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:31.087977  3483 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:31.202857  3552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.011s	sys 0.014s
I20260709 12:22:31.624745  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:31.486189  3678 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.327s	user 0.000s	sys 0.001s
W20260709 12:22:31.421104  3669 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.411s	user 0.000s	sys 0.001s
I20260709 12:22:31.718433  2417 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
W20260709 12:22:31.426512  3662 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.382s	user 0.001s	sys 0.000s
W20260709 12:22:31.630105  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 633510us
I20260709 12:22:31.722106  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:31.609187  3587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.003s	sys 0.006s
W20260709 12:22:31.613955  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 56538.5us
I20260709 12:22:31.136930  3438 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:31.186620  3618 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:31.294003  3652 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:22:31.484939  3658 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:31.558005  1696 leader_election.cc:400] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5.
I20260709 12:22:31.752574  1696 leader_election.cc:403] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260709 12:22:31.753300  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 195526us
I20260709 12:22:31.691321  3364 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:31.754927  1771 leader_election.cc:336] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.515s (SENT)
I20260709 12:22:31.441607  3540 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:31.589140  3593 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:31.312330  3576 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:31.380965  3683 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:31.701603  1806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.027s	sys 0.025s
I20260709 12:22:31.257515  3561 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:31.709329  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84705.9us
I20260709 12:22:31.345081  3616 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:31.615684  3672 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:31.103770  3538 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:31.648926  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:31.752308  3659 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:31.681361  3649 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.028s	sys 0.074s
I20260709 12:22:31.752408  3658 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:31.723557  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:31.137722  3630 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:31.733839  1723 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:31.733775  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2200) took 1972 ms. Trace:
I20260709 12:22:31.808171  1728 rpcz_store.cc:276] 0709 12:22:29.761213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.991511 (+230298us) service_pool.cc:224] Handling call
0709 12:22:31.733524 (+1742013us) spinlock_profiling.cc:243] Waited 584 ms on lock 0x7b54002d19d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:31.733703 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1342753664}
I20260709 12:22:31.620524  3556 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:31.703116  3588 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:31.718652  3552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.040s	user 0.017s	sys 0.033s
I20260709 12:22:31.695504  3687 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:31.700562  3433 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.934s	user 0.022s	sys 0.034s
W20260709 12:22:31.715930  1802 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.003s	sys 0.011s
I20260709 12:22:31.648135  3539 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.755439  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:31.766144  2383 leader_election.cc:336] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.841s (SENT)
W20260709 12:22:31.773684  1806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.027s	sys 0.025s
I20260709 12:22:31.628543  3469 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:31.780028  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:31.693753  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 102343us
W20260709 12:22:31.829653  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49734.6us
I20260709 12:22:31.829952  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:31.752640  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1395) took 1533 ms. Trace:
I20260709 12:22:31.791497  1879 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:31.759552  3364 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.786599  3672 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:31.715261  1800 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.017s	sys 0.029s
I20260709 12:22:31.712239  3483 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:31.777554  3628 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.181s	user 0.015s	sys 0.000s
I20260709 12:22:31.801975  1805 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:31.776678  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:31.811352  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1396) took 1597 ms. Trace:
I20260709 12:22:31.784065  3476 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:31.823685  1802 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.003s	sys 0.012s
I20260709 12:22:31.814380  3691 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:31.837548  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:31.843266  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1401) took 1067 ms. Trace:
W20260709 12:22:31.845196  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 90237us
W20260709 12:22:31.815379  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 92953.1us
I20260709 12:22:31.822301  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 404) took 1602 ms. Trace:
I20260709 12:22:31.796442  3618 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:31.796739  3649 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:32.011762  3649 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:31.811339  3681 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.562s	user 0.001s	sys 0.000s
W20260709 12:22:31.823657  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69374.8us
I20260709 12:22:31.806723  3616 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:31.828037  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 61931.6us
W20260709 12:22:32.018553  2383 leader_election.cc:336] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.914s (SENT)
I20260709 12:22:31.832248  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:22:32.021277  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 189096us
I20260709 12:22:32.022324  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:32.022466  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1330) took 1701 ms. Trace:
I20260709 12:22:32.023077  1723 rpcz_store.cc:276] 0709 12:22:30.321224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.321841 (+   617us) service_pool.cc:224] Handling call
0709 12:22:31.733664 (+1411823us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.807695 (+ 74031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.022123 (+214428us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54002d19d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:32.022372 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3178709760}
I20260709 12:22:31.820072  3588 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:32.024343  3352 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:32.024864  3352 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.024981  3469 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:32.025310  3352 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:32.025723  3469 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:31.953033  2412 rpcz_store.cc:276] 0709 12:22:30.217693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.232533 (+ 14840us) service_pool.cc:224] Handling call
0709 12:22:30.252202 (+ 19669us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.252466 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.252494 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.253004 (+   510us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.103634 (+850630us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:31.156701 (+ 53067us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.190866 (+ 34165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.813355 (+622489us) raft_consensus.cc:1685] finished
0709 12:22:31.813398 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.819520 (+  6122us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":827387,"threads_started":1,"wal-append.queue_time_us":782053}
I20260709 12:22:31.824132  3687 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:32.019876  2383 leader_election.cc:336] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.926s (SENT)
I20260709 12:22:32.029148  3687 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:32.031140  3609 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:32.031739  3609 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:32.033655  3352 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:31.853502  1698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.712s	user 0.004s	sys 0.018s
W20260709 12:22:32.036056  1698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.037s	user 0.004s	sys 0.022s
I20260709 12:22:32.036307  3352 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:32.037163  3688 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.335s	user 0.001s	sys 0.001s
I20260709 12:22:31.836431  3539 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:31.853806  3569 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.133s	user 0.015s	sys 0.001s
W20260709 12:22:31.851622  3692 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:22:31.891924  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1337) took 1318 ms. Trace:
I20260709 12:22:32.041913  1727 rpcz_store.cc:276] 0709 12:22:30.573841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.614070 (+ 40229us) service_pool.cc:224] Handling call
0709 12:22:31.891729 (+1277659us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54002d05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:31.891852 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":297981184}
I20260709 12:22:31.895246  1875 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 3 because replica has last-logged OpId of term: 2 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260709 12:22:31.895546  3715 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:32.047514  3715 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:31.787532  3683 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:31.859411  1806 rpcz_store.cc:276] 0709 12:22:30.775502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.776314 (+   812us) service_pool.cc:224] Handling call
0709 12:22:30.818243 (+ 41929us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.841238 (+ 22995us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.841268 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.894680 (+ 53412us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.828129 (+933449us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:31.828189 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.828283 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.842970 (+ 14687us) raft_consensus.cc:1685] finished
0709 12:22:31.842992 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.843170 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":34524,"thread_start_us":842935,"threads_started":1,"wal-append.queue_time_us":896701}
I20260709 12:22:32.049789  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:32.050377  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:32.051146  1806 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:31.899159  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:31.845506  3672 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:31.904704  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:31.903292  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:31.910764  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:31.936220  2419 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:22:32.061275  1873 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:31.867316  3677 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:22:31.863850  1771 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.551s (SENT)
I20260709 12:22:32.070176  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:22:32.071601  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 207769us
W20260709 12:22:32.072454  1771 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.573s (SENT)
I20260709 12:22:31.876441  1879 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:31.845582  1800 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.804s	user 0.017s	sys 0.030s
W20260709 12:22:32.073561  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.847169  3483 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:31.855093  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1400) took 1080 ms. Trace:
I20260709 12:22:31.944684  2416 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:22:31.850987  1799 rpcz_store.cc:276] 0709 12:22:30.214370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.214620 (+   250us) service_pool.cc:224] Handling call
0709 12:22:30.254217 (+ 39597us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.254572 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.254601 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.255172 (+   571us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.967738 (+712566us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:30.967793 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.967878 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.811073 (+843195us) raft_consensus.cc:1685] finished
0709 12:22:31.811098 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.811280 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":293552,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":712017,"threads_started":1,"wal-append.queue_time_us":798875}
I20260709 12:22:31.856523  3394 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:31.847919  1874 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:31.837064  3364 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:31.921301  3582 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.957s	user 0.008s	sys 0.009s
W20260709 12:22:32.082372  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.904455  3716 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:22:31.899433  3624 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:31.924276  3593 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.903801  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:31.924633  3657 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:31.956236  2418 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:31.936064  3642 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:31.958835  1805 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:31.972570  3618 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:31.950695  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:31.992432  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:32.107758  1880 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:31.997694  3363 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:31.998752  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:31.999927  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:32.004980  2414 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:31.849656  3397 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:31.847484  3666 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.075s	user 0.003s	sys 0.000s
I20260709 12:22:32.063400  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:32.072942  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:32.115772  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52489.3us
I20260709 12:22:32.116307  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:32.117125  1873 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:32.072909  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:32.078433  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1387) took 2387 ms. Trace:
I20260709 12:22:32.054975  3690 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:32.026655  3469 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
W20260709 12:22:32.128113  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55677.5us
W20260709 12:22:32.128881  1771 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.592s (SENT)
I20260709 12:22:32.129304  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:32.131265  3705 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:32.132254  3705 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:32.121346  3397 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:32.137326  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 228) took 1148 ms. Trace:
I20260709 12:22:32.137979  1731 rpcz_store.cc:276] 0709 12:22:30.989197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.053998 (+ 64801us) service_pool.cc:224] Handling call
0709 12:22:31.086665 (+ 32667us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.649556 (+562891us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.649612 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.698030 (+ 48418us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.791314 (+ 93284us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:31.792160 (+   846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.926711 (+134551us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b540015cf54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:31.926805 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.136830 (+210025us) raft_consensus.cc:1685] finished
0709 12:22:32.136862 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.137234 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5040,"raft.queue_time_us":139423,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":9019,"spinlock_wait_cycles":1281567104,"thread_start_us":45503,"threads_started":1,"wal-append.queue_time_us":121915}
I20260709 12:22:31.957647  3556 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:31.851821  3710 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:32.075103  3685 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.353s	user 0.001s	sys 0.001s
W20260709 12:22:32.077157  3684 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.394s	user 0.004s	sys 0.000s
I20260709 12:22:32.118842  3569 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:32.089114  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:31.845295  3658 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:32.079936  3394 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:32.107158  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 156545us
I20260709 12:22:32.114017  1728 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:32.114187  1727 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:22:32.117718  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:32.118124  1800 rpcz_store.cc:276] 0709 12:22:29.686938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.687146 (+   208us) service_pool.cc:224] Handling call
0709 12:22:29.744057 (+ 56911us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.744366 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:30.575127 (+830761us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.036552 (+1461425us) spinlock_profiling.cc:243] Waited 38 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:32.073673 (+ 37121us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:32.073734 (+    61us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:32.073787 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.073875 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.074020 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":288626,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":284323,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":5108,"spinlock_wait_cycles":87326976,"thread_start_us":2314036,"threads_started":3,"wal-append.queue_time_us":1014534}
I20260709 12:22:32.084465  3677 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:32.125172  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:32.125723  3407 raft_consensus.cc:1064] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:32.126245  3690 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:32.146345  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:22:32.082177  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 326091us
I20260709 12:22:32.108352  2410 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:32.150215  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:32.148301  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:32.151338  1771 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.729s (SENT)
W20260709 12:22:32.214890  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63581.9us
I20260709 12:22:32.096977  3702 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:32.153167  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:32.108173  3657 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:32.086288  3582 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:32.164836  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:32.092113  3675 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:32.093731  3593 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:32.171245  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53674.5us
I20260709 12:22:32.169425  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:32.174998  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149801us
W20260709 12:22:32.241326  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 72018.9us
I20260709 12:22:32.178365  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:32.242257  1729 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 1.
I20260709 12:22:32.247347  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1332) took 1924 ms. Trace:
I20260709 12:22:32.247812  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:32.249545  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:32.174607  3450 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.119s	user 0.010s	sys 0.018s
I20260709 12:22:32.178845  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:32.179975  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:32.253239  1800 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:32.208985  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62724.1us
I20260709 12:22:32.153219  3630 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:32.209070  1697 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:32.276185  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:32.163793  3642 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.118408  3671 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 12:22:32.158653  3708 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:32.159590  3613 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:32.218981  1880 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:32.074667  3616 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:32.220854  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:32.175436  3394 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:32.229303  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:32.175324  3683 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:32.180707  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:32.208905  3658 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:32.401479  3582 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:32.335978  3576 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.562s	user 0.010s	sys 0.001s
I20260709 12:22:32.279259  3540 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:32.248045  1732 rpcz_store.cc:276] 0709 12:22:30.322563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.382804 (+ 60241us) service_pool.cc:224] Handling call
0709 12:22:30.383327 (+   523us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.448172 (+ 64845us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.448210 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.452479 (+  4269us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.543638 (+ 91159us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b18003b5a80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:22:31.035508 (+491870us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:31.035554 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.035648 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.013050 (+977402us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:32.246984 (+233934us) raft_consensus.cc:1685] finished
0709 12:22:32.247017 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.247249 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74367,"raft.queue_time_us":64851,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":6655,"spinlock_wait_cycles":74774144,"thread_start_us":491630,"threads_started":1,"wal-append.queue_time_us":1138681}
W20260709 12:22:32.185213  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 169114us
I20260709 12:22:32.219008  3476 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:32.256500  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:32.256361  3590 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.259922  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:32.264164  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:32.266085  3675 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:32.275784  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:32.279052  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:32.280067  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 71074us
I20260709 12:22:32.432955  1805 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:32.268078  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1397) took 2053 ms. Trace:
I20260709 12:22:32.433467  1803 rpcz_store.cc:276] 0709 12:22:30.215024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.462126 (+247102us) service_pool.cc:224] Handling call
0709 12:22:30.462664 (+   538us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.463148 (+   484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.463195 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.477745 (+ 14550us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.068330 (+590585us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:31.068390 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.068490 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.011842 (+943352us) spinlock_profiling.cc:243] Waited 41.3 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:32.267681 (+255839us) raft_consensus.cc:1685] finished
0709 12:22:32.267712 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.267994 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":85244,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":49,"spinlock_wait_cycles":95058688,"thread_start_us":577910,"threads_started":1,"wal-append.queue_time_us":890611}
I20260709 12:22:32.435595  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:32.316812  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:32.252868  3450 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.208s	user 0.010s	sys 0.018s
I20260709 12:22:31.833829  1801 rpcz_store.cc:276] 0709 12:22:30.213991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.216970 (+  2979us) service_pool.cc:224] Handling call
0709 12:22:30.251814 (+ 34844us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.314938 (+ 63124us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:30.359323 (+ 44385us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.680575 (+1321252us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.701414 (+ 20839us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:31.708162 (+  6748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.726130 (+ 17968us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.747012 (+ 20882us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14699,"prepare.queue_time_us":23603,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":135664,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":156,"thread_start_us":1271127,"threads_started":1,"wal-append.queue_time_us":731143}
I20260709 12:22:32.255048  3624 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:32.379099  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:32.385784  3705 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.276115  3539 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:32.077618  1802 rpcz_store.cc:276] 0709 12:22:30.775173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.778562 (+  3389us) service_pool.cc:224] Handling call
0709 12:22:30.782217 (+  3655us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.782488 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.782531 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.789349 (+  6818us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.854532 (+1065183us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:31.854594 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.854700 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.854776 (+    76us) raft_consensus.cc:1685] finished
0709 12:22:31.854823 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.855013 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":874255,"threads_started":1,"wal-append.queue_time_us":943877}
I20260709 12:22:32.390334  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:32.455605  1880 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:32.390506  3674 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:32.389828  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 148930us
W20260709 12:22:32.396503  3729 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:22:32.397625  3407 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:32.387126  3588 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:32.423411  3576 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:32.231230  1771 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.837s (SENT)
I20260709 12:22:32.401270  1874 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:32.406688  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:32.414534  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:32.436614  1876 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:32.226936  3702 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:32.469926  2419 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:32.425168  3723 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:32.432570  2417 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:32.427326  1628 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 253) took 6521 ms. Trace:
I20260709 12:22:32.442301  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:32.478858  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:32.430429  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:32.443148  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:32.422746  3476 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.444648  3590 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:32.486411  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:32.487031  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:32.389230  1806 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:32.430873  1875 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:32.463735  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:32.336974  3630 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:32.490501  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:32.429121  1627 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:32.468144  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:32.491988  3724 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:22:32.492846  1627 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:32.471511  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 240312us
I20260709 12:22:32.384737  3613 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:32.430750  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:32.460429  3728 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:32.451258  3538 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.480518  1800 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:32.504851  1627 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 2, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:32.506181  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:32.507288  1627 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 3, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:32.481120  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:32.483367  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:32.401275  3672 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:32.432132  3540 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:32.510852  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:32.453404  3582 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:32.337078  3642 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:22:32.519372  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.486003  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:32.337380  3708 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:32.525357  1730 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:32.489730  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:32.490687  3730 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:32.486485  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:32.491616  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:32.530872  1805 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:32.477912  3709 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.350s	user 0.001s	sys 0.001s
I20260709 12:22:32.499094  1873 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:32.441617  3671 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260709 12:22:32.465550  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:32.498399  1798 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:32.502247  1731 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:22:32.466822  3539 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:32.493711  3630 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:32.560467  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 95024.5us
I20260709 12:22:32.463312  3705 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:32.508997  1627 catalog_manager.cc:5697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:32.498718  3717 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:32.563120  3717 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:32.563776  3630 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:32.563369  1627 catalog_manager.cc:5697] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 4, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:32.512579  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:32.468042  3718 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.484515  3552 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.486701  3476 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:32.481031  3624 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:32.431743  3677 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:32.529120  3723 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:22:32.536204  3720 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.307s	user 0.002s	sys 0.000s
I20260709 12:22:32.540653  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2051) took 1105 ms. Trace:
I20260709 12:22:32.503829  3450 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.465466  3674 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:32.494024  3725 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:22:32.506897  3616 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:32.510955  3675 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.128s
I20260709 12:22:32.570152  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:32.999415  1800 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:32.999222  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:32.997260  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:33.000463  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:32.589469  1729 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:33.002142  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:32.888821  1731 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:32.478724  1628 rpcz_store.cc:276] 0709 12:22:25.905768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.907543 (+  1775us) service_pool.cc:224] Handling call
0709 12:22:32.427198 (+6519655us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:28.031671 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:28.389187 (+357516us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:28.400074 (+ 10887us) write_op.cc:435] Acquired schema lock
0709 12:22:28.412198 (+ 12124us) tablet.cc:662] Decoding operations
0709 12:22:28.836569 (+424371us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:28.836693 (+   124us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:28.836728 (+    35us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:22:29.021319 (+184591us) tablet.cc:701] Row locks acquired
0709 12:22:29.021395 (+    76us) write_op.cc:260] PREPARE: finished
0709 12:22:29.021705 (+   310us) write_op.cc:270] Start()
0709 12:22:29.021935 (+   230us) write_op.cc:276] Timestamp: P: 1783599749021605 usec, L: 0
0709 12:22:29.021988 (+    53us) op_driver.cc:348] REPLICATION: starting
0709 12:22:29.023138 (+  1150us) log.cc:844] Serialized 4489 byte log entry
0709 12:22:30.758904 (+1735766us) op_driver.cc:464] REPLICATION: finished
0709 12:22:31.766357 (+1007453us) write_op.cc:301] APPLY: starting
0709 12:22:31.787003 (+ 20646us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:31.837143 (+ 50140us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:31.837307 (+   164us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:31.839081 (+  1774us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:31.839447 (+   366us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0709 12:22:31.839498 (+    51us) write_op.cc:312] APPLY: finished
0709 12:22:31.840039 (+   541us) log.cc:844] Serialized 83 byte log entry
0709 12:22:32.081581 (+241542us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:32.088268 (+  6687us) write_op.cc:454] Released schema lock
0709 12:22:32.109640 (+ 21372us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:32.135342 (+ 25702us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:32.135390 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":12,"child_traces":[["op",{"apply.queue_time_us":953803,"apply.run_cpu_time_us":47535,"apply.run_wall_time_us":395694,"num_ops":11,"prepare.queue_time_us":247150,"prepare.run_cpu_time_us":86181,"prepare.run_wall_time_us":1449075,"raft.queue_time_us":900143,"raft.run_cpu_time_us":32914,"raft.run_wall_time_us":457682,"replication_time_us":1735356,"thread_start_us":1939988,"threads_started":5,"wal-append.queue_time_us":960390,"wal-append.run_cpu_time_us":35547,"wal-append.run_wall_time_us":1874846}]]}
W20260709 12:22:32.997221  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 435819us
I20260709 12:22:32.716130  1876 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:33.004710  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:32.580225  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.005829  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.008157  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.006448  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2054) took 1569 ms (client timeout 1992 ms). Trace:
I20260709 12:22:33.009295  2419 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:33.000710  1873 rpcz_store.cc:276] 0709 12:22:31.435637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.404083 (+968446us) service_pool.cc:224] Handling call
0709 12:22:32.540546 (+136463us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.012081  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:33.017184  1805 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
W20260709 12:22:32.993130  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98552.3us
W20260709 12:22:33.011607  1876 rpcz_store.cc:269] 0709 12:22:31.436716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.250829 (+814113us) service_pool.cc:224] Handling call
0709 12:22:33.006383 (+755554us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.023562  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:32.794459  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.169s (SENT) [suppressed 59 similar messages]
I20260709 12:22:33.024367  1803 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
W20260709 12:22:33.029449  1771 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.993s (SENT)
I20260709 12:22:33.065121  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:33.066082  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63992.8us
I20260709 12:22:33.066145  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2070) took 1178 ms. Trace:
I20260709 12:22:33.067039  1876 rpcz_store.cc:276] 0709 12:22:31.888217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.064743 (+1176526us) service_pool.cc:224] Handling call
0709 12:22:33.066076 (+  1333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.067269  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:33.075307  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:33.076143  1873 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:33.076884  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 282460us
I20260709 12:22:33.079689  3612 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:32.518734  3538 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:32.567163  3539 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.080950  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:33.082392  3538 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:33.083191  3539 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:33.083544  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:33.084414  1876 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:33.084707  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 530) took 1185 ms. Trace:
I20260709 12:22:33.084864  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2063) took 1392 ms. Trace:
I20260709 12:22:33.085242  1876 rpcz_store.cc:276] 0709 12:22:31.692384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.083218 (+1390834us) service_pool.cc:224] Handling call
0709 12:22:33.084806 (+  1588us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.086652  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:33.087675  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1505) took 1178 ms. Trace:
I20260709 12:22:33.088137  1876 rpcz_store.cc:276] 0709 12:22:31.909188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.086385 (+1177197us) service_pool.cc:224] Handling call
0709 12:22:33.087616 (+  1231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.089919  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:33.090854  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1488) took 1600 ms. Trace:
I20260709 12:22:33.091255  1876 rpcz_store.cc:276] 0709 12:22:31.490703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.089610 (+1598907us) service_pool.cc:224] Handling call
0709 12:22:33.090759 (+  1149us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:33.093361  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54455.1us
I20260709 12:22:33.087699  1800 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:33.096035  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1501) took 1187 ms. Trace:
W20260709 12:22:33.094920  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.022s	sys 0.059s
I20260709 12:22:33.088410  3702 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:33.096894  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.023s	sys 0.059s
W20260709 12:22:33.097023  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53315.5us
W20260709 12:22:33.097260  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 636208us
I20260709 12:22:33.094293  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:33.099417  1799 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.592s	user 0.010s	sys 0.003s
I20260709 12:22:33.095031  1727 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:22:32.870105  3721 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.506s	user 0.000s	sys 0.002s
W20260709 12:22:32.866914  3716 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
W20260709 12:22:33.107012  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.039s	sys 0.052s
I20260709 12:22:33.107093  1799 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:32.558007  3687 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:32.543756  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:33.110385  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.096575  1873 rpcz_store.cc:276] 0709 12:22:31.908138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.074964 (+1166826us) service_pool.cc:224] Handling call
0709 12:22:33.095948 (+ 20984us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:33.112903  1771 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.110s (SENT)
I20260709 12:22:32.793218  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-ea5d8f652fc448d6af846ff6850004b8 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:22:33.116389  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:32.567230  3683 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:32.718998  3593 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:33.116439  3732 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.611s	user 0.001s	sys 0.001s
I20260709 12:22:33.162242  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:33.110172  3561 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:33.104562  3613 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:32.864866  3735 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.448s
I20260709 12:22:33.163895  3702 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:32.529443  3582 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.170253  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:32.565598  3717 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:33.185525  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-ea5d8f652fc448d6af846ff6850004b8 on tablet ea5d8f652fc448d6af846ff6850004b8 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:22:33.085134  2413 rpcz_store.cc:276] 0709 12:22:31.899785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.900721 (+   936us) service_pool.cc:224] Handling call
0709 12:22:31.901159 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.901668 (+   509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.901703 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.902335 (+   632us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.103864 (+201529us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:32.143769 (+ 39905us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.152364 (+  8595us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.535429 (+383065us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:33.084342 (+548913us) raft_consensus.cc:1685] finished
0709 12:22:33.084371 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.084642 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2444,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"spinlock_wait_cycles":30982016,"thread_start_us":188700,"threads_started":1,"wal-append.queue_time_us":261820}
W20260709 12:22:33.171715  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.216293  3186 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.000s	sys 0.022s
W20260709 12:22:33.176508  1879 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.010s	user 0.008s	sys 0.004s
I20260709 12:22:33.217320  1879 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
W20260709 12:22:33.218093  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1495) took 1593 ms (client timeout 1505 ms). Trace:
W20260709 12:22:33.218523  1879 rpcz_store.cc:269] 0709 12:22:31.625072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.080956 (+455884us) service_pool.cc:224] Handling call
0709 12:22:33.217987 (+1137031us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.219019  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1409) took 2284 ms. Trace:
I20260709 12:22:33.023433  3576 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:33.176333  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1493) took 1649 ms. Trace:
W20260709 12:22:33.161175  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.049s	sys 0.061s
I20260709 12:22:33.202965  3561 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:33.186053  1873 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:33.228833  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2067) took 1535 ms. Trace:
I20260709 12:22:33.229658  1873 rpcz_store.cc:276] 0709 12:22:31.693656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.161876 (+1468220us) service_pool.cc:224] Handling call
0709 12:22:33.228752 (+ 66876us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.159319  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:33.013290  3723 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:33.022459  3690 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:32.543804  3730 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.114163  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:32.745368  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:33.073915  3587 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:33.022285  3552 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:33.036533  3616 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:33.217924  3705 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:33.197389  3691 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.314s	user 0.006s	sys 0.011s
I20260709 12:22:32.797070  3657 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:33.168172  3687 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:33.216800  3186 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.000s	sys 0.022s
I20260709 12:22:33.219825  1807 rpcz_store.cc:276] 0709 12:22:30.934257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.890601 (+956344us) service_pool.cc:224] Handling call
0709 12:22:31.891050 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.045309 (+154259us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.045341 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.045927 (+   586us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.151923 (+105996us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:32.151969 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.406420 (+254451us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b540026fa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:32.406477 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.211494 (+805017us) raft_consensus.cc:1685] finished
0709 12:22:33.211517 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.218946 (+  7429us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6384,"raft.queue_time_us":11765,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":12348,"spinlock_wait_cycles":353559424,"thread_start_us":105539,"threads_started":1,"wal-append.queue_time_us":204971}
I20260709 12:22:33.223235  1876 rpcz_store.cc:276] 0709 12:22:31.527205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.109050 (+1581845us) service_pool.cc:224] Handling call
0709 12:22:33.176206 (+ 67156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:32.701273  3713 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:33.226532  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 113667us
I20260709 12:22:33.072991  3450 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.207074  3702 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:33.278473  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:33.279104  3702 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:33.239055  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 125039us
I20260709 12:22:33.279681  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:33.214422  3731 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:33.209460  3740 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
W20260709 12:22:33.226924  3733 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.721s	user 0.002s	sys 0.000s
I20260709 12:22:33.218264  3710 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:33.253198  3576 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:33.201471  3683 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:33.261588  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 194442us
I20260709 12:22:33.237694  3561 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:33.282308  3687 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:33.279253  1876 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:33.071082  3718 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.239432  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:33.294298  3540 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:33.237171  3672 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:33.286155  3582 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:33.295635  3744 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:33.289335  3364 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.195s	user 0.008s	sys 0.004s
W20260709 12:22:33.339377  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2064) took 1646 ms (client timeout 2110 ms). Trace:
I20260709 12:22:33.302331  3708 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:33.339515  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:33.290527  3609 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.258s	user 0.012s	sys 0.002s
I20260709 12:22:33.251740  3642 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:33.254022  3717 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:33.195149  3588 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.014s	sys 0.029s
W20260709 12:22:33.293182  3659 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.491s	user 0.009s	sys 0.006s
I20260709 12:22:33.260932  3587 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:33.275213  3690 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:33.247083  3649 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.012s	sys 0.012s
I20260709 12:22:33.281277  3691 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5.
W20260709 12:22:33.312429  1723 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.800s	user 0.026s	sys 0.080s
W20260709 12:22:33.373575  1723 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.861s	user 0.027s	sys 0.080s
I20260709 12:22:33.328606  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:33.374166  1799 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:33.291841  3731 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:33.333773  2410 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.106s	user 0.005s	sys 0.014s
I20260709 12:22:33.298156  3705 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:33.303303  3710 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:33.288470  3616 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:33.290287  3658 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.888s	user 0.005s	sys 0.005s
I20260709 12:22:33.351222  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:33.339893  1879 rpcz_store.cc:269] 0709 12:22:31.692692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.219769 (+1527077us) service_pool.cc:224] Handling call
0709 12:22:33.339299 (+119530us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.359190  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:33.390326  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:33.390858  1807 raft_consensus.cc:2368] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3.
W20260709 12:22:33.302922  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1487) took 1807 ms (client timeout 2145 ms). Trace:
I20260709 12:22:33.250849  3624 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:33.368463  1728 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:22:33.358646  3588 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.015s	sys 0.030s
I20260709 12:22:33.377025  2410 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:33.393389  1876 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
W20260709 12:22:33.393944  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2065) took 1700 ms (client timeout 2239 ms). Trace:
W20260709 12:22:33.394246  1876 rpcz_store.cc:269] 0709 12:22:31.693058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.278101 (+1585043us) service_pool.cc:224] Handling call
0709 12:22:33.393891 (+115790us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:33.377555  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.907s	user 0.030s	sys 0.034s
W20260709 12:22:33.396401  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.926s	user 0.030s	sys 0.034s
W20260709 12:22:33.397049  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 982602us
I20260709 12:22:33.351476  3540 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:33.373217  3649 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.910s	user 0.012s	sys 0.012s
I20260709 12:22:33.391346  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1503) took 1482 ms. Trace:
I20260709 12:22:33.335794  3675 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:22:33.397292  1872 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:33.398006  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:33.400597  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 532) took 1493 ms. Trace:
I20260709 12:22:33.404992  2410 rpcz_store.cc:276] 0709 12:22:31.900516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.903501 (+  2985us) service_pool.cc:224] Handling call
0709 12:22:33.393579 (+1490078us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.375151  3674 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:33.401428  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:33.406311  1807 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:33.407387  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:33.408296  1807 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:33.378928  3659 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.377552  3717 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:33.400632  1879 rpcz_store.cc:276] 0709 12:22:31.908677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.390026 (+1481349us) service_pool.cc:224] Handling call
0709 12:22:33.391272 (+  1246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.414332  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:33.423511  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2071) took 1534 ms. Trace:
I20260709 12:22:33.425616  1879 rpcz_store.cc:276] 0709 12:22:31.888630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.413943 (+1525313us) service_pool.cc:224] Handling call
0709 12:22:33.423430 (+  9487us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.418773  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2072) took 1516 ms. Trace:
I20260709 12:22:33.387825  3710 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:33.417297  1799 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:33.426797  1876 rpcz_store.cc:276] 0709 12:22:31.888930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.397736 (+1508806us) service_pool.cc:224] Handling call
0709 12:22:33.405181 (+  7445us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.428202  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:33.357290  3744 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.411194  3352 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.309329  3723 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:33.428879  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 232651us
W20260709 12:22:33.431270  1846 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items
I20260709 12:22:33.431687  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:33.432356  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:33.433008  1879 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:33.435278  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:33.469600  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 174112us
I20260709 12:22:33.367697  3609 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:33.433677  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 209792us
I20260709 12:22:33.364023  3364 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:33.376070  3690 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.397094  3742 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:33.392596  1874 rpcz_store.cc:269] 0709 12:22:31.490351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.541509 (+1051158us) service_pool.cc:224] Handling call
0709 12:22:33.297640 (+756131us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.411654  3540 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:33.473161  3587 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.484098  3717 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.490459  1880 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:33.497226  3723 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:33.441090  1846 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 205)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 207)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 208)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 204)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 214)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 216)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2074)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2075)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2076)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2078)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 209)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 213)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2077)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 223)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43056 (request call id 77)
I20260709 12:22:33.467674  3731 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:33.466750  3740 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:33.430572  3659 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:33.516945  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:33.513716  3552 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:33.829355  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:33.831091  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:33.831969  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2053) took 2395 ms (client timeout 2829 ms). Trace:
I20260709 12:22:33.832113  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:33.832403  1876 rpcz_store.cc:269] 0709 12:22:31.436350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.441301 (+2004951us) service_pool.cc:224] Handling call
0709 12:22:33.831893 (+390592us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.833088  2418 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:33.833524  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:33.835132  3556 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:33.506880  3742 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:33.835922  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:33.837690  3742 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:33.782490  3590 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:33.516692  3690 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:33.537005  3364 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:33.849767  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:33.537902  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:33.548434  3757 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:22:33.545660  3540 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:33.515542  1881 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:33.540104  3397 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.406s	user 0.014s	sys 0.000s
I20260709 12:22:33.541302  3450 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:33.565958  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:33.605378  3363 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.484s	user 0.015s	sys 0.000s
W20260709 12:22:33.845104  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1502) took 1936 ms (client timeout 2243 ms). Trace:
I20260709 12:22:33.563687  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2212) took 1093 ms. Trace:
I20260709 12:22:33.469461  3352 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:33.841009  1801 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.643s	user 0.005s	sys 0.005s
W20260709 12:22:33.851290  3687 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.513s	user 0.009s	sys 0.002s
I20260709 12:22:33.545235  1725 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:33.848481  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 331586us
I20260709 12:22:33.851647  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2215) took 1377 ms. Trace:
I20260709 12:22:33.849900  1724 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:33.540017  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:33.885567  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:33.898380  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70608us
W20260709 12:22:33.857296  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1499) took 1949 ms (client timeout 1738 ms). Trace:
I20260709 12:22:33.834669  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 208) took 1295 ms. Trace:
I20260709 12:22:33.858399  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:33.939042  2418 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 2: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:33.940651  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:33.947477  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1408) took 2985 ms (client timeout 1722 ms). Trace:
W20260709 12:22:33.517652  3727 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.553s	user 0.000s	sys 0.003s
I20260709 12:22:33.979539  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:33.843220  3561 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.000s
W20260709 12:22:33.616850  3569 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.448s	user 0.015s	sys 0.000s
I20260709 12:22:34.219368  1728 rpcz_store.cc:276] 0709 12:22:32.474193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.474339 (+   146us) service_pool.cc:224] Handling call
0709 12:22:33.851397 (+1377058us) spinlock_profiling.cc:243] Waited 877 ms on lock 0x7b540030b454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:33.851568 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2017118592}
I20260709 12:22:34.188478  3186 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:34.219348  1798 rpcz_store.cc:269] 0709 12:22:30.871782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.085956 (+214174us) service_pool.cc:224] Handling call
0709 12:22:33.857100 (+2771144us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001e3f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:33.857293 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2323958016}
I20260709 12:22:33.539510  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:34.046833  1727 rpcz_store.cc:276] 0709 12:22:32.470368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.470532 (+   164us) service_pool.cc:224] Handling call
0709 12:22:33.563394 (+1092862us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:33.563582 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1298148096}
W20260709 12:22:34.046880  1879 rpcz_store.cc:269] 0709 12:22:31.908425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.427832 (+1519407us) service_pool.cc:224] Handling call
0709 12:22:33.845034 (+417202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.845605  3740 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:33.561412  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2050) took 2126 ms (client timeout 1658 ms). Trace:
I20260709 12:22:34.221040  3186 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:33.998214  3397 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.242900  3687 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:34.254925  2417 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
W20260709 12:22:34.057322  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 115545us
I20260709 12:22:34.257652  1876 rpcz_store.cc:276] 0709 12:22:32.539169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.833287 (+1294118us) service_pool.cc:224] Handling call
0709 12:22:33.834593 (+  1306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:33.845827  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2216) took 1371 ms. Trace:
I20260709 12:22:33.539011  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1347) took 1395 ms. Trace:
I20260709 12:22:34.207357  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:34.221652  1729 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:22:33.947014  3394 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.545s	user 0.009s	sys 0.009s
I20260709 12:22:34.427847  1726 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:22:34.429538  3556 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.594s	user 0.011s	sys 0.000s
I20260709 12:22:34.379907  1723 rpcz_store.cc:276] 0709 12:22:32.474491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.494826 (+ 20335us) service_pool.cc:224] Handling call
0709 12:22:32.495188 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.495448 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:33.375609 (+880161us) log.cc:844] Serialized 14 byte log entry
0709 12:22:33.845434 (+469825us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:33.845505 (+    71us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:33.845555 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.845637 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.845766 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":254216,"prepare.queue_time_us":761210,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":254382,"raft.run_cpu_time_us":77,"raft.run_wall_time_us":80,"thread_start_us":1076687,"threads_started":2}
W20260709 12:22:34.430604  3723 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.890s	user 0.010s	sys 0.000s
I20260709 12:22:34.380224  1731 rpcz_store.cc:276] 0709 12:22:32.143343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.143669 (+   326us) service_pool.cc:224] Handling call
0709 12:22:33.538829 (+1395160us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54002b7ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:33.538940 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":634488064}
I20260709 12:22:34.431311  3394 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:34.328933  3659 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:34.310462  3769 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.465s	user 0.001s	sys 0.000s
I20260709 12:22:34.056731  1801 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:34.258096  3687 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:34.284029  3710 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.816s	user 0.011s	sys 0.002s
I20260709 12:22:33.538638  3476 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:34.474920  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.357241  3407 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.015s	sys 0.022s
I20260709 12:22:34.267802  3718 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:34.252532  3561 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:34.437100  3649 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.492555  1878 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:33.540465  3609 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:34.235163  3569 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:34.246361  3540 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:33.968228  3588 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.013s	sys 0.032s
I20260709 12:22:34.268132  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:34.258773  1880 rpcz_store.cc:269] 0709 12:22:31.907497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.525301 (+617804us) service_pool.cc:224] Handling call
0709 12:22:33.857101 (+1331800us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b5400205dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:33.857224 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":915332480}
I20260709 12:22:34.058565  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:34.414242  2411 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:34.425483  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.425730  3552 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.290596  1873 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:34.295923  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:33.937633  3743 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:33.507403  3758 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.706s
W20260709 12:22:34.444936  1696 connection.cc:256] RPC call timeout handler was delayed by -2.1759118449999733s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1505s for a response
I20260709 12:22:34.446192  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2039) took 3483 ms. Trace:
I20260709 12:22:34.467118  3740 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:34.467784  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 229) took 2222 ms (client timeout 1907 ms). Trace:
W20260709 12:22:34.302850  3763 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:22:34.433454  3657 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.261317  1802 raft_consensus.cc:2376] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3. Re-sending same reply.
I20260709 12:22:34.441699  3450 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.641295  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 585) took 1388 ms. Trace:
I20260709 12:22:33.505755  3730 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:33.514261  3683 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:34.411379  3186 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:34.290853  1807 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:33.548225  3702 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:34.242334  1875 rpcz_store.cc:269] 0709 12:22:31.435269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.049448 (+614179us) service_pool.cc:224] Handling call
0709 12:22:33.561058 (+1511610us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:33.561296 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510262144}
I20260709 12:22:34.291157  3760 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.222s
W20260709 12:22:34.524451  1846 service_pool.cc:136] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items
I20260709 12:22:34.551617  1798 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 12:22:34.552165  1696 connection.cc:256] RPC call timeout handler was delayed by -2.1623540529999445s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
I20260709 12:22:34.552735  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1305) took 1181 ms. Trace:
I20260709 12:22:34.559036  1807 rpcz_store.cc:276] 0709 12:22:33.371569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.885241 (+513672us) service_pool.cc:224] Handling call
0709 12:22:34.552657 (+667416us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:34.553282  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 207) took 2042 ms (client timeout 2022 ms). Trace:
W20260709 12:22:34.560078  1873 rpcz_store.cc:269] 0709 12:22:32.511345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.539775 (+1028430us) service_pool.cc:224] Handling call
0709 12:22:34.553195 (+1013420us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.542734  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:34.554307  1727 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:34.542236  1723 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:34.547606  1877 rpcz_store.cc:276] 0709 12:22:30.962915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.963175 (+   260us) service_pool.cc:224] Handling call
0709 12:22:30.963732 (+   557us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:30.997793 (+ 34061us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.997828 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.025050 (+ 27222us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.052679 (+1027629us) spinlock_profiling.cc:243] Waited 1.74 ms on lock 0x7b180036c1c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:22:32.272169 (+219490us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:32.407594 (+135425us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.424678 (+ 17084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.424804 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.517285 (+1092481us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b240023a8d8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d2a09d 00007f16ab95c10e 00007f16ab97388c 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221
0709 12:22:34.445783 (+928498us) raft_consensus.cc:1685] finished
0709 12:22:34.445821 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.446050 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":69669,"prepare.queue_time_us":611344,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":70,"raft.queue_time_us":210518,"raft.run_cpu_time_us":500,"raft.run_wall_time_us":111350,"spinlock_wait_cycles":11250432,"thread_start_us":289935,"threads_started":2,"wal-append.queue_time_us":1587938}
I20260709 12:22:34.565615  3540 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:34.566628  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1296) took 1217 ms. Trace:
I20260709 12:22:34.580506  1798 rpcz_store.cc:276] 0709 12:22:33.349078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.273924 (+924846us) service_pool.cc:224] Handling call
0709 12:22:34.566540 (+292616us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.576400  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 221) took 2332 ms. Trace:
I20260709 12:22:34.582201  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:34.582049  1805 rpcz_store.cc:276] 0709 12:22:32.244248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.068173 (+823925us) service_pool.cc:224] Handling call
0709 12:22:33.068751 (+   578us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.292129 (+223378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:33.292173 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.301812 (+  9639us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.590446 (+288634us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:33.590507 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.590627 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.576081 (+985454us) raft_consensus.cc:1685] finished
0709 12:22:34.576111 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.576312 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12808,"raft.queue_time_us":139909,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":24592,"thread_start_us":413808,"threads_started":2,"wal-append.queue_time_us":531293}
I20260709 12:22:34.587721  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:34.588644  1805 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:34.589289  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1308) took 1213 ms. Trace:
I20260709 12:22:34.589707  1805 rpcz_store.cc:276] 0709 12:22:33.376127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.587389 (+1211262us) service_pool.cc:224] Handling call
0709 12:22:34.589221 (+  1832us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:34.586987  1845 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.020s	sys 0.038s
I20260709 12:22:34.593793  1804 raft_consensus.cc:2410] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:22:34.593977  1845 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.020s	sys 0.038s
W20260709 12:22:34.594513  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 588985us
I20260709 12:22:34.595744  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:34.596877  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 216) took 1506 ms (client timeout 1650 ms). Trace:
W20260709 12:22:34.597352  1880 rpcz_store.cc:269] 0709 12:22:33.090581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.540408 (+1449827us) service_pool.cc:224] Handling call
0709 12:22:34.596802 (+ 56394us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:34.555305  3588 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.125s	user 0.019s	sys 0.037s
I20260709 12:22:34.558264  1846 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 211)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 224)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 212)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2074)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 215)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2075)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 218)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2076)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2078)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 219)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 220)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 217)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2081)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 209)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 213)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2082)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2077)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2089)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2085)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43056 (request call id 77)
I20260709 12:22:34.571575  3739 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.614858  3739 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:34.576759  3758 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:34.616350  3758 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:34.617308  3758 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.617695  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:34.618711  1798 raft_consensus.cc:2368] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4.
I20260709 12:22:34.618564  3758 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:34.619128  3743 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:34.619323  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 237) took 1224 ms. Trace:
I20260709 12:22:34.619666  1798 rpcz_store.cc:276] 0709 12:22:33.395288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.594711 (+1199423us) service_pool.cc:224] Handling call
0709 12:22:34.619251 (+ 24540us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.620270  3743 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:34.607836  3717 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.036s	sys 0.081s
I20260709 12:22:34.497637  3649 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.479569  3710 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:34.525720  3397 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:34.620688  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:34.669179  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2219) took 2186 ms. Trace:
I20260709 12:22:34.695863  1798 raft_consensus.cc:2368] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:34.696573  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1298) took 1346 ms. Trace:
I20260709 12:22:34.672009  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:34.697327  1798 rpcz_store.cc:276] 0709 12:22:33.349718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.620391 (+1270673us) service_pool.cc:224] Handling call
0709 12:22:34.696492 (+ 76101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.696030  1732 rpcz_store.cc:276] 0709 12:22:32.475361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.526395 (+ 51034us) service_pool.cc:224] Handling call
0709 12:22:32.529873 (+  3478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.236079 (+706206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:33.236123 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:33.236277 (+   154us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:33.236347 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.661411 (+1425064us) spinlock_profiling.cc:243] Waited 706 ms on lock 0x7b54002b70d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:34.661470 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.661641 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1623406336}
I20260709 12:22:34.698635  1805 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:34.701697  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 435237us
I20260709 12:22:34.702910  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1306) took 1331 ms. Trace:
I20260709 12:22:34.703269  1800 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:34.579638  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 239) took 1147 ms. Trace:
I20260709 12:22:34.704092  1805 rpcz_store.cc:276] 0709 12:22:33.371933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.590534 (+1218601us) service_pool.cc:224] Handling call
0709 12:22:34.702833 (+112299us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.705374  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:34.710716  1807 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:34.711298  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:34.671119  3540 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:34.693528  3717 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s	user 0.036s	sys 0.087s
I20260709 12:22:34.719238  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:34.712764  1732 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:22:34.550966  1801 rpcz_store.cc:269] 0709 12:22:32.245672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.483602 (+237930us) service_pool.cc:224] Handling call
0709 12:22:34.467666 (+1984064us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.674980  3674 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:34.621536  3569 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:34.716398  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:34.714563  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:34.714618  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:34.565968  3450 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.724462  1732 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:34.576812  3552 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.726640  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:34.790148  1805 raft_consensus.cc:2368] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:34.579773  3657 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:34.725037  3674 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:34.738200  3394 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:34.771981  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 347672us
I20260709 12:22:34.711535  3723 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:34.707692  1802 rpcz_store.cc:276] 0709 12:22:33.419322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.979040 (+559718us) service_pool.cc:224] Handling call
0709 12:22:34.566278 (+587238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.790623  1798 raft_consensus.cc:2376] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3. Re-sending same reply.
I20260709 12:22:34.791985  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1299) took 1442 ms. Trace:
I20260709 12:22:34.827116  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:34.799795  1729 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:34.827237  1805 rpcz_store.cc:276] 0709 12:22:33.350010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.715993 (+1365983us) service_pool.cc:224] Handling call
0709 12:22:34.791924 (+ 75931us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.795612  1731 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:34.797633  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:34.828413  1727 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 1.
W20260709 12:22:34.793673  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 236143us
I20260709 12:22:34.828330  2411 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:22:34.829312  1696 connection.cc:256] RPC call timeout handler was delayed by -2.0885540529998252s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2227s for a response
I20260709 12:22:34.798199  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:34.829732  1696 connection.cc:256] RPC call timeout handler was delayed by -2.0867540529998223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22290000000000001s for a response
W20260709 12:22:34.830094  1696 connection.cc:256] RPC call timeout handler was delayed by -1.9806118450001122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
I20260709 12:22:34.796445  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:34.830488  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:34.554737  2414 rpcz_store.cc:276] 0709 12:22:32.252557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.254478 (+  1921us) service_pool.cc:224] Handling call
0709 12:22:32.399004 (+144526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.642903 (+243899us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.658015 (+ 15112us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:32.694927 (+ 36912us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:32.744160 (+ 49233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.640908 (+896748us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400487454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:33.640987 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.641209 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":369609088}
W20260709 12:22:34.830495  1696 connection.cc:256] RPC call timeout handler was delayed by -1.9662118450000889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
I20260709 12:22:34.832887  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:34.838811  1805 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:34.841661  1696 connection.cc:256] RPC call timeout handler was delayed by -1.95451184500007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1751s for a response
W20260709 12:22:34.816195  3766 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.106s	user 0.002s	sys 0.000s
I20260709 12:22:34.842474  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:34.846014  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:34.846235  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1292) took 1492 ms. Trace:
I20260709 12:22:34.856750  1805 rpcz_store.cc:276] 0709 12:22:33.347772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.838278 (+1490506us) service_pool.cc:224] Handling call
0709 12:22:34.838614 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.839283 (+   669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.839478 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:34.860064  1696 connection.cc:256] RPC call timeout handler was delayed by -1.5117118449998088s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
W20260709 12:22:34.860517  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2814825519999431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1796s for a response
W20260709 12:22:34.860935  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0519825520000268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20510000000000003s for a response
I20260709 12:22:34.861001  1730 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:34.863961  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:34.866122  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 69799.2us
W20260709 12:22:34.866317  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2048) took 3432 ms (client timeout 1678 ms). Trace:
W20260709 12:22:34.866662  1872 rpcz_store.cc:269] 0709 12:22:31.433773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.433919 (+   146us) service_pool.cc:224] Handling call
0709 12:22:34.866121 (+3432202us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b540001efd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:34.866245 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3975916416}
I20260709 12:22:34.865937  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:34.869702  1798 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:34.869900  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:34.870258  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1307) took 1498 ms. Trace:
I20260709 12:22:34.871014  2415 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:34.871567  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:34.872671  1798 rpcz_store.cc:276] 0709 12:22:33.373031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.838212 (+1465181us) service_pool.cc:224] Handling call
0709 12:22:34.870173 (+ 31961us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.833729  3556 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.904403  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:34.843101  3773 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
I20260709 12:22:34.832581  3677 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:34.826284  3659 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.828305  3730 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.833961  3363 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.843338  3476 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:34.839272  3561 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:34.865845  3397 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:34.794193  3612 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.714s	user 0.043s	sys 0.043s
I20260709 12:22:34.820982  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:34.919770  1798 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:34.920466  1872 raft_consensus.cc:2368] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4.
I20260709 12:22:34.920624  2412 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:34.915199  3438 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:34.837796  3540 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260709 12:22:34.904970  3588 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:34.937068  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 70277.4us
I20260709 12:22:34.914759  3642 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:34.933625  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 215) took 1843 ms (client timeout 1907 ms). Trace:
I20260709 12:22:34.805397  3569 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.904004  3776 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
I20260709 12:22:34.934577  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1295) took 1585 ms. Trace:
W20260709 12:22:34.938536  1872 rpcz_store.cc:269] 0709 12:22:33.090278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.903587 (+1813309us) service_pool.cc:224] Handling call
0709 12:22:34.933529 (+ 29942us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.904707  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:34.943332  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:34.943544  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:34.944866  3775 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:22:34.937150  3394 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:34.925992  3677 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.946798  3394 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:34.948022  3394 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:34.949113  3677 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:34.945483  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:34.950868  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2075) took 1568 ms (client timeout 1633 ms). Trace:
I20260709 12:22:34.940450  3642 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:34.951699  3642 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:34.952745  3469 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:34.953277  3469 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:34.955253  1872 rpcz_store.cc:269] 0709 12:22:33.382414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.942400 (+1559986us) service_pool.cc:224] Handling call
0709 12:22:34.950754 (+  8354us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:34.942833  1798 rpcz_store.cc:276] 0709 12:22:33.348777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.873837 (+1525060us) service_pool.cc:224] Handling call
0709 12:22:34.919546 (+ 45709us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.934238 (+ 14692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.934503 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:34.935257  3612 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.855s	user 0.045s	sys 0.044s
I20260709 12:22:34.961726  3612 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:34.963359  3612 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.929754  3363 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.971374  3450 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:34.971951  3710 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:34.929946  3659 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.974521  3363 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:34.975303  3659 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:34.925798  3556 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:34.935956  3397 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:34.934226  3702 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:34.987012  1802 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
W20260709 12:22:34.988821  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.990162  3552 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:34.920991  3674 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:34.992630  1724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2210) took 2523 ms (client timeout 2239 ms). Trace:
I20260709 12:22:34.986568  3556 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:34.994736  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 212) took 1990 ms (client timeout 1947 ms). Trace:
W20260709 12:22:35.018008  1877 rpcz_store.cc:269] 0709 12:22:33.003977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.994374 (+1990397us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:34.994582 (+   208us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:35.020005  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:35.021644  1877 raft_consensus.cc:2410] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
W20260709 12:22:35.008540  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.023175  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.023952  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.998701  3397 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:34.994135  3569 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:35.043213  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2076) took 1660 ms (client timeout 1655 ms). Trace:
W20260709 12:22:35.014304  1724 rpcz_store.cc:269] 0709 12:22:32.469476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.469635 (+   159us) service_pool.cc:224] Handling call
0709 12:22:34.992414 (+2522779us) spinlock_profiling.cc:243] Waited 901 ms on lock 0x7b54002b66d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:34.992554 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2072633600}
W20260709 12:22:35.026913  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.053126  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:35.056612  2419 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:35.057119  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:35.058409  1872 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 2: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:34.940094  1801 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:35.071887  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.073395  1798 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:35.074815  1847 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:43056 (request call id 77) took 1697 ms. Trace:
I20260709 12:22:34.954177  3469 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:35.075114  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 218) took 1983 ms (client timeout 1939 ms). Trace:
W20260709 12:22:35.058403  1877 rpcz_store.cc:269] 0709 12:22:33.382655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.019286 (+1636631us) service_pool.cc:224] Handling call
0709 12:22:35.043058 (+ 23772us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:35.013583  3772 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:22:34.927907  3730 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:35.088696  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 228178us
W20260709 12:22:35.089677  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2078) took 1700 ms (client timeout 1658 ms). Trace:
W20260709 12:22:35.090226  1877 rpcz_store.cc:269] 0709 12:22:33.383341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.083724 (+1700383us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:35.083909 (+   185us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:22:35.091481  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 219) took 1999 ms (client timeout 1961 ms). Trace:
W20260709 12:22:35.091913  1877 rpcz_store.cc:269] 0709 12:22:33.097262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.091189 (+1993927us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:35.091395 (+   206us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:35.093094  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:35.093962  1877 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
W20260709 12:22:35.094261  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.078380  1847 rpcz_store.cc:276] 0709 12:22:33.381471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.074699 (+1693228us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:22:35.081588  1872 rpcz_store.cc:269] 0709 12:22:33.091259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.981299 (+1890040us) service_pool.cc:224] Handling call
0709 12:22:35.075034 (+ 93735us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:35.097868  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:35.085069  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:35.014091  3674 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:35.011798  3352 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:35.094563  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 220) took 1988 ms (client timeout 2022 ms). Trace:
W20260709 12:22:35.106135  1877 rpcz_store.cc:269] 0709 12:22:33.106518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.092656 (+1986138us) service_pool.cc:224] Handling call
0709 12:22:35.094499 (+  1843us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:35.002431  3657 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:35.107573  1798 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:22:35.110359  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 217) took 2009 ms (client timeout 2053 ms). Trace:
W20260709 12:22:35.111438  1872 rpcz_store.cc:269] 0709 12:22:33.090931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.097493 (+2006562us) service_pool.cc:224] Handling call
0709 12:22:35.098810 (+  1317us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:35.086678  3758 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.086395  3723 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:35.121273  1802 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:22:35.129202  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 425) took 1297 ms. Trace:
I20260709 12:22:35.131070  3760 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:35.141041  3657 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:35.133224  3758 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.141578  3657 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:35.141742  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:35.142663  3657 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.143209  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1291) took 1813 ms. Trace:
I20260709 12:22:35.143636  3758 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:35.144008  3657 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:35.144413  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:35.144976  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1300) took 1778 ms. Trace:
I20260709 12:22:35.145426  1802 rpcz_store.cc:276] 0709 12:22:33.368360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.120420 (+1752060us) service_pool.cc:224] Handling call
0709 12:22:35.120966 (+   546us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:35.144627 (+ 23661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.144896 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:35.133343  3777 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.194s	user 0.001s	sys 0.001s
W20260709 12:22:35.147383  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2047) took 3714 ms (client timeout 2223 ms). Trace:
I20260709 12:22:35.147502  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:35.147820  1878 rpcz_store.cc:269] 0709 12:22:31.433207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.433435 (+   228us) service_pool.cc:224] Handling call
0709 12:22:31.998565 (+565130us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:22:35.147195 (+3148630us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:35.147304 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5480933632}
I20260709 12:22:35.148754  3760 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:35.149446  3760 consensus_queue.cc:1243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 6.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:35.149925  3760 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:35.150239  3760 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:35.150609  3760 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:35.151796  3758 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:35.152184  3758 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:35.153154  3758 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:35.154280  3758 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:35.155575  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 214) took 2065 ms (client timeout 1616 ms). Trace:
W20260709 12:22:35.156183  1876 rpcz_store.cc:269] 0709 12:22:33.089910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.155292 (+2065382us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:35.155471 (+   179us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:35.148370  1802 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:22:35.159600  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1304) took 1788 ms. Trace:
I20260709 12:22:35.163564  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:35.164507  2412 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:35.165448  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 593) took 2164 ms. Trace:
I20260709 12:22:35.165900  2417 rpcz_store.cc:276] 0709 12:22:33.000977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.001094 (+   117us) service_pool.cc:224] Handling call
0709 12:22:33.001378 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.255454 (+1254076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.165126 (+909672us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54003c7154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:35.165365 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2882231040}
W20260709 12:22:35.168219  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1491) took 3676 ms (client timeout 1567 ms). Trace:
W20260709 12:22:35.168645  1881 rpcz_store.cc:269] 0709 12:22:31.491683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.739793 (+248110us) service_pool.cc:224] Handling call
0709 12:22:31.904549 (+164756us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:22:35.168017 (+3263468us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:35.168148 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3334353536}
I20260709 12:22:35.171083  3649 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:35.162318  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:35.173748  1728 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:35.174728  3552 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:35.175383  3552 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.175805  3552 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:35.176385  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:35.178953  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:35.179179  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 204) took 2724 ms (client timeout 2221 ms). Trace:
W20260709 12:22:35.179586  1879 rpcz_store.cc:269] 0709 12:22:32.454369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.179009 (+2724640us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:35.179120 (+   111us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:35.181178  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:35.136897  3730 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:35.137940  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1297) took 1788 ms (client timeout 2070 ms). Trace:
W20260709 12:22:35.138806  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 224) took 2893 ms (client timeout 1871 ms). Trace:
I20260709 12:22:35.131137  3715 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:35.138190  3723 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.182659  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:35.143573  1800 rpcz_store.cc:276] 0709 12:22:33.329693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.338511 (+  8818us) service_pool.cc:224] Handling call
0709 12:22:33.338971 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.708348 (+1369377us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.142933 (+434585us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540009fbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:35.143138 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3137055616}
I20260709 12:22:35.183269  2417 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Advancing to term 4
I20260709 12:22:35.241528  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:35.244140  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:35.210841  2419 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260709 12:22:35.247764  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66711us
I20260709 12:22:35.249624  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:35.254174  1807 rpcz_store.cc:269] 0709 12:22:33.349379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.581821 (+1232442us) service_pool.cc:224] Handling call
0709 12:22:35.137686 (+555865us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b5400132254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:35.137863 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287152896}
I20260709 12:22:35.256018  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.159976  1802 rpcz_store.cc:276] 0709 12:22:33.371194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.146331 (+1775137us) service_pool.cc:224] Handling call
0709 12:22:35.146711 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.159229 (+ 12518us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.159528 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:35.257656  1804 rpcz_store.cc:269] 0709 12:22:32.244855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.446679 (+201824us) service_pool.cc:224] Handling call
0709 12:22:35.138590 (+2691911us) spinlock_profiling.cc:243] Waited 2.1 s on lock 0x7b5400167bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:35.138715 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4833537408}
I20260709 12:22:35.233054  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:35.288106  1728 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:35.269423  1732 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:35.239375  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 228) took 2993 ms. Trace:
I20260709 12:22:35.289196  1803 rpcz_store.cc:276] 0709 12:22:32.245497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.080270 (+834773us) service_pool.cc:224] Handling call
0709 12:22:33.080578 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:33.292630 (+212052us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:33.292677 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:33.292744 (+    67us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:33.292809 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.239002 (+1946193us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54001676d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:35.239075 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.239282 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":486997760}
W20260709 12:22:35.298354  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45137.4us
I20260709 12:22:35.309535  2417 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 4 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:35.298935  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.303628  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:35.311636  1800 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:35.316109  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.184134  3710 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.320133  3710 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.322271  3710 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:35.322875  1798 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:35.238678  3782 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:35.323670  3782 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:35.324070  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1312) took 1933 ms. Trace:
I20260709 12:22:35.324440  1804 rpcz_store.cc:276] 0709 12:22:33.390650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.311129 (+1920479us) service_pool.cc:224] Handling call
0709 12:22:35.323562 (+ 12433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.323742 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:35.323767 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:35.323825 (+    58us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:35.323855 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.323908 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.324014 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:35.325798  3782 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:35.328317  3642 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.333456  3744 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:35.335830  1804 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:35.190747  3569 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:35.337427  2417 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:22:35.226437  3450 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:35.346880  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 290531us
I20260709 12:22:35.361467  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:35.364342  2383 connection.cc:256] RPC call timeout handler was delayed by -1.6494938639998509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
W20260709 12:22:35.364748  2383 connection.cc:256] RPC call timeout handler was delayed by -1.5375982139998996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
I20260709 12:22:35.365088  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:35.365293  2383 connection.cc:256] RPC call timeout handler was delayed by -1.4989714430003005s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2053s for a response
W20260709 12:22:35.365739  2383 connection.cc:256] RPC call timeout handler was delayed by -1.4199938639999345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
I20260709 12:22:35.348333  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:35.366148  2383 connection.cc:256] RPC call timeout handler was delayed by -1.3989521009998498s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19390000000000002s for a response
W20260709 12:22:35.366549  2383 connection.cc:256] RPC call timeout handler was delayed by -1.351593863999824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1947s for a response
I20260709 12:22:35.366621  1800 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
W20260709 12:22:35.367029  2383 connection.cc:256] RPC call timeout handler was delayed by -1.351593863999824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1947s for a response
W20260709 12:22:35.367398  2383 connection.cc:256] RPC call timeout handler was delayed by -1.3389938639998036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
W20260709 12:22:35.367720  2383 connection.cc:256] RPC call timeout handler was delayed by -1.2840938639997148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2022s for a response
I20260709 12:22:35.367849  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:35.368134  2383 connection.cc:256] RPC call timeout handler was delayed by -1.2741938640001536s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
W20260709 12:22:35.368533  2383 connection.cc:256] RPC call timeout handler was delayed by -1.2741938640001536s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2033s for a response
W20260709 12:22:35.368876  2383 connection.cc:256] RPC call timeout handler was delayed by -1.10499386399988s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22210000000000002s for a response
I20260709 12:22:35.330574  3642 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:35.372542  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:35.369415  3687 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:35.363765  3743 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 12:22:35.359513  3677 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:35.141194  2413 rpcz_store.cc:276] 0709 12:22:33.831305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.832097 (+   792us) service_pool.cc:224] Handling call
0709 12:22:33.832428 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.303246 (+470818us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:34.709806 (+406560us) log.cc:844] Serialized 14 byte log entry
0709 12:22:34.845184 (+135378us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:34.860827 (+ 15643us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:34.867102 (+  6275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.127219 (+260117us) spinlock_profiling.cc:243] Waited 5.34 ms on lock 0x7b18002e98a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419
0709 12:22:35.128865 (+  1646us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b5400488d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:35.128932 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.129121 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18903,"prepare.queue_time_us":268350,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":115,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":4582,"spinlock_wait_cycles":1059103488,"thread_start_us":501816,"threads_started":2,"wal-append.queue_time_us":266211}
I20260709 12:22:35.378953  3702 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:35.379370  3715 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:35.379997  3742 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:35.380549  3790 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:35.381816  1801 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:35.382807  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:35.314493  3760 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.383344  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.099623  1847 service_pool.cc:137] LeaderStepDown request on kudu.consensus.ConsensusService from 127.0.0.1:43056 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 226)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 225)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2080)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2081)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 221)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 209)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 213)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2082)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2077)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2089)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2085)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
I20260709 12:22:35.384399  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.394088  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:35.400431  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:35.402019  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:35.391953  3552 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:35.409940  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:35.423388  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.429534  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.344017  3561 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20260709 12:22:35.431424  2419 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:35.400274  3782 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:35.436655  1800 raft_consensus.cc:2368] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3.
I20260709 12:22:35.474174  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:35.400108  3677 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:35.475164  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.439940  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:35.476861  1800 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:35.480340  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:35.482895  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.488508  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.433565  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1303) took 2062 ms. Trace:
I20260709 12:22:35.503317  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:35.504732  1728 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:35.434156  3797 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:35.397332  3687 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:35.502957  1801 rpcz_store.cc:276] 0709 12:22:33.370897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.747614 (+1376717us) service_pool.cc:224] Handling call
0709 12:22:35.430659 (+683045us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:35.476702  3782 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.419693  3476 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:22:35.420855  3450 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.419920  3715 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:35.518465  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.557301  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.557883  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.476730  3540 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:35.582756  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:35.430335  3783 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:35.477985  3677 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:35.527880  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:35.599100  2419 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:35.438866  1802 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:35.504278  3657 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.429787  3742 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:35.572108  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.614246  1805 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:35.525763  3782 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:35.639668  3540 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:35.599715  3450 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:35.620774  3739 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:35.594082  3715 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:35.510959  3797 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:35.429868  3702 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:35.438108  3744 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:35.701238  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.435009  3790 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:35.749374  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 309494us
I20260709 12:22:35.752301  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:35.759248  1804 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:35.764353  2383 leader_election.cc:400] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4.
I20260709 12:22:35.795509  2383 leader_election.cc:403] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:22:35.639164  3742 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:35.800827  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:35.801499  2415 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:35.804071  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:35.804692  2419 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:35.802388  1615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 296305us
W20260709 12:22:35.807374  3795 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.295s	user 0.000s	sys 0.001s
W20260709 12:22:35.820281  1847 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55690 dropped due to backpressure: service queue is full with 50 items [suppressed 1 similar messages]
I20260709 12:22:35.824849  3677 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.800901  3744 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:35.831740  3744 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:35.834019  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:35.834839  1806 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:35.835703  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.835543  3797 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.836535  3797 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:35.836830  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:35.837837  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.838354  1806 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:35.842972  3741 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:35.843153  3796 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.843550  3741 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:35.843581  3796 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:35.845769  3741 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:35.845988  3796 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:35.857739  1771 connection.cc:256] RPC call timeout handler was delayed by -2.194645145999857s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2258s for a response
W20260709 12:22:35.858259  1771 connection.cc:256] RPC call timeout handler was delayed by -2.1935837420001008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1575s for a response
I20260709 12:22:35.858428  2419 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
W20260709 12:22:35.861104  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 96960.1us
I20260709 12:22:35.820643  1847 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55690 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 213)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2082)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2077)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2089)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1522)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2085)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526)
I20260709 12:22:35.861793  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:35.866055  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:35.867552  1771 connection.cc:256] RPC call timeout handler was delayed by -2.1413837420000164s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1633s for a response
W20260709 12:22:35.868429  1771 connection.cc:256] RPC call timeout handler was delayed by -1.9843475849997958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22010000000000002s for a response
W20260709 12:22:35.868820  1771 connection.cc:256] RPC call timeout handler was delayed by -1.8185143910000079s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
W20260709 12:22:35.869170  1771 connection.cc:256] RPC call timeout handler was delayed by -1.8158143910000035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
W20260709 12:22:35.869567  1771 connection.cc:256] RPC call timeout handler was delayed by -1.8086143909999919s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1666s for a response
W20260709 12:22:35.869932  1771 connection.cc:256] RPC call timeout handler was delayed by -1.6310837420001008s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22000000000000003s for a response
W20260709 12:22:35.870296  1771 connection.cc:256] RPC call timeout handler was delayed by -1.4237060879995624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1678s for a response
W20260709 12:22:35.870857  1771 connection.cc:256] RPC call timeout handler was delayed by -1.343314391000149s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2183s for a response
W20260709 12:22:35.871245  1771 connection.cc:256] RPC call timeout handler was delayed by -1.2056283500000973s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1768s for a response
W20260709 12:22:35.871630  1771 connection.cc:256] RPC call timeout handler was delayed by -1.1550349600001937s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15460000000000002s for a response
W20260709 12:22:35.871999  1771 connection.cc:256] RPC call timeout handler was delayed by -1.0866349600000831s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
I20260709 12:22:35.641180  3657 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.877992  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:35.613862  3783 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:35.880648  3783 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:35.880939  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1302) took 2511 ms. Trace:
I20260709 12:22:35.881316  1799 rpcz_store.cc:276] 0709 12:22:33.370497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.146174 (+1775677us) service_pool.cc:224] Handling call
0709 12:22:35.146544 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:35.218836 (+ 72292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:35.218883 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:35.316832 (+ 97949us) log.cc:844] Serialized 26 byte log entry
0709 12:22:35.518674 (+201842us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:35.721136 (+202462us) log.cc:844] Serialized 14 byte log entry
0709 12:22:35.746491 (+ 25355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.746571 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:35.880733 (+134162us) raft_consensus.cc:1685] finished
0709 12:22:35.880752 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.880878 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50922,"prepare.queue_time_us":245093,"raft.queue_time_us":58340,"raft.run_cpu_time_us":439,"raft.run_wall_time_us":72981,"spinlock_wait_cycles":4226944,"thread_start_us":318150,"threads_started":2,"wal-append.queue_time_us":539723}
I20260709 12:22:35.883297  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:35.884927  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:35.892982  1696 leader_election.cc:400] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3.
I20260709 12:22:35.893350  1696 leader_election.cc:403] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:35.895298  1804 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:35.895792  2415 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:35.897688  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.900084  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:35.904120  1847 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55690 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 213)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2082)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2077)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2089)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1522)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2085)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526)
I20260709 12:22:35.854684  3687 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:35.865758  3644 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:35.927902  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:35.930616  1806 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:35.877080  3815 raft_consensus.cc:2764] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
W20260709 12:22:35.941138  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.909796  3397 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:35.899626  3747 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4.
I20260709 12:22:35.891419  3810 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:35.909646  3642 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3.
I20260709 12:22:35.925432  3677 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:35.952533  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:35.949313  1847 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55690 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 213)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2082)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2077)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2089)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1522)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2085)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526)
I20260709 12:22:35.949657  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:35.967620  3552 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:35.943056  3796 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.954283  3741 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.917318  3783 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260709 12:22:35.952728  3397 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.987485  3397 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:35.939896  3644 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:35.985680  3657 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:36.015225  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2074) took 2633 ms (client timeout 1575 ms). Trace:
W20260709 12:22:36.015792  1875 rpcz_store.cc:269] 0709 12:22:33.382141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.014981 (+2632840us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.015147 (+   166us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.016463  3644 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:35.974077  3760 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:35.976006  3776 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:35.971020  3810 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:36.022825  3810 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.026350  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:36.026830  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:36.029040  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:36.027922  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:35.971908  1847 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55690 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 213)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2082)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2084)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 222)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2077)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 223)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2087)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2089)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1518)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1522)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2085)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526)
I20260709 12:22:36.030200  1806 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:36.034653  1799 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:36.037281  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:36.038007  1732 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:36.040863  3657 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:36.047012  3657 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.047226  3702 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:36.047462  3657 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:35.986447  3741 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.051347  3469 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:36.051818  3715 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.053217  3741 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.054304  3715 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:35.984959  3796 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:36.056520  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.057109  3796 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.059341  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:36.060346  2419 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:36.074025  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:36.074923  2412 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:22:36.075515  3765 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.505s	user 0.002s	sys 0.012s
I20260709 12:22:36.078346  1804 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:36.079142  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 224) took 2862 ms (client timeout 1616 ms). Trace:
W20260709 12:22:36.079512  3809 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:22:36.081976  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1294) took 2733 ms. Trace:
I20260709 12:22:36.084189  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1318) took 1126 ms. Trace:
I20260709 12:22:36.084607  1798 rpcz_store.cc:276] 0709 12:22:34.958270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.111710 (+153440us) service_pool.cc:224] Handling call
0709 12:22:36.083992 (+972282us) spinlock_profiling.cc:243] Waited 72.3 ms on lock 0x7b54003b88d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:36.084123 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166333184}
I20260709 12:22:36.086989  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1310) took 2710 ms. Trace:
I20260709 12:22:36.087373  1807 rpcz_store.cc:276] 0709 12:22:33.377873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.310601 (+1932728us) service_pool.cc:224] Handling call
0709 12:22:35.311689 (+  1088us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:35.521279 (+209590us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:35.521325 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:35.582589 (+ 61264us) log.cc:844] Serialized 26 byte log entry
0709 12:22:35.951889 (+369300us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:35.951930 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.952011 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:36.086733 (+134722us) raft_consensus.cc:1685] finished
0709 12:22:36.086751 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.086925 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2234,"raft.queue_time_us":6811,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":304,"thread_start_us":502539,"threads_started":2,"wal-append.queue_time_us":355564}
I20260709 12:22:36.091384  3782 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.093344  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:36.094301  3782 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:36.079573  1873 rpcz_store.cc:269] 0709 12:22:33.216647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.078909 (+2862262us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.079042 (+   133us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.110908  1732 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:36.096700  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:36.095147  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:36.095956  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:36.104514  3450 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:36.097450  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:36.134387  1723 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:36.133526  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 240652us
W20260709 12:22:36.072567  3807 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.060s	user 0.000s	sys 0.002s
I20260709 12:22:36.041610  3760 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:36.101132  3790 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:36.082372  1805 rpcz_store.cc:276] 0709 12:22:33.348460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.858148 (+1509688us) service_pool.cc:224] Handling call
0709 12:22:34.858636 (+   488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.615296 (+756660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.081735 (+466439us) spinlock_profiling.cc:243] Waited 755 ms on lock 0x7b5400133154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:36.081917 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1737051008}
I20260709 12:22:36.122695  1724 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:36.133407  1730 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:36.137334  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2082) took 2622 ms (client timeout 1768 ms). Trace:
W20260709 12:22:36.140208  1873 rpcz_store.cc:269] 0709 12:22:33.524134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.137143 (+2613009us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.137266 (+   123us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:22:36.146286  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2084) took 2340 ms (client timeout 1546 ms). Trace:
W20260709 12:22:36.146927  1873 rpcz_store.cc:269] 0709 12:22:33.805880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.146010 (+2340130us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.146196 (+   186us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.135614  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:22:36.151337  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 222) took 2955 ms (client timeout 2221 ms). Trace:
I20260709 12:22:36.151914  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:22:36.151948  1873 rpcz_store.cc:269] 0709 12:22:33.195716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.151087 (+2955371us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.151252 (+   165us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:22:36.157459  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2077) took 2774 ms (client timeout 2183 ms). Trace:
W20260709 12:22:36.158098  1873 rpcz_store.cc:269] 0709 12:22:33.383024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.157198 (+2774174us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.157373 (+   175us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.158495  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:36.159394  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2073) took 2777 ms (client timeout 2200 ms). Trace:
W20260709 12:22:36.159911  1873 rpcz_store.cc:269] 0709 12:22:33.381849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.159088 (+2777239us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.159318 (+   230us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:22:36.160925  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 223) took 2954 ms (client timeout 2615 ms). Trace:
W20260709 12:22:36.162035  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 287443us
W20260709 12:22:36.182070  1873 rpcz_store.cc:269] 0709 12:22:33.206610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.160733 (+2954123us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.160861 (+   128us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.183076  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:36.136616  3760 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.170838  1724 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:36.166052  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:36.177156  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.189909  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:36.191329  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:36.197960  1723 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:36.162956  3783 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:36.149022  3815 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:22:36.183403  3644 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:36.184224  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2087) took 1698 ms (client timeout 1622 ms). Trace:
I20260709 12:22:36.184253  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:36.217206  1873 rpcz_store.cc:269] 0709 12:22:34.486257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.182769 (+1696512us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.184157 (+  1388us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.190099  3760 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:36.145726  3657 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:36.224829  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.201874  3552 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:36.199878  3717 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:36.244120  1873 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:22:36.232640  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 280218us
I20260709 12:22:36.172145  3397 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.203709  3783 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:36.252386  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:36.254501  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2089) took 1730 ms (client timeout 1950 ms). Trace:
W20260709 12:22:36.256093  1873 rpcz_store.cc:269] 0709 12:22:34.523841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.222515 (+1698674us) service_pool.cc:224] Handling call
0709 12:22:36.254409 (+ 31894us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.258106  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 233) took 3991 ms. Trace:
I20260709 12:22:36.258137  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.260072  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1514) took 1226 ms. Trace:
I20260709 12:22:36.258698  1725 rpcz_store.cc:276] 0709 12:22:32.266289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.267931 (+  1642us) service_pool.cc:224] Handling call
0709 12:22:32.278065 (+ 10134us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.422925 (+2144860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.257841 (+1834916us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540044a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:36.258035 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2913614592}
I20260709 12:22:36.262506  1873 rpcz_store.cc:276] 0709 12:22:35.032457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.257300 (+1224843us) service_pool.cc:224] Handling call
0709 12:22:36.259115 (+  1815us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:36.261847  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2081) took 2747 ms (client timeout 1678 ms). Trace:
W20260709 12:22:36.264536  1881 rpcz_store.cc:269] 0709 12:22:33.514267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.261564 (+2747297us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.261744 (+   180us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.254024  3717 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:36.266300  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 208937us
W20260709 12:22:36.268266  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.268816  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.258244  3783 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:36.266961  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 223663us
W20260709 12:22:36.271162  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.256233  3552 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:36.265141  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:36.273701  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.273984  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.261019  3469 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:36.313782  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 440866us
I20260709 12:22:36.270084  3717 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:36.271184  3822 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:36.282255  3476 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:36.286176  3820 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:36.304823  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.297215  3742 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.306282  1873 raft_consensus.cc:2376] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 4. Re-sending same reply.
I20260709 12:22:36.333159  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1513) took 1301 ms. Trace:
I20260709 12:22:36.333686  1873 rpcz_store.cc:276] 0709 12:22:35.032170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.273329 (+1241159us) service_pool.cc:224] Handling call
0709 12:22:36.333064 (+ 59735us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.266556  3702 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:36.317910  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:36.318998  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 134790us
I20260709 12:22:36.320099  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:36.313313  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48298.8us
I20260709 12:22:36.340622  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:36.335415  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:36.317754  3735 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:36.339027  1846 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1522)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1516)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2085)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2090)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2091)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2093)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2097)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2094)
I20260709 12:22:36.345268  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.346041  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:36.350060  2383 leader_election.cc:336] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.616s (SENT)
I20260709 12:22:36.350682  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:36.351107  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.353289  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.353348  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:36.344998  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.359145  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:36.346092  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 156227us
I20260709 12:22:36.346998  3642 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:36.348083  3657 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.353981  3739 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:36.362506  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:36.363061  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:36.346122  3450 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:36.370769  1881 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:36.371479  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1512) took 1339 ms (client timeout 1713 ms). Trace:
W20260709 12:22:36.371826  1881 rpcz_store.cc:269] 0709 12:22:35.031850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.265543 (+1233693us) service_pool.cc:224] Handling call
0709 12:22:36.371423 (+105880us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.346026  3476 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:36.360647  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:36.352526  3394 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:22:36.364562  1771 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Remote error: Timed out: Call waited in the queue past client deadline
I20260709 12:22:36.383963  3397 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:36.363205  3827 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:36.363405  3744 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:36.360471  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.392035  1873 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:36.392333  1845 leader_election.cc:400] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:36.392575  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1516) took 1359 ms. Trace:
I20260709 12:22:36.392951  1873 rpcz_store.cc:276] 0709 12:22:35.033081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.360207 (+1327126us) service_pool.cc:224] Handling call
0709 12:22:36.392514 (+ 32307us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.394330  1845 leader_election.cc:400] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:36.394764  1845 leader_election.cc:403] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:22:36.394965  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:36.391196  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.396356  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.362097  1846 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1524)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1528)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2085)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2104)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2090)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2091)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2093)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2097)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2094)
I20260709 12:22:36.363361  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:36.400249  3824 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:36.400470  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2080) took 2886 ms (client timeout 1666 ms). Trace:
I20260709 12:22:36.400835  3824 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:22:36.401103  1872 rpcz_store.cc:269] 0709 12:22:33.513943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.400215 (+2886272us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.400382 (+   167us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.401870  3824 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.402359  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.402988  3822 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:22:36.407466  1872 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:36.359923  1807 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:36.347136  3735 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:36.365820  3642 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:36.385748  3825 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:22:36.386472  3394 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:36.386832  3742 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.352649  3790 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:36.384763  3827 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:36.416126  3827 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:36.387090  3833 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:22:36.387511  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:36.419440  1724 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:36.387713  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:36.388731  3476 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:36.389217  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 226) took 3144 ms (client timeout 1907 ms). Trace:
I20260709 12:22:36.376128  3810 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:36.411034  3735 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:36.364845  3818 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:22:36.417514  3469 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:36.397172  3397 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:36.404663  3783 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.443097  3397 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:36.443672  3397 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:36.364053  3552 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:36.405807  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 211) took 3588 ms (client timeout 1939 ms). Trace:
I20260709 12:22:36.444900  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:36.445188  1880 rpcz_store.cc:269] 0709 12:22:32.817143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.405607 (+3588464us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.405737 (+   130us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.445930  1798 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:36.446347  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.416365  3642 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.447096  1880 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:36.447636  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1509) took 1417 ms. Trace:
I20260709 12:22:36.447978  1880 rpcz_store.cc:276] 0709 12:22:35.030426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.446027 (+1415601us) service_pool.cc:224] Handling call
0709 12:22:36.447571 (+  1544us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.436973  3790 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:36.449065  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.409142  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2085) took 2438 ms. Trace:
W20260709 12:22:36.409970  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.363838  1723 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:36.410851  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:36.455760  1803 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:36.414443  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.459065  1872 rpcz_store.cc:276] 0709 12:22:33.971015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.402015 (+2431000us) service_pool.cc:224] Handling call
0709 12:22:36.409058 (+  7043us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:36.373610  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.409256  1846 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1520)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1529)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1521)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2107)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2105)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2104)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2090)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2091)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2093)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2097)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2094)
W20260709 12:22:36.421835  1877 rpcz_store.cc:269] 0709 12:22:33.244458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.388999 (+3144541us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.389141 (+   142us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.406257  3823 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:36.396924  1771 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Remote error: Timed out: Call waited in the queue past client deadline
I20260709 12:22:36.473353  1807 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:22:36.473688  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76819us
I20260709 12:22:36.405915  3731 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:36.458410  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.460558  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1417) took 1144 ms. Trace:
W20260709 12:22:36.460816  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66663.8us
I20260709 12:22:36.477421  1803 rpcz_store.cc:276] 0709 12:22:35.316048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.410487 (+1094439us) service_pool.cc:224] Handling call
0709 12:22:36.460489 (+ 50002us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.478260  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:36.413209  1873 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:36.466914  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 92357.6us
W20260709 12:22:36.479300  2383 leader_election.cc:336] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.907s (SENT)
I20260709 12:22:36.471611  1724 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:36.471750  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:36.480741  2383 leader_election.cc:336] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.939s (SENT)
I20260709 12:22:36.480866  1877 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:36.481184  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.481415  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1521) took 1005 ms. Trace:
I20260709 12:22:36.477118  1846 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2098)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2107)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2105)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2108)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2104)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2090)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2091)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2111)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2093)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2109)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2112)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2110)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2094)
W20260709 12:22:36.482630  2383 leader_election.cc:336] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.947s (SENT)
W20260709 12:22:36.483608  2383 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.961s (SENT)
I20260709 12:22:36.481765  1877 rpcz_store.cc:276] 0709 12:22:35.475979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.471467 (+995488us) service_pool.cc:224] Handling call
0709 12:22:36.481341 (+  9874us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.468564  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.475561  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:36.475946  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.476183  1807 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260709 12:22:36.476505  1771 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.546s (SENT)
I20260709 12:22:36.487718  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1517) took 1454 ms. Trace:
W20260709 12:22:36.488430  1771 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.622s (SENT)
I20260709 12:22:36.483973  1846 service_pool.cc:137] LeaderStepDown request on kudu.consensus.ConsensusService from 127.0.0.1:43056 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2099)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2107)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2105)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2101)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2108)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2104)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2090)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2092)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2091)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2095)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2111)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2102)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2093)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2109)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1533)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2096)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1534)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1532)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2097)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2112)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2110)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2094)
I20260709 12:22:36.488934  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.444909  3783 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:36.498121  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:36.490008  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:36.490419  1771 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.666s (SENT)
I20260709 12:22:36.500483  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:36.502359  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:36.504431  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:36.484078  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.503572  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:36.492945  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.506017  1724 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:36.466292  3702 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:36.506192  1873 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:36.495800  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:36.497077  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:36.485639  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.510545  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.511210  1877 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:36.511363  3394 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.512169  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.512890  1877 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:36.488183  1872 rpcz_store.cc:276] 0709 12:22:35.033389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.468256 (+1434867us) service_pool.cc:224] Handling call
0709 12:22:36.487653 (+ 19397us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.504305  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:36.507184  2383 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.022s (SENT)
I20260709 12:22:36.491465  1807 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:36.508184  1803 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:36.482026  3810 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:22:36.515215  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.517457  1872 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:36.508306  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.473057  3823 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:36.520458  1771 leader_election.cc:336] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.678s (SENT)
W20260709 12:22:36.607064  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86944.1us
I20260709 12:22:36.494382  3657 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.496726  3821 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:36.514106  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.497812  3690 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:36.515573  3702 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:36.519426  3796 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:36.503995  3776 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:36.520012  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:36.554473  3823 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:36.626502  1724 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:22:36.561908  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.638417  1877 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:36.514729  3826 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:36.525071  3552 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:36.513029  3815 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:36.513132  3791 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:36.630385  3783 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.011s
I20260709 12:22:36.645939  3791 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:36.646406  3791 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:36.619953  3657 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.497999  3744 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:36.449460  3790 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.604300  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:36.519845  3642 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:36.666188  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 150802us
I20260709 12:22:36.518121  1873 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:36.620551  3810 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:36.608560  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1519) took 1170 ms. Trace:
I20260709 12:22:36.667711  1872 rpcz_store.cc:276] 0709 12:22:35.438431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.514958 (+1076527us) service_pool.cc:224] Handling call
0709 12:22:36.608477 (+ 93519us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:36.615024  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 107867us
W20260709 12:22:36.668322  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64100.3us
W20260709 12:22:36.669484  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.033s (SENT)
W20260709 12:22:36.616230  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 95797.4us
I20260709 12:22:36.617079  1798 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:36.672595  3642 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:36.673411  2413 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 2 LEARNER]: Advancing to term 3
I20260709 12:22:36.609385  3469 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:22:36.673650  3642 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.616470  3715 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.674970  3715 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.676446  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:36.629098  3717 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.606606  3540 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:36.619901  3796 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:36.683725  1724 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:36.684396  2410 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:36.617161  3837 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:36.616680  3841 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.167s	user 0.000s	sys 0.006s
I20260709 12:22:36.603641  3741 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:36.639757  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1511) took 1608 ms. Trace:
I20260709 12:22:36.620250  3760 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.653277  3450 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:36.683936  3849 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:36.610181  3364 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:22:36.656642  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:36.707144  3741 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:36.659086  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:36.707726  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 51189us
I20260709 12:22:36.657796  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.708011  1802 raft_consensus.cc:2376] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 5. Re-sending same reply.
I20260709 12:22:36.707728  3741 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:36.662144  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:36.708812  3552 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:36.708977  3657 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.663300  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:36.709652  3715 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:36.709941  2411 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 2: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:22:36.607384  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 86956.5us
I20260709 12:22:36.641011  3826 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.710753  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:36.711117  3826 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:36.616778  3644 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
W20260709 12:22:36.710819  1771 leader_election.cc:336] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.768s (SENT)
I20260709 12:22:36.682037  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.686532  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:36.696851  1877 rpcz_store.cc:276] 0709 12:22:35.031521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.513509 (+1481988us) service_pool.cc:224] Handling call
0709 12:22:36.639689 (+126180us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:36.679657  2383 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.169s (SENT)
I20260709 12:22:36.698469  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:36.707271  3717 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:36.683398  3540 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:36.708155  3760 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.677125  2415 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:36.709412  3741 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.713565  1771 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.687750  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:36.714985  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.715021  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.735248  1872 raft_consensus.cc:2376] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5. Re-sending same reply.
W20260709 12:22:36.735471  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55861.9us
W20260709 12:22:36.736099  2383 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.221s (SENT)
W20260709 12:22:36.695276  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 85134.7us
I20260709 12:22:36.736351  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.736471  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.737033  1872 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:22:36.737562  2383 leader_election.cc:336] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.615s (SENT)
I20260709 12:22:36.727460  3851 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:36.644318  3735 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.731954  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:36.693940  3796 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:36.737957  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.738001  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:36.708847  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1430) took 1202 ms (client timeout 1582 ms). Trace:
I20260709 12:22:36.739703  3810 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:36.755438  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 209) took 4115 ms (client timeout 2615 ms). Trace:
W20260709 12:22:36.756007  1878 rpcz_store.cc:269] 0709 12:22:32.639894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.755219 (+4115325us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.755359 (+   140us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.757094  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.757812  1878 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:36.739446  3815 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:36.761116  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 213) took 3671 ms (client timeout 2169 ms). Trace:
W20260709 12:22:36.761608  1875 rpcz_store.cc:269] 0709 12:22:33.089456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.760914 (+3671458us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.761043 (+   129us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.709329  3552 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:36.767614  3741 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:36.691148  3776 raft_consensus.cc:2955] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:36.775756  2415 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:36.741441  1872 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:36.751643  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 225) took 3525 ms (client timeout 1947 ms). Trace:
W20260709 12:22:36.777638  1876 rpcz_store.cc:269] 0709 12:22:33.225930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.751431 (+3525501us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.751570 (+   139us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.751116  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.779395  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.779524  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.706694  3849 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:36.783248  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.783876  1876 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:36.792747  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:36.793666  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82898.5us
W20260709 12:22:36.794464  1771 leader_election.cc:336] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.950s (SENT)
I20260709 12:22:36.795194  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:36.796340  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:36.797307  1800 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:22:36.799288  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 221) took 3633 ms (client timeout 2033 ms). Trace:
W20260709 12:22:36.799878  1879 rpcz_store.cc:269] 0709 12:22:33.165554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.799002 (+3633448us) service_pool.cc:211] Skipping call since client already timed out
0709 12:22:36.799193 (+   191us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:22:36.801506  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:36.802371  1872 raft_consensus.cc:2376] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 4. Re-sending same reply.
I20260709 12:22:36.803822  3810 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:36.804301  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:36.805287  1800 raft_consensus.cc:2376] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 5. Re-sending same reply.
I20260709 12:22:36.807590  3850 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:36.808483  3850 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:36.808619  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 76750.9us
I20260709 12:22:36.787959  1881 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:36.809589  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1524) took 1306 ms. Trace:
I20260709 12:22:36.810050  1881 rpcz_store.cc:276] 0709 12:22:35.503008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.648472 (+1145464us) service_pool.cc:224] Handling call
0709 12:22:36.809505 (+161033us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.793726  1876 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:36.789072  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 101459us
I20260709 12:22:36.815382  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:36.815430  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:36.789657  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 103267us
I20260709 12:22:36.817277  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:36.807639  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.818835  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.750396  3735 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.723163  3469 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260709 12:22:36.729156  3698 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260709 12:22:36.773360  3552 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:36.830956  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1427) took 1399 ms. Trace:
I20260709 12:22:36.825171  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.827167  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 550) took 1446 ms. Trace:
I20260709 12:22:36.831528  1804 rpcz_store.cc:276] 0709 12:22:35.431167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.079283 (+648116us) service_pool.cc:224] Handling call
0709 12:22:36.079630 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.079836 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:36.311547 (+231711us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.511984 (+200437us) spinlock_profiling.cc:243] Waited 2.01 ms on lock 0x7b18002d6d00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16afcac58f 00007f16a8ceb571 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd
0709 12:22:36.765719 (+253735us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b18002d6d00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:22:36.830461 (+ 64742us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.830535 (+    74us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:36.830597 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.830672 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.830871 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68797,"prepare.queue_time_us":239759,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":62404,"spinlock_wait_cycles":30643200,"thread_start_us":257729,"threads_started":2,"wal-append.queue_time_us":78733}
I20260709 12:22:36.832662  2417 rpcz_store.cc:276] 0709 12:22:35.380499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.380729 (+   230us) service_pool.cc:224] Handling call
0709 12:22:35.381056 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.587271 (+206215us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:35.753837 (+166566us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.145697 (+391860us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.145766 (+    69us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:36.145840 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.826925 (+681085us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b540046a354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:36.826975 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.827117 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":175930,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":108113,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":6978,"spinlock_wait_cycles":473530240,"thread_start_us":541002,"threads_started":2,"wal-append.queue_time_us":501658}
I20260709 12:22:36.829523  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:36.822944  3731 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:36.729346  3582 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
I20260709 12:22:36.809856  3739 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:36.829777  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.789577  3687 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:36.748359  3797 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260709 12:22:36.836385  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:36.823966  3657 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:36.839198  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.700846  3837 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.837682  1879 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:36.841881  1881 raft_consensus.cc:2368] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 2. Current term is 3.
I20260709 12:22:36.828146  3540 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.843238  3850 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.844450  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.843706  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.846138  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.847090  3849 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:36.850382  1880 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:36.852046  2413 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 3 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:22:36.852210  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.853437  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.851375  1876 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:36.824374  3710 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:36.854087  1881 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:36.841677  3657 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.855401  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.855585  3657 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:36.857375  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.859067  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.859337  3837 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:36.831620  3364 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:36.834486  3742 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:22:36.860335  1878 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:36.862511  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.860152  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.864355  1876 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:22:36.865124  1876 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:36.839018  3556 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:36.865738  3556 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.866161  3556 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:36.867604  3747 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:36.868077  3747 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:36.869022  3747 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.870072  3747 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:36.834909  3674 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:36.839594  3739 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.871799  3739 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:36.866255  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:36.872872  1875 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:36.865713  1876 raft_consensus.cc:606] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:36.874751  1876 raft_consensus.cc:614] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:36.875593  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:37240 (request call id 9) took 4623 ms. Trace:
I20260709 12:22:36.876082  1876 rpcz_store.cc:276] 0709 12:22:32.268527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.865003 (+4596476us) service_pool.cc:224] Handling call
0709 12:22:36.875475 (+ 10472us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.878235  3397 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.880406  3397 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:36.880214  3657 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.881592  1878 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:36.882263  3540 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:36.882376  3826 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.883901  1878 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:36.884555  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1486) took 5394 ms. Trace:
I20260709 12:22:36.884713  3826 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.884896  1878 rpcz_store.cc:276] 0709 12:22:31.490090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.883330 (+5393240us) service_pool.cc:224] Handling call
0709 12:22:36.883709 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.884295 (+   586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.884464 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.885802  1877 raft_consensus.cc:2368] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6.
I20260709 12:22:36.885670  3702 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.886978  1881 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:36.887542  1877 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:36.888192  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1490) took 5396 ms. Trace:
I20260709 12:22:36.888522  1877 rpcz_store.cc:276] 0709 12:22:31.491433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.887032 (+5395599us) service_pool.cc:224] Handling call
0709 12:22:36.887369 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.887974 (+   605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.888141 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.889612  1881 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:36.890141  3702 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:36.874037  1880 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:36.891856  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1494) took 5291 ms. Trace:
I20260709 12:22:36.891942  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2052) took 5456 ms. Trace:
I20260709 12:22:36.892412  1880 rpcz_store.cc:276] 0709 12:22:31.435991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.873385 (+5437394us) service_pool.cc:224] Handling call
0709 12:22:36.873835 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.891698 (+ 17863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.891888 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.892310  1881 rpcz_store.cc:276] 0709 12:22:31.600749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.887757 (+5287008us) service_pool.cc:224] Handling call
0709 12:22:36.889377 (+  1620us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.891616 (+  2239us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.891800 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.854658  3796 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.752976  3827 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:36.898276  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:36.957365  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.747948  1802 rpcz_store.cc:269] 0709 12:22:35.506394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.647619 (+1141225us) service_pool.cc:224] Handling call
0709 12:22:36.708775 (+ 61156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:36.896108  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1484) took 5406 ms. Trace:
I20260709 12:22:36.896653  3552 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.874087  3556 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:36.895138  3657 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.894682  3717 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:36.925323  3823 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:36.924513  3827 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:37.060153  1872 raft_consensus.cc:2368] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5.
I20260709 12:22:37.044549  1802 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:22:36.975617  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 181050us
W20260709 12:22:37.118243  1771 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.200s (SENT)
I20260709 12:22:37.118937  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.028573  1880 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:37.032320  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 134005us
I20260709 12:22:37.062036  3556 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:37.025890  1876 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:37.067487  3687 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:37.069852  3717 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:37.041695  3450 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:37.020216  1873 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:37.055967  3552 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:36.970687  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:36.971349  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 259403us
I20260709 12:22:37.057868  3674 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.986327  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:37.108009  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.023058   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:22:37.069343  3710 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
W20260709 12:22:37.017462  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 181687us
I20260709 12:22:36.993589  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.034015  3364 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:36.990760  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:36.991446  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:36.982437  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:37.122082  3823 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:36.898304  3791 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:37.123895  3717 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:37.135195  1805 raft_consensus.cc:2376] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5. Re-sending same reply.
I20260709 12:22:37.136547  3717 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:37.137291  3791 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:37.139256  1803 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:37.138974  3674 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:37.139716  1798 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:37.143312  1804 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:37.144097  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:37.144341  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:37.146282  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:37.148609  3674 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:37.147127  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1498) took 5424 ms. Trace:
I20260709 12:22:37.164852  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.167196  3782 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.167708  3450 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:37.168382  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 186415us
I20260709 12:22:37.168308  3782 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:37.168744  3450 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:37.169538  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:37.169561  3450 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:37.172508  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:37.173425  3744 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:37.025885  1874 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:37.162058  1880 rpcz_store.cc:276] 0709 12:22:31.701695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.027182 (+5325487us) service_pool.cc:224] Handling call
0709 12:22:37.028019 (+   837us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.120210 (+ 92191us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.125804 (+  5594us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.169636  3540 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:37.193183  3869 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.194111  3869 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:37.196902  3869 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:37.198936  3790 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:37.200639  3723 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.201350  3723 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:37.203322  1877 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:37.207195  3644 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:37.207894  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89738.3us
I20260709 12:22:37.207746  3857 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:37.207938  3644 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:37.208856  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:37.210350  1872 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 12:22:37.216387  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 245850us
W20260709 12:22:37.224592  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59976.5us
I20260709 12:22:37.045176  1875 rpcz_store.cc:276] 0709 12:22:31.489342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.873491 (+5384149us) service_pool.cc:224] Handling call
0709 12:22:36.894822 (+ 21331us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:36.895318 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.895586 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.895637 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.895718 (+    81us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:36.895775 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.895858 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.896015 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":38185728}
I20260709 12:22:37.214704  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:37.226737  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:37.227517  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.228856  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 246) took 1874 ms. Trace:
I20260709 12:22:37.208972  3644 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:37.229899  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1485) took 5740 ms. Trace:
I20260709 12:22:37.231182  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1496) took 5592 ms. Trace:
I20260709 12:22:37.232587  3796 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:37.233454  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:37.224232  3861 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:37.246827  1876 rpcz_store.cc:276] 0709 12:22:31.489703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.876956 (+5387253us) service_pool.cc:224] Handling call
0709 12:22:36.877422 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.149816 (+272394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.229555 (+ 79739us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400074254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:37.229815 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":339950848}
I20260709 12:22:37.234879  3723 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:37.235731  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.220181  3702 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:37.239424  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47433.8us
W20260709 12:22:37.234690  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.242664  1803 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:37.238035  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.247146  1877 rpcz_store.cc:276] 0709 12:22:31.639255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.889249 (+5249994us) service_pool.cc:224] Handling call
0709 12:22:36.889567 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.203983 (+314416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.230854 (+ 26871us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54001f7554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:37.231112 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":591714048}
I20260709 12:22:37.253279  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.245038  1799 rpcz_store.cc:276] 0709 12:22:35.354463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.023178 (+668715us) service_pool.cc:224] Handling call
0709 12:22:36.034324 (+ 11146us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:36.035168 (+   844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.223311 (+1188143us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b5400066650. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:37.228591 (+  5280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24598400}
I20260709 12:22:37.246858  3674 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 12:22:37.271925  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:37.273767  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:37.276191  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:37.276710  1627 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 269) took 5383 ms. Trace:
I20260709 12:22:37.279366  1877 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:37.281332  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1424) took 1879 ms (client timeout 2220 ms). Trace:
W20260709 12:22:37.281767  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 205) took 4811 ms (client timeout 1961 ms). Trace:
W20260709 12:22:37.282032  1801 rpcz_store.cc:269] 0709 12:22:35.401803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.223793 (+1821990us) service_pool.cc:224] Handling call
0709 12:22:37.281234 (+ 57441us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:37.282538  1874 rpcz_store.cc:269] 0709 12:22:32.470297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.539170 (+1068873us) service_pool.cc:224] Handling call
0709 12:22:37.281412 (+3742242us) spinlock_profiling.cc:243] Waited 2.43 s on lock 0x7b5400223654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:37.281651 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5598402432}
I20260709 12:22:37.284410  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:37.284775  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:37.285501  1875 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:37.286415  1874 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:37.183377  3450 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:37.278578   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:37.306421  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:37.277319  1627 rpcz_store.cc:276] 0709 12:22:31.892905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.893196 (+   291us) service_pool.cc:224] Handling call
0709 12:22:32.001584 (+108388us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd42090 00007f16abde0f3a 00007f16b1b96e9c 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:22:33.556318 (+1554734us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b54001564a8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0d289b4 00007f16af80c3dc 00007f16af7cde0b 00007f16af7cdbe4 00007f16b1ccdfa1 00007f16b1cce7ff 00007f16b1b8da1b 00007f16b1c9590e 00007f16b01562a4 00007f16b0156232
0709 12:22:37.276397 (+3720079us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:34.468314 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:34.469075 (+   761us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:34.469121 (+    46us) write_op.cc:435] Acquired schema lock
0709 12:22:34.469147 (+    26us) tablet.cc:662] Decoding operations
0709 12:22:34.523672 (+ 54525us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:34.523739 (+    67us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:34.523771 (+    32us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:22:34.524303 (+   532us) tablet.cc:701] Row locks acquired
0709 12:22:34.524322 (+    19us) write_op.cc:260] PREPARE: finished
0709 12:22:34.524508 (+   186us) write_op.cc:270] Start()
0709 12:22:34.524632 (+   124us) write_op.cc:276] Timestamp: P: 1783599754524449 usec, L: 0
0709 12:22:34.524655 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:22:34.525430 (+   775us) log.cc:844] Serialized 2449 byte log entry
0709 12:22:35.394315 (+868885us) op_driver.cc:464] REPLICATION: finished
0709 12:22:35.985427 (+591112us) write_op.cc:301] APPLY: starting
0709 12:22:35.986938 (+  1511us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:36.044701 (+ 57763us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:36.058811 (+ 14110us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:36.074125 (+ 15314us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:36.074357 (+   232us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0709 12:22:36.074410 (+    53us) write_op.cc:312] APPLY: finished
0709 12:22:36.074715 (+   305us) log.cc:844] Serialized 53 byte log entry
0709 12:22:36.074948 (+   233us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:36.075113 (+   165us) write_op.cc:454] Released schema lock
0709 12:22:36.075158 (+    45us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:36.075600 (+   442us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:36.075630 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":277840,"reported_tablets":13,"spinlock_wait_cycles":12552320,"child_traces":[["op",{"apply.queue_time_us":564464,"apply.run_cpu_time_us":4014,"apply.run_wall_time_us":104249,"num_ops":6,"prepare.queue_time_us":819261,"prepare.run_cpu_time_us":22716,"prepare.run_wall_time_us":398744,"raft.queue_time_us":430033,"raft.run_cpu_time_us":9358,"raft.run_wall_time_us":357001,"replication_time_us":869457,"thread_start_us":1107343,"threads_started":4,"wal-append.queue_time_us":407883,"wal-append.run_cpu_time_us":47956,"wal-append.run_wall_time_us":2138061}]]}
I20260709 12:22:37.300151  1626 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 3 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:37.302533  3782 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:37.303331  3815 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:37.305593  3851 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:37.305889  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69466.8us
I20260709 12:22:37.307116  3810 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:37.313149  3612 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.316481  2413 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:22:37.319875  2417 raft_consensus.cc:2955] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:37.316677  3739 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:37.323542  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 118813us
I20260709 12:22:37.324244  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.277498  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:37.341214  3744 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:37.359977   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:37.363075  3744 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:37.363888  1626 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 2 to 4, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 4 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:37.364786  3739 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:37.372635  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:37.371037  1873 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:37.373571  1876 raft_consensus.cc:2376] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 4. Re-sending same reply.
I20260709 12:22:37.358088  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:37.351784  3810 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:37.400365  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 228076us
I20260709 12:22:37.402614  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.397493  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1489) took 5884 ms. Trace:
I20260709 12:22:37.405458  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:37.394107  3810 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:37.410058  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:37.396306  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:37.413095  1771 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items
I20260709 12:22:37.418258  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.418596  1732 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:37.418594  1731 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:37.404816  1878 rpcz_store.cc:276] 0709 12:22:31.491167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.886013 (+5394846us) service_pool.cc:224] Handling call
0709 12:22:36.886326 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:37.088409 (+202083us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.088479 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:37.114297 (+ 25818us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.247683 (+133386us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:37.247754 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.261082 (+ 13328us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:37.366181 (+105099us) raft_consensus.cc:1685] finished
0709 12:22:37.366200 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.375676 (+  9476us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":140438,"raft.queue_time_us":153260,"raft.run_cpu_time_us":408,"raft.run_wall_time_us":408,"thread_start_us":190613,"threads_started":2,"wal-append.queue_time_us":242652}
I20260709 12:22:37.425767  3820 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:37.426506  3820 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:37.427016  3820 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:37.438133  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 160770us
I20260709 12:22:37.439822  1873 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:37.435640  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.459692  1728 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:22:37.462188  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 106543us
I20260709 12:22:37.479696  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.480616  1725 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:37.484934  1880 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:37.483784  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1506) took 5440 ms. Trace:
I20260709 12:22:37.497390  1876 rpcz_store.cc:276] 0709 12:22:32.043507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.411042 (+5367535us) service_pool.cc:224] Handling call
0709 12:22:37.482916 (+ 71874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.483254 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.483294 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.483356 (+    62us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:37.483436 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.483517 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.483683 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.499451  1876 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:37.504364  1873 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:22:37.505288  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 202) took 5081 ms. Trace:
I20260709 12:22:37.505653  1873 rpcz_store.cc:276] 0709 12:22:32.424260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.465252 (+5040992us) service_pool.cc:224] Handling call
0709 12:22:37.504004 (+ 38752us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:37.504934 (+   930us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.505192 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.506116  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1497) took 5855 ms. Trace:
I20260709 12:22:37.506691  1879 rpcz_store.cc:276] 0709 12:22:31.650742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.898763 (+5248021us) service_pool.cc:224] Handling call
0709 12:22:36.899136 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:37.132132 (+232996us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.132236 (+   104us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:37.133277 (+  1041us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.245600 (+112323us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:37.245656 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.255684 (+ 10028us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:37.428907 (+173223us) spinlock_profiling.cc:243] Waited 5.45 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:37.505574 (+ 76667us) raft_consensus.cc:1685] finished
0709 12:22:37.505598 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.505890 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":128222,"raft.queue_time_us":141048,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":194,"spinlock_wait_cycles":12530944,"thread_start_us":173146,"threads_started":2,"wal-append.queue_time_us":232824}
I20260709 12:22:37.511900  1873 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:37.520794  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2083) took 3761 ms. Trace:
I20260709 12:22:37.521364  1876 rpcz_store.cc:276] 0709 12:22:33.750284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.498551 (+3748267us) service_pool.cc:224] Handling call
0709 12:22:37.499072 (+   521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.499941 (+   869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.511464 (+ 11523us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.521664  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1507) took 3213 ms. Trace:
I20260709 12:22:37.522640  1873 rpcz_store.cc:276] 0709 12:22:34.306840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.507659 (+3200819us) service_pool.cc:224] Handling call
0709 12:22:37.508074 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.513536 (+  5462us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.520124 (+  6588us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.407397  3612 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:37.529954  1725 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:37.530895  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1504) took 5622 ms. Trace:
I20260709 12:22:37.436090  3851 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:37.525530  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:37.535620  3876 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.074s	user 0.001s	sys 0.001s
I20260709 12:22:37.537398  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.538091  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.538976  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.539420  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.539862  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.544137  1800 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:37.544605  1800 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:37.544925  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 206) took 5056 ms. Trace:
I20260709 12:22:37.545442  1880 rpcz_store.cc:276] 0709 12:22:32.488570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.518151 (+5029581us) service_pool.cc:224] Handling call
0709 12:22:37.544320 (+ 26169us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.544587 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.544615 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.544659 (+    44us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:37.544694 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.544752 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.544861 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:37.547395  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 180835us
I20260709 12:22:37.547976  1731 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:37.549188  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:37.549882  1731 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:22:37.550634  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.541055  1729 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:22:37.527138  3710 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:37.550745  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.555600  1880 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:37.583442  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1523) took 2104 ms. Trace:
I20260709 12:22:37.583719  1880 rpcz_store.cc:276] 0709 12:22:35.479484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.546488 (+2067004us) service_pool.cc:224] Handling call
0709 12:22:37.555066 (+  8578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.583201 (+ 28135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.583387 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.568956  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.570523  1875 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:37.572849  1876 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:37.557132  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 203) took 5132 ms. Trace:
W20260709 12:22:37.581022  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.561829  1873 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:37.531512  1872 rpcz_store.cc:276] 0709 12:22:31.908941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.407573 (+5498632us) service_pool.cc:224] Handling call
0709 12:22:37.502978 (+ 95405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.525781 (+ 22803us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.525840 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.530452 (+  4612us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:37.530525 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.530620 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.530814 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18723,"raft.queue_time_us":24661}
I20260709 12:22:37.564182  1729 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:37.588496  1878 rpcz_store.cc:276] 0709 12:22:32.424613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.510109 (+5085496us) service_pool.cc:224] Handling call
0709 12:22:37.556289 (+ 46180us) spinlock_profiling.cc:243] Waited 8.29 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:37.556493 (+   204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.556719 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.556752 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.556840 (+    88us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:37.556878 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.556931 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.557067 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19059968}
I20260709 12:22:37.539932  1727 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:37.592859  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1515) took 2560 ms. Trace:
I20260709 12:22:37.593351  1873 rpcz_store.cc:276] 0709 12:22:35.032758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.558160 (+2525402us) service_pool.cc:224] Handling call
0709 12:22:37.560733 (+  2573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.592459 (+ 31726us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.592753 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.566018  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.601275  1877 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:37.603489  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1525) took 2079 ms. Trace:
I20260709 12:22:37.603984  1875 rpcz_store.cc:276] 0709 12:22:35.508422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.569965 (+2061543us) service_pool.cc:224] Handling call
0709 12:22:37.570330 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.587107 (+ 16777us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.587393 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.588456  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1508) took 2558 ms. Trace:
I20260709 12:22:37.605203  1879 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:22:37.604929  1876 rpcz_store.cc:276] 0709 12:22:35.030036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.526461 (+2496425us) service_pool.cc:224] Handling call
0709 12:22:37.572462 (+ 46001us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.588149 (+ 15687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.588383 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.579744  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.615180  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.555361  1800 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:37.627178  1724 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:37.566395  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:37.569522  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:37.634851  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.571619  1874 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:22:37.647157  1800 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:37.650758  3702 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:37.651499  1724 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:37.653029  1724 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:37.653741  1877 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:37.654162  1878 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:37.655259  1877 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:37.655371  1878 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:37.656692  1878 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:37.564038  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:37.658932  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.661855  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.604053  3861 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:37.659757  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 109100us
I20260709 12:22:37.574178  3612 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:37.666208  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:37.662595  1726 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:37.667050  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:37.658515  1874 pending_rounds.cc:85] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 5
I20260709 12:22:37.670718  2415 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:37.667178  1728 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:37.678510  1725 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:37.678917  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:37.682212  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 283983us
I20260709 12:22:37.688372  1727 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:37.696785  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.688283  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.714201  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.663321  3815 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:37.722177  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.683226  1874 pending_rounds.cc:107] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:22:37.734754  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.734022  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.705025  3861 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:37.743316  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:37.751830  1771 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items
I20260709 12:22:37.752753  1729 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:37.737500  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.718613  3612 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.765048  1800 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:37.768497  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:37.769306  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47504.5us
I20260709 12:22:37.769732  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:37.771216  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.771126  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.775435  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.776228  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:37.776536  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60907.1us
I20260709 12:22:37.781549  1880 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:22:37.781987  3715 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:22:37.798251  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.798255  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2094) took 1469 ms. Trace:
I20260709 12:22:37.799255  1880 rpcz_store.cc:276] 0709 12:22:36.328453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.631755 (+1303302us) service_pool.cc:224] Handling call
0709 12:22:37.720100 (+ 88345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.795393 (+ 75293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.798194 (+  2801us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:37.783111  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:37.787022  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 223197us
W20260709 12:22:37.788277  3878 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:22:37.787942  3810 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:37.788808  3612 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:37.738394  1874 raft_consensus.cc:2889] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:37.789656  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.790706  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:37.792659  1678 auto_rebalancer.cc:893] Removing replace marker failed: Leader has not yet committed an operation in its own term
I20260709 12:22:37.801481  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.803375  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.736246  2413 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:37.719723  3644 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:37.823153  1805 raft_consensus.cc:2410] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:37.814109  1807 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
W20260709 12:22:37.817903  1771 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:55680 dropped due to backpressure: service queue is full with 50 items [suppressed 36 similar messages]
I20260709 12:22:37.802553  1802 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:37.832504  2411 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:37.828327  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.833716  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.828473  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2086) took 3352 ms. Trace:
I20260709 12:22:37.835501  1879 rpcz_store.cc:276] 0709 12:22:34.476095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.508567 (+3032472us) service_pool.cc:224] Handling call
0709 12:22:37.602063 (+ 93496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:37.629586 (+ 27523us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.828293 (+198707us) spinlock_profiling.cc:243] Waited 93.1 ms on lock 0x7b5400026f50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:37.828420 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":214097152}
W20260709 12:22:37.832280  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.837520  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:37.848059  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.851090  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.853427  1800 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:37.854956  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.855904  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.847131  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:37.862706  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.870641  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.871778  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.876792  1798 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:37.877274  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.876025  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:37.880684  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:37.887101  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.895231  3875 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260709 12:22:37.896610  3875 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:37.897032  3875 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:37.898926  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.899916  3820 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:37.900789  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:37.875861  1678 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: LeaderStepDown request on kudu.consensus.ConsensusService from 127.0.0.1:43056 dropped due to backpressure: service queue is full with 50 items
I20260709 12:22:37.905622  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.860172  3851 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:37.908435  3698 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:22:37.909597  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.901604  1801 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:37.918973  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1500) took 5939 ms. Trace:
I20260709 12:22:37.952894  1729 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:37.921772  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.953161  1881 rpcz_store.cc:276] 0709 12:22:31.907860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.904801 (+4996941us) service_pool.cc:224] Handling call
0709 12:22:36.905218 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:37.254018 (+348800us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.254075 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:37.254864 (+   789us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.392100 (+137236us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:37.392156 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.397259 (+  5103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:37.837299 (+440040us) raft_consensus.cc:1685] finished
0709 12:22:37.837333 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.847596 (+ 10263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":123202,"raft.queue_time_us":135330,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":204,"thread_start_us":294274,"threads_started":2,"wal-append.queue_time_us":359006}
I20260709 12:22:37.955024  3612 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:37.956445  2414 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:37.957281  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 102485us
I20260709 12:22:37.924085  3717 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:37.958874  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:37.960006  3744 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:37.923846  3810 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:37.964591  3810 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.958665  3717 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:37.927527  3710 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:37.969615  3717 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:37.928215  3715 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:37.888325  3815 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:22:37.972431  3815 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:37.973057  3815 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:37.975248  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.924000  3644 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:37.930024  3702 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:37.930341  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:37.930847  1805 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:37.931417  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.934872  3820 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:37.935166  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.936650  2417 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:37.992808  1803 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:22:37.937032  1678 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: LeaderStepDown request on kudu.consensus.ConsensusService from 127.0.0.1:43056 dropped due to backpressure: service queue is full with 50 items
I20260709 12:22:37.937763  3450 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:37.938359  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:37.938716  3782 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:37.939815  3861 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:37.943384  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:37.947072  3857 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:37.948148  1724 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:37.896632  1802 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:37.948668  1731 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:37.948874  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:37.949754  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:37.951220  1723 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:37.951617  1728 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:37.952211  2382 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:38.007241  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55315.8us
I20260709 12:22:37.955188  1806 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:38.007619  1798 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:37.929126  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1526) took 2110 ms. Trace:
I20260709 12:22:38.010535  1727 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
W20260709 12:22:37.979117  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.976208  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:38.008600  1726 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:37.898198  3875 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:38.007488  1800 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:38.012813  3556 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260709 12:22:38.023456  3875 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:38.023885  3739 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:38.024645  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:38.024856  1798 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:37.984577  3810 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.024385  3739 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:38.044401  3739 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.045166  3810 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:38.045974  1807 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:38.033268  1806 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
W20260709 12:22:38.034709  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 104608us
I20260709 12:22:38.035053  1798 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:38.038620  1723 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:38.049098  1806 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 3
I20260709 12:22:38.010996  1874 rpcz_store.cc:276] 0709 12:22:35.819046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.570836 (+1751790us) service_pool.cc:224] Handling call
0709 12:22:37.571178 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:37.888272 (+317094us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.929050 (+ 40778us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:38.018908  3874 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:38.017373  3796 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:38.041563  1728 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:38.024577  3820 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:38.049491  1806 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:22:38.053649  1806 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:38.051712  3739 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:38.055488  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:37.974377  3815 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.061782  3815 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:38.061453  3820 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:38.052080  3874 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.068082  3874 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:38.070842  3556 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:38.050258  3810 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:22:38.085879  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:38.091136  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.090147  3796 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:22:38.221097  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45900.9us
I20260709 12:22:38.242641  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:38.243420  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:38.172456  3820 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260709 12:22:38.219775  3796 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:38.252604  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78378.6us
W20260709 12:22:38.250821  2410 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.015s	sys 0.029s
W20260709 12:22:38.256831  2410 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.015s	sys 0.029s
I20260709 12:22:38.244566  3702 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.255362  3810 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.286057  3702 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.293998  1799 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
W20260709 12:22:38.306346  3887 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:22:38.295140  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:38.309746  1805 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:38.314456  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:38.247504  3556 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20260709 12:22:38.322899  1800 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260709 12:22:38.326054  1800 pending_rounds.cc:85] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 3
W20260709 12:22:38.326413  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.326442  1800 pending_rounds.cc:107] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20260709 12:22:38.327172  1800 raft_consensus.cc:2889] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:38.314019  3644 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20260709 12:22:38.322381  3702 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:38.299152  3810 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:38.341799  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 433633us
W20260709 12:22:38.346050  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 479158us
I20260709 12:22:38.291373  3796 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:38.353245  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:38.353466  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 394747us
I20260709 12:22:38.325765  3698 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:22:38.355757  2417 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:38.341071  1801 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:22:38.362736  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 335385us
W20260709 12:22:38.368886  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.354176  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:38.382532  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:38.380254  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:38.364820  1803 raft_consensus.cc:2955] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:38.419629  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:38.410317  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.429488  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 186125us
W20260709 12:22:38.441530  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 195533us
W20260709 12:22:38.452220  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 137871us
I20260709 12:22:38.454717  1723 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:38.461359  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:22:38.459695  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48007.7us
I20260709 12:22:38.430481  3715 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:38.451294  3744 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:38.431047  3851 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:22:38.483229  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:38.464793  3450 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:38.484133  1800 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:38.488623  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:38.489506  1799 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:38.467195  3715 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:38.499835  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 118523us
I20260709 12:22:38.501653  3911 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:38.502213  3911 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:38.503336  3911 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.503531  3715 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:38.505266  3810 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:38.505607  3910 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:38.505839  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:38.506222  3910 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.507475  3894 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:38.508019  3810 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:38.508122  3890 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:38.508677  3810 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.508791  3890 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.509585  1723 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:38.509794  3698 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:22:38.510648  3888 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:22:38.509397  3890 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.511960  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:38.504974  3911 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:38.509334  3810 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:38.486718  3450 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.532766  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:38.512775  3889 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:22:38.479321  3815 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:22:38.508069  3894 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:38.508627  3910 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:38.553761  3908 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:22:38.542588  3450 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:38.545634  3698 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:38.568930  3782 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:38.549803  3715 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:38.549420  1801 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:22:38.565078  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86962.2us
I20260709 12:22:38.539016  3851 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:22:38.569707  3782 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.595224  2413 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:38.543133  3717 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260709 12:22:38.571768  3698 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:38.635422  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46317.4us
W20260709 12:22:38.636081  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 119210us
W20260709 12:22:38.640822  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 108198us
I20260709 12:22:38.588948  3644 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:38.638469  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:22:38.629805  3874 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.541s	user 0.003s	sys 0.010s
I20260709 12:22:38.655915  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:22:38.659933  3739 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.600s	user 0.010s	sys 0.003s
W20260709 12:22:38.699533  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 79602us
I20260709 12:22:38.683699  3710 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.629454  3891 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:38.603438  3901 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:38.649113  3556 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
W20260709 12:22:38.709784  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 197894us
I20260709 12:22:38.500765  3912 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:38.690234  3920 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:38.693769  3923 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:38.712730  3891 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.718600  3702 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:38.718005  3698 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:38.685843  3922 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 12:22:38.686120  3909 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.706238  3904 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:38.604007  3898 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.127s
I20260709 12:22:38.725150  3891 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:38.664382  3874 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:38.746196  3874 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:38.722836  3893 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:38.725026  3702 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:38.732722  3698 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:38.733268  3902 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:38.694442  3914 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:22:38.706370  3739 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
W20260709 12:22:38.779078  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59604.2us
I20260709 12:22:38.756410  3893 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:22:38.758585  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58220.5us
I20260709 12:22:38.735702  3815 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.713296  3796 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
W20260709 12:22:38.764357  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 58936.4us
W20260709 12:22:38.765798  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 335164us
W20260709 12:22:38.767956  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64382.8us
I20260709 12:22:38.737524  3904 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:38.814143  3904 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:38.796414  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:38.817543  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.818943  3815 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.819458  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:38.819887  2416 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 3 LEARNER]: Advancing to term 4
I20260709 12:22:38.738866  3909 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:38.703342  3894 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.802873  1804 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:38.803364  3875 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:38.804065  3928 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260709 12:22:38.804481  3920 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:22:38.804842  3900 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.211s	user 0.001s	sys 0.001s
I20260709 12:22:38.805840  3922 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:38.803773  3612 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:22:38.809823  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 165653us
W20260709 12:22:38.713421  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 111731us
I20260709 12:22:38.810829  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1342) took 1020 ms. Trace:
I20260709 12:22:38.736192  3926 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260709 12:22:38.761477  3556 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:38.737054  3710 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:38.795993  3702 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.822472  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:38.826613  1804 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:38.827984  3898 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:38.830336  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:38.830904  3918 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:22:38.831290  3810 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.828652  3875 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:38.837895  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:38.842998  1803 rpcz_store.cc:276] 0709 12:22:37.790389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.993582 (+203193us) service_pool.cc:224] Handling call
0709 12:22:37.994230 (+   648us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:37.994671 (+   441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.994705 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:38.030862 (+ 36157us) log.cc:844] Serialized 556 byte log entry
0709 12:22:38.364652 (+333790us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:38.707205 (+342553us) log.cc:844] Serialized 14 byte log entry
0709 12:22:38.810262 (+103057us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.810402 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:38.810469 (+    67us) raft_consensus.cc:1685] finished
0709 12:22:38.810482 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:38.810715 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30022,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":8995,"thread_start_us":415544,"threads_started":2,"wal-append.queue_time_us":402471}
I20260709 12:22:38.831493  3903 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 12:22:38.848961  3739 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.829824  3644 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:38.828322  3744 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:38.837288  3922 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:38.850996  3923 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.840512  3612 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:38.836319  3920 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:38.914446  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.837855  1728 raft_consensus.cc:2955] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:38.861729  3782 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:38.925027  3934 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.925252  1804 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:38.925868  3934 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.926877  1804 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:38.927105  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:38.927649  3924 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:22:38.928516  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78583us
I20260709 12:22:38.903800  3702 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:38.943397  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 113201us
W20260709 12:22:38.936949  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 117543us
W20260709 12:22:38.976100  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46893us
I20260709 12:22:38.928159  3924 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:38.985381  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:38.988744  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:38.988276  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:38.884584  3556 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.903741  3739 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:38.939103  3782 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:38.884572  3898 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.911661  3744 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.004629  3896 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:38.980919  3923 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:38.993598  3926 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:22:39.008037  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.025280  3922 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.042657  3739 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:39.042806  3927 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:39.043239  3922 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:39.043556  3927 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:39.044584  3927 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.045800  3942 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:39.046144  3902 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.046600  3942 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.046806  3902 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.047163  3942 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:38.996295  3612 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.050056  3612 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.042728  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:39.051205  1804 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:22:39.025437  3815 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:39.042694  3925 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.224s	user 0.002s	sys 0.002s
I20260709 12:22:39.051797  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1350) took 1172 ms. Trace:
I20260709 12:22:39.057226  3851 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:39.058163  3947 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:39.058771  3947 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.059892  3644 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:39.061537  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 87399.7us
I20260709 12:22:39.061905  3947 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.064011  3810 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:39.064257  3898 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.064931  3930 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:39.026751  3920 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.065970  3917 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.167s
I20260709 12:22:39.067129  3920 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:39.069893  3940 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:38.994086  3875 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.070844  3938 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.150s
I20260709 12:22:39.071719  3875 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.071907  3938 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:39.072381  3938 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:39.073354  3938 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.003257  3782 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:39.075201  3782 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.075291  3857 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:39.075714  3857 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:39.076736  3857 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.077854  3857 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:39.054064  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:39.055214  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.057405  1804 rpcz_store.cc:276] 0709 12:22:37.879525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.001589 (+122064us) service_pool.cc:224] Handling call
0709 12:22:38.013628 (+ 12039us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:38.802738 (+789110us) spinlock_profiling.cc:243] Waited 421 ms on lock 0x7b5400408e20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:22:39.051570 (+248832us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.051745 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":968005888}
W20260709 12:22:39.083427  3899 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:22:39.083870  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:39.084873  3744 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.056133  3924 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.085997  3924 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.091215  3810 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:39.074687  3938 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:39.091988  3810 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.097848  3940 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:39.086028  3910 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:39.094273  3890 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.582s	user 0.012s	sys 0.000s
I20260709 12:22:39.099710  3890 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:39.105645  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.106739  3937 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:39.107717  3923 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:39.108651  3936 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:39.109131  3936 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:39.097870  3810 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:39.110026  3936 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.110952  3810 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:39.056339  3909 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:39.104220  3942 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.100253  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:39.104228  3556 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.113571  3556 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:39.114965  3890 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.119392  3929 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:39.119844  3929 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:39.120802  3929 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.121918  3929 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.056911  3710 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:39.060642  3939 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:22:39.113795  3891 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.156914  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:39.157410  3891 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.158244  3913 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:39.110031  2416 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 4 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260709 12:22:39.169438  2416 pending_rounds.cc:85] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 3
I20260709 12:22:39.169854  2416 pending_rounds.cc:107] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 3.4
I20260709 12:22:39.170182  2416 raft_consensus.cc:2889] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 4 LEARNER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:39.126503  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:39.115751  3894 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:39.133219  1804 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 12:22:39.146873  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.151921  3450 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.001s
I20260709 12:22:39.174566  3450 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.151786  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:39.177021  3934 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.177451  1800 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.152633  3782 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.180902  3782 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.154385  3810 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:39.181945  3810 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.184937  3810 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:39.154958  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:39.095742  3911 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.558s	user 0.012s	sys 0.000s
I20260709 12:22:39.163327  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:39.165210  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:39.198510  3875 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:39.199371  3875 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:39.199898  3875 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:39.202999  3782 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:39.203779  3782 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:39.206148  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.113734  3893 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.212081  3893 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:39.215767  3902 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.177016  3450 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:39.222970  1771 leader_election.cc:400] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 2. Current term is 3.
I20260709 12:22:39.223610  1771 leader_election.cc:403] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:39.225376  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:39.226135  3450 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.234383  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:39.235810  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:39.237197  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:39.239693  3902 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.240372  3902 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:39.247072  3782 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:39.236121  2418 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:39.210357  3904 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:39.267123  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 275256us
I20260709 12:22:39.115407  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:39.273706  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:39.279884  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:39.280611  1804 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:39.291729  1800 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:39.307753  1803 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:39.311944  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:39.313251  2418 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:22:39.314649  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 199349us
I20260709 12:22:39.320705  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:39.325567  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 483194us
W20260709 12:22:39.338981  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.340884  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 341317us
I20260709 12:22:39.309346  3934 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:39.306402  3891 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:39.282132  3911 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260709 12:22:39.238577  3875 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.313848  3710 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:39.315915  3922 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:39.350463  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.310884  3918 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:39.322153  3902 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:39.355389  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 130126us
I20260709 12:22:39.356232  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:39.361227  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.359014  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.359614  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.332374  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:39.363548  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.364207  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.367849  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:39.362584  1730 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:39.371423  3909 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:39.371232  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.372335  1732 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:39.363036  3918 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.374783  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:39.375490  1725 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:39.379417  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45760.2us
I20260709 12:22:39.380359  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:39.377334  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:39.373994  1729 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:39.348246  3934 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:39.358978  3952 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:39.382100  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:39.395347  2416 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:39.381695  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:39.357323  3710 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:39.406585  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60631.1us
I20260709 12:22:39.407438  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:39.351032  3810 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:22:39.406253  3710 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:39.385432  3540 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.416819  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.420164  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:39.421197  1729 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.422133  1725 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:39.423420  1732 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:39.354033  3891 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.396976  3934 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:39.398317  3909 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:39.451714  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:39.381632  3918 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:39.428671  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.460407  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.409238  3901 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:39.411530  3952 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:39.430603  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.430155  1771 leader_election.cc:400] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6.
I20260709 12:22:39.492745  2416 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:39.443271  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:39.420542  3904 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:39.430248  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63124.4us
I20260709 12:22:39.430979  3739 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:39.500537  3739 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:39.496089  3556 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:39.426599  3540 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.501533  3739 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.502079  3556 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:39.502513  3556 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:39.496436  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72967.1us
I20260709 12:22:39.503814  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:39.503942  3540 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:39.507097  3901 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:39.507153  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55562us
I20260709 12:22:39.497754  3936 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:39.515043  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71872.1us
I20260709 12:22:39.517346  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:22:39.511998  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.509537  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.430449  3922 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:22:39.527603  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.506330  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:39.531898  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:39.431414  3875 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:39.533252  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:39.460515  3903 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:39.537679  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68462.8us
I20260709 12:22:39.544260  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:39.539934  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:39.550355  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.552533  1731 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
W20260709 12:22:39.554913  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48656.2us
I20260709 12:22:39.558094  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.515300  3918 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:39.531646  3911 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:39.565490  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.538817  3912 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:39.544955  3910 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:39.572606  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.573385  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:39.541301  3923 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:39.556180  3909 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:22:39.575632  3912 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:39.559988  3815 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:39.587831  3905 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:39.622366  3952 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:39.624600  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51348.6us
I20260709 12:22:39.627311  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:39.628569  1800 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:39.633420  1729 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:39.616758  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:39.571328  3960 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20260709 12:22:39.619119  1874 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:22:39.618963  3815 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.619656  3959 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:22:39.620157  3911 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:22:39.620903  3961 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:22:39.580034  2413 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:39.580132  3923 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:39.643191  1803 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:22:39.647192  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.643565  2419 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:39.639814  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:39.573261  3796 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:22:39.611372  3912 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:39.639437  1874 pending_rounds.cc:85] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 2
I20260709 12:22:39.550200  3903 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.653594  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:39.593340  3922 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260709 12:22:39.575250  3918 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.652647  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.648448  3905 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:39.664975  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:39.672603  1727 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:39.638720  3874 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:39.623916  3910 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:39.643657  3904 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.664805  3875 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:39.672969  1881 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:39.673625  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:39.687362  1800 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:39.661962  1874 pending_rounds.cc:107] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:22:39.684455  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:39.659745  3923 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:39.690959  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:39.692257  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:39.693035  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:39.699268  1807 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:39.700253  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:39.697674  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:39.704562  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:39.705439  1878 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:39.707003  1873 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:39.711555  1798 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:22:39.713603  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:39.689746  3875 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:39.715462  3875 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:39.719141  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:39.719623  3966 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
W20260709 12:22:39.720907  3965 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:22:39.705327  3953 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:22:39.725767  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:39.730748  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.736402  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.737726  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:39.737546  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 158731us
I20260709 12:22:39.738649  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:39.739703  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:39.739950  1803 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:39.740661  1725 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:39.695647  1874 raft_consensus.cc:2889] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:39.737977  1800 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:39.738476  1877 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:39.711632  3954 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:22:39.747767  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:39.748915  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.771416  1872 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:39.738994  3874 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:39.701197  3904 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:39.805227  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:39.812423  1803 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
W20260709 12:22:39.773121  1696 connection.cc:256] RPC call timeout handler was delayed by -1.41522695499998s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1582s for a response
W20260709 12:22:39.814191  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2892269550002311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17220000000000002s for a response
W20260709 12:22:39.814574  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1478575220003222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2198s for a response
I20260709 12:22:39.807410  1877 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:39.803826  1875 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:39.808435  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.816344  1725 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:39.807940  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:39.817395  1876 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:39.821033  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:39.821987  1798 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:39.774940  3951 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:39.778234  3918 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:39.918005  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.943282  1806 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:39.949476  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:39.950320  1798 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:39.943063  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 135270us
I20260709 12:22:39.955828  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:39.869516  3893 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:39.986541  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:39.933702  3891 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:39.933025  3941 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:39.999552  3911 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:39.957242  3450 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.723s	user 0.010s	sys 0.003s
W20260709 12:22:40.002519  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.000731  3969 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.195s	user 0.000s	sys 0.000s
W20260709 12:22:40.007529  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 60907.3us
W20260709 12:22:40.019603  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 335261us
I20260709 12:22:40.021359  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:39.932335  3951 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:40.014935  3556 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.511s	user 0.012s	sys 0.000s
I20260709 12:22:39.862493  3910 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.958756  3934 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.014460  3450 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 2. Current term is 3.
W20260709 12:22:40.039764  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.982985  3710 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:39.993216  3796 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:40.095592  1800 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:40.094897   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:39.995230  3893 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:40.101878  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 80654.1us
I20260709 12:22:40.041158  3815 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.134054  1805 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:40.122828  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:40.121089  1798 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:40.123623  1799 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:40.141474  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 112996us
I20260709 12:22:40.022747  3941 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.108690  3744 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:40.066032  3556 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.156138  1881 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:40.114611  3973 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.041008  3911 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:22:40.122947  3698 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.004s	sys 0.008s
W20260709 12:22:40.183264  3698 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.004s	sys 0.009s
I20260709 12:22:40.110832  3796 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:40.164356  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 177940us
I20260709 12:22:40.123294  3891 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:40.185570  3891 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:40.186815  3891 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.182276  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:40.150564  3450 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:40.116173  3739 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260709 12:22:40.197631  1723 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 2: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:40.193938  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:40.154439  3981 raft_consensus.cc:2764] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260709 12:22:40.203704  1873 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:22:40.163681  3744 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:40.165179  3904 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:40.226727  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.166247  3903 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.235073  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:40.154901  3874 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260709 12:22:40.238296  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:40.179483  3905 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:22:40.220005  3744 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:40.241496  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:40.252106  2410 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:40.263962  3978 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.246750  3911 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:40.235666  3904 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 12:22:40.282321  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:40.249256  3710 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:40.288820  1879 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.001s
I20260709 12:22:40.269356  3981 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:40.273227  3820 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.289724  1879 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
W20260709 12:22:40.275692  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72537.3us
I20260709 12:22:40.277642  3976 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.239410  3796 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.277168  3874 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:22:40.285027  3971 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.187s	user 0.002s	sys 0.000s
I20260709 12:22:40.285295  3739 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.289158  3975 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:40.313889  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 222886us
I20260709 12:22:40.302321  3927 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:40.319393  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:40.302824  3934 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:40.320024  3988 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:22:40.301743  3874 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:40.308905  3984 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.309314  3940 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:40.315625  3974 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:22:40.337723  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:40.301035  3820 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.341141  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:40.348945  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.346138  3932 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:22:40.273195  3986 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.071s
I20260709 12:22:40.264617  3978 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.352605  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1415) took 1003 ms. Trace:
I20260709 12:22:40.313225  3739 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.357870  3893 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.203538  3973 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.329072  3951 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.301607  3976 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.360293  2410 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:40.357224  3932 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:40.353941  3556 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:40.368206  3977 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 12:22:40.360036  1724 rpcz_store.cc:276] 0709 12:22:39.349117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.349315 (+   198us) service_pool.cc:224] Handling call
0709 12:22:39.363417 (+ 14102us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:39.410414 (+ 46997us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.410448 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:39.420096 (+  9648us) log.cc:844] Serialized 26 byte log entry
0709 12:22:39.742945 (+322849us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:39.742991 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.743060 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.352332 (+609272us) raft_consensus.cc:1685] finished
0709 12:22:40.352359 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.352531 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63443,"raft.queue_time_us":47470,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":156,"thread_start_us":282166,"threads_started":1,"wal-append.queue_time_us":572504}
W20260709 12:22:40.363082  3989 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:22:40.368642  3874 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.369194  3941 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:40.369720  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.335850  3981 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260709 12:22:40.370383  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:40.374218  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.374330  3739 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:40.337212  3927 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:40.381628  1877 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:22:40.386772  1879 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:40.374866  3744 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:40.411208  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.387621  3986 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:40.318331  3975 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:40.374531  3912 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.697s	user 0.005s	sys 0.006s
I20260709 12:22:40.413691  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:40.431262  1875 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:40.389907  3932 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:40.414937  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47309.3us
I20260709 12:22:40.374927  3998 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:40.425863  1872 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:40.412288  3901 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.905s	user 0.009s	sys 0.006s
I20260709 12:22:40.428566  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:40.414472  1876 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:40.418974  3904 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.442351  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.442764  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:40.410389  3893 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:40.338843  3984 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.347635  3940 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:40.431402  3980 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:22:40.484817  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.452946  1877 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:40.492705  1876 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 2: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:22:40.472919  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59305.3us
I20260709 12:22:40.430953  3992 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:40.493242  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50565.1us
I20260709 12:22:40.429723  3976 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.485428  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.496315  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:40.467622  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:40.427992  3905 consensus_queue.cc:1243] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:40.430884  3996 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.429833  3973 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:40.479938  3978 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:40.493180  3875 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.777s	user 0.013s	sys 0.000s
I20260709 12:22:40.498304  1872 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:40.431334  3975 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:40.479269  4005 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:22:40.497372  3936 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.964s	user 0.014s	sys 0.000s
I20260709 12:22:40.474443  3901 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:40.496745  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:40.496917  3937 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:40.472544  3927 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.526763  3937 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:40.527324  3937 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:40.507633  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:40.507931  3987 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20260709 12:22:40.466715  3904 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.529438  1876 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:40.508351  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:40.498756  3932 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:40.530989  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 206398us
I20260709 12:22:40.531457  3932 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:40.509162  3996 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.510375  3820 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.512157  3744 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.513001  1874 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:40.536791  3996 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.530500  1724 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:40.538661  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:40.539966  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:40.540568  1724 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:40.542552  3904 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.550182  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:40.551060  1724 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:40.552947  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.553611  1876 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:40.552241  1730 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
W20260709 12:22:40.558904  4001 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:22:40.561782  3984 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:40.519774  3875 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.562525  3937 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.563263  1802 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:40.443471  3912 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.452358  3959 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.073s
W20260709 12:22:40.570084  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 73423.9us
I20260709 12:22:40.446462  3986 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.572743  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:40.501489  3893 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.576215  1878 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:40.577405  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:40.583364  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:40.573453  3932 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:40.587267  1807 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:40.588609  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.589315  1877 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:40.591360  1806 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:22:40.497221  3998 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.587374  1798 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:40.598034  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:40.596652  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2266) took 1046 ms. Trace:
I20260709 12:22:40.584882  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.598443  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:40.599879  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:40.599027  1726 rpcz_store.cc:276] 0709 12:22:39.550508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.931776 (+381268us) service_pool.cc:224] Handling call
0709 12:22:39.957198 (+ 25422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.115392 (+158194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.115438 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.115501 (+    63us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:40.115547 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.596411 (+480864us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b5400435dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:40.596466 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.596590 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":363171072}
I20260709 12:22:40.616865  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:40.587143  3931 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:40.578913  3936 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 12:22:40.583874  3875 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:40.622953  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:40.626569  1800 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:40.598021  4005 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.629921  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:40.620746  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:40.628965  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:40.630909  1798 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:40.625228  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:40.630929  1806 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:40.615355  3820 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:40.634053  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:40.635069  1800 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:40.636178  4000 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:40.636736  4000 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.637086  4000 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:40.639341  3991 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.319s	user 0.001s	sys 0.002s
I20260709 12:22:40.640195  4009 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.641108  4009 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.642952  3983 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:40.643757  3937 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.619078  3931 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:40.645238  3931 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.647966  3893 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.648789  3927 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.649358  3927 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:40.651680  3923 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.920s	user 0.019s	sys 0.000s
W20260709 12:22:40.652436  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.652160  3923 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.653163  3912 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:40.654470  1880 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:40.655547  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:40.634297  3903 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:40.660256  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:40.660617  3903 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.661084  3903 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:40.662055  1724 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:40.659217  3952 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:40.663403  3952 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:40.663818  3952 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:40.619364  3815 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:40.659526  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:40.667029  1723 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:40.670806  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.673569  3927 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.674469  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.675460  1879 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:40.675992  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:40.677357  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:40.678329  1770 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:40.679162  3875 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:40.679584  3905 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:40.659276  4008 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.680059  3744 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:40.680608  3744 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.680583  4008 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.681250  3744 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:40.659482  3810 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:40.664327  3983 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:40.683539  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:40.679796  3986 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:40.687994  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.688957  1879 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:40.685973  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:40.679759  3875 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:40.672498  3815 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:40.709328  1798 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:40.697129  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:40.677778  3931 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:40.698331  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:40.770990  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73293us
I20260709 12:22:40.786846  4000 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:40.690205  4007 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:22:40.795023  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:40.795192  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:40.732815  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.796245  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:40.793810  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:40.790350  3952 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.815726  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.817368  1876 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:40.700594  3940 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:40.683120  3810 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:40.798379  4000 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:40.769182  3815 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:40.803680  3710 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:40.824836  3710 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.827011  3710 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:40.827111  3903 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.829120  3810 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:40.829691  3903 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:40.831645  3983 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:40.821378  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.842133  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.842170  3875 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.846607  1879 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:40.848678  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:40.854367  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:40.855098  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:40.855921  1879 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:40.814908  3905 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:40.870724  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.909476  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:40.910563  1798 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:40.667969  4005 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.798986  3904 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:40.978413  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:40.988018  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:40.991720  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:40.992482  1879 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:40.994208  3955 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.349s	user 0.009s	sys 0.004s
I20260709 12:22:40.997409  1798 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:40.999298  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:41.006036  1798 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:41.002032  1878 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:40.868343  3905 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.046891  3905 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:41.050678  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:41.052780  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:41.053869  1798 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:41.041194  3998 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:40.978413  3744 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:40.984272  3904 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.057250  3744 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.058393  3952 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.083046  3904 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:41.054826  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:41.059715  4005 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.016721  3810 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.089686  3810 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.090969  4008 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.026640  3815 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:41.138693  3957 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.196s	user 0.011s	sys 0.011s
I20260709 12:22:41.144913  1874 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 12:22:41.145503  1874 pending_rounds.cc:85] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 4
I20260709 12:22:41.145911  1874 pending_rounds.cc:107] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:22:41.146256  1874 raft_consensus.cc:2889] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:41.085893  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:41.098524  3820 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:41.183099  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 273) took 1502 ms. Trace:
I20260709 12:22:41.183696  1802 rpcz_store.cc:276] 0709 12:22:39.680240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.681493 (+  1253us) service_pool.cc:224] Handling call
0709 12:22:39.710774 (+ 29281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.563765 (+852991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.182755 (+618990us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b54001676d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:41.183015 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1959926656}
I20260709 12:22:41.106580  3710 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.195122  3815 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.203096  3710 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.205991  3920 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:41.214488  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:41.215749  2410 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:41.219929  1798 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:41.222996  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:41.224032  2413 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:41.225293  3710 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.217844  3810 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:41.225880  3710 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.206591  3920 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:41.227089  1872 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.016s	sys 0.025s
W20260709 12:22:41.227464  1872 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.016s	sys 0.025s
W20260709 12:22:41.259613  1877 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.026s	sys 0.028s
I20260709 12:22:41.179452  3905 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.219206  3820 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:41.226886  3810 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.526566  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 244) took 1178 ms. Trace:
I20260709 12:22:41.635519  1875 rpcz_store.cc:276] 0709 12:22:40.348460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.348630 (+   170us) service_pool.cc:224] Handling call
0709 12:22:41.526278 (+1177648us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b54002058d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:41.526476 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187547904}
W20260709 12:22:41.261960  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.010s	sys 0.016s
I20260709 12:22:41.527272  1878 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:22:41.332729  1725 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.013s	sys 0.033s
W20260709 12:22:41.369444  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.279405  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:41.636953  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.020s	user 0.018s	sys 0.037s
W20260709 12:22:41.637624  1725 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.021s	user 0.014s	sys 0.057s
I20260709 12:22:41.471194  1801 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:41.436894  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 467) took 1908 ms. Trace:
I20260709 12:22:41.505798  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 466) took 1988 ms. Trace:
I20260709 12:22:41.639658  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:41.518424  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1548) took 1848 ms (client timeout 2405 ms). Trace:
I20260709 12:22:41.270749  1879 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:22:41.644410  1880 rpcz_store.cc:269] 0709 12:22:39.669812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.677860 (+  8048us) service_pool.cc:224] Handling call
0709 12:22:41.518133 (+1840273us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:41.518314 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2192164736}
W20260709 12:22:41.461882  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.434027  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1556) took 1692 ms. Trace:
W20260709 12:22:41.458915  2419 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 613) took 2332 ms (client timeout 1775 ms). Trace:
I20260709 12:22:41.363596  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:41.650076  2416 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
W20260709 12:22:41.263353  1877 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.027s	sys 0.028s
I20260709 12:22:41.646727  1873 rpcz_store.cc:276] 0709 12:22:39.743049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.744174 (+  1125us) service_pool.cc:224] Handling call
0709 12:22:39.775552 (+ 31378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.212966 (+437414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.433667 (+1220701us) spinlock_profiling.cc:243] Waited 428 ms on lock 0x7b5400205dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:41.433943 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":984032128}
I20260709 12:22:41.638654  2410 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:41.514176  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1463) took 1845 ms. Trace:
I20260709 12:22:41.666375  1799 rpcz_store.cc:276] 0709 12:22:39.668984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.669119 (+   135us) service_pool.cc:224] Handling call
0709 12:22:39.669430 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.139851 (+470421us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.513804 (+1373953us) spinlock_profiling.cc:243] Waited 447 ms on lock 0x7b54001680d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:41.514086 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1027231488}
W20260709 12:22:41.475023  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 271) took 1952 ms (client timeout 1651 ms). Trace:
I20260709 12:22:41.641449  2415 rpcz_store.cc:276] 0709 12:22:39.528980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.529155 (+   175us) service_pool.cc:224] Handling call
0709 12:22:41.435935 (+1906780us) spinlock_profiling.cc:243] Waited 6.14 ms on lock 0x7b5800003f98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:22:41.436187 (+   252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.436431 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.436469 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.436526 (+    57us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:41.436569 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.436639 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.436783 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14131456}
W20260709 12:22:41.496585  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 257) took 2078 ms (client timeout 1716 ms). Trace:
I20260709 12:22:41.264420  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:41.642023  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:22:41.255293  3932 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.038s	sys 0.043s
I20260709 12:22:41.728799  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2272) took 1264 ms. Trace:
W20260709 12:22:41.440886  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.774s	user 0.009s	sys 0.008s
W20260709 12:22:41.730237  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.064s	user 0.009s	sys 0.009s
W20260709 12:22:41.730718  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 1.07518e+06us
I20260709 12:22:41.680621  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:41.732447  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:41.733125  1799 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:41.675179  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:41.734252  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1374) took 1062 ms. Trace:
W20260709 12:22:41.647606  2419 rpcz_store.cc:269] 0709 12:22:39.125991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.126154 (+   163us) service_pool.cc:224] Handling call
0709 12:22:41.458581 (+2332427us) spinlock_profiling.cc:243] Waited 471 ms on lock 0x7b5400489754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:41.458745 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1084357504}
I20260709 12:22:41.734717  1799 rpcz_store.cc:276] 0709 12:22:40.671789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.732517 (+1060728us) service_pool.cc:224] Handling call
0709 12:22:41.732969 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.733939 (+   970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.734199 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:41.739308  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2128) took 1069 ms. Trace:
I20260709 12:22:41.641914  2417 rpcz_store.cc:276] 0709 12:22:39.517215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.517438 (+   223us) service_pool.cc:224] Handling call
0709 12:22:39.517777 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:39.688360 (+170583us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.688400 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:39.688444 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:39.688477 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.505454 (+1816977us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b540049eb54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:41.505526 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.505718 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":391800576}
W20260709 12:22:41.473866  1626 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 169) took 14149 ms (client timeout 15000 ms). Trace:
W20260709 12:22:41.668047  1805 rpcz_store.cc:269] 0709 12:22:39.522605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.524944 (+  2339us) service_pool.cc:224] Handling call
0709 12:22:41.474693 (+1949749us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b54001d5954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:41.474891 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1354067328}
I20260709 12:22:41.741416  2418 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:41.729460  1725 rpcz_store.cc:276] 0709 12:22:40.377375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.500030 (+122655us) service_pool.cc:224] Handling call
0709 12:22:40.502476 (+  2446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.502688 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:40.564562 (+ 61874us) log.cc:844] Serialized 14 byte log entry
0709 12:22:41.640899 (+1076337us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:41.640968 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:41.641021 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.641165 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.641439 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13897,"prepare.queue_time_us":55215,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":46006,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"thread_start_us":1061870,"threads_started":2,"wal-append.queue_time_us":295972}
I20260709 12:22:41.739842  1878 rpcz_store.cc:276] 0709 12:22:40.669586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.047398 (+377812us) service_pool.cc:224] Handling call
0709 12:22:41.526809 (+479411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.637309 (+110500us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.739080 (+101771us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54002058d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:41.739235 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1101519616}
I20260709 12:22:41.736961  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1472) took 1932 ms. Trace:
I20260709 12:22:41.742256  1801 rpcz_store.cc:276] 0709 12:22:39.805075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.860596 (+ 55521us) service_pool.cc:224] Handling call
0709 12:22:39.948183 (+ 87587us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.653464 (+1705281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.730690 (+ 77226us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400133b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:41.736887 (+  6197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3484722944}
W20260709 12:22:41.736313  3932 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.118s	user 0.038s	sys 0.079s
I20260709 12:22:41.746160  3932 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:41.504516  3941 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:41.429481  3644 raft_consensus.cc:1064] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:41.264986  3922 raft_consensus.cc:1064] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:41.345321  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:41.755165  2417 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260709 12:22:41.438149  3897 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.009s	sys 0.008s
I20260709 12:22:41.756596  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:41.355458  1802 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:41.747553  3904 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.664s	user 0.002s	sys 0.010s
W20260709 12:22:41.701253  3911 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.007s	sys 0.011s
W20260709 12:22:41.440709  3875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.007s	sys 0.010s
I20260709 12:22:41.778460  1727 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:22:41.624830  1629 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 3 to 4, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 4 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:41.634828  3874 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:41.448869  3998 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:41.783249  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:41.765245  2419 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:41.758944  3956 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:41.466091  3702 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.734416  2411 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:41.472698  3905 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.464587  3990 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:41.754254  3941 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:41.766702  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:41.766952  2418 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:41.730937  3975 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.369233  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:41.731235  4017 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.816035  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1362) took 1214 ms. Trace:
I20260709 12:22:41.785432  4008 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 12:22:41.767506  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:41.278833  3920 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.820698  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:41.291553  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:41.529091  4009 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:41.512470  3923 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:41.673555  1730 rpcz_store.cc:269] 0709 12:22:39.418222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.418417 (+   195us) service_pool.cc:224] Handling call
0709 12:22:41.496285 (+2077868us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002aa154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:41.496444 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2434974080}
I20260709 12:22:41.771626  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:41.737183  3931 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:41.747936  3970 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.773232  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:41.773571  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:22:41.774084  3909 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.958s	user 0.000s	sys 0.026s
I20260709 12:22:41.736485  3999 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:41.757705  4000 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:41.769029  3897 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.067s	user 0.015s	sys 0.009s
W20260709 12:22:41.773952  3698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.147s	user 0.015s	sys 0.011s
I20260709 12:22:41.782951  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:41.470542  3450 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:41.762001  3901 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.792657  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:41.287870  3820 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.794691  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:41.843719  3909 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.036s	user 0.000s	sys 0.026s
I20260709 12:22:41.857028  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:41.783957  3875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.012s	sys 0.012s
I20260709 12:22:41.811837  3702 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.868324  3810 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.816169  4017 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.869616  3702 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.870239  3702 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:41.778234  3911 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.804s	user 0.009s	sys 0.012s
I20260709 12:22:41.673105  3815 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:41.466969  3951 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:41.816318  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:41.794332  3922 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.780843  4005 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:41.824060  1629 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:41.816607  1802 rpcz_store.cc:276] 0709 12:22:40.601320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.185092 (+583772us) service_pool.cc:224] Handling call
0709 12:22:41.185495 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.778139 (+592644us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.815744 (+ 37605us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:41.815952 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":352935808}
I20260709 12:22:41.766729  3938 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260709 12:22:41.870754  3702 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.878296  3938 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.817411  2412 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:41.747586  3932 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:41.827270  1772 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.938s	user 0.023s	sys 0.055s
I20260709 12:22:41.879230  3702 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:41.879528  1772 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.991s	user 0.023s	sys 0.055s
I20260709 12:22:41.879936  3702 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:41.879987  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 991361us
I20260709 12:22:41.788023  3956 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:41.870884  4017 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:41.881803  3956 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:41.882100  4017 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:41.882687  4017 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.883292  3938 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.885082  4017 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.886091  3938 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:41.837087  2417 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:41.887176  3941 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.888244  3941 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:41.841933  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:41.871205  3815 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.811290  3710 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:41.785732  3904 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.820231  3782 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:41.827945  3891 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.832438  4009 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.812588  1728 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:41.844340  3920 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:41.847149  3698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.227s	user 0.015s	sys 0.011s
I20260709 12:22:41.818441  3990 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.868384  4008 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:41.890126  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:41.834882  2413 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:41.789739  3998 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.821983  3975 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.847378  3999 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.876309  1629 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 2 to 3, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:41.909822  3782 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.897382  1801 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:22:41.961858  3998 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:41.956751  3923 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.963382  3923 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:41.965696  3923 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:41.871430  3951 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.968086  3975 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:41.880669  3702 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.853283  2414 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:41.795970  3960 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.906728  1725 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
W20260709 12:22:41.741012  1626 rpcz_store.cc:269] 0709 12:22:27.325055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.325556 (+   501us) service_pool.cc:224] Handling call
0709 12:22:41.473755 (+14148199us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:37.907066 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:37.907904 (+   838us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:37.907989 (+    85us) write_op.cc:435] Acquired schema lock
0709 12:22:37.908020 (+    31us) tablet.cc:662] Decoding operations
0709 12:22:37.908557 (+   537us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:37.908638 (+    81us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:37.908667 (+    29us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:22:37.909143 (+   476us) tablet.cc:701] Row locks acquired
0709 12:22:37.909187 (+    44us) write_op.cc:260] PREPARE: finished
0709 12:22:37.909430 (+   243us) write_op.cc:270] Start()
0709 12:22:37.909591 (+   161us) write_op.cc:276] Timestamp: P: 1783599757909336 usec, L: 0
0709 12:22:37.909606 (+    15us) op_driver.cc:348] REPLICATION: starting
0709 12:22:37.992120 (+ 82514us) log.cc:844] Serialized 913 byte log entry
0709 12:22:39.544379 (+1552259us) op_driver.cc:464] REPLICATION: finished
0709 12:22:40.029346 (+484967us) write_op.cc:301] APPLY: starting
0709 12:22:40.048387 (+ 19041us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:40.114921 (+ 66534us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:40.144015 (+ 29094us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:40.144611 (+   596us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:40.144883 (+   272us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:22:40.144929 (+    46us) write_op.cc:312] APPLY: finished
0709 12:22:40.145249 (+   320us) log.cc:844] Serialized 29 byte log entry
0709 12:22:40.145470 (+   221us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:40.145637 (+   167us) write_op.cc:454] Released schema lock
0709 12:22:40.145713 (+    76us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:40.146172 (+   459us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:40.146200 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":65361,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":422850,"apply.run_cpu_time_us":3144,"apply.run_wall_time_us":139030,"num_ops":2,"prepare.queue_time_us":359445,"prepare.run_cpu_time_us":23098,"prepare.run_wall_time_us":462812,"raft.queue_time_us":398925,"raft.run_cpu_time_us":39715,"raft.run_wall_time_us":345584,"replication_time_us":1551281,"thread_start_us":883130,"threads_started":4,"wal-append.queue_time_us":458583,"wal-append.run_cpu_time_us":10321,"wal-append.run_wall_time_us":2692540}]]}
I20260709 12:22:41.842941  3931 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.853020  3450 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:41.910286  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:41.958137  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:41.905822  2417 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:41.852559  3970 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:41.855600  4000 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.866978  3901 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.875202  4005 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:42.112605  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:42.114212  3710 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:22:41.989778  1807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.400s	user 0.033s	sys 0.119s
I20260709 12:22:42.114877  3710 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:42.115170  3970 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:42.116294  3710 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:42.118328  1805 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:42.115334  1807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.525s	user 0.038s	sys 0.124s
I20260709 12:22:41.923645  3891 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:41.970284  4008 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:42.068601  3990 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:42.131341  4008 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:42.123522  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1480) took 1538 ms. Trace:
I20260709 12:22:41.970029  4009 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:42.136323  3901 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:42.135267  1807 rpcz_store.cc:276] 0709 12:22:40.585367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.588835 (+  3468us) service_pool.cc:224] Handling call
0709 12:22:40.589143 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.589366 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:40.589677 (+   311us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.123096 (+1533419us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:42.123156 (+    60us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:42.123225 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.123310 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.123453 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3370,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":93,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":1529785,"threads_started":1}
I20260709 12:22:42.088434  3999 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:42.107573  3702 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:42.138036  4009 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:42.007535  3960 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:42.139209  3951 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:42.140802  3960 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:42.141755  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:42.000875  1874 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.819s	user 0.030s	sys 0.081s
W20260709 12:22:42.143198  1874 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.034s	sys 0.081s
I20260709 12:22:42.134509  4000 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:41.972628  3810 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:42.157097  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.157850  3810 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:42.158600  3810 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:42.159797  3782 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:42.161559  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:41.972162  3612 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.005s	sys 0.076s
I20260709 12:22:42.172205  1728 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:42.089752  3904 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:42.177652  3612 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.989s	user 0.014s	sys 0.081s
I20260709 12:22:42.237375  1805 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
W20260709 12:22:42.243350  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.195713  3983 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.087s	user 0.006s	sys 0.041s
W20260709 12:22:42.282627  1804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.094s	user 0.011s	sys 0.028s
I20260709 12:22:42.180709  2412 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
W20260709 12:22:42.290716  1804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.102s	user 0.011s	sys 0.028s
I20260709 12:22:42.321509  3956 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:22:42.293098  3983 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.182s	user 0.006s	sys 0.041s
W20260709 12:22:42.388803  3861 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.932s	user 0.015s	sys 0.017s
I20260709 12:22:42.324069  3910 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.282377  4019 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:22:42.324483  3931 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:42.312880  3820 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:42.446146  3910 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:42.439121  3861 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.004s	user 0.015s	sys 0.022s
I20260709 12:22:42.476596  3931 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.485975  3931 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:42.495643  3810 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:42.500465  2416 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.019s	sys 0.020s
W20260709 12:22:42.501003  2416 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.719s	user 0.019s	sys 0.020s
I20260709 12:22:42.490592  3891 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:42.523701  3450 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:42.524124  4005 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:42.554970  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.739s	user 0.015s	sys 0.028s
W20260709 12:22:42.559756  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.016s	sys 0.029s
W20260709 12:22:42.578323  3922 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.009s	sys 0.018s
W20260709 12:22:42.590061  1727 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.012s	sys 0.016s
W20260709 12:22:42.605727  1727 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.012s	sys 0.016s
W20260709 12:22:42.525534  3920 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.596s	user 0.008s	sys 0.002s
W20260709 12:22:42.606760  1800 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.836s	user 0.012s	sys 0.017s
W20260709 12:22:42.612270  1800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.841s	user 0.013s	sys 0.017s
W20260709 12:22:42.608666  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.854s	user 0.008s	sys 0.017s
W20260709 12:22:42.617151  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.009s	sys 0.017s
W20260709 12:22:42.524730  3984 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.008s	sys 0.015s
W20260709 12:22:42.621968  3922 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.010s	sys 0.018s
I20260709 12:22:42.605868  3941 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.103s
W20260709 12:22:43.124365  4015 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
W20260709 12:22:42.664386  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.900s	user 0.000s	sys 0.033s
I20260709 12:22:43.154699  4021 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:42.689942  1698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.018s	sys 0.040s
W20260709 12:22:43.155907  1698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.318s	user 0.028s	sys 0.077s
I20260709 12:22:43.155790  4021 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:42.792980  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.040s	user 0.008s	sys 0.022s
W20260709 12:22:43.158706  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.406s	user 0.011s	sys 0.029s
I20260709 12:22:43.159197  3951 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:43.159370  4021 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:43.159694  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
W20260709 12:22:43.160234  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.396s	user 0.000s	sys 0.049s
I20260709 12:22:43.159885  3951 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:42.932490  3796 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.845s	user 0.007s	sys 0.021s
W20260709 12:22:43.160979  3796 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.073s	user 0.008s	sys 0.022s
W20260709 12:22:42.713989  1806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.894s	user 0.016s	sys 0.012s
W20260709 12:22:42.883323  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.112s	user 0.020s	sys 0.020s
W20260709 12:22:42.777303  1798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.138s	user 0.016s	sys 0.013s
I20260709 12:22:43.162241  3951 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:42.950527  3932 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.069s	user 0.006s	sys 0.027s
W20260709 12:22:43.163434  3932 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.282s	user 0.006s	sys 0.034s
I20260709 12:22:43.163681  3810 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.622222  3996 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:43.164773  3998 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.152107  3931 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:42.766309  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.035s	user 0.018s	sys 0.018s
I20260709 12:22:42.622274  3960 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.544s
I20260709 12:22:43.165874  3998 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:43.023478  3710 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.903s	user 0.022s	sys 0.023s
W20260709 12:22:43.016878  1723 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.232s	user 0.021s	sys 0.061s
W20260709 12:22:42.768854  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.016s	user 0.002s	sys 0.028s
W20260709 12:22:43.167542  1723 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.383s	user 0.021s	sys 0.074s
W20260709 12:22:43.167886  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.415s	user 0.011s	sys 0.029s
I20260709 12:22:43.168398  3998 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:42.703169  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.972s	user 0.012s	sys 0.023s
W20260709 12:22:42.726406  1877 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.978s	user 0.010s	sys 0.018s
W20260709 12:22:42.931155  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.172s	user 0.000s	sys 0.036s
W20260709 12:22:42.931566  3899 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.708s	user 0.009s	sys 0.022s
W20260709 12:22:43.170152  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.411s	user 0.000s	sys 0.043s
W20260709 12:22:42.897365  1803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.078s	user 0.011s	sys 0.048s
I20260709 12:22:43.166635  3931 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:43.163825  3815 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:42.777714  3874 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.991s	user 0.007s	sys 0.007s
I20260709 12:22:43.171096  3931 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:42.931710  1695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.166s	user 0.018s	sys 0.018s
I20260709 12:22:43.171705  3931 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:43.172072  1695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.406s	user 0.019s	sys 0.018s
I20260709 12:22:42.964707  3956 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:42.653716  4023 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.555s
W20260709 12:22:43.172734  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.165027  3996 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:43.169615  1877 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.421s	user 0.015s	sys 0.025s
W20260709 12:22:43.168542  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.52645e+06us
W20260709 12:22:43.162297  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.391s	user 0.020s	sys 0.023s
I20260709 12:22:43.175778  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1563) took 2592 ms. Trace:
W20260709 12:22:43.166520  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.435s	user 0.019s	sys 0.029s
I20260709 12:22:43.176623  1877 rpcz_store.cc:276] 0709 12:22:40.583509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.589960 (+  6451us) service_pool.cc:224] Handling call
0709 12:22:40.590301 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:40.590692 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.590724 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:40.593561 (+  2837us) log.cc:844] Serialized 26 byte log entry
0709 12:22:41.651291 (+1057730us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:41.746042 (+ 94751us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.175407 (+1429365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.175501 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.175552 (+    51us) raft_consensus.cc:1685] finished
0709 12:22:43.175563 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.175689 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2186,"prepare.queue_time_us":94813,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":35,"raft.queue_time_us":73935,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":4972,"thread_start_us":2483981,"threads_started":2,"wal-append.queue_time_us":826427,"wal-append.run_cpu_time_us":3043,"wal-append.run_wall_time_us":1327372}
W20260709 12:22:43.178962  3710 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.058s	user 0.022s	sys 0.028s
W20260709 12:22:43.154546  3984 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.397s	user 0.017s	sys 0.020s
W20260709 12:22:43.179622  1803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.361s	user 0.013s	sys 0.055s
I20260709 12:22:43.212386  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:43.162566  1798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.524s	user 0.017s	sys 0.023s
W20260709 12:22:42.932027  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.173s	user 0.033s	sys 0.030s
W20260709 12:22:43.214623  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.455s	user 0.034s	sys 0.030s
I20260709 12:22:43.215225  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:43.215425  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 1.53479e+06us
I20260709 12:22:43.216086  1729 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:42.935592  1629 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.972s	user 0.016s	sys 0.013s
W20260709 12:22:43.218142  1629 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.258s	user 0.016s	sys 0.020s
I20260709 12:22:43.164382  3450 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:43.161935  1806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.342s	user 0.017s	sys 0.024s
W20260709 12:22:43.169346  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.438s	user 0.012s	sys 0.036s
I20260709 12:22:43.183061  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:22:43.170332  3899 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.947s	user 0.009s	sys 0.024s
W20260709 12:22:43.153903  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.192585  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.193641  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:43.221783  1725 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:43.194494  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:43.222277  1731 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:43.224996  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:43.225639  1732 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:43.213501  1730 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
W20260709 12:22:43.198987  2416 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.000s	sys 0.015s
W20260709 12:22:43.201678  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.247172  2416 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.000s	sys 0.015s
W20260709 12:22:43.247454  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.203246  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.248255  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 588) took 1865 ms. Trace:
I20260709 12:22:43.151492  3910 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:43.171787  3874 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:43.209794  2384 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.750s	user 0.011s	sys 0.009s
W20260709 12:22:43.272152  2384 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.011s	sys 0.009s
I20260709 12:22:43.214334  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:43.165114  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.274173  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.274055  1726 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
W20260709 12:22:43.191860  1845 connection.cc:256] RPC call timeout handler was delayed by -1.1058955500002412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
W20260709 12:22:43.197306  3612 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.008s	sys 0.008s
W20260709 12:22:43.250509  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67568.2us
I20260709 12:22:43.248713  2416 rpcz_store.cc:276] 0709 12:22:41.382634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.732661 (+350027us) service_pool.cc:224] Handling call
0709 12:22:41.764431 (+ 31770us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.764657 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:42.512061 (+747404us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.247762 (+735701us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:43.247831 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:43.247901 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.248003 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.248183 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1454078,"threads_started":2}
I20260709 12:22:43.255388  3932 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:43.256122  3960 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.248489  1724 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
W20260709 12:22:43.207123  4008 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.073s	user 0.009s	sys 0.012s
W20260709 12:22:43.210992  3861 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.003s	sys 0.015s
W20260709 12:22:43.275449  1845 connection.cc:256] RPC call timeout handler was delayed by -1.0360501790000853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
I20260709 12:22:43.307793  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:43.285185  1846 connection.cc:256] RPC call timeout handler was delayed by -1.0514619499999753s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
I20260709 12:22:43.164788  3810 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:43.204872  3922 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:43.307019  3861 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.003s	sys 0.015s
I20260709 12:22:43.312000  3922 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:43.301702  3920 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:43.313709  3920 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:43.316748  1732 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:43.320920  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1367) took 2702 ms. Trace:
W20260709 12:22:43.307423  4008 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.167s	user 0.009s	sys 0.013s
I20260709 12:22:43.311223  3910 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:43.320858  3960 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:43.306797  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 90393.5us
I20260709 12:22:43.325681  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:43.311336  3810 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:43.310608  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:43.321738  1806 rpcz_store.cc:276] 0709 12:22:40.618743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.519715 (+900972us) service_pool.cc:224] Handling call
0709 12:22:41.520178 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.771231 (+251053us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.771259 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.776295 (+  5036us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.250393 (+1474098us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:43.250471 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.250607 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.320645 (+ 70038us) raft_consensus.cc:1685] finished
0709 12:22:43.320673 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.320846 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":250591,"raft.queue_time_us":264568,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":7776,"thread_start_us":1399701,"threads_started":1,"wal-append.queue_time_us":1435195}
W20260709 12:22:43.286520  3612 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.844s	user 0.008s	sys 0.009s
I20260709 12:22:43.341681  3909 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:43.338888  4023 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.284415  4000 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.079s
I20260709 12:22:43.336867  4050 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:43.349253  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:43.348268  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 187013us
I20260709 12:22:43.440315  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:43.322736  3932 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:43.304111  3874 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:43.397102  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54462.5us
I20260709 12:22:43.344980  3891 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.075s
I20260709 12:22:43.397934  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1434) took 1629 ms. Trace:
I20260709 12:22:43.379599  1731 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:43.411815  3983 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
W20260709 12:22:43.474004  1846 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.538s (SENT) [suppressed 4 similar messages]
I20260709 12:22:43.476418  1732 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:43.481326  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.494482  1728 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:43.481408  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.472889  3922 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:43.491218  2416 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:43.497015  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56857.4us
W20260709 12:22:43.493644  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 168041us
I20260709 12:22:43.499744  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:43.501224  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:43.420220  4050 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:43.500751  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:43.510882  2419 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:43.510946  2416 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 12:22:43.511646  2416 pending_rounds.cc:85] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 4
I20260709 12:22:43.506078  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:43.509455  1727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.903s	user 0.005s	sys 0.009s
I20260709 12:22:43.511901  2416 pending_rounds.cc:107] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:22:43.513057  2416 raft_consensus.cc:2889] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:43.377758  4053 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.513626  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:43.514074  1727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.907s	user 0.005s	sys 0.009s
W20260709 12:22:43.514267  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 165076us
I20260709 12:22:43.490394  1723 rpcz_store.cc:276] 0709 12:22:41.768207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.768404 (+   197us) service_pool.cc:224] Handling call
0709 12:22:41.768696 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.768966 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:43.168264 (+1399298us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.397351 (+229087us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:43.397435 (+    84us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:43.397545 (+   110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.397653 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.397843 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55700,"prepare.queue_time_us":910175,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":21897,"raft.queue_time_us":56128,"thread_start_us":1555616,"threads_started":2}
I20260709 12:22:43.520416  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:43.385004  3710 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:43.533420  4028 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.044s	user 0.009s	sys 0.006s
W20260709 12:22:43.536314  4029 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.013s	user 0.005s	sys 0.011s
I20260709 12:22:43.538239  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:43.539987  4008 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:43.540591  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.541392  1725 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:22:43.355063  4047 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.160s
I20260709 12:22:43.550963  1729 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:22:43.555079  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.560635  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:43.564224  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:43.565022  2419 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 1.
I20260709 12:22:43.568818  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:43.578369  2418 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 12:22:43.427675  4023 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:43.583838  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:43.594482  4027 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.108s	user 0.005s	sys 0.013s
I20260709 12:22:43.348716  1877 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:37240
W20260709 12:22:43.636484  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67778.4us
I20260709 12:22:43.541769  3810 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:43.635824  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1437) took 1854 ms. Trace:
I20260709 12:22:43.645959  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:43.632987  1724 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
W20260709 12:22:43.640664  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 125851us
I20260709 12:22:43.635123  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:43.635989  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.648487  2419 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:43.649950  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.647239  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:43.651886  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.652848  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:43.653383  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.645215  1877 raft_consensus.cc:606] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:43.565567  4059 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.175s	user 0.001s	sys 0.001s
I20260709 12:22:43.667344  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:43.654270  1877 raft_consensus.cc:614] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:43.661841  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:43.590377  3612 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:22:43.673261  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 133755us
I20260709 12:22:43.678646  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.681700  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.685626  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:43.696276  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 196592us
I20260709 12:22:43.579084  3951 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:43.701066  1731 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:22:43.548656  4049 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:43.565658  4043 raft_consensus.cc:2764] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260709 12:22:43.686659  2416 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:43.645675  1727 rpcz_store.cc:276] 0709 12:22:41.781628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.781865 (+   237us) service_pool.cc:224] Handling call
0709 12:22:41.799379 (+ 17514us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:41.799642 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:42.606335 (+806693us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.635428 (+1029093us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:43.635490 (+    62us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:43.635551 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.635619 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.635760 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49828,"prepare.queue_time_us":1076585,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":95,"raft.queue_time_us":72353,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"thread_start_us":1759443,"threads_started":3}
I20260709 12:22:43.704514  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.704818  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.721096  1726 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:43.706284  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.703862  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:43.703992  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:43.629937  4048 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:43.711580  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:43.526489  4053 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:43.715780  2418 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:43.722618  1724 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:43.630064  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:43.710896  4034 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:22:43.725709  1723 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:43.662002  4063 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:43.729928  3998 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.750555  1727 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:43.666531  3810 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:43.699199  3910 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.679982  3956 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:22:43.707448  3951 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:43.699502  3909 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:43.698179  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2141) took 2866 ms. Trace:
I20260709 12:22:43.747182  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:43.749017  1730 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:22:43.579005  3710 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:43.675787  3612 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:43.740585  4048 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:43.754994  3891 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:43.755867  3998 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:43.757351  3810 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:43.757917  3910 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:43.805331  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:43.805810  1728 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:22:43.716670  2413 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 4 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:22:43.718094  4040 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.765311  3815 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260709 12:22:43.767102  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 91061.5us
I20260709 12:22:43.751243  4063 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:43.775192  4023 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:43.776425  3932 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:43.759248  3956 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:43.779656  4043 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:43.780382  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:43.781680  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:43.785282  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 132580us
W20260709 12:22:43.836710  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55134.8us
I20260709 12:22:43.837308  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:43.837893  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:43.730556  3984 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:43.786932  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 138538us
I20260709 12:22:43.792439  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:43.793462  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 321809us
I20260709 12:22:43.788868  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:43.802054  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1364) took 3200 ms. Trace:
I20260709 12:22:43.792780  4049 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:43.815471  1730 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260709 12:22:43.817754  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.784063  3951 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:43.824286  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:43.900538  2415 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:43.802698  3899 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:43.815660  2412 raft_consensus.cc:2410] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 12:22:43.848145  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 101045us
I20260709 12:22:43.759966  3941 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.065s
I20260709 12:22:43.923799  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:43.924310  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:43.799520  3909 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:43.736606  4038 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:43.767197  4072 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:43.859246  3996 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:43.853742  4071 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:43.854460  4051 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:43.855070  1771 connection.cc:256] RPC call timeout handler was delayed by -1.7400277000001552s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
W20260709 12:22:43.856105  4033 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
W20260709 12:22:43.857398  4061 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:22:43.823225  3891 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:43.699438  4050 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:43.870405  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:43.871274  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:43.876564  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 87822.5us
I20260709 12:22:43.947409  1801 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:43.918944  3984 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:43.879998  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.876535  1798 rpcz_store.cc:276] 0709 12:22:40.601866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.288536 (+686670us) service_pool.cc:224] Handling call
0709 12:22:41.289058 (+   522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.289637 (+   579us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.289676 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.335570 (+ 45894us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.213892 (+1878322us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:43.559741 (+345849us) log.cc:844] Serialized 14 byte log entry
0709 12:22:43.801532 (+241791us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.801642 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.801701 (+    59us) raft_consensus.cc:1685] finished
0709 12:22:43.801714 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.801931 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":537686,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":208052,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":214,"thread_start_us":2099003,"threads_started":3,"wal-append.queue_time_us":2117474}
I20260709 12:22:43.766232  4039 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.116s
I20260709 12:22:43.876564  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:43.864979  4009 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:43.859408  4065 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.367s	user 0.001s	sys 0.001s
I20260709 12:22:43.835623  3956 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:43.912580  1696 connection.cc:256] RPC call timeout handler was delayed by -1.8696807030000855s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
I20260709 12:22:43.913677  1877 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:22:43.961308  1696 connection.cc:256] RPC call timeout handler was delayed by -1.3044807030000811s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2195s for a response
I20260709 12:22:43.913955  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:43.954950  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:22:43.964188  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50349.6us
W20260709 12:22:43.965515  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 89064.5us
I20260709 12:22:43.913677  1879 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:43.973615  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:43.911481  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 131157us
I20260709 12:22:43.910223  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:43.924662  1875 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:43.925616  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:43.932217  1881 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:22:43.858233  3796 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:43.781716  4083 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.878973  4049 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.903829  4023 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:43.924876  4019 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.447s	user 0.005s	sys 0.010s
W20260709 12:22:43.939085  1771 connection.cc:256] RPC call timeout handler was delayed by -1.7301277000001392s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
I20260709 12:22:43.940454  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:43.937093  4081 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:43.945640  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:43.951905  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:43.924578  4043 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:43.933008  4072 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.838851  4040 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:43.957515  1801 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:43.937724  4038 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.915052  3899 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:43.908140  3644 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:44.101626  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 128259us
I20260709 12:22:44.038996  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:43.947085  3983 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:44.157196  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:44.091449  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 97808.8us
I20260709 12:22:44.000974  2415 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:44.036509  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:44.160209  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:43.937580  4000 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:44.161835  1771 connection.cc:256] RPC call timeout handler was delayed by -1.7157277000001159s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
W20260709 12:22:44.162469  1771 connection.cc:256] RPC call timeout handler was delayed by -1.6491277000000082s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1605s for a response
W20260709 12:22:44.162993  1771 connection.cc:256] RPC call timeout handler was delayed by -1.6212276999999631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1636s for a response
W20260709 12:22:44.163455  1771 connection.cc:256] RPC call timeout handler was delayed by -1.4232277000000977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18560000000000001s for a response
W20260709 12:22:44.163931  1771 connection.cc:256] RPC call timeout handler was delayed by -1.3917277000000468s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18910000000000002s for a response
I20260709 12:22:44.158003  1879 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
W20260709 12:22:44.164362  1771 connection.cc:256] RPC call timeout handler was delayed by -1.3791277000000264s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1905s for a response
W20260709 12:22:44.164829  1771 connection.cc:256] RPC call timeout handler was delayed by -1.2945276999998896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20260709 12:22:44.165395  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:44.166214  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 256098us
I20260709 12:22:44.166703  1881 pending_rounds.cc:85] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 2
I20260709 12:22:43.973721  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:43.948365  4077 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.173460  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:43.941643  4071 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:44.177330  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.161051  1801 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:43.992409  3932 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.188s
I20260709 12:22:43.957479  4009 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.954998  4008 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:43.956943  3984 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:44.097795  3941 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:44.180922  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:44.707757  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:44.089676  3909 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:44.393353  3810 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.563s	user 0.008s	sys 0.000s
I20260709 12:22:44.441555  4085 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.697435  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:43.992519  4048 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:44.185381  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 141901us
I20260709 12:22:44.161048  4023 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.367759  4078 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:44.735026  4023 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:44.735272  4078 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.735910  3941 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:44.736413  3941 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:44.729492  4009 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:44.737339  3941 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:44.124122  4053 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:44.366348  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1369) took 3742 ms. Trace:
I20260709 12:22:44.739899  4050 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:44.739820  1803 rpcz_store.cc:276] 0709 12:22:40.624076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.527408 (+903332us) service_pool.cc:224] Handling call
0709 12:22:41.527988 (+   580us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.758474 (+230486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.758510 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.763825 (+  5315us) log.cc:844] Serialized 26 byte log entry
0709 12:22:43.213288 (+1449463us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:43.213343 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.213436 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.366030 (+1152594us) raft_consensus.cc:1685] finished
0709 12:22:44.366062 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.366252 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":229971,"raft.queue_time_us":232255,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":11574,"thread_start_us":1394122,"threads_started":1,"wal-append.queue_time_us":2575478}
I20260709 12:22:43.952507  4086 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:44.365013  4052 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:22:44.444519  4095 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.267s
I20260709 12:22:44.119081  3891 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:44.661324  4087 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.143519  3998 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:44.206622  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 369380us
W20260709 12:22:44.493906  4056 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.007s	sys 0.014s
I20260709 12:22:44.082748  4039 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:44.713928  4008 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:43.993204  4031 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.503s	user 0.009s	sys 0.012s
W20260709 12:22:44.715106  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.004s	sys 0.005s
W20260709 12:22:44.438453  4091 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.268s	user 0.000s	sys 0.001s
I20260709 12:22:44.169292  3983 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:44.218542  1806 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:22:44.394490  4036 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:44.453203  3922 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:44.692929  4092 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:22:44.581866  1807 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.007s	sys 0.020s
I20260709 12:22:44.016256  4020 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.706s
I20260709 12:22:44.018364  1802 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:22:44.117767  3920 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:44.194113  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 220990us
I20260709 12:22:44.180454  1881 pending_rounds.cc:107] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:22:44.139752  4069 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:44.709507  4079 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:44.310068  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:44.801052  1877 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:44.110522  4022 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.672s	user 0.014s	sys 0.005s
I20260709 12:22:44.725540  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:44.805299  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1363) took 4203 ms. Trace:
W20260709 12:22:44.726004  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.227347  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:44.805773  1804 rpcz_store.cc:276] 0709 12:22:40.601602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.186509 (+584907us) service_pool.cc:224] Handling call
0709 12:22:41.186955 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.187405 (+   450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.187440 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.188080 (+   640us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.293161 (+1105081us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:42.293217 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.293305 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.836685 (+1543380us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:44.791100 (+954415us) raft_consensus.cc:1685] finished
0709 12:22:44.791119 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.805214 (+ 14095us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":439,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":136,"spinlock_wait_cycles":38766976,"thread_start_us":1104558,"threads_started":1,"wal-append.queue_time_us":1301319}
W20260709 12:22:44.179343  1771 connection.cc:256] RPC call timeout handler was delayed by -1.19282770000018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21120000000000003s for a response
W20260709 12:22:44.807438  1771 connection.cc:256] RPC call timeout handler was delayed by -1.1289277000000766s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2183s for a response
W20260709 12:22:44.807782  1771 connection.cc:256] RPC call timeout handler was delayed by -1.110027700000046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22040000000000004s for a response
I20260709 12:22:43.961498  3815 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:44.115294  3710 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:44.728026  2383 connection.cc:256] RPC call timeout handler was delayed by -1.3048035300002994s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.39260000000000006s for a response
I20260709 12:22:44.097470  4076 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:44.037436  3931 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.865s	user 0.011s	sys 0.003s
I20260709 12:22:44.015782  4051 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:44.181855  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:44.181892  4043 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:44.175195  4081 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:44.836608  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:44.744410  4085 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.837337  1804 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:44.038985  3874 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:44.397858  3951 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:44.839569  4085 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:44.841609  3951 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:44.214572  4083 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:44.843408  4078 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:44.844902  4083 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:44.133325  4063 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:44.169922  3899 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:44.846109  4092 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:44.847061  3983 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.849072  3983 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:44.169119  4019 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:44.132771  3861 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:44.851708  3861 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.852311  3861 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:44.852595  3956 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.815s	user 0.008s	sys 0.003s
I20260709 12:22:44.853298  3956 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.855036  2417 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
W20260709 12:22:44.220692  3450 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.962s	user 0.012s	sys 0.000s
I20260709 12:22:43.997052  4089 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:44.856925  4089 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:44.857547  4089 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:44.169675  4072 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:44.574944  4021 raft_consensus.cc:1027] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 4 LEADER]: attempt to promote peer 123095127f334445889c21a42ba1d41e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:22:44.859761  4082 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:44.742550  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.167083  3796 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.813135  3815 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:44.860599  4082 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.444051  4090 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.268s
I20260709 12:22:44.752605  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:44.717295  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:44.755479  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:44.756230  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:44.757222  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.170428  4049 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:44.219677  3960 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:44.759704  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:44.078917  4041 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.663s
I20260709 12:22:44.716797  3984 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:44.723479  4071 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:44.757560  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.008s	sys 0.008s
I20260709 12:22:44.394119  3910 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:44.763064  2410 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:44.838613  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:44.865892  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:44.866266  1804 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:22:44.753679  4035 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.004s	sys 0.013s
I20260709 12:22:44.768450  2411 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260709 12:22:44.780082  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.004s	sys 0.006s
I20260709 12:22:44.235519  4000 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:44.766170  4056 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.009s	sys 0.018s
W20260709 12:22:44.787602  1807 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.840s	user 0.012s	sys 0.022s
I20260709 12:22:44.761433  3941 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:44.789060  1881 raft_consensus.cc:2889] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:44.791652  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:44.795624  4040 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:44.795549  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:44.796334  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1366) took 4189 ms. Trace:
I20260709 12:22:44.234570  4077 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.765087  3998 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.743181  3810 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.808390  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:44.786195  4036 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:44.819031  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 637902us
I20260709 12:22:44.785966  4039 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:44.809226  3909 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 12:22:43.783906  1874 rpcz_store.cc:276] 0709 12:22:40.831452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.174653 (+343201us) service_pool.cc:224] Handling call
0709 12:22:41.177702 (+  3049us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:41.180404 (+  2702us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.180442 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:41.181210 (+   768us) log.cc:844] Serialized 52 byte log entry
0709 12:22:42.143720 (+962510us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:42.144165 (+   445us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.144828 (+   663us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.145206 (+   378us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.145388 (+   182us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:43.535594 (+1390206us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:43.629344 (+ 93750us) raft_consensus.cc:1685] finished
0709 12:22:43.643684 (+ 14340us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:43.677897 (+ 34213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61,"prepare.queue_time_us":707,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":83,"raft.queue_time_us":1816,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":301,"spinlock_wait_cycles":5260032,"thread_start_us":962015,"threads_started":1,"wal-append.queue_time_us":1247442,"wal-append.run_cpu_time_us":19975,"wal-append.run_wall_time_us":2196655}
I20260709 12:22:44.827268  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:44.868996  4088 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.787400  3922 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:44.818940  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:44.811908  4094 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:44.751499  4086 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.833063  4051 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:44.834869  4076 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:44.835672  3931 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.836128  4043 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:44.861009  3796 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:44.158983  4038 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:44.862325  3815 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:44.812110  3920 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:22:44.763636  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
W20260709 12:22:44.866533  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.008s	sys 0.008s
W20260709 12:22:44.867126  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 107537us
I20260709 12:22:44.867723  1803 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:44.865376  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:44.870348  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 714322us
I20260709 12:22:44.918682  1806 raft_consensus.cc:2410] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:22:44.935532  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:44.873184  3450 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:44.940330  1802 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:44.846724  4092 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.863690  4090 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:22:44.872757  4035 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.725s	user 0.004s	sys 0.013s
I20260709 12:22:44.829711  4069 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:44.943670  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1571) took 1043 ms. Trace:
I20260709 12:22:44.934966  1800 rpcz_store.cc:276] 0709 12:22:40.607269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.430970 (+823701us) service_pool.cc:224] Handling call
0709 12:22:41.431431 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.730091 (+298660us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.730121 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.730643 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.612847 (+882204us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:42.612906 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.613016 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:44.796106 (+2183090us) raft_consensus.cc:1685] finished
0709 12:22:44.796129 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.796267 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9759,"raft.queue_time_us":209960,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":80164,"thread_start_us":841732,"threads_started":1,"wal-append.queue_time_us":2463472}
I20260709 12:22:44.766373  4087 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:44.962152  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 153843us
I20260709 12:22:44.960107  3998 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.157734  1801 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:45.158084  2414 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:44.963429  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:44.997987  4077 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.114916  4084 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:45.026957  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.163803  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.163727  3922 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.164801  3922 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:45.165946  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 1.00711e+06us
I20260709 12:22:45.166087  4086 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.169292  1877 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:45.171471  4069 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.173238  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.139417  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1482) took 1270 ms. Trace:
I20260709 12:22:44.994041  2418 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:44.957610  4036 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.179323  4087 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.139178  3996 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:45.200013  1805 rpcz_store.cc:276] 0709 12:22:43.868554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.899757 (+ 31203us) service_pool.cc:224] Handling call
0709 12:22:43.932049 (+ 32292us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:44.255523 (+323474us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.255557 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:44.303161 (+ 47604us) log.cc:844] Serialized 52 byte log entry
0709 12:22:44.786112 (+482951us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:45.127220 (+341108us) log.cc:844] Serialized 14 byte log entry
0709 12:22:45.134109 (+  6889us) log.cc:844] Serialized 14 byte log entry
0709 12:22:45.138887 (+  4778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.139046 (+   159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.139106 (+    60us) raft_consensus.cc:1685] finished
0709 12:22:45.139124 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.139325 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17538,"prepare.queue_time_us":352020,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":112,"raft.queue_time_us":442889,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":279,"thread_start_us":691084,"threads_started":3,"wal-append.queue_time_us":585081}
I20260709 12:22:45.201732  3996 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.202965  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.204556  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.171478  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.180150  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.205797  1877 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:45.174750  4084 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.175112  3951 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:45.175339  1875 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:44.985127  4081 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.125728  4072 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.175081  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.173640  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.138557  4009 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.177281  4076 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.177757  3815 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:44.981974  4040 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:44.993624  3810 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.008471  4088 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.005661  4000 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.175467  4092 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:45.180140  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.180943  4077 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:45.234848  1878 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:45.181573  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 362797us
I20260709 12:22:45.184239  2412 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 4 LEARNER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:22:45.245579  3983 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.246187  3983 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.246698  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:45.247313  2415 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:45.184516  4119 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:22:45.184921  1881 rpcz_store.cc:276] 0709 12:22:43.900179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.900357 (+   178us) service_pool.cc:224] Handling call
0709 12:22:43.931828 (+ 31471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:44.943345 (+1011517us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.943606 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:45.248293  3983 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.248792  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.248911  3450 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:45.250455  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.250834  3450 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.188627  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:45.158701  4090 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:45.253058  4090 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:44.986641  4023 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.257457  4083 raft_consensus.cc:2764] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:22:45.192756  3998 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.215040  1806 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:45.215301  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.259850  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.260690  1805 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
W20260709 12:22:45.259222  4050 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.519s	user 0.009s	sys 0.004s
I20260709 12:22:45.261669  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.262399  1805 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:45.222522  1802 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:45.223213  1803 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:45.264159  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1382) took 1068 ms. Trace:
I20260709 12:22:45.264477  1803 rpcz_store.cc:276] 0709 12:22:44.195800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.174733 (+978933us) service_pool.cc:224] Handling call
0709 12:22:45.264102 (+ 89369us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:45.176872  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.224895  1801 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:45.228029  4072 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.229981  3710 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.236799  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.239298  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.248911  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1483) took 1380 ms. Trace:
I20260709 12:22:45.035449  3956 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:44.995510  4039 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:45.113303  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 306985us
I20260709 12:22:45.191442  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.257570  4078 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:45.259693  3861 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:45.262545  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82491.1us
I20260709 12:22:45.283699  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.265977  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1492) took 1043 ms. Trace:
I20260709 12:22:45.269532  1807 rpcz_store.cc:276] 0709 12:22:43.868866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.869020 (+   154us) service_pool.cc:224] Handling call
0709 12:22:43.912602 (+ 43582us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:44.875321 (+962719us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.875362 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:44.875988 (+   626us) log.cc:844] Serialized 26 byte log entry
0709 12:22:45.114252 (+238264us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:45.114297 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.114394 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.248640 (+134246us) raft_consensus.cc:1685] finished
0709 12:22:45.248671 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.248835 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34432,"raft.queue_time_us":454354,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":124,"thread_start_us":1058698,"threads_started":2,"wal-append.queue_time_us":150153}
W20260709 12:22:45.274240  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 85726.6us
I20260709 12:22:45.285233  1806 rpcz_store.cc:276] 0709 12:22:44.222647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.179821 (+957174us) service_pool.cc:224] Handling call
0709 12:22:45.265933 (+ 86112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:45.277325  1800 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:45.279983  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.280143  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.287174  1805 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:45.289565  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1384) took 1093 ms. Trace:
I20260709 12:22:45.289970  1800 rpcz_store.cc:276] 0709 12:22:44.196393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.191045 (+994652us) service_pool.cc:224] Handling call
0709 12:22:45.289505 (+ 98460us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:45.290290  3983 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.290920  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.290983  3983 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.292178  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.292784  1800 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:45.293901  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.294371  1731 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:45.294595  1800 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
W20260709 12:22:45.277236  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.061s (SENT)
W20260709 12:22:45.262971  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 499431us
I20260709 12:22:45.293920  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.296351  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.296782  1726 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
W20260709 12:22:45.297821  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.295984  1727 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:22:45.298704  1727 pending_rounds.cc:85] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 6
I20260709 12:22:45.299044  1727 pending_rounds.cc:107] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:22:45.299310  1727 raft_consensus.cc:2889] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:45.302259  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.303843  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.304281  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.300133  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.303892  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.305868  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1390) took 1107 ms. Trace:
I20260709 12:22:45.295647  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:45.306231  1807 rpcz_store.cc:276] 0709 12:22:44.529066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.303618 (+774552us) service_pool.cc:224] Handling call
0709 12:22:45.305790 (+  2172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15360}
I20260709 12:22:45.159179  3931 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.306710  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:45.307332  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.308163  1723 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:45.308377  1800 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:22:45.062464  3899 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.311483  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.316847  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.312124  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:45.317588  1725 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:45.318135  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:45.319088  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.279238  4009 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.315238  1875 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:45.322960  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.323235  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.276755  4090 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:45.319875  1728 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:45.279683  4050 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:45.316992  1731 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:45.323923  1724 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:45.331903  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:45.332386  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:45.322000  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.303678  3983 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.324558  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.325891  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1385) took 1129 ms. Trace:
I20260709 12:22:45.327234  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.335649  1800 rpcz_store.cc:276] 0709 12:22:44.196655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.303557 (+1106902us) service_pool.cc:224] Handling call
0709 12:22:45.325823 (+ 22266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:45.330384  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:45.338913  1877 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:45.326678  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1583) took 1269 ms. Trace:
I20260709 12:22:45.330982  4050 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.340003  4050 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:45.339883  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.317379  3951 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:22:45.309903  3810 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:45.342204  3899 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.342952  1799 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:45.281057  4089 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.279770  3710 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.325964  4019 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:45.325433  3941 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:45.346130  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.327961  4063 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.364713  4063 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.365127  4063 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.365837  4081 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.366355  4081 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:45.366748  4081 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.359941  4019 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:45.339793  1874 rpcz_store.cc:276] 0709 12:22:44.052814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.176567 (+1123753us) service_pool.cc:224] Handling call
0709 12:22:45.322024 (+145457us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:45.367331  4019 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:45.368230  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.368996  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:45.369693  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.370419  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.328372  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:45.348598  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 405627us
I20260709 12:22:45.371372  2412 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:45.351070  3909 consensus_queue.cc:1243] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:22:45.328384  4071 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.351330  3612 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.353178  3996 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.322335  4077 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:45.331049  4043 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260709 12:22:45.380914  1804 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:45.354883  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1491) took 1243 ms. Trace:
I20260709 12:22:45.381875  1800 rpcz_store.cc:276] 0709 12:22:44.111243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.353843 (+1242600us) service_pool.cc:224] Handling call
0709 12:22:45.354260 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.354493 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:45.354526 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:45.354571 (+    45us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:45.354634 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.354686 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.354815 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:45.376174  4097 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.359054  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:45.307552  4072 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:45.362462  3941 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.363875  3710 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.393097  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:45.373942  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 58070.4us
I20260709 12:22:45.362097  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.284309  4049 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.377619  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.339119  4023 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.389448  3941 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:45.407682  3861 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.402560  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:45.355940  2415 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:45.374987  4086 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:45.380401  4077 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.372529  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:45.384773  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.400112  3983 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.422561  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.398787  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.423233  4072 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:45.423607  4072 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:45.422755  4078 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:45.424118  1875 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:45.389111  4035 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:22:45.420562  4077 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:45.410430  3899 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260709 12:22:45.432094  1875 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:22:45.432389  3899 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:45.432829  3899 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:45.433105  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.433630  3899 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.434559  3899 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:45.385527  4097 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:45.420161  3710 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:45.403723  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:45.405213  1802 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:45.409096  4084 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.414371  1800 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:22:45.357120  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:45.415063  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:22:45.415408  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53413.8us
I20260709 12:22:45.422734  3983 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.439389  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:45.435789  1881 raft_consensus.cc:2368] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5.
I20260709 12:22:45.435621  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.388616  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:45.399289  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.441327  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:45.458278  4116 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:22:45.438263  4097 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.418864  3612 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:45.459267  2415 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:45.446566  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45761.8us
W20260709 12:22:45.458416  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69882.1us
I20260709 12:22:45.460271  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1381) took 1264 ms. Trace:
I20260709 12:22:45.460526  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:45.458581  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48196.8us
I20260709 12:22:45.460640  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:45.424149  4072 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:45.414836  3920 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 12:22:45.473958  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.474623  1877 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:45.472548  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.461786  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.468598  1800 rpcz_store.cc:276] 0709 12:22:44.195508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.383070 (+1187562us) service_pool.cc:224] Handling call
0709 12:22:45.410542 (+ 27472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:45.460066 (+ 49524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.460225 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:45.436919  2412 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:45.465731  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 108454us
I20260709 12:22:45.478039  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:45.479710  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.460263  3983 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.482523  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.461748  4069 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:45.464928  3932 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.486104  3810 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:45.463811  4009 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260709 12:22:45.483568  3450 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:45.474875  3815 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.489077  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.490047  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:45.492444  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:45.505051  4120 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:22:45.506102  3891 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:45.506916  4083 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:45.506981  4076 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:45.507927  3909 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:45.508355  3909 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:45.509157  3909 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.510336  3909 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:45.495102  4051 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:22:45.512997  4071 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:45.514889  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 73680us
I20260709 12:22:45.437063  3796 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.475010  3710 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.497838  2415 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:45.440264  3874 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:22:45.488533  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1573) took 1572 ms. Trace:
I20260709 12:22:45.487805  3996 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:45.478863  4084 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:45.500495  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.494270  4094 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:45.495038  2410 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:45.459115  4086 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 12:22:45.496870  1874 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:45.538702  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.539671  1877 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:45.540673  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.532424  1873 rpcz_store.cc:276] 0709 12:22:43.915991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.922926 (+  6935us) service_pool.cc:224] Handling call
0709 12:22:43.939663 (+ 16737us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:44.760450 (+820787us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.760476 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:44.767389 (+  6913us) log.cc:844] Serialized 52 byte log entry
0709 12:22:44.881591 (+114202us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:44.940352 (+ 58761us) log.cc:844] Serialized 14 byte log entry
0709 12:22:45.348287 (+407935us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b180025a3e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:22:45.350310 (+  2023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.488174 (+137864us) spinlock_profiling.cc:243] Waited 774 ms on lock 0x7b5400026054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:45.488240 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.488286 (+    46us) raft_consensus.cc:1685] finished
0709 12:22:45.488299 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.488468 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35772,"prepare.queue_time_us":75621,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":479856,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":236,"spinlock_wait_cycles":1784600192,"thread_start_us":127520,"threads_started":2,"wal-append.queue_time_us":388099}
I20260709 12:22:45.516536  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:45.491957  3932 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:45.528343  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.463929  4036 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.491123  4069 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:45.492990  4035 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:22:45.549974  3932 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.565454  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 489) took 1774 ms. Trace:
I20260709 12:22:45.565980  2414 rpcz_store.cc:276] 0709 12:22:43.790690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.794221 (+  3531us) service_pool.cc:224] Handling call
0709 12:22:43.837070 (+ 42849us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.158548 (+1321478us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.565157 (+406609us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400572e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:45.565380 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3016957312}
I20260709 12:22:45.567451  4049 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.568307  1729 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:22:45.549629  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.568816  4092 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.500304  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:45.570479  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:45.571135  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 70937.4us
I20260709 12:22:45.544492  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.503650  3815 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:45.534703  4041 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:45.546167  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:45.528883  4000 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:45.531978  3710 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.533970  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.571231  1873 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:45.537885  4084 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.574335  2419 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:45.546394  1872 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:45.575313  1730 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:45.574854  1724 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:45.547039  3891 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:45.571075  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.538702  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:22:45.580587  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46708.6us
I20260709 12:22:45.612190  2415 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:45.613080  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:45.547094  4051 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:45.559978  4069 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:45.570084  4040 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:45.610055  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:45.614293  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.561753  4036 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.579442  3710 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.616312  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.581161  4089 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.575986  3815 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.562340  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.624118  1875 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:45.625140  1730 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
W20260709 12:22:45.617470  4121 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:22:45.561425  3796 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.575893  3922 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.628360  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.615541  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.609393  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:45.632316  1877 raft_consensus.cc:2376] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5. Re-sending same reply.
I20260709 12:22:45.671433  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.672185  1877 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:45.537901  4047 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 12:22:45.631978  3996 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:45.625218  1873 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:22:45.610656  4023 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:45.629680  3956 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:45.627537  3861 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:22:45.671347  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 62010.8us
I20260709 12:22:45.607581  4084 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:45.633876  4083 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:45.625433  4051 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:45.677896  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49603.1us
W20260709 12:22:45.695091  1846 connection.cc:256] RPC call timeout handler was delayed by -1.0039869789998193s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
I20260709 12:22:45.696178  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.664112  2415 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:45.674422  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:45.611068  3874 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:22:45.611569  3920 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260709 12:22:45.742043  3920 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:45.683271  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.742422  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.742816  1873 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:45.743218  1874 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:45.616465  4049 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:22:45.742466  3920 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:45.746673  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 279) took 1214 ms (client timeout 1523 ms). Trace:
I20260709 12:22:45.747148  3920 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.748030  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1488) took 1808 ms. Trace:
I20260709 12:22:45.748814  3920 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:45.747154  1804 rpcz_store.cc:269] 0709 12:22:44.532014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.936604 (+404590us) service_pool.cc:224] Handling call
0709 12:22:45.746459 (+809855us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b540026f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:45.746607 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":510674816}
W20260709 12:22:45.751717  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1574) took 1826 ms (client timeout 1930 ms). Trace:
I20260709 12:22:45.752270  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.622244  4009 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260709 12:22:45.752878  4082 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.627030  3960 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.057s
I20260709 12:22:45.667424  4086 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.755618  4086 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.756351  4081 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.758085  4086 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.695528  4023 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:45.677486  4036 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:45.697481  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.759541  2415 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:45.760532  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:45.760591  4072 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.761430  2415 raft_consensus.cc:2368] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6.
I20260709 12:22:45.762233  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:45.698376  4124 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:22:45.764240  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.764917  1876 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
W20260709 12:22:45.765769  4096 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.944s	user 0.004s	sys 0.011s
I20260709 12:22:45.698429  4083 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.700489  1730 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:45.747128  4063 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.767774  4020 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:45.768244  4020 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:45.769361  4063 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.769273  4020 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.770488  2414 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:45.770983  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.771560  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.700883  4051 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:45.772627  4051 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.765805  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.773062  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:45.773516  1876 raft_consensus.cc:2376] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4. Re-sending same reply.
I20260709 12:22:45.774320  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.703370  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.793298  1806 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 2.
I20260709 12:22:45.677491  3922 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.692543  3996 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:45.719496  4092 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.687595  3861 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:45.807616  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.818745  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.812970  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.822209  2416 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:45.823328  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:45.813622  2417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 593) took 2313 ms (client timeout 1662 ms). Trace:
W20260709 12:22:45.825681  2417 rpcz_store.cc:269] 0709 12:22:43.500668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.500849 (+   181us) service_pool.cc:224] Handling call
0709 12:22:45.813436 (+2312587us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54005f05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:45.813553 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3111109888}
I20260709 12:22:45.724220  4077 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.725210  4019 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.728467  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.736595  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.726984  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.835291  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.836118  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.739387  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1380) took 1544 ms. Trace:
W20260709 12:22:45.744024  1799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1486) took 1874 ms (client timeout 2133 ms). Trace:
W20260709 12:22:45.752143  1878 rpcz_store.cc:269] 0709 12:22:43.925493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.931709 (+  6216us) service_pool.cc:224] Handling call
0709 12:22:45.751528 (+1819819us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54001f7554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:45.751656 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2458479872}
I20260709 12:22:45.753722  3941 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.758172  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.774859  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.634359  4038 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.842170  1876 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:45.702575  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.843358  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.776957  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.844282  2411 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:45.778153  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.844331  1697 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:45.845425  1807 raft_consensus.cc:2410] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:22:45.778455  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:45.779664  1728 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:45.780599  1730 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:22:45.780843  4087 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.748462  1798 rpcz_store.cc:276] 0709 12:22:43.939903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.956738 (+ 16835us) service_pool.cc:224] Handling call
0709 12:22:43.957204 (+   466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.164661 (+1207457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:45.164691 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:45.164730 (+    39us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:45.164768 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.747767 (+582999us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002f2e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:45.747819 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.747965 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2776328192}
I20260709 12:22:45.803043  3922 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.803517  4047 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:45.806944  4049 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.784837  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:45.785710  2418 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 490) took 1994 ms (client timeout 2071 ms). Trace:
W20260709 12:22:45.786484  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1502) took 1394 ms (client timeout 1676 ms). Trace:
I20260709 12:22:45.753713  1804 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:45.790591  2415 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:22:45.856086  1801 rpcz_store.cc:269] 0709 12:22:44.392441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.172942 (+780501us) service_pool.cc:224] Handling call
0709 12:22:45.786299 (+613357us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b54001eb9d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:45.786424 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54720768}
I20260709 12:22:45.792038  1873 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:45.858645  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.689697  1877 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:45.859977  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.860622  1798 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:45.860956  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.861482  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.861763  1877 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:45.782485  4000 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:45.796641  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 100556us
W20260709 12:22:45.798401  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 140349us
I20260709 12:22:45.864253  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.862689  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.867229  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.868012  1879 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:45.838662  4072 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.872416  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.803157  4040 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
I20260709 12:22:45.873519  1804 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:45.877115  2415 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:45.862092  1798 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:45.717465  2412 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:45.766368  3932 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.883757  1723 raft_consensus.cc:2410] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:45.884366  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.884727  2417 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:45.831123  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:45.832695  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1572) took 1923 ms (client timeout 2133 ms). Trace:
W20260709 12:22:45.838313  1799 rpcz_store.cc:269] 0709 12:22:43.869836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.871009 (+  1173us) service_pool.cc:224] Handling call
0709 12:22:45.743817 (+1872808us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54001d4f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:45.743941 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3209102592}
I20260709 12:22:45.836536  2414 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:45.887099  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.836163  1880 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
W20260709 12:22:45.886474  1872 rpcz_store.cc:269] 0709 12:22:43.909518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.909714 (+   196us) service_pool.cc:224] Handling call
0709 12:22:45.832420 (+1922706us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400026554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:45.832611 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3196086144}
I20260709 12:22:45.888528  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.889811  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.889987  3815 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.892710  3815 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:45.847477  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.895026  1772 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:45.895695  4083 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:45.863148  4049 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.896365  4083 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.896767  4072 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:45.898109  4049 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.898828  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.758674  4081 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.898553  4072 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:45.899406  1881 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:45.852030  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.899897  4077 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.857991  4087 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:45.900384  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.901038  1881 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:45.852701  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.837682  1802 rpcz_store.cc:276] 0709 12:22:44.195208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.371802 (+1176594us) service_pool.cc:224] Handling call
0709 12:22:45.372135 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.445687 (+ 73552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.739001 (+293314us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:45.739283 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31906560}
I20260709 12:22:45.902765  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.903345  1805 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:45.794586  4020 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:45.828024  3861 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:45.793320  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 118996us
W20260709 12:22:45.855370  2418 rpcz_store.cc:269] 0709 12:22:43.791566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.791752 (+   186us) service_pool.cc:224] Handling call
0709 12:22:45.785482 (+1993730us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400840154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:45.785602 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2731803904}
I20260709 12:22:45.863197  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:45.907656  2419 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:45.878679  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.868058  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.909639  1727 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:45.777205  4097 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.896212  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:45.866125  4063 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:45.908771  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.914314  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:45.901834  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.916742  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.917702  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:45.917944  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.918866  1881 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:45.904553  1799 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:45.781755  4088 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:45.890395  1880 raft_consensus.cc:2376] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7. Re-sending same reply.
I20260709 12:22:45.909770  1877 raft_consensus.cc:2368] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7.
I20260709 12:22:45.928108  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.929153  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:45.876108  4050 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.501s	user 0.011s	sys 0.000s
I20260709 12:22:45.928115  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:45.933475  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:45.910405  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.915738  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:45.874534  3612 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:45.854172  3710 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:45.904120  4036 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.899545  4083 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.932621  1877 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:45.954958  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.956496  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.957604  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:45.912870  2413 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:45.937841  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:45.938282  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:45.918506  1798 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:45.941165  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.925366  3815 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:45.942939  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:45.882246  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:45.926023  4094 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:45.850960  4095 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:22:45.931550  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.948717  1803 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:45.936290  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:45.953073  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:45.982559  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 118931us
I20260709 12:22:45.958642  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:46.038908  1880 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:46.039165  1877 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:46.039848  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:46.040108  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:46.041049  1877 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:45.925091  4086 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:45.943023  1805 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:45.978693  1806 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:46.019541  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:45.887523  3960 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.057s
W20260709 12:22:46.081156  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:46.021960  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 84194.6us
W20260709 12:22:46.081781  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.927591  3450 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:45.927397  4126 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:46.024442  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 142268us
I20260709 12:22:45.944047  3612 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.094133  3612 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:46.095981  4019 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.944916  4082 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.134s
I20260709 12:22:46.097589  4082 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:46.097975  4082 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:45.958442  4087 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:45.936314  4063 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:46.042593  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:46.041638  4023 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:46.049727  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 106944us
I20260709 12:22:46.107645  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:46.108937  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:46.056334  2417 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260709 12:22:46.110407  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.111047  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:46.058960  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:45.972945  4132 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:22:46.112062  1798 raft_consensus.cc:2410] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 12:22:46.083475  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.950806  4078 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:46.065701  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:46.067488  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:46.069222  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:45.968536  4097 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.971077  3710 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.949759  4089 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:46.151979  4089 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:46.153038  4089 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.114506  4023 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.161676  2410 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:45.952333  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:45.972216  3932 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:45.964532  4043 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.946020  4050 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.085808  1881 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:46.088515  4038 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.020678  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:46.099922  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:46.176839  1872 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:46.090225  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:46.178481  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 88338.3us
I20260709 12:22:46.048760  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:46.180096  1880 raft_consensus.cc:2368] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4.
W20260709 12:22:46.084617  3983 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.586s	user 0.012s	sys 0.000s
I20260709 12:22:46.043419  4083 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:46.038508  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:22:46.089457  3909 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.579s	user 0.010s	sys 0.002s
I20260709 12:22:46.099613  4063 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.115852  4019 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:46.116503  3941 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.116900  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.117251  4069 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.058763  1799 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:46.191869  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:46.192471  1799 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:46.192883  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 76046.6us
I20260709 12:22:46.194037  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:46.195501  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:46.119107  3796 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:46.124101  3998 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:22:46.130112  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.984463  4036 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.133751  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:45.936829  3996 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.068158  4094 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:46.137251  3922 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.126338  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:46.229269  1804 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:46.134420  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.073004  4088 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:46.141145  1727 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:46.145423  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:45.980010  4084 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.144708  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:46.150246  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:46.235713  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 91068us
I20260709 12:22:46.236557  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:46.237231  1877 raft_consensus.cc:2368] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7.
W20260709 12:22:46.101622  4090 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.772s	user 0.011s	sys 0.001s
I20260709 12:22:45.978637  4047 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:46.155723  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.156867  2382 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.162335  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.169019  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:46.239308  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 82573us
I20260709 12:22:46.239526  1881 raft_consensus.cc:2376] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5. Re-sending same reply.
I20260709 12:22:46.240548  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:46.169502  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:46.241081  1881 raft_consensus.cc:2376] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3. Re-sending same reply.
I20260709 12:22:46.140565  1874 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:46.241469  1807 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:46.241859  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:46.101183  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:46.163703  4023 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:46.245383  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.241482  4051 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:46.248842  4051 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.249207  4051 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:46.181901  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:46.242328  4019 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:46.243070  1878 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:46.251220  1880 raft_consensus.cc:2376] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7. Re-sending same reply.
I20260709 12:22:46.202086  3909 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:46.184995  3956 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:46.262295  3956 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.262774  3956 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:46.268431  4069 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.170605  2414 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:46.188287  1875 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:46.275367  4050 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.098874  4082 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.227409  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.229669  1873 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:46.232370  4088 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.277212  4088 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:46.277832  4000 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:46.277947  1873 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:22:46.230576  1799 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:46.278734  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1587) took 2148 ms. Trace:
W20260709 12:22:46.230695  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 97039.2us
I20260709 12:22:46.189574  4008 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:22:46.237671  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:46.279465  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:46.171937  4043 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.280172  4008 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:46.190891  4083 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:46.281132  4008 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.281958  4043 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.228607  1802 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:46.237946  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:46.282974  4097 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:46.283229  1877 raft_consensus.cc:2376] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7. Re-sending same reply.
I20260709 12:22:46.283699  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:46.256595  3796 raft_consensus.cc:2764] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:22:46.281908  4083 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.286254  4084 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.183653  3983 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.243677  1881 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:46.288621  1881 raft_consensus.cc:606] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:46.288990  1881 raft_consensus.cc:614] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:22:46.185212  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 146815us
I20260709 12:22:46.279161  1873 rpcz_store.cc:276] 0709 12:22:44.130374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.277316 (+2146942us) service_pool.cc:224] Handling call
0709 12:22:46.277675 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.278493 (+   818us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.278679 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.290112  1881 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:46.289896  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:46.246522  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.256853  1804 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:46.229225  3920 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:46.259442  1880 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:46.277165  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 75786.7us
I20260709 12:22:46.296011  1879 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:46.296773  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:46.252530  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.231236  3941 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:46.289088  1846 leader_election.cc:400] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6.
I20260709 12:22:46.199079  3998 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:46.290704  1873 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:46.302865  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2147) took 1025 ms. Trace:
W20260709 12:22:46.302937  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50500us
I20260709 12:22:46.293902  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:46.232105  4049 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:46.303251  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.316109  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.270655  3956 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.305065  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:46.290704  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 264) took 1497 ms. Trace:
I20260709 12:22:46.303594  1873 rpcz_store.cc:276] 0709 12:22:45.277849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.290298 (+1012449us) service_pool.cc:224] Handling call
0709 12:22:46.290551 (+   253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.302501 (+ 11950us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.302769 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.316254  1697 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:46.319873  3810 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:22:46.320433  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.320660  1878 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:22:46.322558  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:46.322899  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:46.319711  1881 rpcz_store.cc:276] 0709 12:22:44.793748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.289686 (+1495938us) service_pool.cc:224] Handling call
0709 12:22:46.289956 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.290491 (+   535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.290650 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.327288  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:46.327932  1881 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:46.330710  1881 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:46.335667  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.336706  1730 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:46.337054  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.339322  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1576) took 2401 ms. Trace:
I20260709 12:22:46.339900  1878 rpcz_store.cc:276] 0709 12:22:43.937689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.251880 (+2314191us) service_pool.cc:224] Handling call
0709 12:22:46.314443 (+ 62563us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.338870 (+ 24427us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.339225 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.345153  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.346351  1723 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:46.319264  4147 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:46.351840  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.352515  1723 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:22:46.301492  3920 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.371515  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.372825  1728 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:46.348035  4147 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:46.279121  2414 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:46.301394  4050 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:46.357811  1730 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:22:46.315163  4019 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:46.314240  4000 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:46.320785  4038 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:46.321026  3909 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:46.380633  3920 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:46.393335  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:46.395301  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.343282  4090 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:46.349412  4077 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:46.358526  4129 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:46.414870  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.360939  3983 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.414860  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.346918  3932 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:46.423790  1807 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:46.332171  4047 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.094s
I20260709 12:22:46.427794  1730 raft_consensus.cc:2368] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4.
I20260709 12:22:46.332523  4043 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260709 12:22:46.321017  3941 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.428983  4000 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:46.388463  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:46.410764  3861 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.358251  3450 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 12:22:46.381723  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:46.454730  3956 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.422508  3998 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.430773  4090 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.415596  4050 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:46.370704  4040 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260709 12:22:46.539669  4090 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:46.539863  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.541391  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.491329  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:46.543201  4036 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:46.543987  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:46.544241  3998 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:46.452235  4136 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
W20260709 12:22:46.435796  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 108559us
I20260709 12:22:46.552950  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:46.446451  4145 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:22:46.431972  4038 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:46.481535  1872 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:46.446636  4097 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:46.527511  4088 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.528656  1724 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:22:46.528851  4043 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:46.566987  1877 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:22:46.532570  1723 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:46.496841  4078 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:46.438550  4077 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:46.424813  4039 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:46.454039  3922 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:46.543228  1807 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:46.542564  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.546586  3932 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:46.526583  3861 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.482399  4129 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.575616  4089 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:46.583729  3932 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.584491  1798 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:46.584916  4129 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.446622  3612 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.585410  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1383) took 2389 ms. Trace:
I20260709 12:22:46.585899  1798 rpcz_store.cc:276] 0709 12:22:44.196103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.574985 (+2378882us) service_pool.cc:224] Handling call
0709 12:22:46.584265 (+  9280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.585076 (+   811us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.585333 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.570104  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.571672  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:46.587167  3612 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:46.587577  4038 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.571903  1872 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:46.577916  1728 raft_consensus.cc:2410] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:22:46.438474  1879 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:37240
W20260709 12:22:46.581260  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 258525us
I20260709 12:22:46.559455  4094 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:46.589624  3920 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.567137  4043 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:46.574712  4077 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:46.617893  1798 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:46.613513  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.614686  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:22:46.620146  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48583.9us
I20260709 12:22:46.544373  4050 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.616906  1725 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:46.633212  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1500) took 2241 ms. Trace:
I20260709 12:22:46.591456  1879 raft_consensus.cc:606] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:22:46.634033  1798 rpcz_store.cc:276] 0709 12:22:44.391885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.591926 (+2200041us) service_pool.cc:224] Handling call
0709 12:22:46.617598 (+ 25672us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.623213 (+  5615us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.633115 (+  9902us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.638556  1725 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:46.621709  1872 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:46.622153  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:22:46.613462  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.622355  1728 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:46.612470  4000 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.619712  4089 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:46.627756  4158 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:46.643911  1728 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:46.592242  4039 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:46.645314  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.651822  1798 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:46.655098  1728 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:46.655642  1728 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:46.656896  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 281) took 2124 ms. Trace:
I20260709 12:22:46.657537  1798 rpcz_store.cc:276] 0709 12:22:44.532617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.636280 (+2103663us) service_pool.cc:224] Handling call
0709 12:22:46.651636 (+ 15356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.656452 (+  4816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.656802 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.659973  1728 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.661335  1728 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:46.637607  4094 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:46.665005  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1391) took 2135 ms. Trace:
I20260709 12:22:46.637257  4088 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.646945  3922 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:46.647115  4151 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:22:46.740911  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 78676us
I20260709 12:22:46.746225  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 286) took 2001 ms. Trace:
I20260709 12:22:46.656262  3920 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:46.749246  1798 rpcz_store.cc:276] 0709 12:22:44.744740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.674390 (+1929650us) service_pool.cc:224] Handling call
0709 12:22:46.745409 (+ 71019us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.745721 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.745766 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:46.745828 (+    62us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:46.745906 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.745985 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.746149 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.753216  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:46.758292  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:22:46.760044  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 463352us
I20260709 12:22:46.761559  1728 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 12:22:46.761991  1728 pending_rounds.cc:85] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 3
I20260709 12:22:46.762260  1728 pending_rounds.cc:107] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:22:46.762512  1728 raft_consensus.cc:2889] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:46.762897  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:46.763644  1872 raft_consensus.cc:2368] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4.
I20260709 12:22:46.651661  4023 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.764583  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:46.766095  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 287) took 1377 ms. Trace:
I20260709 12:22:46.766763  1872 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:22:46.766862  1801 rpcz_store.cc:276] 0709 12:22:45.388188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.765249 (+1377061us) service_pool.cc:224] Handling call
0709 12:22:46.765550 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.765743 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.765770 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:46.765812 (+    42us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:46.765873 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.765931 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.766038 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.768321  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:46.765745  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:46.769716  4086 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:46.747172  3796 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:22:46.739732  4157 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:46.801411  1725 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:22:46.801858  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.809675  2419 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:46.766122  4069 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:46.812885  1798 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260709 12:22:46.698940  4043 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.805172  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:46.767295  4051 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.515s	user 0.013s	sys 0.000s
I20260709 12:22:46.770241  4086 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.769392  1801 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:46.750213  4158 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:46.802263  1771 leader_election.cc:400] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7.
I20260709 12:22:46.764024  4151 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:46.740674  4094 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:46.813020  1723 raft_consensus.cc:2368] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8.
I20260709 12:22:46.813649  1879 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:46.807252  3941 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:46.811970  4157 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:46.815414  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1398) took 1427 ms. Trace:
I20260709 12:22:46.801296  4036 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:46.664193  4150 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.149s
I20260709 12:22:46.831677  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1586) took 2743 ms. Trace:
I20260709 12:22:46.831406  4151 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.832221  1880 rpcz_store.cc:276] 0709 12:22:44.088435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.252274 (+2163839us) service_pool.cc:224] Handling call
0709 12:22:46.252700 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.295571 (+ 42871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.831433 (+535862us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b5400223b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:46.831621 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4934912}
W20260709 12:22:46.834645  4154 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:22:46.740785  1805 rpcz_store.cc:276] 0709 12:22:44.529697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.615705 (+2086008us) service_pool.cc:224] Handling call
0709 12:22:46.616070 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.616304 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:46.637917 (+ 21613us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.664285 (+ 26368us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:46.664528 (+   243us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:46.664616 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.664719 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.664921 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1554,"prepare.queue_time_us":46939,"raft.queue_time_us":36601,"spinlock_wait_cycles":10881792,"thread_start_us":24766,"threads_started":1}
I20260709 12:22:46.837051  1805 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:46.838279  4043 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:46.839015  1800 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:22:46.814682  3909 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.758124  3922 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 12:22:46.819695  4086 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:46.819542  4051 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.851667  3922 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:46.851369  4090 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.852160  3922 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:46.853097  4051 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:46.853662  1799 raft_consensus.cc:2368] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6.
I20260709 12:22:46.854156  4090 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:46.856446  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1509) took 1379 ms. Trace:
I20260709 12:22:46.856904  1800 rpcz_store.cc:276] 0709 12:22:45.477053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.838280 (+1361227us) service_pool.cc:224] Handling call
0709 12:22:46.838747 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.856134 (+ 17387us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.856382 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.814953  1728 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:22:46.822352  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 288) took 1434 ms. Trace:
I20260709 12:22:46.860868  1801 rpcz_store.cc:276] 0709 12:22:45.388335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.768840 (+1380505us) service_pool.cc:224] Handling call
0709 12:22:46.769155 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.820800 (+ 51645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.822268 (+  1468us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.862948  1879 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:46.864678  1800 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:46.823475  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:46.829303  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1397) took 1441 ms (client timeout 1875 ms). Trace:
I20260709 12:22:46.831727  4150 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:22:46.855561  4158 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.840466  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 503) took 1344 ms. Trace:
I20260709 12:22:46.840755  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1507) took 1368 ms. Trace:
I20260709 12:22:46.869618  1805 rpcz_store.cc:276] 0709 12:22:45.472414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.836401 (+1363987us) service_pool.cc:224] Handling call
0709 12:22:46.836832 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.840473 (+  3641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.840678 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:46.818302  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48100.8us
I20260709 12:22:46.840670  1798 rpcz_store.cc:276] 0709 12:22:45.387712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.763689 (+1375977us) service_pool.cc:224] Handling call
0709 12:22:46.810605 (+ 46916us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.815140 (+  4535us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.815359 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.845778  3909 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:46.847360  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.877528  1726 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:46.865382  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1512) took 1383 ms. Trace:
W20260709 12:22:46.865783  1803 rpcz_store.cc:269] 0709 12:22:45.387502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.411066 (+ 23564us) service_pool.cc:224] Handling call
0709 12:22:46.829089 (+1418023us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b540035f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:46.829236 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1208001280}
I20260709 12:22:46.869441  2410 rpcz_store.cc:276] 0709 12:22:45.495812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.542101 (+ 46289us) service_pool.cc:224] Handling call
0709 12:22:46.161421 (+619320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.162124 (+   703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.840272 (+678148us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b54006b0150. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:46.840409 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1416850816}
I20260709 12:22:46.877907  1800 rpcz_store.cc:276] 0709 12:22:45.481786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.861449 (+1379663us) service_pool.cc:224] Handling call
0709 12:22:46.864468 (+  3019us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.865118 (+   650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.865326 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.840364  4157 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:46.880533  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1469) took 1039 ms. Trace:
I20260709 12:22:46.880960  1726 rpcz_store.cc:276] 0709 12:22:45.841498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.847073 (+1005575us) service_pool.cc:224] Handling call
0709 12:22:46.880467 (+ 33394us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.853217  3922 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.867594  4150 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:46.818256  4069 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.821017  4094 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:46.875546  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:46.770313  4089 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.901327  1771 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:46.886997  1800 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:46.839651  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2169) took 1138 ms. Trace:
I20260709 12:22:46.891501  1798 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:22:46.905313  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1515) took 1400 ms. Trace:
I20260709 12:22:46.905417  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1406) took 1212 ms. Trace:
I20260709 12:22:46.862485  4088 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:46.915804  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.915805  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2311) took 1165 ms. Trace:
I20260709 12:22:46.894990  3922 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:46.896202  3909 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:46.885742  1728 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:46.905797  1798 rpcz_store.cc:276] 0709 12:22:45.504442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.876231 (+1371789us) service_pool.cc:224] Handling call
0709 12:22:46.891298 (+ 15067us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.904954 (+ 13656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.905240 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.906155  1800 rpcz_store.cc:276] 0709 12:22:45.692653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.882139 (+1189486us) service_pool.cc:224] Handling call
0709 12:22:46.886806 (+  4667us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.902639 (+ 15833us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.905318 (+  2679us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.902688  1878 rpcz_store.cc:276] 0709 12:22:45.701562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.341200 (+639638us) service_pool.cc:224] Handling call
0709 12:22:46.390394 (+ 49194us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.390885 (+   491us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.390926 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.391549 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 12:22:46.561573 (+170024us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:46.561620 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.561700 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.839392 (+277692us) raft_consensus.cc:1685] finished
0709 12:22:46.839421 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.839587 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":27715,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":4847,"thread_start_us":131011,"threads_started":1,"wal-append.queue_time_us":201867}
I20260709 12:22:46.919762  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:46.893054  4023 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:46.920590  1724 rpcz_store.cc:276] 0709 12:22:45.750384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.750547 (+   163us) service_pool.cc:224] Handling call
0709 12:22:45.750983 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.563841 (+812858us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.915562 (+351721us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b54002b7fd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:46.915745 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1787751296}
I20260709 12:22:46.936506  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:46.936451  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.937311  1800 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260709 12:22:46.938316  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1407) took 1243 ms. Trace:
I20260709 12:22:46.938370  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.939402  1800 rpcz_store.cc:276] 0709 12:22:45.695163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.932851 (+1237688us) service_pool.cc:224] Handling call
0709 12:22:46.936776 (+  3925us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.937917 (+  1141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.938221 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.822669  4147 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260709 12:22:46.939810  1724 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:46.879559  4077 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.941768  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:46.938428  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:46.934803  4088 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.817153  4039 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:46.910828  3874 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:46.946182  1800 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:22:46.908188  4051 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:46.947353  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.949383  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.955152  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1408) took 1258 ms. Trace:
I20260709 12:22:46.955564  1800 rpcz_store.cc:276] 0709 12:22:45.696549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.940471 (+1243922us) service_pool.cc:224] Handling call
0709 12:22:46.944327 (+  3856us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.954855 (+ 10528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.955091 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:46.956024  1724 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:46.949477  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:46.942204  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.957859  1800 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:46.958586  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.959913  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.963336  3909 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:46.963956  3909 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:46.964303  4069 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:46.964819  3909 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:46.966477  4162 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:22:46.968412  4079 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:46.952909  4126 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:46.952409  4150 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:46.970337  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 790876us
I20260709 12:22:46.944362  4072 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:46.972994  4072 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:46.973381  4072 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:46.975947  2411 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:46.976315  1732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1448) took 3445 ms (client timeout 1662 ms). Trace:
W20260709 12:22:46.976797  1732 rpcz_store.cc:269] 0709 12:22:43.531198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.635348 (+104150us) service_pool.cc:224] Handling call
0709 12:22:46.976053 (+3340705us) spinlock_profiling.cc:243] Waited 2.48 s on lock 0x7b540030af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:46.976201 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5697160576}
I20260709 12:22:46.977995  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:46.978929  4147 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:46.979218  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.978942  1732 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:46.981179  1728 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 2 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 1. Current term is 2. Ops: []
I20260709 12:22:46.987810  1724 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 6 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:46.995060  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:46.995396  1798 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260709 12:22:46.997915  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48539.4us
I20260709 12:22:46.998840  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:47.002439  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:47.005371  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.007449  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.009826  1876 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:22:46.965520  1802 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:22:47.014724  1731 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:46.996047  4050 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:22:47.016207  4050 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:47.016585  4050 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:47.016934  4155 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.274s	user 0.004s	sys 0.000s
W20260709 12:22:47.017589  4163 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 12:22:47.000824  4147 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:46.968931  4079 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:47.020855  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:47.026495  1798 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:47.022089  1876 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:22:47.031466  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.034860  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:47.038096  4036 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:47.040237  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:47.040871  1800 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.013777  2414 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:47.008957  4126 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:47.014402  3998 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:47.021667  3951 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.050482  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:47.052474  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1405) took 1368 ms. Trace:
I20260709 12:22:47.057605  1872 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: [3.3-4.4]
I20260709 12:22:47.053880  1803 rpcz_store.cc:276] 0709 12:22:45.683883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.878279 (+1194396us) service_pool.cc:224] Handling call
0709 12:22:46.891262 (+ 12983us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.891490 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:46.933407 (+ 41917us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.052051 (+118644us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:47.052112 (+    61us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:47.052161 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.052229 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.052372 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":41684,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":122,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"thread_start_us":118126,"threads_started":1,"wal-append.queue_time_us":118465}
I20260709 12:22:47.005097  1771 leader_election.cc:400] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4.
I20260709 12:22:47.055702  3710 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:47.042112  4147 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
I20260709 12:22:47.061357  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:47.063133  4079 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.064404  4079 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:47.020035  3796 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 12:22:47.009141  4023 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
W20260709 12:22:47.068936  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 57564.5us
I20260709 12:22:47.078543  1798 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:47.060046  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55161.9us
I20260709 12:22:47.085981  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:47.014070  4092 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:22:47.095090  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:47.099270  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.104723  1872 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:47.099623  4169 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 12:22:47.107964  4153 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.369s	user 0.000s	sys 0.001s
I20260709 12:22:47.032178  4170 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 12:22:47.068691  3951 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.110114  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.110989  1873 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:47.111352  3951 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:47.111877  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.112668  1873 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:47.113312  2412 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:47.114743  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1466) took 1296 ms. Trace:
I20260709 12:22:47.124505  1725 rpcz_store.cc:276] 0709 12:22:45.817843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.639515 (+821672us) service_pool.cc:224] Handling call
0709 12:22:46.639838 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.809627 (+169789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.114463 (+304836us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540035d754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:47.114679 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":370282880}
I20260709 12:22:47.126047  1725 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:47.125762  4050 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.081888  3815 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260709 12:22:47.127444  4050 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:47.133805  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.107053  4035 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:47.123206  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1589) took 2624 ms. Trace:
I20260709 12:22:47.137490  1877 rpcz_store.cc:276] 0709 12:22:44.499176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.283991 (+1784815us) service_pool.cc:224] Handling call
0709 12:22:46.562909 (+278918us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.567650 (+  4741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.122976 (+555326us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b5400117e50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.123135 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":640472448}
I20260709 12:22:47.135478  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:47.113603  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.123685  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:47.139698  1873 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.137519  1800 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:47.124003  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:47.138801  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:47.141310  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:47.142387  1800 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.145354  1872 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:47.138370  1725 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 12:22:47.146302  1725 pending_rounds.cc:85] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 1
I20260709 12:22:47.146664  1725 pending_rounds.cc:107] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:22:47.147060  1725 raft_consensus.cc:2889] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:47.147719  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:47.105486  1771 leader_election.cc:400] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7.
W20260709 12:22:47.118074  4172 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:22:47.123988  4086 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:47.162501  1801 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:22:47.162909  4035 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:47.119051  4126 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:47.158808  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:47.158809  3874 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.119758  4077 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:47.161015  4072 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:47.164561  4150 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:22:47.116360  4140 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.729s	user 0.003s	sys 0.000s
I20260709 12:22:47.158039  1803 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:47.165095  2412 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.123812  3612 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:22:47.169633  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 64328us
I20260709 12:22:47.172165  1873 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:47.181066  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:47.185556  2411 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:22:47.188040  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:47.189735  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:47.190500  1872 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:22:47.182641  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.192238  1873 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:47.215468  4084 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:47.183773  4077 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:22:47.216305  4084 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:47.216948  4077 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:47.217445  4077 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:47.178009  4158 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:47.184331  4088 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:47.217898  4084 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.180279  3612 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:47.116168  4157 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.219448  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.223835  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 611) took 1130 ms. Trace:
I20260709 12:22:47.226617  1877 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:47.221649  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:47.165202  3874 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:47.221720  4072 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:47.237325  3998 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 12:22:47.248641  1876 raft_consensus.cc:606] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:22:47.263861  1876 raft_consensus.cc:614] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:47.236658  1800 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:47.257989  3874 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:47.239439  2418 rpcz_store.cc:276] 0709 12:22:46.093117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.093298 (+   181us) service_pool.cc:224] Handling call
0709 12:22:46.107959 (+ 14661us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.108417 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.108445 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.130594 (+ 22149us) log.cc:844] Serialized 556 byte log entry
0709 12:22:46.198578 (+ 67984us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:46.198616 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.198689 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:47.115598 (+916909us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:47.223501 (+107903us) raft_consensus.cc:1685] finished
0709 12:22:47.223525 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.223752 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7821568,"thread_start_us":67656,"threads_started":1,"wal-append.queue_time_us":184490}
I20260709 12:22:47.258986  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.273439  1872 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:47.260668  4157 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:47.218688  4077 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.226683  4126 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:47.261593  3450 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:47.222304  4086 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:47.222918  4089 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260709 12:22:47.224804  3612 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.276261  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:47.261467  1878 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:47.328018  2418 raft_consensus.cc:2368] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:47.255632  1873 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:47.318017  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.330672  1876 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:47.258360  4094 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.303081  3450 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.332161  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.301919  4147 consensus_queue.cc:1243] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:22:47.341910  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:47.344043  1873 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.307416  4051 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:47.351764  4094 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:47.355417  3612 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:47.358560  3922 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:47.344944  3450 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:47.354888  4035 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:47.330946  4077 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:47.327217  4126 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:47.374362  4160 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.565s	user 0.002s	sys 0.003s
I20260709 12:22:47.376569  3920 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:47.379860  3920 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:47.381233  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 507) took 1055 ms. Trace:
I20260709 12:22:47.381024  3920 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.381748  2413 rpcz_store.cc:276] 0709 12:22:46.325370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.333532 (+  8162us) service_pool.cc:224] Handling call
0709 12:22:47.380951 (+1047419us) spinlock_profiling.cc:243] Waited 743 ms on lock 0x7b5400573ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.381150 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1708289920}
I20260709 12:22:47.382656  3920 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:47.384107  3796 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:47.384555  3796 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:47.385460  3796 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:47.379890  4126 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:47.390931  4035 consensus_queue.cc:1243] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:22:47.398028  2412 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:47.400736  4000 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:47.401239  4000 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:47.402163  4000 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.386720  3796 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.408629  4000 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.495033  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2318) took 1162 ms. Trace:
I20260709 12:22:47.496088  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 288) took 1367 ms. Trace:
I20260709 12:22:47.496331  1724 rpcz_store.cc:276] 0709 12:22:46.332694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.003458 (+670764us) service_pool.cc:224] Handling call
0709 12:22:47.003772 (+   314us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:47.005572 (+  1800us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:47.005616 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:47.059174 (+ 53558us) log.cc:844] Serialized 52 byte log entry
0709 12:22:47.493696 (+434522us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:47.493859 (+   163us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.494019 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:47.494124 (+   105us) raft_consensus.cc:1685] finished
0709 12:22:47.494149 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.494571 (+   422us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1403,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":17530,"thread_start_us":344476,"threads_started":1,"wal-append.queue_time_us":372275}
W20260709 12:22:47.501353  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.496611  1728 rpcz_store.cc:276] 0709 12:22:46.128180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.982009 (+853829us) service_pool.cc:224] Handling call
0709 12:22:46.982275 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.982465 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:47.103466 (+121001us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.495162 (+391696us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:47.495297 (+   135us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:47.495432 (+   135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.495561 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.495836 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":164487,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"thread_start_us":491609,"threads_started":2,"wal-append.queue_time_us":247267}
I20260709 12:22:47.274474  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:47.321421  1803 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:47.320489  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 179857us
I20260709 12:22:47.535588  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:47.537523  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:47.333196  1846 leader_election.cc:400] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:47.319979  1872 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:22:47.350745  1876 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:47.568848  1876 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 12:22:47.516183  4199 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:22:47.506274  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 284477us
I20260709 12:22:47.529691  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:47.533950  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:47.557760  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.559049  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1514) took 2057 ms. Trace:
I20260709 12:22:47.573557  4000 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:47.547271  1846 leader_election.cc:403] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:22:47.579257  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2167) took 1878 ms. Trace:
W20260709 12:22:47.579622  1799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 298) took 1435 ms (client timeout 1771 ms). Trace:
W20260709 12:22:47.580045  1799 rpcz_store.cc:269] 0709 12:22:46.143883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.279187 (+135304us) service_pool.cc:224] Handling call
0709 12:22:47.579136 (+1299949us) spinlock_profiling.cc:243] Waited 573 ms on lock 0x7b5400133154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.579434 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1318230272}
I20260709 12:22:47.579773  1881 rpcz_store.cc:276] 0709 12:22:45.701061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.331748 (+630687us) service_pool.cc:224] Handling call
0709 12:22:46.332072 (+   324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.457948 (+125876us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.457980 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.494232 (+ 36252us) log.cc:844] Serialized 26 byte log entry
0709 12:22:46.617442 (+123210us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:46.757224 (+139782us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.764956 (+  7732us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.833018 (+ 68062us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54002058d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:46.833070 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:47.373367 (+540297us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:47.548995 (+175628us) raft_consensus.cc:1685] finished
0709 12:22:47.549048 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.579012 (+ 29964us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4828,"prepare.queue_time_us":205782,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"raft.queue_time_us":8658,"raft.run_cpu_time_us":399,"raft.run_wall_time_us":50112,"spinlock_wait_cycles":292402816,"thread_start_us":190051,"threads_started":2,"wal-append.queue_time_us":224461}
W20260709 12:22:47.583173  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 296) took 1439 ms (client timeout 1823 ms). Trace:
W20260709 12:22:47.583918  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 250908us
W20260709 12:22:47.583770  1802 rpcz_store.cc:269] 0709 12:22:46.143543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.283411 (+139868us) service_pool.cc:224] Handling call
0709 12:22:47.582714 (+1299303us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b5400133654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.582956 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":666845440}
I20260709 12:22:47.585319  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:47.586243  1802 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
W20260709 12:22:47.590996  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.002s	sys 0.017s
W20260709 12:22:47.591521  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.002s	sys 0.017s
W20260709 12:22:47.593526  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 543107us
I20260709 12:22:47.595657  1807 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:47.581574  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:47.598603  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1400) took 2210 ms. Trace:
I20260709 12:22:47.599031  1804 rpcz_store.cc:276] 0709 12:22:45.388028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.936927 (+548899us) service_pool.cc:224] Handling call
0709 12:22:47.598111 (+1661184us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54001ea5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.598403 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489364736}
I20260709 12:22:47.600454  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:47.601848  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:47.603816  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2144) took 2326 ms (client timeout 2130 ms). Trace:
W20260709 12:22:47.604347  1875 rpcz_store.cc:269] 0709 12:22:45.277038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.757854 (+480816us) service_pool.cc:224] Handling call
0709 12:22:47.603274 (+1845420us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.603552 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":795948288}
I20260709 12:22:47.607770  1727 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: [5.4-7.5]
I20260709 12:22:47.610081  1874 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:22:47.610549  1874 pending_rounds.cc:85] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 2
I20260709 12:22:47.610939  1874 pending_rounds.cc:107] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:22:47.611243  1874 raft_consensus.cc:2889] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:47.574678  1805 rpcz_store.cc:276] 0709 12:22:45.501436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.871874 (+1370438us) service_pool.cc:224] Handling call
0709 12:22:46.872446 (+   572us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:46.873109 (+   663us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.873178 (+    69us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:46.873989 (+   811us) log.cc:844] Serialized 52 byte log entry
0709 12:22:47.301586 (+427597us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:47.301634 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.301716 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:47.558296 (+256580us) raft_consensus.cc:1685] finished
0709 12:22:47.558350 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.558753 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":5183,"thread_start_us":412189,"threads_started":1,"wal-append.queue_time_us":483836}
I20260709 12:22:47.622980  3922 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:47.623762  3922 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:47.623859  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:47.619622  3899 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260709 12:22:47.628538  3899 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:47.629033  3899 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:47.623432  4037 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:47.594637  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1513) took 2104 ms. Trace:
I20260709 12:22:47.602643  1804 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:47.633296  1801 rpcz_store.cc:276] 0709 12:22:45.490228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.861900 (+1371672us) service_pool.cc:224] Handling call
0709 12:22:46.862349 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:47.162978 (+300629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.593963 (+430985us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b5400133b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:47.594439 (+   476us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":689597824}
I20260709 12:22:47.618697  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:47.632927  4037 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:47.651506  1696 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.025s	sys 0.048s
I20260709 12:22:47.651301  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:47.653143  2411 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:47.654851  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:47.665102  1696 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.026s	sys 0.048s
I20260709 12:22:47.668999  1876 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:47.630369  3899 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:47.671154  1803 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:47.672097  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:47.678923  1805 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
W20260709 12:22:47.680757  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.682930  4083 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:47.683802  4083 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:47.685385  4083 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:47.686455  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.687233  4038 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:47.688220  4038 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.677297  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:47.693053  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:47.695737  2418 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:47.699258  1799 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:22:47.700047  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 162938us
I20260709 12:22:47.675683  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:47.707408  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:47.707939  1806 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:22:47.712661  3450 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.714890  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1458) took 1761 ms. Trace:
I20260709 12:22:47.715560  1723 rpcz_store.cc:276] 0709 12:22:45.815564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.569819 (+754255us) service_pool.cc:224] Handling call
0709 12:22:47.576064 (+1006245us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b540030a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.576423 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":519084288}
I20260709 12:22:47.716290  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:47.718084  3899 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:47.728698  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.725327  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 721569us
W20260709 12:22:47.737126  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.743072  4191 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.746682  4191 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:47.731689  4000 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:47.749532  4191 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:47.751623  4149 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:47.753314  4149 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:47.753854  4149 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:22:47.755520  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.767369  4149 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.769305  4149 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.773214  1802 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:47.774555  1801 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:47.775326  1806 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:47.779057  1800 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:22:47.770522  4189 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.441s	user 0.005s	sys 0.006s
W20260709 12:22:47.788246  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.799297  2411 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:47.800149  3922 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:47.801831  2411 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:47.774927  4037 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.804382  4037 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:47.804874  3612 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.805694  3612 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:47.806193  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:47.806979  4036 consensus_queue.cc:1175] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "075704438992440697732181e49ccd09" responder_term: 3 status { last_received { term: 2 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:47.808516  1723 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260709 12:22:47.798224  2418 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:47.811607  2419 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:47.790426  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2497461930001919s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20710000000000003s for a response
I20260709 12:22:47.787763  3450 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:47.814373  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 195970us
I20260709 12:22:47.837728  4179 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:47.813063  3922 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:47.841434  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2074461930001235s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
I20260709 12:22:47.844010  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:47.847986  1729 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:47.849828  4082 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:47.860014  3612 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:47.861423  1726 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:47.867940  1730 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:47.868566  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2323) took 1010 ms. Trace:
I20260709 12:22:47.868690  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2173) took 1928 ms. Trace:
I20260709 12:22:47.869155  1727 rpcz_store.cc:276] 0709 12:22:46.858158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.109863 (+251705us) service_pool.cc:224] Handling call
0709 12:22:47.607021 (+497158us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:47.608316 (+  1295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.868288 (+259972us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b540030a550. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.868477 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1072414080}
I20260709 12:22:47.869261  1874 rpcz_store.cc:276] 0709 12:22:45.939859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.511198 (+571339us) service_pool.cc:224] Handling call
0709 12:22:47.609149 (+1097951us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.611910 (+  2761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.868501 (+256591us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540021c0d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.868631 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2469265664}
I20260709 12:22:47.872700  4151 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:47.874642  4151 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:47.876423  4063 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:47.877436  4063 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:47.878186  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 647) took 1419 ms. Trace:
I20260709 12:22:47.878392  4039 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:47.879076  4039 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:47.880358  4039 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.881863  4039 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:47.882022  4168 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:47.882539  4168 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:47.882673  1801 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: []
I20260709 12:22:47.892517  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:47.886488  4000 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:22:47.893289  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1939461930001016s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21330000000000002s for a response
I20260709 12:22:47.839542  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2321) took 1381 ms. Trace:
I20260709 12:22:47.882081  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:47.897647  1804 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
W20260709 12:22:47.904825  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.903911  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1495) took 3530 ms. Trace:
I20260709 12:22:47.885893  4168 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.906695  1807 rpcz_store.cc:276] 0709 12:22:44.369504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.581535 (+2212031us) service_pool.cc:224] Handling call
0709 12:22:47.594970 (+1013435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.618951 (+ 23981us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.893157 (+274206us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400133650. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:47.899589 (+  6432us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2329442176}
I20260709 12:22:47.902360  1873 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:47.897825  4179 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:47.906137  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 91097.1us
I20260709 12:22:47.897199  4000 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:47.918876  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:47.919911  1832 heartbeater.cc:646] Failed to heartbeat to 127.0.140.126:43371 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.140.126:43371 timed out after 15.000s (SENT)
I20260709 12:22:47.920083  4190 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:47.921981  4200 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:22:47.926061  3796 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.002s
I20260709 12:22:47.912783  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:47.911547  1878 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:47.933308  4168 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:47.895874  1732 rpcz_store.cc:276] 0709 12:22:46.457877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.019206 (+561329us) service_pool.cc:224] Handling call
0709 12:22:47.019545 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.019783 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:47.499952 (+480169us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.792278 (+292326us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.821334 (+ 29056us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:47.833851 (+ 12517us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:47.838916 (+  5065us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.839054 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.839287 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":254178,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":126,"raft.queue_time_us":6101,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":14842,"thread_start_us":646232,"threads_started":2,"wal-append.queue_time_us":257909}
I20260709 12:22:47.940106  4193 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:47.941309  4151 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.942205  4151 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.944881  4151 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:47.947687  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 99302.3us
W20260709 12:22:47.948765  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.951601  4077 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.952497  4077 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:47.955439  1844 leader_election.cc:400] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8.
I20260709 12:22:47.955843  1844 leader_election.cc:403] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:47.973315  1726 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:22:47.973771  1726 pending_rounds.cc:85] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 6
I20260709 12:22:47.981581  1726 pending_rounds.cc:107] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:22:47.982091  1726 raft_consensus.cc:2889] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:47.982765  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 146291us
I20260709 12:22:47.878623  2414 rpcz_store.cc:276] 0709 12:22:46.458373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.482187 (+ 23814us) service_pool.cc:224] Handling call
0709 12:22:46.531857 (+ 49670us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.048422 (+516565us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.877605 (+829183us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54004560d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:47.877943 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055975808}
W20260709 12:22:47.986887  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.978427  1728 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:47.990902  1729 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:47.993309  1771 leader_election.cc:400] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4.
I20260709 12:22:47.993757  1771 leader_election.cc:403] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:22:47.991499  1729 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:47.999274  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.012306  4050 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.013362  4050 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.019596  3796 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.972834  3450 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.034s
I20260709 12:22:47.994238  4190 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:48.006642  4191 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:22:48.035949  4063 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:48.031692  4072 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4.
W20260709 12:22:48.041272  4202 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:22:48.052035  4063 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.042515  4036 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8.
I20260709 12:22:48.062704  1770 leader_election.cc:400] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4.
I20260709 12:22:48.066485  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:48.068902  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:48.022629  4179 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.064700  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:48.076478  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 171981us
I20260709 12:22:48.046877  4166 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:48.080703  1629 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 250) took 15680 ms (client timeout 15000 ms). Trace:
I20260709 12:22:48.070600  4051 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:48.083778  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:48.044867  3796 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:48.091982  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 155529us
I20260709 12:22:48.074973  4083 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:48.054872  4151 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.033s
I20260709 12:22:48.050997  4000 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.102348  4009 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.121104  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 99974.8us
I20260709 12:22:48.099438  4063 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:48.043702  4190 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.127194  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.127781  4193 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:48.131256  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101548us
I20260709 12:22:48.111325  1729 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.122469  3899 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:22:48.148941  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 114118us
I20260709 12:22:48.096138  4072 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:22:48.128110  4063 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:48.128209  4036 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:48.102670  4126 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 12:22:48.164556  4208 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.111627  4166 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:48.170274  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 126232us
W20260709 12:22:48.194245  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47001.3us
I20260709 12:22:48.155598  4193 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:48.154919  4077 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:48.220741  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:48.193295  4051 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:48.157150  3612 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 12:22:48.225708  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:48.235437  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:48.153415  4009 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.201563  4050 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:48.246865  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.247514  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.251304  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 154064us
I20260709 12:22:48.217073  3922 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.200063  4063 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:48.254364  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 133421us
I20260709 12:22:48.240525  4035 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.205474  4072 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
W20260709 12:22:48.255231  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.240816  4151 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:22:48.269905  4193 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.266469  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:48.274315  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52159.1us
I20260709 12:22:48.222111  4036 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.250629  3899 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:48.257797  4000 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:48.283881  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.244517  4077 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:22:48.225360  4208 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.270483  4151 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:22:48.290251  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:48.081815  1629 rpcz_store.cc:269] 0709 12:22:32.399866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.462588 (+ 62722us) service_pool.cc:224] Handling call
0709 12:22:33.856086 (+1393498us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b54001564a8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1fca331 00007f16b1fc53fc 00007f16b1b96e84 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:22:48.080389 (+14224303us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:43.355051 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:43.360272 (+  5221us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:43.374510 (+ 14238us) write_op.cc:435] Acquired schema lock
0709 12:22:43.383327 (+  8817us) tablet.cc:662] Decoding operations
0709 12:22:43.681991 (+298664us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:43.699891 (+ 17900us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:43.711801 (+ 11910us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:22:43.780166 (+ 68365us) tablet.cc:701] Row locks acquired
0709 12:22:43.780189 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:22:43.780389 (+   200us) write_op.cc:270] Start()
0709 12:22:43.780549 (+   160us) write_op.cc:276] Timestamp: P: 1783599763780321 usec, L: 0
0709 12:22:43.780581 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 12:22:43.830836 (+ 50255us) log.cc:844] Serialized 1308 byte log entry
0709 12:22:45.764861 (+1934025us) spinlock_profiling.cc:243] Waited 4.52 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:46.151743 (+386882us) op_driver.cc:464] REPLICATION: finished
0709 12:22:46.244601 (+ 92858us) write_op.cc:301] APPLY: starting
0709 12:22:46.244698 (+    97us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:46.244900 (+   202us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:46.244924 (+    24us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:46.245499 (+   575us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:46.245757 (+   258us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0709 12:22:46.245802 (+    45us) write_op.cc:312] APPLY: finished
0709 12:22:46.246028 (+   226us) log.cc:844] Serialized 35 byte log entry
0709 12:22:46.246211 (+   183us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:46.246319 (+   108us) write_op.cc:454] Released schema lock
0709 12:22:46.246363 (+    44us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:46.246864 (+   501us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:46.246892 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":181958,"reported_tablets":11,"spinlock_wait_cycles":8039296,"child_traces":[["op",{"apply.queue_time_us":92530,"apply.run_cpu_time_us":2185,"apply.run_wall_time_us":2185,"num_ops":3,"prepare.queue_time_us":1395035,"prepare.run_cpu_time_us":39045,"prepare.run_wall_time_us":1766269,"raft.queue_time_us":295873,"raft.run_cpu_time_us":30188,"raft.run_wall_time_us":154814,"replication_time_us":2333501,"spinlock_wait_cycles":10391936,"thread_start_us":2485557,"threads_started":4,"wal-append.queue_time_us":944615,"wal-append.run_cpu_time_us":105485,"wal-append.run_wall_time_us":2448580}]]}
W20260709 12:22:48.295403  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.301240  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:48.283756  3922 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.333328  1732 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:48.333920  1731 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:48.325964  3796 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:48.287847  3612 consensus_queue.cc:1243] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:48.284847  4179 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:48.341418  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 115804us
I20260709 12:22:48.345350  4097 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.122s
W20260709 12:22:48.349090  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.350116  4214 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:48.350630  4214 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:48.351703  4214 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.353214  4214 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:48.348477  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:48.358834  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.359809  1723 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:48.374462  4221 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:48.375092  4221 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:48.376304  4221 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.379009  4220 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:48.379601  4220 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.380034  4220 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:48.382938  4051 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:48.379030  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 199645us
I20260709 12:22:48.286610  4035 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.404901  1723 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:48.407169  4218 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:48.419616  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 71293.1us
I20260709 12:22:48.385135  4215 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:48.410653  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.424468  4213 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.401921  4179 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:48.428043  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 233508us
I20260709 12:22:48.412657  4190 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:48.438289  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 278) took 1423 ms. Trace:
I20260709 12:22:48.438889  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:48.440622  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.439273  1878 rpcz_store.cc:276] 0709 12:22:47.008418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.008764 (+   346us) service_pool.cc:224] Handling call
0709 12:22:48.420653 (+1411889us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b5400206cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:48.431472 (+ 10819us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":524704640}
I20260709 12:22:48.408692  3899 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.438587  4166 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:48.444584  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 117992us
W20260709 12:22:48.457824  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 130436us
I20260709 12:22:48.443792  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:48.407938  4218 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.441793  4190 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:48.441707  4038 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:48.473039  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 229100us
I20260709 12:22:48.487543  4063 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:48.519027  4036 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:48.489530  4216 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.173s
I20260709 12:22:48.519730  4226 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
W20260709 12:22:48.525823  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87059.6us
I20260709 12:22:48.473510  4179 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:48.476778  4190 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:48.521250  4083 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:48.451517  4213 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:48.534181  4207 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.252s	user 0.001s	sys 0.001s
I20260709 12:22:48.591121  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:48.547005  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 130387us
I20260709 12:22:48.546216  4035 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:48.530746  4009 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:48.592423  3922 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:48.605649  4208 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:48.608424  4179 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:48.628449  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 93656.5us
I20260709 12:22:48.635288  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:48.636674  4235 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:48.636900  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:48.637283  4235 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:48.638000  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:48.638337  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:48.638993  4223 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.639199  4051 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:48.639611  4051 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:48.639747  4223 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:48.640671  4051 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.641995  4051 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:48.642244  4223 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:48.642755  4227 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20260709 12:22:48.643235  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:48.643925  1807 raft_consensus.cc:2376] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7. Re-sending same reply.
I20260709 12:22:48.644536  3922 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:22:48.643229  4222 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:48.646996  4231 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.647576  4231 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.650171  4231 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:48.653590  4126 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.627432  4035 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:48.655995  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.691510  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.667187  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.694948  4038 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.695188  1727 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:48.695645  4038 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.696578  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.697907  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.697960  4038 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:48.667303  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.667824  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.676385  3612 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:48.678341  4037 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.873s	user 0.013s	sys 0.000s
W20260709 12:22:48.680575  4149 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.911s	user 0.010s	sys 0.005s
I20260709 12:22:48.681604  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.682638  4126 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.683682  4218 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:48.685683  4082 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.835s	user 0.013s	sys 0.000s
I20260709 12:22:48.687916  4213 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:48.689734  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.667009  4050 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.666402  1729 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:48.725481  1724 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:48.725351  4050 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.727401  3612 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.728483  4050 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:48.698562  1727 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:48.638515  4235 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.730660  4228 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.629963  4083 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.102s
I20260709 12:22:48.731402  4228 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:48.690398  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:22:48.734603  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45282.5us
W20260709 12:22:48.702638  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 218185us
I20260709 12:22:48.694542  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:48.698586  4038 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.702083  4208 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:48.717348  4179 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.723481   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:22:48.735334  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 512) took 1085 ms. Trace:
W20260709 12:22:48.748202  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53753.2us
I20260709 12:22:48.745545  1731 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:48.730690  4235 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:48.777336  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.778384  1725 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:48.743474  4050 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:48.797188  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:48.786839  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:48.814429  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58337.2us
W20260709 12:22:48.813913  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62232.6us
I20260709 12:22:48.749379  4213 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.818182  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:48.797391  1729 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:48.824803  1798 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:48.817623  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:48.814677  4050 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:48.832748  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.850001  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:48.746028  4038 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:48.849335  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.787758  4063 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:48.908726  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:48.754561  4220 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.947332  2416 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:48.948865  4221 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:48.949363  4220 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:48.943281  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:48.948900   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:48.951718  4037 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:48.952185  4037 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.952553  4037 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:48.898140  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:48.955523  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:48.897655  1725 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:48.907807  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:48.899068  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 81547.9us
I20260709 12:22:48.818833  4083 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.745633  2412 rpcz_store.cc:276] 0709 12:22:47.649528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.653476 (+  3948us) service_pool.cc:224] Handling call
0709 12:22:48.255693 (+602217us) spinlock_profiling.cc:243] Waited 2.2 ms on lock 0x7b54006000a8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0d7cebc 00007f16b0db208c 00007f16b0d9c93b 00007f16b0d9c3d2 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c
0709 12:22:48.274855 (+ 19162us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.327454 (+ 52599us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.329823 (+  2369us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:48.412318 (+ 82495us) log.cc:844] Serialized 26 byte log entry
0709 12:22:48.590663 (+178345us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:48.590709 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.590804 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:48.734839 (+144035us) raft_consensus.cc:1685] finished
0709 12:22:48.734869 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.735158 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5064576,"thread_start_us":148595,"threads_started":1,"wal-append.queue_time_us":307651}
I20260709 12:22:48.955760  4219 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:22:48.829393  4213 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:48.963747  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:48.964536  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:48.965180  3922 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.965101  4220 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.965879  1872 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:48.967432  1876 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:48.967684  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.967916  4229 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.200s	user 0.002s	sys 0.002s
I20260709 12:22:48.968300  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:48.969089  1874 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:48.951736   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:48.970701  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:48.957536  1725 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:48.972383  1807 raft_consensus.cc:2368] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:48.851284  4050 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:49.006848  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.972152  2414 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:48.941409  4215 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:48.830077  3612 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:48.959645  4039 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.061s	user 0.008s	sys 0.008s
I20260709 12:22:48.909494  4228 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:48.997553  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:48.914027  4166 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:48.823918  4208 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.011709  1880 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:49.015404  2410 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:49.016680  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:48.825937  4035 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.016242  4009 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.019377  1873 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:49.000993  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:49.009238  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51320.3us
I20260709 12:22:49.023447  1805 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:48.987294  4219 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:48.996749  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.011020  4149 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.066869  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:49.050808  4251 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.261s	user 0.000s	sys 0.005s
I20260709 12:22:49.052640  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:48.915161  4241 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.073220  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.058971  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:49.079058  1877 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:49.059715  4083 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.085350  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.063369  4247 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:49.093082  4247 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.093284  1874 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: [3.3-7.5]
I20260709 12:22:49.094049  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:49.095101  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.095431  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:49.095872  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 201355us
I20260709 12:22:49.096159  2417 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:49.096912  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.097126  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:49.099308  3612 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:49.099848  3612 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:49.100286  3612 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:49.062906  4220 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.103037  4211 raft_consensus.cc:2764] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:22:49.103852  4126 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:49.065820  4227 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.104802  4220 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:49.105232  4227 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.093505  4219 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.078346  4035 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.015939  4237 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.001619  4190 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:49.091504  1875 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:49.091042  4149 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.092106  4009 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.087584  2416 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260709 12:22:49.109881  1873 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:49.134848  2416 pending_rounds.cc:85] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 4
I20260709 12:22:49.095160  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:49.135317  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:49.093577  4247 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:49.107877  4219 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:22:49.116617  4245 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.244s	user 0.001s	sys 0.001s
I20260709 12:22:49.085651  3899 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:49.136129  1881 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:22:49.159157  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.170s (SENT) [suppressed 37 similar messages]
I20260709 12:22:49.135169  2416 pending_rounds.cc:107] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.5
I20260709 12:22:49.161430  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:49.188347  2416 raft_consensus.cc:2917] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Aborting config change with OpId 4.5: Aborted: Op aborted by new leader
W20260709 12:22:49.130584  4000 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.827s	user 0.012s	sys 0.002s
I20260709 12:22:49.163769  4222 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:49.163625  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.188907  2416 raft_consensus.cc:2889] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:49.164538  4248 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.165122  4227 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:49.190387  1807 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:49.165556  4230 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.166718  4009 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:49.166982  4220 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:49.167310  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:49.167830  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.168565  4239 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.161454  4097 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:49.069294  3922 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.194379  4222 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.218308  1726 raft_consensus.cc:2410] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:49.118033  4035 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:49.181062  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:49.221361  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.169309  4256 raft_consensus.cc:2955] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:49.186371  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.171059  1725 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.131115  4226 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:49.193262  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 58052.9us
I20260709 12:22:49.106889  4250 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
I20260709 12:22:49.106665  4050 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.133853  4063 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:49.273267  4063 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.274269  4050 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:49.161532  4126 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:49.275147  4126 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:49.168751  4243 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:22:49.276310  4258 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.212235  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:49.276952  4258 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.277558  4126 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:49.215370  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 116574us
I20260709 12:22:49.279656  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:49.191316  3899 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.280474  3899 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:49.162304  1874 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:22:49.198141  4215 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.282457  4239 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.284056  4083 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:49.284941  4239 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:49.285748  1807 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:49.288920  2419 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:49.223132  4248 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.263630  4038 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:49.264358  4236 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.077s
W20260709 12:22:49.265748  4246 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:22:49.204660  4179 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:49.161734  4237 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.268328  4213 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:49.222962  4097 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.207557  2414 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:22:49.213490  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 87087us
I20260709 12:22:49.221467  2417 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:49.302665  1877 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
W20260709 12:22:49.378818  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 99241.4us
W20260709 12:22:49.378219  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 90111.3us
W20260709 12:22:49.287647  4214 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.934s	user 0.012s	sys 0.005s
W20260709 12:22:49.293695  4051 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.647s	user 0.008s	sys 0.008s
I20260709 12:22:49.220865  4222 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:49.318329  1724 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
W20260709 12:22:49.262992  4242 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:22:49.271183  4226 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.379804  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:49.195030  4037 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.224426  4009 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:49.134261  4241 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.378844  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:49.304471  2413 raft_consensus.cc:2410] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 12:22:49.383162  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 186788us
W20260709 12:22:49.383589  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.383684  1803 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:49.223539  3796 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.169s
I20260709 12:22:49.262662  4220 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.216356  4230 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.374647  1725 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:49.385037  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.275564  3612 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.377905  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:49.398998  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:49.417840  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.482870  1874 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: [2.5-4.6]
I20260709 12:22:49.385860  1724 pending_rounds.cc:85] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 4
I20260709 12:22:49.591490  1724 pending_rounds.cc:107] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 4.5
I20260709 12:22:49.414956  1802 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:49.496186  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:49.487995  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:49.482992  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 346002us
I20260709 12:22:49.389632  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:49.388911  1806 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260709 12:22:49.423439  1877 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 12:22:49.523212  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.590238  2416 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:22:49.596232  1727 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:49.495211  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:49.597970  2418 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:49.307363  4035 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:49.588845  1726 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:49.349669  4193 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:49.491500  4259 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:22:49.594075  1806 pending_rounds.cc:85] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 5
I20260709 12:22:49.614892  1806 pending_rounds.cc:107] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260709 12:22:49.615509  1806 raft_consensus.cc:2889] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:49.605863  1730 raft_consensus.cc:2410] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:49.600924  1724 raft_consensus.cc:2917] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Aborting config change with OpId 4.5: Aborted: Op aborted by new leader
I20260709 12:22:49.593432  1804 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
W20260709 12:22:49.532912  4235 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.731s	user 0.012s	sys 0.000s
I20260709 12:22:49.602516  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:49.429940  4150 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.613754  1872 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:22:49.422207  4232 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.349946  4239 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:49.382580  4223 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:49.623157  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:49.396255  4231 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:49.387579  4126 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.422896  4257 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.433320  2414 raft_consensus.cc:2955] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:49.498127  4237 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:49.542029  4248 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:49.631687  1879 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:49.489790  3922 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:49.472653  4258 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:49.521229  4190 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:49.630141  1724 raft_consensus.cc:2889] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:49.495447  4149 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:49.429003  4268 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.635741  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:49.553025  4216 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.635737  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:49.625699  1878 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:49.691093  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:49.644358  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:49.319057  4082 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.308s
I20260709 12:22:49.599057  4267 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.423619  4221 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.214s
I20260709 12:22:49.653034  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:49.609820  4097 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:49.609357  4219 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:49.538938  1626 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:49.651955  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:49.422626  4083 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.213s
I20260709 12:22:49.674142  1730 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:49.635351  4215 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20260709 12:22:49.679771  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:49.606276  4247 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:49.679467  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:49.689510  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53889.9us
I20260709 12:22:49.657246  4038 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.609304  4009 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.695024  1873 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:49.694856  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:49.705263  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:49.677098  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:49.723865  1727 raft_consensus.cc:2410] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:49.706561  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 315) took 1070 ms. Trace:
I20260709 12:22:49.696712  3922 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:49.698185  4216 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:49.698927  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.717648  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:49.645262  3899 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:49.716288  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 110666us
I20260709 12:22:49.766180  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:49.680358  4126 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:49.724874  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 96822.9us
I20260709 12:22:49.630017  4193 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.725047  1874 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:49.660725  4231 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.705317  4239 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:49.681685  4228 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:49.638945  4232 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:49.764099  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 161693us
W20260709 12:22:49.765262  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47736.9us
I20260709 12:22:49.724429  1800 rpcz_store.cc:276] 0709 12:22:48.636523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.686907 (+ 50384us) service_pool.cc:224] Handling call
0709 12:22:48.687337 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.687789 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.687816 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:48.688325 (+   509us) log.cc:844] Serialized 26 byte log entry
0709 12:22:48.880133 (+191808us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:48.880173 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.880240 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:49.706312 (+826072us) raft_consensus.cc:1685] finished
0709 12:22:49.706334 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.706494 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11656,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":10572,"thread_start_us":191296,"threads_started":1,"wal-append.queue_time_us":329101}
I20260709 12:22:49.735543  1805 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:49.644394  4035 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.704964  4268 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.660250  4257 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.710700  4215 consensus_queue.cc:1243] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:22:49.728525  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:49.697279  4267 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.679620  4150 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:49.725329  4190 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.735541  4097 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:49.815987  1799 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:49.786569  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:49.778515  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2334) took 1035 ms. Trace:
I20260709 12:22:49.867632  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:49.728637  3922 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.733032  4235 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.804776  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:22:49.927930  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 123268us
I20260709 12:22:49.903383  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:49.815364  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:49.789492  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:49.903280  2415 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
W20260709 12:22:49.911496  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 43965us
I20260709 12:22:49.878860  1626 catalog_manager.cc:5697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 3, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:22:49.909112  1729 rpcz_store.cc:276] 0709 12:22:48.743268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.824959 (+ 81691us) service_pool.cc:224] Handling call
0709 12:22:48.895671 (+ 70712us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.003206 (+107535us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.003248 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.040439 (+ 37191us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.088724 (+ 48285us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:22:49.088780 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.088878 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:49.778275 (+689397us) raft_consensus.cc:1685] finished
0709 12:22:49.778299 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.778450 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13679,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":26613,"thread_start_us":33966,"threads_started":1,"wal-append.queue_time_us":150465}
I20260709 12:22:49.721643  4000 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.732985  4248 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:49.915685  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:49.916702  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 144649us
W20260709 12:22:49.914228  4072 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.399s	user 0.008s	sys 0.021s
I20260709 12:22:49.945262  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.878129  3796 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.878540  4082 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.724076  4219 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.788049  4265 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:49.938917  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:49.926312  3922 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:49.787227  4266 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.807107  3612 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:49.878011  4239 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.941845  1873 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:49.809844  3899 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:49.941900  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61496.2us
I20260709 12:22:49.786517  4224 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:49.790058  4220 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:49.817891  4272 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.946563  2419 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:22:49.793484  4263 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:49.916028  4222 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.502s	user 0.004s	sys 0.008s
I20260709 12:22:49.936568  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:49.810532  4208 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:49.935701  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:49.796435  4241 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:49.949369  1729 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:49.732761  3450 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.201s
I20260709 12:22:49.952307  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:49.952907  1730 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:22:49.809784  4230 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:49.961396  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:49.923410  4213 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.961684  1876 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 5. (term mismatch)
I20260709 12:22:49.962639  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:50.115902  1727 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:49.916886  4097 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.925256  4036 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:49.969882  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:49.945864  4072 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.430s	user 0.008s	sys 0.021s
I20260709 12:22:49.733920  4038 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.976640  4247 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:49.973631  1626 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 6, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 6 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:49.979138  4228 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:49.981972  4239 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:49.957901  3796 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:49.953802  4083 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:49.795257  4234 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.326s
I20260709 12:22:49.907374  4237 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 12:22:49.962236  4250 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.960914  4265 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.916457  4190 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:49.987713  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 117259us
I20260709 12:22:49.991139  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:49.973459  2413 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:49.998054  4272 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:50.017455  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71986.8us
I20260709 12:22:49.953218  4000 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:50.028435  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:22:50.050308  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.958158  4082 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:49.977505  4235 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:50.058593  1874 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:49.969579  4037 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:50.085510  1800 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:22:50.104461  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:50.058018  4256 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.623s	user 0.013s	sys 0.000s
I20260709 12:22:50.105158  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:50.115386  1876 pending_rounds.cc:85] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 4
I20260709 12:22:50.064244  4220 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:49.998456  4223 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:49.980602  4266 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:50.058452  4222 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:50.065192  3899 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260709 12:22:50.064704  4263 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:50.702075  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:50.506242  1807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.005s	sys 0.001s
W20260709 12:22:50.703730  1807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.005s	sys 0.004s
I20260709 12:22:50.703962  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 291) took 1205 ms. Trace:
I20260709 12:22:50.704387  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1441) took 1098 ms. Trace:
I20260709 12:22:50.168843  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:50.568374  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.003s	sys 0.012s
W20260709 12:22:50.145685  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:50.707063  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.006s	sys 0.012s
W20260709 12:22:50.708158  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 769297us
I20260709 12:22:50.704746  1874 rpcz_store.cc:276] 0709 12:22:49.498086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.903050 (+404964us) service_pool.cc:224] Handling call
0709 12:22:50.703894 (+800844us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.712558  1730 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:50.714354  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:50.715056  1876 pending_rounds.cc:107] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 4.5
W20260709 12:22:50.458029  1725 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.868s	user 0.013s	sys 0.000s
I20260709 12:22:50.716190  1876 raft_consensus.cc:2889] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:50.199640  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 432845us
I20260709 12:22:50.705148  1800 rpcz_store.cc:276] 0709 12:22:49.605910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.951978 (+346068us) service_pool.cc:224] Handling call
0709 12:22:50.704295 (+752317us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.718572  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 293) took 1219 ms. Trace:
I20260709 12:22:50.719293  1873 rpcz_store.cc:276] 0709 12:22:49.498738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.729298 (+230560us) service_pool.cc:224] Handling call
0709 12:22:50.718304 (+989006us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:50.718443 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":402065792}
I20260709 12:22:50.381062  1626 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } }
I20260709 12:22:50.723191  1725 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:50.729045  1724 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:50.731455  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:50.732760  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:50.403285  1728 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s	user 0.000s	sys 0.004s
W20260709 12:22:50.733726  1728 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.003s	sys 0.006s
I20260709 12:22:50.722678  1626 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:22:50.146438  4278 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.211s	user 0.002s	sys 0.000s
I20260709 12:22:50.121639  4097 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:50.393553  4293 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.193s	user 0.001s	sys 0.000s
W20260709 12:22:50.408766  4287 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260709 12:22:50.393432  4268 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:50.742117  1873 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:50.738457  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1425) took 1894 ms. Trace:
W20260709 12:22:50.411532  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.407212  4257 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:50.743608  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 752493us
W20260709 12:22:50.199038  4284 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
W20260709 12:22:50.250398  4282 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
W20260709 12:22:50.752631  4291 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.320s	user 0.001s	sys 0.000s
I20260709 12:22:50.137799  4009 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:50.750355  1732 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:22:50.352255  4275 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:22:50.741106  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2332) took 2023 ms. Trace:
I20260709 12:22:50.161871  4050 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:50.407449  4231 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:50.122751  4213 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:50.403431  4126 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:50.743072  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:50.913877  2414 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:22:50.758214  1626 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 4, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:50.775911  1874 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:50.897364  1725 rpcz_store.cc:276] 0709 12:22:48.700230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.799836 (+ 99606us) service_pool.cc:224] Handling call
0709 12:22:48.826771 (+ 26935us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.723682 (+1896911us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.723901 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.737825  4218 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:50.861642  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:50.759806  1879 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
W20260709 12:22:50.925637  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 64121.2us
I20260709 12:22:50.926744  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:50.927819  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:50.928656  1725 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:50.786383  1806 rpcz_store.cc:276] 0709 12:22:48.843917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.844059 (+   142us) service_pool.cc:224] Handling call
0709 12:22:48.905496 (+ 61437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.616260 (+710764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.729403 (+1113143us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b54001d4f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:50.738399 (+  8996us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1110178688}
I20260709 12:22:50.249001  4226 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:50.729743  4286 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.749634  4222 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:50.193889  4292 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.721596  4221 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.863323  4083 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:50.403139  4219 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:50.753034  4247 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.715538  4220 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.396803  4232 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:50.720636  4265 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:50.882515  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 284) took 1386 ms. Trace:
I20260709 12:22:50.891647  4038 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:50.882894  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:50.896965  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 286) took 1397 ms. Trace:
I20260709 12:22:51.078593  1873 rpcz_store.cc:276] 0709 12:22:49.496614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.720144 (+1223530us) service_pool.cc:224] Handling call
0709 12:22:50.720441 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.893062 (+172621us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.893870 (+   808us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.898213  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:50.729561  4294 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:50.898932  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 661) took 1400 ms. Trace:
I20260709 12:22:51.085122  1626 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 5, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:50.897415  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2202) took 1860 ms. Trace:
I20260709 12:22:50.922351  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 285) took 1426 ms. Trace:
I20260709 12:22:51.086624  1872 rpcz_store.cc:276] 0709 12:22:49.041176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.068228 (+ 27052us) service_pool.cc:224] Handling call
0709 12:22:49.077760 (+  9532us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.630012 (+552252us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.897198 (+1267186us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b5400073854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:50.897357 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1231841664}
I20260709 12:22:51.086764  1874 rpcz_store.cc:276] 0709 12:22:49.496331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.726378 (+1230047us) service_pool.cc:224] Handling call
0709 12:22:50.775632 (+ 49254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.922050 (+146418us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.922283 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.927821  4218 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.935875  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:51.109447  2419 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:50.759758  4290 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.707669  4035 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:50.945243  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:50.735023  4250 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:50.949512  1724 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:50.864264  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 633) took 1134 ms. Trace:
I20260709 12:22:50.901355  4306 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.945904  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:50.771754  4212 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.202s
I20260709 12:22:50.880262  4295 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:51.076344  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.199721  4283 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.966s
I20260709 12:22:50.751574  4266 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:51.077266  1876 rpcz_store.cc:276] 0709 12:22:49.496041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.945341 (+449300us) service_pool.cc:224] Handling call
0709 12:22:49.961234 (+ 15893us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.882150 (+920916us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.882434 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:50.860231  4037 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
W20260709 12:22:50.967685  4072 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.032s	sys 0.067s
W20260709 12:22:51.195519  4072 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.995s	user 0.032s	sys 0.067s
W20260709 12:22:51.113781  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 219189us
I20260709 12:22:51.117913  2413 rpcz_store.cc:276] 0709 12:22:49.730054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.734680 (+  4626us) service_pool.cc:224] Handling call
0709 12:22:49.907166 (+172486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:50.721347 (+814181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.770276 (+ 48929us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.112133  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1508) took 1123 ms. Trace:
I20260709 12:22:50.951915  4292 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.115121  1626 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 4, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:50.958503  4221 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.115777  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:51.088398  4222 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.087842  4097 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.163494  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1503) took 1214 ms. Trace:
I20260709 12:22:51.165617  1872 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:50.892508  4036 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.195s
I20260709 12:22:51.111248  4218 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:51.167563  1874 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: []
W20260709 12:22:51.167762  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 436609us
I20260709 12:22:50.862672  4223 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.316123  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.316859  1730 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:51.317368  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 309) took 1327 ms. Trace:
I20260709 12:22:51.174546  1725 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:51.318238  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2350) took 1375 ms. Trace:
I20260709 12:22:51.318590  1725 rpcz_store.cc:276] 0709 12:22:49.942939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.927572 (+984633us) service_pool.cc:224] Handling call
0709 12:22:51.318169 (+390597us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.010713  4226 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.156s
I20260709 12:22:51.333227  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 663) took 1402 ms. Trace:
I20260709 12:22:51.259424  4009 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.074s
I20260709 12:22:51.333634  2417 rpcz_store.cc:276] 0709 12:22:49.931300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.932197 (+   897us) service_pool.cc:224] Handling call
0709 12:22:49.941421 (+  9224us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.955894 (+ 14473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.955942 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.976418 (+ 20476us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.359869 (+383451us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:50.359934 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.244862 (+884928us) spinlock_profiling.cc:243] Waited 7.27 ms on lock 0x7b54005f28d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:51.244917 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.332909 (+ 87992us) raft_consensus.cc:1685] finished
0709 12:22:51.332930 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.333156 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6661,"raft.queue_time_us":7489,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":12670,"spinlock_wait_cycles":16725376,"thread_start_us":331523,"threads_started":1,"wal-append.queue_time_us":1190132}
I20260709 12:22:51.337524  4193 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:51.340595  2383 connection.cc:256] RPC call timeout handler was delayed by -2.3193564670000342s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19230000000000003s for a response
W20260709 12:22:51.341292  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:51.341688  2383 connection.cc:256] RPC call timeout handler was delayed by -2.1339564670001891s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2129s for a response
I20260709 12:22:51.342516  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 288) took 1845 ms. Trace:
I20260709 12:22:51.342950  1874 rpcz_store.cc:276] 0709 12:22:49.497225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.110513 (+1613288us) service_pool.cc:224] Handling call
0709 12:22:51.164136 (+ 53623us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.342241 (+178105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.342458 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:51.181217  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 253419us
I20260709 12:22:51.344660  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:51.346312  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:51.088140  4247 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.352059  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.353024  4247 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.195986  4037 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:51.353586  4037 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:51.354512  4037 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.355700  4037 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:51.356335  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2198) took 2392 ms (client timeout 1972 ms). Trace:
I20260709 12:22:51.356707  4072 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:51.357445  4072 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.358495  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 522) took 2355 ms. Trace:
I20260709 12:22:51.359053  2412 rpcz_store.cc:276] 0709 12:22:49.003581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.003751 (+   170us) service_pool.cc:224] Handling call
0709 12:22:49.028104 (+ 24353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.229169 (+2201065us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:51.229206 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:51.229250 (+    44us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:51.229285 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.358153 (+128868us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54006009d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:51.358214 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.358388 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5061365504}
I20260709 12:22:50.955384  4220 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:51.364933  4126 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.365682  4126 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.367465  4272 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.313594  4219 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:51.369179  4219 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.199055  1723 rpcz_store.cc:276] 0709 12:22:49.989029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.897915 (+908886us) service_pool.cc:224] Handling call
0709 12:22:51.112069 (+214154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.372776  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.373732  1723 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:51.107741  4083 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.217463  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:51.218557  3922 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.235s	user 0.012s	sys 0.001s
I20260709 12:22:51.177059  4000 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.219223  4283 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:51.202467  4237 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:51.223582  2419 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:51.223295  1626 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 5, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:51.114310  4303 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:22:51.230120  1806 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:51.085717  2410 rpcz_store.cc:276] 0709 12:22:49.498572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.498757 (+   185us) service_pool.cc:224] Handling call
0709 12:22:49.499184 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.499618 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.499648 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.500321 (+   673us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.905103 (+404782us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:49.905165 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.905268 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.898710 (+993442us) raft_consensus.cc:1685] finished
0709 12:22:50.898730 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.898874 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3345,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":4645,"thread_start_us":404212,"threads_started":1,"wal-append.queue_time_us":477341}
I20260709 12:22:50.945741  4286 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.234472  1724 rpcz_store.cc:276] 0709 12:22:49.948931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.880052 (+931121us) service_pool.cc:224] Handling call
0709 12:22:51.163402 (+283350us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.234026  1800 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:22:51.237341  4312 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.238247  4232 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.399780  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.242165  4267 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:51.243952  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:51.244326  4296 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:51.245752  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.405622  1731 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:51.247601  4035 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.248301  4266 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:51.249373  4263 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.249903  4038 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:51.250273  4213 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:51.163033  4294 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.250368  4222 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.251444  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 287) took 1754 ms. Trace:
I20260709 12:22:51.253059  4097 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.253551  4190 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.201001  4292 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:51.254464  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 317) took 2159 ms (client timeout 2298 ms). Trace:
W20260709 12:22:51.254849  4239 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.502s	user 0.007s	sys 0.003s
W20260709 12:22:51.255334  1799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1433) took 2186 ms (client timeout 1526 ms). Trace:
I20260709 12:22:51.255522  4221 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.256384  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1436) took 2161 ms. Trace:
I20260709 12:22:51.258118  1845 leader_election.cc:400] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4.
I20260709 12:22:51.258970  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1429) took 2324 ms. Trace:
I20260709 12:22:51.115797  4290 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:51.198685  4305 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:22:51.230628  4295 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:51.262081  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:51.455163  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:51.262459  1727 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.135s	user 0.013s	sys 0.005s
W20260709 12:22:51.306901  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:51.243860  4298 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:22:51.317660  1730 rpcz_store.cc:276] 0709 12:22:49.989994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.315831 (+1325837us) service_pool.cc:224] Handling call
0709 12:22:51.317314 (+  1483us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:51.320816  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 148696us
I20260709 12:22:51.330535  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.485925  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 310) took 1398 ms. Trace:
I20260709 12:22:51.486276  1725 rpcz_store.cc:276] 0709 12:22:50.113627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.319425 (+1205798us) service_pool.cc:224] Handling call
0709 12:22:51.485861 (+166436us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.206866  4306 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.487226  4082 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.465094  4290 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.466279  4295 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.489239  4306 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.490561  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 632) took 1773 ms. Trace:
I20260709 12:22:51.188350  4250 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.347191  4281 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:51.347803  4166 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.348421  4314 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:51.348866  2383 connection.cc:256] RPC call timeout handler was delayed by -1.3265079070001775s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
W20260709 12:22:51.505306  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1321079069998632s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1942s for a response
I20260709 12:22:51.350610  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:51.353170  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.371536  4219 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.399102  3922 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.377357  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:51.514479  4166 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.515029  1873 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:22:51.516301  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:51.379470  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:51.393446  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1444) took 1747 ms (client timeout 1796 ms). Trace:
I20260709 12:22:51.397707  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2208) took 2292 ms. Trace:
W20260709 12:22:51.342530  4299 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.163s	user 0.000s	sys 0.000s
I20260709 12:22:51.404007  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:51.405071  1802 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:22:51.366186  4265 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.456969  4266 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.461073  4239 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.424678  4312 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.424952  4296 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:51.507037  1729 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:51.519227  1877 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:51.461231  1798 rpcz_store.cc:276] 0709 12:22:48.934292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.934460 (+   168us) service_pool.cc:224] Handling call
0709 12:22:48.974898 (+ 40438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.169280 (+194382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.169330 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.170030 (+   700us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.381578 (+211548us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:49.381634 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.258553 (+1876919us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54001eb4d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:51.258597 (+    44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.258646 (+    49us) raft_consensus.cc:1685] finished
0709 12:22:51.258657 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.258888 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27757,"raft.queue_time_us":27989,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":138,"spinlock_wait_cycles":366950784,"thread_start_us":210926,"threads_started":1,"wal-append.queue_time_us":205010,"wal-append.run_cpu_time_us":2157,"wal-append.run_wall_time_us":1005179}
I20260709 12:22:51.546620  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.491046  2411 rpcz_store.cc:276] 0709 12:22:49.716899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.717582 (+   683us) service_pool.cc:224] Handling call
0709 12:22:49.717936 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.773739 (+ 55803us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.773794 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.879406 (+105612us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.168584 (+289178us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:50.168635 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.168738 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.490279 (+1321541us) raft_consensus.cc:1685] finished
0709 12:22:51.490310 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.490486 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":352,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":13154,"thread_start_us":188069,"threads_started":1,"wal-append.queue_time_us":1023649}
I20260709 12:22:51.548015  1729 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
W20260709 12:22:51.549618  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1440) took 1984 ms (client timeout 2096 ms). Trace:
W20260709 12:22:51.551451  1802 rpcz_store.cc:269] 0709 12:22:49.565744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.243620 (+1677876us) service_pool.cc:224] Handling call
0709 12:22:51.549557 (+305937us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:51.556914  1800 rpcz_store.cc:269] 0709 12:22:49.646291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.864475 (+1218184us) service_pool.cc:224] Handling call
0709 12:22:51.393370 (+528895us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.557289  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.516981  1873 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:51.559036  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:22:51.559497  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.783s	user 0.002s	sys 0.039s
W20260709 12:22:51.560024  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.003s	sys 0.039s
I20260709 12:22:51.559036  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:51.560407  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 783671us
W20260709 12:22:51.421882  1807 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.674s	user 0.010s	sys 0.010s
W20260709 12:22:51.422953  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.007s	sys 0.014s
I20260709 12:22:51.424700  1872 rpcz_store.cc:276] 0709 12:22:49.496917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.112641 (+1615724us) service_pool.cc:224] Handling call
0709 12:22:51.164038 (+ 51397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.234652 (+ 70614us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.251369 (+ 16717us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.564586  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:51.564826  1807 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.010s	sys 0.010s
I20260709 12:22:51.390540  4247 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:22:51.428974  4237 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.412760  4232 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.431778  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.435220  1626 catalog_manager.cc:5697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:51.441148  4234 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:51.446132  4294 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.447250  4072 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.449100  4222 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.424463  4038 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.451014  4288 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:51.452196  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:51.455714  1723 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:22:51.455466  1799 rpcz_store.cc:269] 0709 12:22:49.073133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.082157 (+  9024us) service_pool.cc:224] Handling call
0709 12:22:51.255126 (+2172969us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b54001671d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:51.255255 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1651389696}
I20260709 12:22:51.396963  4283 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:51.425282  1803 rpcz_store.cc:269] 0709 12:22:49.095344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.141393 (+ 46049us) service_pool.cc:224] Handling call
0709 12:22:51.254272 (+2112879us) spinlock_profiling.cc:243] Waited 87.1 ms on lock 0x7b54002f2e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:51.254376 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":200241536}
I20260709 12:22:51.460891  1806 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:51.460916  1801 rpcz_store.cc:276] 0709 12:22:49.094694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.431319 (+336625us) service_pool.cc:224] Handling call
0709 12:22:49.431726 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.946955 (+515229us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.947002 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:49.947062 (+    60us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:49.947116 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.256145 (+1309029us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b5400132c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:51.256191 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.256326 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1168626048}
W20260709 12:22:51.423732  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.010s	sys 0.024s
I20260709 12:22:51.467988  1727 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
W20260709 12:22:51.463483  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 205504us
W20260709 12:22:51.356703  1878 rpcz_store.cc:269] 0709 12:22:48.963663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.991312 (+ 27649us) service_pool.cc:224] Handling call
0709 12:22:51.356115 (+2364803us) spinlock_profiling.cc:243] Waited 550 ms on lock 0x7b54001f8454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:51.356233 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1265875840}
I20260709 12:22:51.522349  1874 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:51.566154  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.007s	sys 0.014s
I20260709 12:22:51.397673  4303 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.427800  4035 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.575889  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.568563  1730 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:22:51.561587  4126 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.587435  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:51.588106  4272 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.587977  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:51.588802  4281 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:22:51.590875  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 295) took 2028 ms (client timeout 1550 ms). Trace:
I20260709 12:22:51.591807  4218 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:51.592423  1732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2347) took 1917 ms (client timeout 2382 ms). Trace:
I20260709 12:22:51.591840  4216 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.555032  1880 rpcz_store.cc:276] 0709 12:22:49.093067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.131208 (+ 38141us) service_pool.cc:224] Handling call
0709 12:22:49.155726 (+ 24518us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.387179 (+231453us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:49.610051 (+222872us) log.cc:844] Serialized 14 byte log entry
0709 12:22:49.926188 (+316137us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.945168 (+ 18980us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.945356 (+   188us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:49.945567 (+   211us) log.cc:844] Serialized 14 byte log entry
0709 12:22:51.253374 (+1307807us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b18003eaac0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:22:51.380597 (+127223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.380701 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.385642 (+  4941us) raft_consensus.cc:1685] finished
0709 12:22:51.385671 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.385845 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43539,"prepare.queue_time_us":18108,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":4065,"raft.queue_time_us":217631,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":65795,"spinlock_wait_cycles":17549056,"thread_start_us":539409,"threads_started":3,"wal-append.queue_time_us":460182}
I20260709 12:22:51.715188  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:51.852838  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:51.880822  1731 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:51.831892  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2354) took 1096 ms. Trace:
W20260709 12:22:51.832602  1727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1495) took 1891 ms (client timeout 1950 ms). Trace:
W20260709 12:22:51.883527  1727 rpcz_store.cc:269] 0709 12:22:49.941687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.941847 (+   160us) service_pool.cc:224] Handling call
0709 12:22:51.832529 (+1890682us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:51.833077  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1438) took 2269 ms (client timeout 1950 ms). Trace:
I20260709 12:22:51.655361  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:51.884889  1881 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:22:51.885080  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45658us
W20260709 12:22:51.766031  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.709438  1725 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
W20260709 12:22:51.713690  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1422) took 2893 ms (client timeout 1727 ms). Trace:
I20260709 12:22:51.886296  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1513) took 1140 ms. Trace:
W20260709 12:22:51.886240  1804 rpcz_store.cc:269] 0709 12:22:48.820589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.820982 (+   393us) service_pool.cc:224] Handling call
0709 12:22:51.713444 (+2892462us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54002f2454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:51.713598 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1134372608}
I20260709 12:22:51.886765  1725 rpcz_store.cc:276] 0709 12:22:50.746090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.486980 (+740890us) service_pool.cc:224] Handling call
0709 12:22:51.886224 (+399244us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:51.834847  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.969s	user 0.012s	sys 0.031s
I20260709 12:22:51.657876  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:51.835110  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:51.835203  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:51.660151  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:51.890754  1800 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:51.892647  1723 rpcz_store.cc:276] 0709 12:22:50.735896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.374499 (+638603us) service_pool.cc:224] Handling call
0709 12:22:51.831792 (+457293us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.836161  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1514) took 1076 ms. Trace:
I20260709 12:22:51.835794  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:51.893862  1730 rpcz_store.cc:276] 0709 12:22:50.760112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.564229 (+804117us) service_pool.cc:224] Handling call
0709 12:22:51.836091 (+271862us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.895073  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:51.656201  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1432) took 2653 ms (client timeout 3148 ms). Trace:
I20260709 12:22:51.808928  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:51.593091  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:51.839184  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:51.839777  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:51.711658  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 133746us
I20260709 12:22:51.603698  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:51.854651  1732 rpcz_store.cc:269] 0709 12:22:49.674908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.681547 (+  6639us) service_pool.cc:224] Handling call
0709 12:22:51.592173 (+1910626us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400119ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:51.592330 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2362191232}
I20260709 12:22:51.897710  1873 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:51.897013  1805 rpcz_store.cc:269] 0709 12:22:49.002747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.079537 (+ 76790us) service_pool.cc:224] Handling call
0709 12:22:51.655974 (+2576437us) spinlock_profiling.cc:243] Waited 639 ms on lock 0x7b54001680d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:51.656131 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1470506368}
I20260709 12:22:51.520485  4314 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.713075  4150 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:51.907750  1724 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:51.908411  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 315) took 1209 ms. Trace:
I20260709 12:22:51.516774  4286 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:51.668013  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 289) took 2170 ms (client timeout 2643 ms). Trace:
W20260709 12:22:51.853767  1877 rpcz_store.cc:269] 0709 12:22:49.562255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.629747 (+ 67492us) service_pool.cc:224] Handling call
0709 12:22:51.590589 (+1960842us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b5400223654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:51.590762 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4200861568}
I20260709 12:22:51.908882  1724 rpcz_store.cc:276] 0709 12:22:50.699433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.414153 (+714720us) service_pool.cc:224] Handling call
0709 12:22:51.908335 (+494182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.882761  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1509) took 1173 ms. Trace:
I20260709 12:22:51.566560  4083 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:51.892128  1806 rpcz_store.cc:269] 0709 12:22:49.564033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.085055 (+1521022us) service_pool.cc:224] Handling call
0709 12:22:51.832991 (+747936us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.601539  1875 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:51.858564  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 461817us
I20260709 12:22:51.657121  4266 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:51.650187  3922 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:51.563932  4228 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.803s	user 0.029s	sys 0.060s
W20260709 12:22:51.902891  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 44567.8us
W20260709 12:22:51.924494  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 84832.1us
I20260709 12:22:51.904337  1874 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:51.897958  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:51.906136  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:51.907006  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:51.908310  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:51.875088  4038 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:51.840267  4303 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:51.913599  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.900936  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:51.909276  1879 rpcz_store.cc:269] 0709 12:22:49.497510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.814961 (+317451us) service_pool.cc:224] Handling call
0709 12:22:51.667817 (+1852856us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54001f7f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:51.667939 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1900737920}
I20260709 12:22:51.925961  1799 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:51.915815  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:51.930552  1803 raft_consensus.cc:2410] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:51.926239  1804 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:51.928184  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.931353  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1439) took 2366 ms. Trace:
I20260709 12:22:51.931813  1803 rpcz_store.cc:276] 0709 12:22:49.564826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.859707 (+2294881us) service_pool.cc:224] Handling call
0709 12:22:51.931284 (+ 71577us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.927853  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:51.721138  4235 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.909721  1731 rpcz_store.cc:276] 0709 12:22:50.709258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.714812 (+1005554us) service_pool.cc:224] Handling call
0709 12:22:51.882690 (+167878us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.667548  3796 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.593794  4166 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.842173  4227 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.921082  2419 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:51.923753  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.657485  4226 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:51.842537  4234 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.906312  1626 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 4, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:51.913482  1806 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:51.928437  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:51.832139  4265 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:51.928427  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:51.937717  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:51.939472  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:51.717299  4312 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.947299  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:51.950125  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1437) took 2387 ms. Trace:
I20260709 12:22:51.950338  1727 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:51.905737  4222 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:51.661850  4220 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:51.913018  4281 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:51.876844  4283 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.030539  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1499) took 2087 ms. Trace:
I20260709 12:22:52.031282  1805 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
W20260709 12:22:52.030875  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.004451  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.059715  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 112167us
I20260709 12:22:52.027112  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:52.032866  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 104508us
I20260709 12:22:52.033157  1799 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:52.061125  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1443) took 2433 ms. Trace:
I20260709 12:22:52.059504  1727 rpcz_store.cc:276] 0709 12:22:49.942939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.913309 (+1970370us) service_pool.cc:224] Handling call
0709 12:22:52.030469 (+117160us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:51.934053  4228 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.188s	user 0.036s	sys 0.063s
W20260709 12:22:52.034880  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 95520.9us
I20260709 12:22:52.060786  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:52.068996  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1510) took 1347 ms. Trace:
I20260709 12:22:52.032482  1806 rpcz_store.cc:276] 0709 12:22:49.563191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.912713 (+2349522us) service_pool.cc:224] Handling call
0709 12:22:51.913113 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:51.949802 (+ 36689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.950052 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.923620  4288 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:51.949105  1723 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
W20260709 12:22:51.975124  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.037955  1731 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:52.076900  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1504) took 2120 ms. Trace:
I20260709 12:22:51.952363  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.038028  1725 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:52.078231  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2356) took 1330 ms. Trace:
I20260709 12:22:52.078567  1725 rpcz_store.cc:276] 0709 12:22:50.748330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.914467 (+1166137us) service_pool.cc:224] Handling call
0709 12:22:52.078153 (+163686us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.039767  1626 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 4, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:52.025518  4227 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:51.948524  4219 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:51.938619  4213 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:52.043452  4235 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:52.037580  4037 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:51.925769  4316 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.131s
I20260709 12:22:52.047343  1872 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:52.048496  4220 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:52.089313  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:51.655637  4296 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.035768  4226 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.048584  4234 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:52.046614  4314 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:51.957177  4267 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:51.949815  4272 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:51.947451  4317 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.126s
W20260709 12:22:51.927039  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.038576  1877 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:52.061690  1799 rpcz_store.cc:276] 0709 12:22:49.628157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.930317 (+2302160us) service_pool.cc:224] Handling call
0709 12:22:52.026195 (+ 95878us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.060749 (+ 34554us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.061046 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.953413  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:52.107367  1626 catalog_manager.cc:5697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 4, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:52.108990  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 155648us
I20260709 12:22:52.109846  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:52.069483  1724 rpcz_store.cc:276] 0709 12:22:50.722054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.929647 (+1207593us) service_pool.cc:224] Handling call
0709 12:22:52.068919 (+139272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.111155  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:52.111505  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:52.112258  1724 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:22:52.112816  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:52.113202  4330 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:22:52.113718  4266 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:52.113946  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:52.114526  4266 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:52.098583  4037 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:52.115116  4220 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:52.115422  4266 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:52.115950  4266 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:52.116079  3796 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:52.116986  4283 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:52.116119  4037 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.119890  4218 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:52.120870  4307 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.401s	user 0.005s	sys 0.005s
I20260709 12:22:52.121944  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.122277  4286 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:52.123212  4286 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:52.123595  4294 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:52.123695  4286 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:52.124212  4272 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:52.125814  4232 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:52.124887  4286 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.099093  4267 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.126470  4288 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.127228  4267 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:52.127979  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:52.126066  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:51.948520  2410 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:52.074879  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:52.077335  1731 rpcz_store.cc:276] 0709 12:22:49.958401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.948111 (+1989710us) service_pool.cc:224] Handling call
0709 12:22:52.037775 (+ 89664us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.076606 (+ 38831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.076836 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:51.951926  1874 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:52.084239  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.090081  2417 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:52.065999  4228 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:52.071130  4222 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:52.097440  1800 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:52.098045  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:52.135870  2411 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:52.100562  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.083158  4265 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:52.103739  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:52.104398  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:52.138121  1725 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:52.105971  1727 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:22:52.093086  4312 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:52.132232  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48083.1us
I20260709 12:22:52.114135  4281 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.095690  4219 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:52.138317  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.136157  4222 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.139299  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1505) took 2175 ms. Trace:
I20260709 12:22:52.116434  4266 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:52.142966  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.143067  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.113252  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:52.144821  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70066.9us
I20260709 12:22:52.124137  4294 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:52.139109  4232 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:52.109714  1626 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 5, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:52.155246  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:52.181212  1731 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:52.145867  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:52.182512  1725 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 2.
I20260709 12:22:52.156721  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.157656  1874 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:52.148574  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:52.150152  3922 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:52.163976  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:52.140378  4265 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.161273  4035 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:52.138862  3796 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:22:52.149726  4312 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.167371  4228 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.170357  4237 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:52.171413  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:52.201382  1724 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:22:52.201421  1873 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:52.175071  4263 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:52.151587  1727 rpcz_store.cc:276] 0709 12:22:49.964260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.071496 (+2107236us) service_pool.cc:224] Handling call
0709 12:22:52.092335 (+ 20839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.138961 (+ 46626us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.139221 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.177037  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:52.187338  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 294) took 2659 ms (client timeout 2601 ms). Trace:
I20260709 12:22:52.204629  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:52.204875  1875 rpcz_store.cc:269] 0709 12:22:49.528872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.789149 (+260277us) service_pool.cc:224] Handling call
0709 12:22:52.187082 (+2397933us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b5400051754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:52.187250 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3834172672}
I20260709 12:22:52.205618  1731 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:22:52.189997  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1434) took 3096 ms. Trace:
I20260709 12:22:52.202080  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.183410  1879 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:22:52.191833  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:52.176074  4297 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:52.195901  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:52.211808  1874 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:52.196918  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:52.197829  1626 catalog_manager.cc:5697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:22:52.212617  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50329.8us
I20260709 12:22:52.199517  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.208971  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:52.214150  1725 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:22:52.216490  1881 pending_rounds.cc:85] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 1
I20260709 12:22:52.207535  1872 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:52.201606  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.193243  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.191004  4232 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.190446  3922 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.190698  4265 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:52.222123  1730 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:52.226585  1626 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 5, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:52.216991  1881 pending_rounds.cc:107] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:22:52.237710  1881 raft_consensus.cc:2889] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:52.239091  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.217967  1877 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:52.218887  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.242698  3450 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:52.221809  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.243096  3450 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:52.243360  1878 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:52.243477  3450 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:52.240489  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.200699  4316 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:52.246430  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 312) took 1548 ms. Trace:
I20260709 12:22:52.246870  1727 rpcz_store.cc:276] 0709 12:22:50.698411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.215328 (+1516917us) service_pool.cc:224] Handling call
0709 12:22:52.245940 (+ 30612us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.246111 (+   171us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.246135 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.246176 (+    41us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:52.246213 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.246264 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.246377 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.257683  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:52.210723  4263 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:52.209079  4310 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.206598  1807 rpcz_store.cc:276] 0709 12:22:49.094059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.382656 (+288597us) service_pool.cc:224] Handling call
0709 12:22:49.395952 (+ 13296us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.674812 (+278860us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.674866 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.708825 (+ 33959us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.716153 (+1007328us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:51.569514 (+853361us) log.cc:844] Serialized 14 byte log entry
0709 12:22:51.612268 (+ 42754us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.120137 (+507869us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:52.188530 (+ 68393us) spinlock_profiling.cc:243] Waited 59.8 ms on lock 0x7b18003bd340. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419
0709 12:22:52.189487 (+   957us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b54003dbdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:52.189545 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.189621 (+    76us) raft_consensus.cc:1685] finished
0709 12:22:52.189639 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.189897 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20150,"prepare.queue_time_us":1003285,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":58172,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":67811,"spinlock_wait_cycles":745138560,"thread_start_us":1771710,"threads_started":2,"wal-append.queue_time_us":993072}
I20260709 12:22:52.211524  4297 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.271219  4313 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:52.239360  3922 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:52.273370  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.303886  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.274037  4320 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:52.239097  4179 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.166539  4219 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.239060  4317 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:52.280792  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.227653  4035 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:52.238547  4232 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:52.283847  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:52.268018  1626 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 4, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:52.339458  2411 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:52.299659  4297 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:52.296025  4191 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:52.308355  1724 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:52.257200  4220 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:52.262118  4316 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:52.313937  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2349) took 2359 ms (client timeout 1753 ms). Trace:
I20260709 12:22:52.276805  4263 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:52.331574  1876 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:22:52.284281  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:52.290935  1725 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:22:52.309302  4320 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:52.301229  4208 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:52.345875  1626 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 4, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:52.368288  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:52.311511  4319 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:52.377005  4150 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:52.383213  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:52.382340  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2351) took 1656 ms. Trace:
I20260709 12:22:52.336212  4268 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.335454  4035 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.405581  1725 rpcz_store.cc:276] 0709 12:22:50.725729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.381253 (+1655524us) service_pool.cc:224] Handling call
0709 12:22:52.381667 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.381889 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.381925 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.381980 (+    55us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:52.382032 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.382099 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.382259 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:52.301584  4310 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.407200  2411 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
W20260709 12:22:52.407136  1726 rpcz_store.cc:269] 0709 12:22:49.942608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.942837 (+   229us) service_pool.cc:224] Handling call
0709 12:22:52.281501 (+2338664us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b5400435dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:52.302250 (+ 20749us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4428330240}
I20260709 12:22:52.330732  4218 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:52.424327  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:52.429173  2411 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:52.368393  4308 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.632s	user 0.005s	sys 0.005s
I20260709 12:22:52.426367  1626 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:52.329272  4179 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.431149  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:52.368276  4316 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:52.445513  2417 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:52.379681  4320 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:52.433519  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 375433us
I20260709 12:22:52.421164  4035 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:52.311851  4313 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.402761  4220 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:52.403074  4208 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.405108  4319 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.371061  4191 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.461027  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:52.428130  4218 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:52.470002  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:52.469676  1626 catalog_manager.cc:5697] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 5, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:52.475247  1626 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 3, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 3 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:52.475989  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.477411  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:52.477365  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:52.482934  1771 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:52.426173  4267 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.412695  4150 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.503788  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:52.512354  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:52.467811  4208 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:52.426714  4268 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.507460  1626 catalog_manager.cc:5697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:52.523042  1878 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:52.455896  4166 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:22:52.525874  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1502) took 2577 ms. Trace:
I20260709 12:22:52.536926  1732 rpcz_store.cc:276] 0709 12:22:49.947985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.947612 (+1999627us) service_pool.cc:224] Handling call
0709 12:22:52.034575 (+ 86963us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.034797 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:52.263604 (+228807us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.525483 (+261879us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.525543 (+    60us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:52.525606 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.525668 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.525813 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":336139,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":20319,"thread_start_us":461891,"threads_started":2}
I20260709 12:22:52.448298  4303 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.475414  4218 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:52.473230  4319 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:52.477473  4234 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:52.550045  1876 raft_consensus.cc:2410] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:22:52.526556  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:52.551438  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:52.462311  4038 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:22:52.485544  4235 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:52.521826  4212 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:52.528378  4334 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:22:52.556653  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:52.567860  1804 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:52.557125  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:52.569272  1807 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:52.479941  4228 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
W20260709 12:22:52.561424  1696 connection.cc:256] RPC call timeout handler was delayed by -5.1425522630001979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
I20260709 12:22:52.577898  3450 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:52.592234  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.597294  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.598272  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:52.599112  1802 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:52.597146  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2340) took 3276 ms. Trace:
I20260709 12:22:52.602001  1728 rpcz_store.cc:276] 0709 12:22:49.289063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.289202 (+   139us) service_pool.cc:224] Handling call
0709 12:22:49.350824 (+ 61622us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.689642 (+338818us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.689790 (+   148us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.707346 (+ 17556us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.734350 (+1027004us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:50.734409 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.734508 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.275348 (+1540840us) spinlock_profiling.cc:243] Waited 19.1 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:52.553584 (+278236us) raft_consensus.cc:1685] finished
0709 12:22:52.553612 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.565268 (+ 11656us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":134557,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":8488,"spinlock_wait_cycles":44038528,"thread_start_us":1249623,"threads_started":2,"wal-append.queue_time_us":986604}
I20260709 12:22:52.616377  3450 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:52.621647  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 161343us
I20260709 12:22:52.552192  4248 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:52.549458  4234 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:52.627449  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:52.630842  1798 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:52.626430  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.633960  1878 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:52.635242  4303 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:52.639392  4302 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.911s	user 0.006s	sys 0.000s
I20260709 12:22:52.641018  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 665) took 2709 ms. Trace:
I20260709 12:22:52.643435  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:52.668748  1805 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:52.662648  4329 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.316s	user 0.002s	sys 0.000s
I20260709 12:22:52.640504  4234 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:52.621320  4220 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:52.593696  4342 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:22:52.627357  4306 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.659006  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:52.631068  4248 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.635174  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:52.641475  2416 rpcz_store.cc:276] 0709 12:22:49.932055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.949849 (+ 17794us) service_pool.cc:224] Handling call
0709 12:22:49.970128 (+ 20279us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:49.977086 (+  6958us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.977132 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:49.984630 (+  7498us) log.cc:844] Serialized 26 byte log entry
0709 12:22:50.297684 (+313054us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:50.297730 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.297822 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.535302 (+2237480us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:52.640659 (+105357us) raft_consensus.cc:1685] finished
0709 12:22:52.640682 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.640935 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3044,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":3207,"spinlock_wait_cycles":6110080,"thread_start_us":269328,"threads_started":1,"wal-append.queue_time_us":739209}
I20260709 12:22:52.698359  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:52.661055  4312 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:52.684834  4337 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
W20260709 12:22:52.708586  1696 connection.cc:256] RPC call timeout handler was delayed by -3.9731098910001492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15560000000000002s for a response
I20260709 12:22:52.713369  1798 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:52.721892  1696 connection.cc:256] RPC call timeout handler was delayed by -3.9596098910001274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
I20260709 12:22:52.706211  1874 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:22:52.722394  1696 connection.cc:256] RPC call timeout handler was delayed by -3.877709890999995s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
W20260709 12:22:52.722915  1696 connection.cc:256] RPC call timeout handler was delayed by -3.8345098909999251s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.171s for a response
W20260709 12:22:52.723352  1696 connection.cc:256] RPC call timeout handler was delayed by -3.7931098910003129s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1756s for a response
W20260709 12:22:52.723788  1696 connection.cc:256] RPC call timeout handler was delayed by -3.7562098910002533s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
W20260709 12:22:52.724321  1696 connection.cc:256] RPC call timeout handler was delayed by -3.6878098910001427s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18730000000000002s for a response
W20260709 12:22:52.724723  1696 connection.cc:256] RPC call timeout handler was delayed by -3.5483098909999171s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2028s for a response
W20260709 12:22:52.725140  1696 connection.cc:256] RPC call timeout handler was delayed by -3.5312098910003442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20470000000000002s for a response
W20260709 12:22:52.725643  1696 connection.cc:256] RPC call timeout handler was delayed by -3.5069098910003049s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2074s for a response
W20260709 12:22:52.726063  1696 connection.cc:256] RPC call timeout handler was delayed by -3.3782098910000968s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2217s for a response
W20260709 12:22:52.726475  1696 connection.cc:256] RPC call timeout handler was delayed by -2.4818098910000117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32130000000000003s for a response
I20260709 12:22:52.716595  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 313) took 2017 ms. Trace:
W20260709 12:22:52.698885  4333 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
I20260709 12:22:52.771301  4310 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:52.781039  4316 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.713585  4248 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:52.795372  4347 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
W20260709 12:22:52.797272  4346 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:22:52.797683  3922 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:52.805451  4335 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20260709 12:22:52.695423  4342 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:52.832063  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.832480  1805 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
I20260709 12:22:52.832943  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.808735  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 669) took 1074 ms. Trace:
I20260709 12:22:52.809880  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 321) took 1413 ms. Trace:
I20260709 12:22:52.811168  2411 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:22:52.812230  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2229) took 1150 ms. Trace:
W20260709 12:22:52.813328  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 330515us
I20260709 12:22:52.834324  1800 rpcz_store.cc:276] 0709 12:22:51.396462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.914327 (+517865us) service_pool.cc:224] Handling call
0709 12:22:52.809660 (+895333us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54001e3f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:52.809795 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":377832704}
I20260709 12:22:52.835438  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:52.834860  1873 rpcz_store.cc:276] 0709 12:22:51.661591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.208761 (+547170us) service_pool.cc:224] Handling call
0709 12:22:52.209083 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:52.222868 (+ 13785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.222899 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.257056 (+ 34157us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.483062 (+226006us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:52.483122 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.483206 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.779139 (+295933us) raft_consensus.cc:1685] finished
0709 12:22:52.779163 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.812162 (+ 32999us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5011,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":9784,"thread_start_us":162148,"threads_started":1,"wal-append.queue_time_us":337893}
I20260709 12:22:52.813856  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.817530  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:52.838393  1874 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:52.825239  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:52.831524  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.837402  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.841521  1880 raft_consensus.cc:2368] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8.
I20260709 12:22:52.834113  2419 rpcz_store.cc:276] 0709 12:22:51.734260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.734542 (+   282us) service_pool.cc:224] Handling call
0709 12:22:52.808452 (+1073910us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b5400487454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:52.808649 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55094656}
I20260709 12:22:52.853168  3922 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:52.853861  3922 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:52.850835  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:52.802222  4297 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.859287  3450 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:52.867276  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:52.834089  4150 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:52.869989  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:52.870705  1805 raft_consensus.cc:2368] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8.
W20260709 12:22:52.872747  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 99307.3us
I20260709 12:22:52.841394  3796 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:52.832448  4313 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:52.838130  4342 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:52.720947  4306 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.874435  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:52.856055  4312 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:52.859818  4265 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.597s	user 0.013s	sys 0.000s
I20260709 12:22:52.898749  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.903606  1800 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:52.861367  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 316) took 2161 ms. Trace:
I20260709 12:22:52.906366  4038 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:52.914510  4038 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.914949  4038 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:52.915524  4265 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.899361  4037 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:52.918967  4037 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:52.918432  4265 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:52.919998  4037 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.920481  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.921938  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.863265  4179 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:52.930387  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 317) took 2230 ms. Trace:
I20260709 12:22:52.933151  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:52.843992  4208 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.934079  1872 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:52.912031  1730 rpcz_store.cc:276] 0709 12:22:50.699728 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.228128 (+1528400us) service_pool.cc:224] Handling call
0709 12:22:52.228500 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.228741 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:52.478437 (+249696us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.857613 (+379176us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.857736 (+   123us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:52.861118 (+  3382us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.861188 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.861302 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":251393,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":12284,"thread_start_us":587405,"threads_started":2}
I20260709 12:22:52.931365  1727 rpcz_store.cc:276] 0709 12:22:50.699981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.247776 (+1547795us) service_pool.cc:224] Handling call
0709 12:22:52.275289 (+ 27513us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.275638 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:52.502735 (+227097us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.929699 (+426964us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.929821 (+   122us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:52.929901 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.930027 (+   126us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.930272 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72660,"prepare.queue_time_us":388457,"prepare.run_cpu_time_us":175,"prepare.run_wall_time_us":767,"raft.queue_time_us":105892,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":78,"thread_start_us":563231,"threads_started":3,"wal-append.queue_time_us":378930}
W20260709 12:22:52.923645  4351 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:22:52.904976  4191 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:52.848330  4320 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:52.924106  4267 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:52.945808  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 99777.7us
I20260709 12:22:52.951757  2419 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:22:52.944677  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:52.963564  1805 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:22:52.966287  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:52.967172  1807 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:52.935580  4268 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:52.976315  4306 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:52.977195  4316 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:52.980525  4212 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:52.979993  4035 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:52.982216  4297 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:52.983392  4344 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.262s	user 0.001s	sys 0.000s
I20260709 12:22:52.985697  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2220) took 1599 ms. Trace:
I20260709 12:22:52.988765  1877 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:52.990480  4352 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:22:52.991597  1772 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:52.995883  4191 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:52.997797  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.003435  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.986186  1881 rpcz_store.cc:276] 0709 12:22:51.386478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.197457 (+810979us) service_pool.cc:224] Handling call
0709 12:22:52.216159 (+ 18702us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:52.340411 (+124252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.340467 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:52.356220 (+ 15753us) log.cc:844] Serialized 52 byte log entry
0709 12:22:52.718902 (+362682us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:52.778994 (+ 60092us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.782846 (+  3852us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.785261 (+  2415us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.785758 (+   497us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.985429 (+199671us) raft_consensus.cc:1685] finished
0709 12:22:52.985452 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.985626 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":125,"prepare.queue_time_us":63575,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":95,"raft.queue_time_us":24330,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":13027,"thread_start_us":263136,"threads_started":1,"wal-append.queue_time_us":473627}
I20260709 12:22:53.006130  1880 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 3.
W20260709 12:22:53.008991  2417 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.563s	user 0.010s	sys 0.005s
I20260709 12:22:53.000285  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.009716  2417 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:22:53.011597  2411 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:53.012301  1726 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
I20260709 12:22:53.017761  2414 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:53.009153  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 169770us
I20260709 12:22:53.008913  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:52.992187  4212 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.986472  4226 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:53.012728  1726 pending_rounds.cc:85] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 3
I20260709 12:22:53.022996  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:53.024531  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:53.024976  4358 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:22:53.026981  4295 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:22:53.028173  4357 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:22:52.747726  1723 rpcz_store.cc:276] 0709 12:22:50.698736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.226854 (+1528118us) service_pool.cc:224] Handling call
0709 12:22:52.267782 (+ 40928us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.268054 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:52.516108 (+248054us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.715835 (+199727us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:52.716118 (+   283us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:52.716206 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.716323 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.716509 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32285,"prepare.queue_time_us":278028,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":6112,"thread_start_us":404641,"threads_started":2}
I20260709 12:22:53.033182  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:53.517728  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 475190us
I20260709 12:22:53.066516  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.033552  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:53.108140  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.512602  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2357) took 2617 ms. Trace:
W20260709 12:22:53.501029  1798 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.779s	user 0.012s	sys 0.003s
W20260709 12:22:53.519111  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 485602us
I20260709 12:22:53.520207  1798 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:53.519943  1725 rpcz_store.cc:276] 0709 12:22:50.894958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.414321 (+1519363us) service_pool.cc:224] Handling call
0709 12:22:52.451905 (+ 37584us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:52.472579 (+ 20674us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.472613 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.481677 (+  9064us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.881039 (+399362us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:52.881120 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.881238 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.512328 (+631090us) raft_consensus.cc:1685] finished
0709 12:22:53.512354 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.512529 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20078,"raft.queue_time_us":12085,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":20432,"thread_start_us":335478,"threads_started":1,"wal-append.queue_time_us":988833}
W20260709 12:22:53.112466  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 89049.5us
I20260709 12:22:53.521957  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 318) took 2821 ms. Trace:
I20260709 12:22:53.112174  1802 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:53.072520  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 92832.4us
I20260709 12:22:53.522414  1724 rpcz_store.cc:276] 0709 12:22:50.700483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.371284 (+1670801us) service_pool.cc:224] Handling call
0709 12:22:52.371705 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:52.404566 (+ 32861us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.404613 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.414770 (+ 10157us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.883842 (+469072us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:52.884042 (+   200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.884154 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.374874 (+490720us) raft_consensus.cc:1685] finished
0709 12:22:53.374910 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.521880 (+146970us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":99,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":14422,"thread_start_us":414637,"threads_started":1,"wal-append.queue_time_us":845683}
I20260709 12:22:53.202103  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:53.525200  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 359919us
I20260709 12:22:53.065480  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.526049  1726 pending_rounds.cc:107] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:22:53.062592  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.526583  1805 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:22:53.526062  1730 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:53.526973  1726 raft_consensus.cc:2889] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:53.503855  4248 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.716s	user 0.008s	sys 0.003s
W20260709 12:22:53.300359  4263 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.898s	user 0.012s	sys 0.010s
W20260709 12:22:53.434350  4367 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:22:53.533347  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 510402us
W20260709 12:22:53.438246  4232 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.071s	user 0.018s	sys 0.000s
W20260709 12:22:53.116794  4266 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.914s	user 0.013s	sys 0.005s
W20260709 12:22:53.529994  2383 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.550s (SENT)
I20260709 12:22:53.535737  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.535666  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:53.139732  4319 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.573s	user 0.007s	sys 0.006s
I20260709 12:22:53.544672  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.535019  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.112670  4035 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:53.537664  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:53.508448  4234 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.799s	user 0.007s	sys 0.002s
I20260709 12:22:53.539656  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.539777  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.065480  4208 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:53.541189  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:53.541472  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.062321  4227 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:53.563798  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:53.523991  4295 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.526087  4072 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:53.524717  4303 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:53.544611  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:53.537314  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.545766  1802 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:53.110796  4239 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:53.043128  4247 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:53.573012  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.573940  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:53.574273  1802 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:53.537765  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.546665  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.547444  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.547907  4319 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:53.535794  4226 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:53.545725  4232 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:53.114401  4038 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:53.550936  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:53.582093  1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2353) took 2855 ms (client timeout 2461 ms). Trace:
I20260709 12:22:53.565363  4312 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:53.582600  1729 rpcz_store.cc:269] 0709 12:22:50.726229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.563311 (+837082us) service_pool.cc:224] Handling call
0709 12:22:53.581759 (+2018448us) spinlock_profiling.cc:243] Waited 692 ms on lock 0x7b54002a9754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:53.581998 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1591106432}
I20260709 12:22:53.192212  4294 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:53.040465  4218 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:53.040997  4272 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.512s
I20260709 12:22:53.047037  4212 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:53.596360  4232 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:53.596632  4212 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:53.597321  2416 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:53.599283  2410 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 533) took 2035 ms (client timeout 2214 ms). Trace:
W20260709 12:22:53.599542  4353 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.930s	user 0.003s	sys 0.001s
W20260709 12:22:53.599772  2410 rpcz_store.cc:269] 0709 12:22:51.563377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.563537 (+   160us) service_pool.cc:224] Handling call
0709 12:22:53.599083 (+2035546us) spinlock_profiling.cc:243] Waited 56.4 ms on lock 0x7b5400783054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:53.599221 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":129716096}
I20260709 12:22:53.603608  4319 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:53.604087  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 646) took 1131 ms. Trace:
I20260709 12:22:53.604470  2418 rpcz_store.cc:276] 0709 12:22:52.472592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.513504 (+ 40912us) service_pool.cc:224] Handling call
0709 12:22:53.601943 (+1088439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.602151 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:53.602180 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:53.602234 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:53.602268 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.602322 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.603910 (+  1588us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400783050. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:53.604024 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2502263552}
I20260709 12:22:53.573323  4247 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.553086  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.542250  4248 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:53.561682  1725 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:53.449186  4286 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.907s	user 0.005s	sys 0.011s
I20260709 12:22:53.198299  4317 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:53.584192  4208 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:53.634493  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:53.567345  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.569044  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.571709  1807 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:22:53.565932  4342 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.634s	user 0.007s	sys 0.003s
I20260709 12:22:53.578999  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.580054  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:53.580811  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50863.1us
I20260709 12:22:53.639791  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:53.640637  1802 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:53.641451  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:53.641894  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:53.581561  1804 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:22:53.643474  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.644171  1804 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:53.608659  4312 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:53.645094  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.581249  1723 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:53.645956  4312 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:53.647229  4342 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:53.647778  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.648536  1723 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:53.649369  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.649703  4342 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:53.613574  4286 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.076s	user 0.005s	sys 0.014s
I20260709 12:22:53.551375  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.584650  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.570328  4227 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:53.585477  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:53.582509  1801 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:53.541877  4263 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:53.567968  4072 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:53.569416  4295 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.561998  4266 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:53.563556  4234 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:53.598474  4218 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:53.611511  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46315.3us
I20260709 12:22:53.677211  1876 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:53.615187  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2226) took 2117 ms. Trace:
I20260709 12:22:53.583585  4320 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:53.562814  1726 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.619338  4038 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:53.620244  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.593384  4222 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:53.616055  4247 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.625460  4037 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.630272  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.651645  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.678078  1875 rpcz_store.cc:276] 0709 12:22:51.498155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.221077 (+722922us) service_pool.cc:224] Handling call
0709 12:22:52.246277 (+ 25200us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:52.263140 (+ 16863us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.263181 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:52.309982 (+ 46801us) log.cc:844] Serialized 52 byte log entry
0709 12:22:52.468505 (+158523us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:52.535146 (+ 66641us) log.cc:844] Serialized 14 byte log entry
0709 12:22:52.535459 (+   313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.535566 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:53.598835 (+1063269us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:53.609264 (+ 10429us) raft_consensus.cc:1685] finished
0709 12:22:53.614994 (+  5730us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:53.615134 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17779,"prepare.queue_time_us":67128,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":8559,"raft.queue_time_us":11883,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":15595,"spinlock_wait_cycles":7388928,"thread_start_us":100522,"threads_started":1,"wal-append.queue_time_us":270254}
I20260709 12:22:53.637217  1727 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:22:53.684762  1729 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:53.656406  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.658660  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.658974  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:53.650476  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.646404  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.661669  4072 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.654603  4212 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:53.663353  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:53.664203  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:53.664247  4377 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.051s	user 0.002s	sys 0.000s
I20260709 12:22:53.656311  4232 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 12:22:53.667428  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:53.723977  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:53.724079  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:53.724341  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 57006.4us
I20260709 12:22:53.724646  1881 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:53.672308  4208 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:53.670056  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.734189  1874 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:53.636015  4317 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:53.673676  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.682258  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:53.720000  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:53.741004  1880 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:22:53.618286  4319 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:53.725780  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:53.743705  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:53.745437  1801 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:53.657946  4227 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:53.728130  1799 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:53.725054  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.730063  1872 raft_consensus.cc:2368] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5.
I20260709 12:22:53.729961  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.724692  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:53.749619  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:53.731504  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.752367  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.753245  1725 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
W20260709 12:22:53.730686  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49135.8us
I20260709 12:22:53.676623  4286 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.685849  4272 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.737654  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.739200  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:53.761310  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:53.761840  1874 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:53.761983  1880 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.741158  1805 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.745924  1804 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:53.736374  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.756444  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.764580  1730 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.756784  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.757251  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:53.754381  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.726068  4212 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.766955  1876 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:53.732509  4232 consensus_queue.cc:1243] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:53.738405  4208 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.762706  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.770278  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.770529  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.729986  4342 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:53.777104  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.781507  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.782397  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.785640  1874 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:53.785938  1875 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:22:53.786772  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.791056  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.791990  1727 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.791002  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.794945  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:53.767836  4248 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:53.797827  4248 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:53.800128  4248 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:53.800623  4248 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.801064  4248 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.803246  4342 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:53.804041  4342 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:53.803145  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:53.760576  4035 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:53.827929  1804 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:22:53.829183  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.829975  1804 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:53.832329  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61313.1us
I20260709 12:22:53.830680  1805 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
W20260709 12:22:53.825234  4297 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.000s	sys 0.027s
I20260709 12:22:53.776398  4212 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:53.842869  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 176745us
I20260709 12:22:53.791163  4208 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.805286  4342 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:53.864682  1880 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
W20260709 12:22:53.862452  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 203524us
I20260709 12:22:53.761093  4319 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:53.767431  4286 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:53.876925  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:53.776614  4272 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:53.878041  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 313415us
I20260709 12:22:53.873870  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.883519  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:53.884142  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 242322us
I20260709 12:22:53.886292  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:53.863600  4297 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.000s	sys 0.028s
I20260709 12:22:53.886199  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:53.888970  1725 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:53.890599  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:53.891633  2419 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:53.896173  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:53.899343  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:53.903766  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:53.904253  1727 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:53.906293  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:53.906361  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.907317  1727 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:53.914938  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:53.882400  4219 raft_consensus.cc:2764] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:22:53.912740  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:53.925170  1724 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:22:53.925729  1724 pending_rounds.cc:85] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 6
I20260709 12:22:53.926033  1724 pending_rounds.cc:107] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:22:53.926344  1724 raft_consensus.cc:2889] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:53.907508  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 158021us
I20260709 12:22:53.908218  4166 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:53.938884  1727 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:22:53.940562  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:53.945037  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.949681  1727 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.967845  4035 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:22:53.968977  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70915.4us
I20260709 12:22:53.972437  1727 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:53.982651  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:53.985970  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 54157us
I20260709 12:22:53.943409  4166 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:53.940793  4385 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:53.903342  4227 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:53.989711  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49239.5us
I20260709 12:22:53.991262  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:53.960029  4263 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:53.966770  4290 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:53.997665  1724 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:53.989830  4166 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:53.996778  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:53.992784  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:53.966545  4234 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:53.970999  4266 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:53.977974  4220 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:54.013025  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:22:54.011777  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 253924us
W20260709 12:22:54.016822  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 101989us
W20260709 12:22:53.984189  2413 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 532) took 2419 ms (client timeout 2221 ms). Trace:
W20260709 12:22:54.020058  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 125903us
W20260709 12:22:54.028941  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.990675  4386 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:54.029122  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59235.6us
I20260709 12:22:54.002942  4290 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:54.039134  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2372) took 1847 ms. Trace:
I20260709 12:22:54.054153  1728 rpcz_store.cc:276] 0709 12:22:52.191606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.604499 (+412893us) service_pool.cc:224] Handling call
0709 12:22:52.665275 (+ 60776us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:52.665541 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:52.699826 (+ 34285us) log.cc:844] Serialized 14 byte log entry
0709 12:22:53.804957 (+1105131us) spinlock_profiling.cc:243] Waited 8.95 ms on lock 0x7b18003892c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:22:54.038553 (+233596us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:54.038615 (+    62us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:54.038677 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.038858 (+   181us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.039045 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54348,"prepare.queue_time_us":34106,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":34629,"spinlock_wait_cycles":20583296,"thread_start_us":210360,"threads_started":1}
I20260709 12:22:54.028234  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:54.036794  4386 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:54.028522  2413 rpcz_store.cc:269] 0709 12:22:51.563059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.565782 (+  2723us) service_pool.cc:224] Handling call
0709 12:22:53.982549 (+2416767us) spinlock_profiling.cc:243] Waited 631 ms on lock 0x7b54006b0154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:53.982742 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1452384384}
I20260709 12:22:54.063001  4386 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:54.064992  4272 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:54.071278  4386 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.009672  4395 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:54.083385  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2228) took 2518 ms. Trace:
I20260709 12:22:54.089335  4386 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:54.089668  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 81887.4us
I20260709 12:22:54.091892  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.092907  4272 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:54.090112  4386 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:22:54.090298  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62153.8us
I20260709 12:22:54.094451  4248 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.094745  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:54.096086  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:54.097278  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:54.099200  1804 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:54.098665  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:54.090770  4319 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:53.971822  4035 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.062189  4290 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:54.083873  1879 rpcz_store.cc:276] 0709 12:22:51.565277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.222243 (+656966us) service_pool.cc:224] Handling call
0709 12:22:52.245121 (+ 22878us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:52.276464 (+ 31343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:52.276497 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.291556 (+ 15059us) log.cc:844] Serialized 26 byte log entry
0709 12:22:53.601606 (+1310050us) spinlock_profiling.cc:243] Waited 7.01 ms on lock 0x7b18002e1920. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:22:53.777159 (+175553us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:53.777221 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.777329 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:54.078762 (+301433us) raft_consensus.cc:1685] finished
0709 12:22:54.078822 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.083304 (+  4482us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16215,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"spinlock_wait_cycles":16119552,"thread_start_us":175243,"threads_started":1,"wal-append.queue_time_us":1617509}
W20260709 12:22:54.126564  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 129885us
I20260709 12:22:54.079651  4220 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:54.096769  4248 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:54.082623  4286 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:54.079401  4166 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:54.140496  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45845.5us
W20260709 12:22:54.149610  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 236930us
I20260709 12:22:54.150808  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:54.153411  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:54.158766  1726 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:22:54.159557  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2379) took 1136 ms. Trace:
I20260709 12:22:54.159940  1726 rpcz_store.cc:276] 0709 12:22:53.023084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.539484 (+516400us) service_pool.cc:224] Handling call
0709 12:22:54.159475 (+619991us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:54.160580  4286 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:22:54.161505  4263 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:54.161928  4263 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:54.163007  4263 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.164463  4263 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:54.167829  4166 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:54.168690  4166 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:54.169157  4166 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:54.170910  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:54.171648  2413 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:54.130038  4395 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:54.167944  1724 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:22:54.150342  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:54.159085  1881 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:22:54.145394  1801 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:54.186846  1730 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:54.126151  4386 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:54.187813  1805 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20260709 12:22:54.170081  4166 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.189944  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:54.096910  4390 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:54.134503  4220 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.061548  4389 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:22:54.195878  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:54.130685  4319 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.056s
W20260709 12:22:54.205199  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54965us
I20260709 12:22:54.157171  4266 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:54.173151  4272 raft_consensus.cc:2764] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260709 12:22:54.204357  4220 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:54.202637  4286 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:54.229270  1878 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:54.210664  4395 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:54.229817  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 190611us
I20260709 12:22:54.230638  4286 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:54.239178  4286 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:54.239435  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:54.256608  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 222957us
I20260709 12:22:54.272305  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:54.272308  4412 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:22:54.277190  4412 raft_consensus.cc:2764] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
W20260709 12:22:54.291791  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 193161us
I20260709 12:22:54.233830  4402 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:54.292629  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:54.296669  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 106641us
W20260709 12:22:54.299794  4317 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.533s	user 0.006s	sys 0.008s
I20260709 12:22:54.304430  4398 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:54.268594  4037 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:54.308535  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69183.9us
I20260709 12:22:54.246663  4286 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:54.343214  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69007.4us
I20260709 12:22:54.297618  4380 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:54.299647  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:54.353734  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:54.354411  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54851us
I20260709 12:22:54.356200  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:54.356602  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:54.238471  4395 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:54.357553  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.359599  4395 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:54.306907  4405 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 12:22:54.357103  1724 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:54.361428  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.350176  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:54.367280  1730 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:54.298985  4403 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:54.360348  4395 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:54.371248  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 217968us
I20260709 12:22:54.373682  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:54.372833  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 114796us
W20260709 12:22:54.351226  4247 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.001s
I20260709 12:22:54.378301  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.374878  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:54.262486  4386 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:54.386262  2383 leader_election.cc:336] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.726s (SENT)
I20260709 12:22:54.388111  1724 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:54.376523  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:54.391605  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:54.401507  1880 raft_consensus.cc:2376] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4. Re-sending same reply.
I20260709 12:22:54.401767  2419 raft_consensus.cc:2410] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:22:54.402554  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.402940  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.393074  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.403353  1880 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:54.407497  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.408387  1880 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:54.409534  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.410519  1880 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:22:54.406231  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:54.411739  1874 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:22:54.416729  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.418820  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.425534  1880 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:54.427502  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.429062  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:54.429416  2383 leader_election.cc:336] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.923s (SENT)
I20260709 12:22:54.429848  1872 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:54.429886  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.428120  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.431185  1876 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:54.418031  2413 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:54.436421  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.437367  1879 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:54.438273  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.418138  4219 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:54.439011  1875 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:54.439234  4402 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:54.390708  4395 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:54.439313  4219 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.442611  4286 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:54.390070  4247 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.444303  1798 raft_consensus.cc:2368] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:54.444689  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:54.443131  4219 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:54.445703  4400 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 12:22:54.420075  4403 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:54.447292  4266 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:54.448067  4266 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:54.447793  4403 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.367079  4398 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:54.449249  4266 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.449770  4398 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.451474  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:54.449973  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 88612us
I20260709 12:22:54.372432  4297 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:22:54.466630  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.380604  4380 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:54.488256  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:54.444603  4392 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 12:22:54.363821  4290 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.469050  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:54.491935  4389 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:54.498564  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 647) took 1981 ms. Trace:
I20260709 12:22:54.366849  4037 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.351712  4317 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.501461  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.519383  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.455677  1696 leader_election.cc:400] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8.
I20260709 12:22:54.520171  1696 leader_election.cc:403] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:22:54.520700  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 65218.2us
I20260709 12:22:54.470433  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:54.465154  1874 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:22:54.511312  2416 rpcz_store.cc:276] 0709 12:22:52.517500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.779313 (+261813us) service_pool.cc:224] Handling call
0709 12:22:52.779690 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.597920 (+818230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.498226 (+900306us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b540046a354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:54.498483 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1879823744}
I20260709 12:22:54.522058  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.520455  1876 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:54.523000  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:54.513491  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84101.7us
W20260709 12:22:54.449762  4072 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.715s	user 0.011s	sys 0.003s
I20260709 12:22:54.452451  4406 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.099s
I20260709 12:22:54.523181  1872 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:22:54.524050  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:54.526885  2383 leader_election.cc:336] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.942s (SENT)
I20260709 12:22:54.529542  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.515754  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:54.526371  1881 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
W20260709 12:22:54.533428  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.042s (SENT) [suppressed 26 similar messages]
I20260709 12:22:54.490483  4219 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:54.441068  4395 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.498647  4292 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:54.542120  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:54.520951  4385 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.514s	user 0.005s	sys 0.010s
I20260709 12:22:54.500480  4409 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:22:54.550978  1873 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:22:54.552634  4385 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:54.537999  4208 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.648s	user 0.013s	sys 0.000s
W20260709 12:22:54.560472  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 60105.2us
I20260709 12:22:54.522013  4413 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:54.539930  4380 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:54.540417  4219 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.555512  4292 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8.
I20260709 12:22:54.512066  4419 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.077s
I20260709 12:22:54.560467  4297 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:54.607345  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 73936.1us
W20260709 12:22:54.608804  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.170s (SENT)
I20260709 12:22:54.613517  4409 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:54.537339  4072 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.535218  4390 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 12:22:54.604396  4219 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:54.627580  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 105597us
I20260709 12:22:54.632097  1802 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:54.618515  2383 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.129s (SENT)
I20260709 12:22:54.634966  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:54.638319  2383 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.241s (SENT)
I20260709 12:22:54.638898  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.624068  4419 consensus_queue.cc:1243] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:54.640677  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:54.598618  4208 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:54.632616  4212 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.735s	user 0.004s	sys 0.013s
I20260709 12:22:54.644104  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.631553  4035 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:22:54.661782  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.614351  4409 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.642660  4380 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:54.625725  4433 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 12:22:54.648388  4292 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:22:54.687026  4403 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:54.649128  4297 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.650029  4417 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:22:54.600174  4413 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.700412  4417 consensus_queue.cc:1243] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:54.700991  4417 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:54.701295  4417 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:54.641713  4247 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:54.691236  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 638) took 2380 ms. Trace:
I20260709 12:22:54.706079  2414 rpcz_store.cc:276] 0709 12:22:52.310731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.310893 (+   162us) service_pool.cc:224] Handling call
0709 12:22:52.341959 (+ 31066us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.018327 (+676368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.690916 (+1672589us) spinlock_profiling.cc:243] Waited 675 ms on lock 0x7b54000e6d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:54.691164 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1553431808}
W20260709 12:22:54.707475  4420 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:22:54.665230  4212 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.693588  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 538) took 1848 ms. Trace:
I20260709 12:22:54.644625  4038 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 LEARNER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:54.710299  2411 rpcz_store.cc:276] 0709 12:22:52.845652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.845808 (+   156us) service_pool.cc:224] Handling call
0709 12:22:53.011382 (+165574us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.011957 (+   575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.693311 (+1681354us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54005609d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:54.693471 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379910016}
I20260709 12:22:54.713245  4212 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:54.713856  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52194.5us
I20260709 12:22:54.664825  4419 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:54.713466  4317 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:54.702178  4417 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.716122  2411 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:54.727731  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.728714  4417 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:54.623274  4389 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:54.717615  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 73572.5us
I20260709 12:22:54.750679  2414 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:54.752163  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:54.753547  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:54.755683  2419 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:22:54.774115  2419 pending_rounds.cc:85] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 5
I20260709 12:22:54.774459  2419 pending_rounds.cc:107] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:22:54.774745  2419 raft_consensus.cc:2889] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:54.758925  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:54.727365  4290 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:54.771777  4380 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:54.784016  4368 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:54.784704  4368 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:54.787628  4072 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:54.736037  4037 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:54.792860  4387 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.949s	user 0.002s	sys 0.000s
I20260709 12:22:54.794551  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 648) took 1157 ms. Trace:
I20260709 12:22:54.795185  2418 rpcz_store.cc:276] 0709 12:22:53.636775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.636927 (+   152us) service_pool.cc:224] Handling call
0709 12:22:53.637273 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:53.637623 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:53.637651 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:53.658372 (+ 20721us) log.cc:844] Serialized 26 byte log entry
0709 12:22:53.747161 (+ 88789us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:53.747221 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.747314 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:54.791983 (+1044669us) spinlock_profiling.cc:243] Waited 7.11 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:54.794193 (+  2210us) raft_consensus.cc:1685] finished
0709 12:22:54.794216 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.794481 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13120,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":3209,"spinlock_wait_cycles":16364288,"thread_start_us":79919,"threads_started":1,"wal-append.queue_time_us":167715}
W20260709 12:22:54.798053  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.794543  4419 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:54.715238  4413 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:54.736992  4038 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
W20260709 12:22:54.646652  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78342.6us
I20260709 12:22:54.785832  4368 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.829620  4368 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:54.821383  4419 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.832962  4419 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:54.770655  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:54.837220  2411 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:54.827278  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:54.844403  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 300879us
I20260709 12:22:54.847745  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:54.699287  4288 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
W20260709 12:22:54.842044  4436 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:22:54.852146  2419 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:22:54.853713  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 478887us
I20260709 12:22:54.842044  4413 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:54.859298  2411 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:22:54.865237  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.896123  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.862654  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:54.851636  4433 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:54.827682  4037 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 12:22:54.888645  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:54.899641  1771 leader_election.cc:400] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5.
I20260709 12:22:54.921068  1771 leader_election.cc:403] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
W20260709 12:22:54.900972  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58225.4us
I20260709 12:22:54.903196  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:54.870189  4388 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.920857  1800 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:54.921476  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0314842809998481s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
I20260709 12:22:54.922053  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:54.895478  4394 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.006s	sys 0.024s
I20260709 12:22:54.929769  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:54.934458  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.890350  4428 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.896099  4398 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.936882  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:54.963152  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:54.920835  4405 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:54.920497  4222 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:54.933373  4394 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.006s	sys 0.025s
I20260709 12:22:54.930022  4290 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:54.975332  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45717.5us
I20260709 12:22:54.980844  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:54.921319  4429 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:54.973119  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:54.963935  4072 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:54.920176  4431 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
W20260709 12:22:54.978542  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1449) took 1444 ms (client timeout 1525 ms). Trace:
I20260709 12:22:54.925976  4433 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.988481  4429 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.937894  4388 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.975495  4288 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:54.973395  4239 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:54.973296  4413 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:55.016134  4413 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:55.015868  4219 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.990234  4419 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:22:54.998999  1798 rpcz_store.cc:269] 0709 12:22:53.534597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.535478 (+   881us) service_pool.cc:224] Handling call
0709 12:22:54.978324 (+1442846us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b5400132254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:54.978476 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2067977472}
I20260709 12:22:54.976068  4212 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:54.985298  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:55.005334  1800 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:54.981366  4222 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:54.973976  4398 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:55.003899  4342 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.939s	user 0.000s	sys 0.012s
I20260709 12:22:55.011314  4239 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:55.016707  4413 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:54.973582  4232 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:55.023756  4432 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:55.024505  4212 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.984350  4290 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:54.969542  4428 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:55.029330  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:55.030644  2413 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
W20260709 12:22:55.019239  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47400.5us
W20260709 12:22:55.032240  4220 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.794s	user 0.004s	sys 0.008s
I20260709 12:22:55.032902  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1528) took 1474 ms. Trace:
I20260709 12:22:55.053159  1725 rpcz_store.cc:276] 0709 12:22:53.558484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.770243 (+211759us) service_pool.cc:224] Handling call
0709 12:22:55.032565 (+1262322us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b540044aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:55.032824 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":235843840}
W20260709 12:22:55.042053  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56839us
I20260709 12:22:55.044034  4222 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:55.032258  4126 raft_consensus.cc:1064] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:22:55.050920  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:55.054288  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:55.057462  1873 pending_rounds.cc:85] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 5
I20260709 12:22:55.064122  1873 pending_rounds.cc:107] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:22:55.064482  1873 raft_consensus.cc:2889] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:55.038763  4239 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:55.015969  4441 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:55.075568  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:55.076869  1876 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:55.077641  1798 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:55.056382  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:55.065259  1800 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:55.039633  4409 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:55.081310  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:55.083163  1872 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:55.049240  4212 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:55.047200  4432 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:55.052933  4288 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:55.063380  4295 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.378s	user 0.014s	sys 0.002s
W20260709 12:22:55.035888  4342 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.971s	user 0.000s	sys 0.013s
I20260709 12:22:55.056780  4208 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:55.105650  2383 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.787s (SENT)
I20260709 12:22:55.109927  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:55.113376  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:55.114199  1725 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:22:55.090901  4359 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.107764  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:55.126832  1876 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:55.062175  4220 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:55.150254  1800 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:22:55.326756  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 334) took 1769 ms. Trace:
I20260709 12:22:55.327364  1800 rpcz_store.cc:276] 0709 12:22:53.557350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.104259 (+1546909us) service_pool.cc:224] Handling call
0709 12:22:55.143567 (+ 39308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.326391 (+182824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.326666 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:55.297943  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 143328us
I20260709 12:22:55.266695  1725 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:22:55.284385  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 331) took 1727 ms. Trace:
I20260709 12:22:55.332779  1798 rpcz_store.cc:276] 0709 12:22:53.556516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.077020 (+1520504us) service_pool.cc:224] Handling call
0709 12:22:55.077392 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.078252 (+   860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.284299 (+206047us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:55.325500  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 269167us
I20260709 12:22:55.346334  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:55.358948  1877 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:55.383893  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54460.3us
I20260709 12:22:55.360863  4220 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:55.225904  4428 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:55.103854  4295 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:55.159547  4432 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:55.364526  4383 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.233s	user 0.002s	sys 0.000s
W20260709 12:22:55.369779  4397 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.307s	user 0.002s	sys 0.001s
I20260709 12:22:55.371156  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1455) took 1832 ms. Trace:
I20260709 12:22:55.416710  1805 rpcz_store.cc:276] 0709 12:22:53.538615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.913325 (+374710us) service_pool.cc:224] Handling call
0709 12:22:54.187550 (+274225us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:54.188364 (+   814us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.370921 (+1182557us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54002c0450. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:55.371086 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":606426880}
I20260709 12:22:55.419261  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1519) took 2401 ms. Trace:
I20260709 12:22:55.402061  4290 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:55.419782  1732 rpcz_store.cc:276] 0709 12:22:53.017994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.793466 (+775472us) service_pool.cc:224] Handling call
0709 12:22:53.835824 (+ 42358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:53.849646 (+ 13822us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:53.849701 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:53.850268 (+   567us) log.cc:844] Serialized 26 byte log entry
0709 12:22:53.971542 (+121274us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:53.971580 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:53.971646 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:55.369162 (+1397516us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:22:55.418876 (+ 49714us) raft_consensus.cc:1685] finished
0709 12:22:55.418907 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.419189 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7549,"raft.queue_time_us":28332,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":9809,"spinlock_wait_cycles":33608064,"thread_start_us":64039,"threads_started":1,"wal-append.queue_time_us":133809}
I20260709 12:22:55.373358  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:55.423305  2411 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:55.418709  1805 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:22:55.435191  1872 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:55.436869  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:55.437903  1872 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:55.101728  4219 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:55.307746  4292 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.027s	sys 0.107s
I20260709 12:22:55.123976  4409 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:22:55.125070  4281 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:55.290088  4386 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.044s	sys 0.077s
W20260709 12:22:55.540200  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 193933us
I20260709 12:22:55.397812  1876 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
W20260709 12:22:55.400863  2383 leader_election.cc:336] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.657s (SENT)
I20260709 12:22:55.130494  4288 raft_consensus.cc:2764] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:22:55.282477  4388 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:55.507933  1805 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:55.509128  1800 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:55.296504  4433 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:55.526296  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:55.542152  1696 leader_election.cc:400] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8.
W20260709 12:22:55.543009  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 142174us
W20260709 12:22:55.578351  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52182.1us
W20260709 12:22:55.578877  2383 leader_election.cc:336] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.743s (SENT)
I20260709 12:22:55.579447  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:55.579602  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:55.579522  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:22:55.555044  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.011s	sys 0.017s
W20260709 12:22:55.555464  1695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.006s	sys 0.022s
I20260709 12:22:55.529518  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:55.581848  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:55.543948  1723 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
W20260709 12:22:55.532393  4292 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.833s	user 0.032s	sys 0.108s
I20260709 12:22:55.582747  1723 pending_rounds.cc:85] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 5
I20260709 12:22:55.587558  1723 pending_rounds.cc:107] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:22:55.587937  1723 raft_consensus.cc:2889] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:55.581871  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.011s	sys 0.017s
W20260709 12:22:55.583356  1695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.006s	sys 0.022s
I20260709 12:22:55.584333  2413 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:22:55.530982  4281 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.591388  4281 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:55.591238  4212 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:55.591913  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1543) took 2581 ms (client timeout 2214 ms). Trace:
W20260709 12:22:55.592427  1807 rpcz_store.cc:269] 0709 12:22:53.010229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.010411 (+   182us) service_pool.cc:224] Handling call
0709 12:22:55.591634 (+2581223us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b540009fbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:55.591823 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":110335360}
I20260709 12:22:55.593672  4212 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:55.594866  1806 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:55.596840  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1459) took 2043 ms. Trace:
I20260709 12:22:55.597194  1806 rpcz_store.cc:276] 0709 12:22:53.553795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.984066 (+430271us) service_pool.cc:224] Handling call
0709 12:22:55.594629 (+1610563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.595383 (+   754us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.596605 (+  1222us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b540009fbd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:55.596777 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3677870336}
W20260709 12:22:55.540288  4386 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.045s	sys 0.079s
I20260709 12:22:55.602360  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:55.585153  4294 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:55.585340  4239 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.519s	user 0.010s	sys 0.002s
I20260709 12:22:55.586328  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:55.143831  4359 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:22:55.590005  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.600684  1800 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:55.601051  4413 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.547s	user 0.011s	sys 0.000s
W20260709 12:22:55.614003  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.586984  4218 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:55.604346  1847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.000s	sys 0.029s
I20260709 12:22:55.604807  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:55.577421  4290 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:55.586474  4215 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:55.586371  4388 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:22:55.615490  1879 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.011s	sys 0.018s
I20260709 12:22:55.587417  4292 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:55.616796  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:22:55.617497  1847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.000s	sys 0.030s
W20260709 12:22:55.617798  1875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.012s	sys 0.086s
I20260709 12:22:55.621836  1723 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:55.628556  1723 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:55.611760  4288 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:55.631102  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.002s	sys 0.020s
I20260709 12:22:55.609257  4398 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:55.629539  1723 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:55.631244  4288 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:55.640254  1723 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:55.640555  4288 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:55.641991  4232 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.595s	user 0.010s	sys 0.001s
I20260709 12:22:55.611724  4239 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:55.644984  4295 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:55.645341  4239 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:55.646188  1798 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:55.648658  4359 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:22:55.630323  4388 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:55.621445  4413 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:55.652171  4388 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:55.653751  4413 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:55.623255  4286 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:55.655468  4215 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:55.656720  4215 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:55.663148  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:55.664074  1872 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:55.675590  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 335) took 1317 ms. Trace:
W20260709 12:22:55.630806  1875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.012s	sys 0.087s
I20260709 12:22:55.613449  4272 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:55.642529  4232 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5.
W20260709 12:22:55.631718  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.003s	sys 0.020s
W20260709 12:22:55.680341  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63665us
I20260709 12:22:55.681735  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:22:55.624336  1879 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.011s	sys 0.018s
W20260709 12:22:55.683503  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 703001us
I20260709 12:22:55.692096  1872 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:55.707331  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:55.716341  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.775s	user 0.006s	sys 0.030s
W20260709 12:22:55.717947  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.006s	sys 0.030s
W20260709 12:22:55.718690  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 796657us
I20260709 12:22:55.719568  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:55.710711  4389 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.869s	user 0.005s	sys 0.022s
I20260709 12:22:55.695874  4290 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:55.740998  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:55.742102  1873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.011s	sys 0.017s
I20260709 12:22:55.716917  4226 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:55.749504  1873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.011s	sys 0.017s
I20260709 12:22:55.712041  4454 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:55.721228  4239 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:55.670356  4292 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:55.751295  4239 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:55.752548  4239 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:55.758392  2417 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.867s	user 0.001s	sys 0.029s
W20260709 12:22:55.722008  4431 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.014s	sys 0.021s
W20260709 12:22:55.744665  4389 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.901s	user 0.006s	sys 0.022s
W20260709 12:22:55.760607  2417 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.002s	sys 0.029s
I20260709 12:22:55.797698  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 681) took 1037 ms. Trace:
I20260709 12:22:55.762346  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:55.800663  1872 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:55.798308  2417 rpcz_store.cc:276] 0709 12:22:54.760241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.864774 (+104533us) service_pool.cc:224] Handling call
0709 12:22:54.865274 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.865541 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:54.890685 (+ 25144us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.797086 (+906401us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:55.797187 (+   101us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:55.797273 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.797390 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.797603 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":29051,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"thread_start_us":905848,"threads_started":1}
W20260709 12:22:55.745186  4394 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.746s	user 0.024s	sys 0.047s
W20260709 12:22:55.764585  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 83008.2us
W20260709 12:22:55.765849  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.717605  4417 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:22:55.809831  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 334) took 1501 ms. Trace:
I20260709 12:22:55.810283  4386 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.745461  4290 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:55.747677  4413 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:55.755657  4226 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:55.811954  4394 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.024s	sys 0.048s
I20260709 12:22:55.793290  1723 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:55.806643  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:55.810454  1726 rpcz_store.cc:276] 0709 12:22:54.308883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.309071 (+   188us) service_pool.cc:224] Handling call
0709 12:22:54.309376 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.309581 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:54.425578 (+115997us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.809214 (+1383636us) spinlock_profiling.cc:243] Waited 4.5 ms on lock 0x7b180035f6e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287
0709 12:22:55.809350 (+   136us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:55.809418 (+    68us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:55.809488 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.809587 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.809755 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8309,"prepare.queue_time_us":321363,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":60,"raft.queue_time_us":84937,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":6097,"spinlock_wait_cycles":10340096,"thread_start_us":157372,"threads_started":2}
I20260709 12:22:55.833895  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:55.835686  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:55.755736  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:55.813687  4386 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:55.814086  4422 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:55.780026  4281 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:55.860798  4281 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:55.818636  4290 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:55.871361  4452 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:22:55.873301  4283 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.873940  4283 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:55.877980  4283 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:55.794051  4431 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.776s	user 0.017s	sys 0.024s
W20260709 12:22:56.451833  1770 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.850s	user 0.000s	sys 0.071s
W20260709 12:22:55.889645  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 83139.5us
W20260709 12:22:56.523981  1770 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.000s	sys 0.077s
W20260709 12:22:56.524358  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 923111us
W20260709 12:22:56.067572  1874 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.987s	user 0.021s	sys 0.049s
W20260709 12:22:56.525920  1874 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.445s	user 0.035s	sys 0.069s
I20260709 12:22:56.526705  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2260) took 1469 ms. Trace:
I20260709 12:22:55.884033  1724 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:56.119632  4396 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s	user 0.014s	sys 0.030s
W20260709 12:22:56.532564  4396 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.311s	user 0.014s	sys 0.041s
I20260709 12:22:56.538486  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:22:56.310009  1804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.008s	sys 0.054s
W20260709 12:22:56.120703  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1451) took 2584 ms (client timeout 1711 ms). Trace:
W20260709 12:22:56.539778  1804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.945s	user 0.012s	sys 0.066s
W20260709 12:22:56.539964  1803 rpcz_store.cc:269] 0709 12:22:53.536384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.536520 (+   136us) service_pool.cc:224] Handling call
0709 12:22:56.120285 (+2583765us) spinlock_profiling.cc:243] Waited 858 ms on lock 0x7b540013a6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:56.120568 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1972905216}
W20260709 12:22:56.059930  1727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.956s	user 0.015s	sys 0.043s
I20260709 12:22:55.676340  1730 rpcz_store.cc:276] 0709 12:22:54.357970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.376707 (+ 18737us) service_pool.cc:224] Handling call
0709 12:22:54.451216 (+ 74509us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.511450 (+ 60234us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:54.713526 (+202076us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b18003456a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf 00007f16abdf2038
0709 12:22:54.904638 (+191112us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.610366 (+705728us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.611074 (+   708us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.611308 (+   234us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:55.611359 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.611459 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:55.675164 (+ 63705us) raft_consensus.cc:1685] finished
0709 12:22:55.675289 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.675501 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":149138,"prepare.queue_time_us":141613,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":108403,"raft.run_cpu_time_us":493,"raft.run_wall_time_us":31762,"spinlock_wait_cycles":7602048,"thread_start_us":874686,"threads_started":2,"wal-append.queue_time_us":650665}
W20260709 12:22:56.541414  1727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.445s	user 0.025s	sys 0.068s
W20260709 12:22:55.985435  1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2380) took 2450 ms (client timeout 1667 ms). Trace:
I20260709 12:22:56.528823  2418 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:56.542207  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:55.883235  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:56.178902  1880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.050s	user 0.002s	sys 0.030s
W20260709 12:22:56.545109  1880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.416s	user 0.017s	sys 0.040s
W20260709 12:22:56.544840  1729 rpcz_store.cc:269] 0709 12:22:53.535307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.584250 (+ 48943us) service_pool.cc:224] Handling call
0709 12:22:55.985137 (+2400887us) spinlock_profiling.cc:243] Waited 30.5 ms on lock 0x7b540030b454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:55.985344 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":70150528}
W20260709 12:22:56.271692  1732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.005s	sys 0.043s
W20260709 12:22:56.546697  1732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.017s	user 0.008s	sys 0.046s
I20260709 12:22:56.546890  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:56.547540  2416 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:55.986534  1801 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:55.980785  2385 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.006s	sys 0.015s
I20260709 12:22:56.519788  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1538) took 3771 ms. Trace:
W20260709 12:22:56.548460  2385 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.236s	user 0.013s	sys 0.025s
I20260709 12:22:56.548735  1799 rpcz_store.cc:276] 0709 12:22:52.748788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.773113 (+ 24325us) service_pool.cc:224] Handling call
0709 12:22:52.794114 (+ 21001us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.756728 (+962614us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.119764 (+2363036us) spinlock_profiling.cc:243] Waited 934 ms on lock 0x7b540004d8d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:56.519718 (+399954us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2147413376}
W20260709 12:22:55.961825  4426 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:22:55.964932  4226 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:56.552469  4226 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:56.527140  1874 rpcz_store.cc:276] 0709 12:22:55.056928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.057073 (+   145us) service_pool.cc:224] Handling call
0709 12:22:55.057342 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.057553 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:55.080539 (+ 22986us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.526361 (+1445822us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.526420 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:56.526472 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.526539 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.526653 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":335530,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":17137,"thread_start_us":1445391,"threads_started":1}
I20260709 12:22:55.879141  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:56.306324  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:56.522949  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 767252us
W20260709 12:22:56.109488  4382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.987s	user 0.011s	sys 0.017s
I20260709 12:22:56.543833  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:56.245507  1881 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.014s	sys 0.033s
I20260709 12:22:56.025496  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:56.157842  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:55.985999  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 336) took 2346 ms (client timeout 1513 ms). Trace:
W20260709 12:22:56.331578  4457 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
W20260709 12:22:56.245034  4456 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20260709 12:22:56.522446  1726 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:56.547667  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1549) took 1550 ms. Trace:
W20260709 12:22:55.952107  4297 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.918s	user 0.016s	sys 0.035s
W20260709 12:22:55.893191  4395 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.868s	user 0.022s	sys 0.023s
I20260709 12:22:56.543582  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:55.889895  4368 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:22:56.557778  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:56.560189  4382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.445s	user 0.015s	sys 0.024s
I20260709 12:22:56.603210  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:56.603996  2415 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:22:56.562278  1881 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.169s	user 0.019s	sys 0.035s
I20260709 12:22:56.580375  2412 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:22:56.589370  1732 rpcz_store.cc:276] 0709 12:22:54.996888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.421880 (+424992us) service_pool.cc:224] Handling call
0709 12:22:55.422287 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.422545 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:55.529193 (+106648us) log.cc:844] Serialized 14 byte log entry
0709 12:22:56.547213 (+1018020us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:56.547273 (+    60us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:56.547409 (+   136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.547479 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.547611 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":116404,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":114,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":123,"thread_start_us":1017649,"threads_started":1}
I20260709 12:22:56.595284  1730 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:22:56.567683  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:56.567900  1802 rpcz_store.cc:269] 0709 12:22:53.640111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.641215 (+  1104us) service_pool.cc:224] Handling call
0709 12:22:55.985785 (+2344570us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54003b74d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:55.985926 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2276091648}
I20260709 12:22:56.560686  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:56.579238  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:56.617933  2417 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:56.556968  2416 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:56.600040  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:22:56.621467  1725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.037s	user 0.017s	sys 0.019s
W20260709 12:22:56.622587  1725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.038s	user 0.017s	sys 0.019s
I20260709 12:22:56.558454  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:56.624126  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63518.4us
I20260709 12:22:56.538563  4386 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:56.562561  4288 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.921s	user 0.014s	sys 0.002s
W20260709 12:22:56.596545  4395 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.569s	user 0.027s	sys 0.032s
W20260709 12:22:56.576539  4290 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.709s	user 0.013s	sys 0.000s
I20260709 12:22:56.583631  4426 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.625411  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:56.555333  4226 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.719503  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:56.741089  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2275) took 1078 ms. Trace:
I20260709 12:22:56.594496  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:56.662762  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:56.602551  4368 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:56.586269  4419 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.538s	user 0.040s	sys 0.038s
I20260709 12:22:56.728921  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:56.741592  1874 rpcz_store.cc:276] 0709 12:22:55.662434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.631252 (+968818us) service_pool.cc:224] Handling call
0709 12:22:56.740981 (+109729us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:56.725778  4312 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.875s	user 0.044s	sys 0.043s
W20260709 12:22:56.738847  1873 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.984s	user 0.000s	sys 0.039s
W20260709 12:22:56.770233  1873 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.016s	user 0.000s	sys 0.040s
I20260709 12:22:56.724258  4431 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:56.735148  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 109910us
W20260709 12:22:56.593478  4297 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.559s	user 0.022s	sys 0.051s
I20260709 12:22:56.628254  4459 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:22:56.677690  4388 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.011s	user 0.014s	sys 0.040s
I20260709 12:22:56.778061  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:56.749711  4456 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:22:56.770083  4312 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.937s	user 0.045s	sys 0.044s
W20260709 12:22:56.758047  4419 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.717s	user 0.043s	sys 0.052s
I20260709 12:22:56.805408  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:56.807133  1874 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:56.732705  4288 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:56.795320  4388 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.128s	user 0.014s	sys 0.040s
I20260709 12:22:56.740191  4290 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:56.820807  1872 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:56.755259  4426 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:56.850016  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 313) took 2409 ms. Trace:
I20260709 12:22:56.845826  4368 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:56.845571  4463 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.062s	user 0.002s	sys 0.000s
I20260709 12:22:56.770903  4226 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:56.788748  4431 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:56.776567  4395 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:56.853781  4422 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.997s	user 0.005s	sys 0.010s
W20260709 12:22:56.851024  4292 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.073s	user 0.008s	sys 0.039s
I20260709 12:22:56.860658  1875 rpcz_store.cc:276] 0709 12:22:54.440823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.467240 (+ 26417us) service_pool.cc:224] Handling call
0709 12:22:54.498172 (+ 30932us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.904821 (+406649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:54.904862 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:54.905501 (+   639us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.677489 (+771988us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:55.738805 (+ 61316us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.739174 (+   369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.739285 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.848069 (+1108784us) raft_consensus.cc:1685] finished
0709 12:22:56.848110 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.849901 (+  1791us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8573,"prepare.queue_time_us":70297,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":2736,"raft.queue_time_us":105095,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":15359,"spinlock_wait_cycles":16831744,"thread_start_us":1135385,"threads_started":2,"wal-append.queue_time_us":1153627}
I20260709 12:22:56.856642  4458 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:56.911486  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1631) took 4120 ms (client timeout 2214 ms). Trace:
W20260709 12:22:56.912087  1878 rpcz_store.cc:269] 0709 12:22:52.791345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:52.921610 (+130265us) service_pool.cc:224] Handling call
0709 12:22:56.911198 (+3989588us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:56.911390 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3004347648}
W20260709 12:22:56.897032  4292 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.143s	user 0.008s	sys 0.040s
I20260709 12:22:56.916644  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:56.882458  4422 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:56.920930  1875 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:22:56.917155  4469 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:22:56.930889  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1548) took 1934 ms. Trace:
I20260709 12:22:56.944983  1725 rpcz_store.cc:276] 0709 12:22:54.996564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.399960 (+403396us) service_pool.cc:224] Handling call
0709 12:22:55.511488 (+111528us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:55.566129 (+ 54641us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.566166 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.581516 (+ 15350us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.715631 (+1134115us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:56.717353 (+  1722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.717475 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:56.930560 (+213085us) raft_consensus.cc:1685] finished
0709 12:22:56.930585 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:56.930806 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61046,"raft.queue_time_us":54382,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":8887,"thread_start_us":1131011,"threads_started":1,"wal-append.queue_time_us":1240205}
I20260709 12:22:56.924351  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:22:56.925105  1770 connection.cc:256] RPC call timeout handler was delayed by -1.1947489240001232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15280000000000002s for a response
I20260709 12:22:56.924376  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:56.949257  1878 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:56.948536  1770 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:56.954331  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:56.958930  4312 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:22:56.959617  4475 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:22:56.959877  4312 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:22:56.960933  4467 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:22:56.964275  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:56.965312  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2273) took 1303 ms. Trace:
I20260709 12:22:56.965721  1878 rpcz_store.cc:276] 0709 12:22:55.661594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.950000 (+1288406us) service_pool.cc:224] Handling call
0709 12:22:56.965234 (+ 15234us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:56.952543  4239 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.085s	user 0.014s	sys 0.028s
I20260709 12:22:57.042950  4288 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.020651  4382 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:56.973768  4468 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:22:56.961617  4248 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
W20260709 12:22:57.014127  4472 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:22:56.943444  4471 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:56.953591  4388 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.928464  4166 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:22:56.968387  1872 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:22:57.037673  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 474168us
I20260709 12:22:56.928877  4234 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:22:57.041342  4465 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.229s	user 0.001s	sys 0.001s
I20260709 12:22:56.985366  1874 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:22:57.141782  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2270) took 1609 ms. Trace:
I20260709 12:22:57.142212  1874 rpcz_store.cc:276] 0709 12:22:55.532966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.791735 (+1258769us) service_pool.cc:224] Handling call
0709 12:22:57.141720 (+349985us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:57.142761  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.002s	sys 0.014s
W20260709 12:22:57.038254  4470 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
W20260709 12:22:57.145296  1881 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.000s	sys 0.017s
W20260709 12:22:57.131245  4474 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.250s	user 0.001s	sys 0.001s
W20260709 12:22:57.145828  1881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.000s	sys 0.017s
I20260709 12:22:57.022382  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1463) took 3467 ms. Trace:
W20260709 12:22:56.996228  4215 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.333s	user 0.064s	sys 0.097s
I20260709 12:22:56.996812  4290 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.088279  4426 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:57.129783  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.000s	sys 0.013s
I20260709 12:22:57.131475  4431 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:57.129419  1695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.005s	sys 0.008s
W20260709 12:22:57.115846  4239 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.249s	user 0.015s	sys 0.029s
W20260709 12:22:57.151363  1695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.006s	sys 0.008s
W20260709 12:22:57.139497  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:57.151822  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 371010us
W20260709 12:22:57.143646  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.003s	sys 0.014s
W20260709 12:22:57.161537  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 569608us
W20260709 12:22:57.117641  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.003s	sys 0.005s
I20260709 12:22:57.164108  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2269) took 2085 ms. Trace:
W20260709 12:22:57.147787  4215 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.484s	user 0.064s	sys 0.128s
W20260709 12:22:57.159427  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 201452us
I20260709 12:22:57.166916  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:57.140637  4471 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:57.160657  2418 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.007s	sys 0.006s
W20260709 12:22:57.158313  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.147293  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2262) took 2087 ms. Trace:
I20260709 12:22:57.156324  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:57.146699  1804 rpcz_store.cc:276] 0709 12:22:53.554927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.528328 (+973401us) service_pool.cc:224] Handling call
0709 12:22:54.528672 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.528913 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:54.559437 (+ 30524us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.594192 (+1034755us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b180044fd40. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:22:57.020339 (+1426147us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.020399 (+    60us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:57.020450 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.022113 (+  1663us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.022305 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":24561,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":8640,"raft.queue_time_us":139577,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":9953,"spinlock_wait_cycles":16938752,"thread_start_us":1425794,"threads_started":2,"wal-append.queue_time_us":2305324}
W20260709 12:22:57.164456  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.003s	sys 0.006s
I20260709 12:22:57.164853  1874 rpcz_store.cc:276] 0709 12:22:55.078343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.147649 (+2069306us) service_pool.cc:224] Handling call
0709 12:22:57.147990 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.148207 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:57.157687 (+  9480us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.163714 (+  6027us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.163778 (+    64us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:57.163833 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.163905 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.164038 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":9301,"thread_start_us":5073,"threads_started":1}
I20260709 12:22:57.164814  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2394) took 1383 ms. Trace:
I20260709 12:22:57.147832  4413 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:57.186393  1729 rpcz_store.cc:276] 0709 12:22:55.781996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.546489 (+764493us) service_pool.cc:224] Handling call
0709 12:22:56.546877 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.547123 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:56.921452 (+374329us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.164431 (+242979us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.164496 (+    65us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:57.164538 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.164601 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.164739 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":569836,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":2032,"raft.queue_time_us":272,"thread_start_us":613401,"threads_started":2}
I20260709 12:22:57.187026  1881 rpcz_store.cc:276] 0709 12:22:55.059863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.060018 (+   155us) service_pool.cc:224] Handling call
0709 12:22:55.060429 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.060707 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:56.611394 (+1550687us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.146710 (+535316us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.146839 (+   129us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:57.146913 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.147019 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.147216 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20063,"prepare.queue_time_us":358299,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"raft.queue_time_us":1382,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":4319,"thread_start_us":1752096,"threads_started":2}
I20260709 12:22:57.189561  1697 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:57.190212  4466 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:22:57.190940  1772 connection.cc:256] RPC call timeout handler was delayed by -1.5810835429997496s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1585s for a response
W20260709 12:22:57.204084  1772 connection.cc:256] RPC call timeout handler was delayed by -1.2633835429996907s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1938s for a response
W20260709 12:22:57.204630  1772 connection.cc:256] RPC call timeout handler was delayed by -1.1872392909999689s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
W20260709 12:22:57.205021  1772 connection.cc:256] RPC call timeout handler was delayed by -1.1256835429999228s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20910000000000004s for a response
W20260709 12:22:57.205423  1772 connection.cc:256] RPC call timeout handler was delayed by -1.0922279060000619s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2053s for a response
I20260709 12:22:57.191375  4487 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:57.188905  2418 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.008s	sys 0.006s
I20260709 12:22:57.183748  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:57.196041  4481 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.197s	user 0.000s	sys 0.000s
W20260709 12:22:57.197161  4483 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:22:57.208791  1880 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.000s	sys 0.015s
I20260709 12:22:57.144593  4234 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.158432  4371 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:57.158358  4263 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:57.164108  4292 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.218640  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1635) took 1592 ms. Trace:
I20260709 12:22:57.231532  1881 rpcz_store.cc:276] 0709 12:22:55.626553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.202450 (+1575897us) service_pool.cc:224] Handling call
0709 12:22:57.202895 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.203150 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.203199 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.203260 (+    61us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:57.218229 (+ 14969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.218351 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.218537 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.140101  4388 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.237370  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2258) took 2182 ms. Trace:
W20260709 12:22:57.238708  1880 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.000s	sys 0.016s
I20260709 12:22:57.233628  1881 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:57.138262  4382 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.226747  1772 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:57.274570  4226 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.234480  4371 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.287238  4266 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:57.287565  4506 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:22:57.268713  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1636) took 1641 ms. Trace:
I20260709 12:22:57.237964  1873 rpcz_store.cc:276] 0709 12:22:55.055377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.055518 (+   141us) service_pool.cc:224] Handling call
0709 12:22:55.057228 (+  1710us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:55.075476 (+ 18248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.075505 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:55.092490 (+ 16985us) log.cc:844] Serialized 52 byte log entry
0709 12:22:55.751700 (+659210us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:55.964708 (+213008us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b48000ce7d8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:22:56.775452 (+810744us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.131250 (+355798us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.131573 (+   323us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.131651 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.237059 (+105408us) raft_consensus.cc:1685] finished
0709 12:22:57.237087 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.237288 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":99828,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":116,"raft.queue_time_us":49279,"raft.run_cpu_time_us":2812,"raft.run_wall_time_us":897683,"spinlock_wait_cycles":33611008,"thread_start_us":1993003,"threads_started":3,"wal-append.queue_time_us":1519073}
I20260709 12:22:57.265715  4263 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.163352  4290 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 12:22:57.291595  4263 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.292042  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:57.289266  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 60203.1us
I20260709 12:22:57.296766  4272 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:57.297384  4272 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:57.219476  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:57.299036  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:57.271004  2414 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:57.276546  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2257) took 2222 ms. Trace:
I20260709 12:22:57.207831  4466 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.303918  4466 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:57.273114  4500 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:57.224449  4487 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.284215  1731 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 5 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:57.309684  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.301064  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:57.301865  4502 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:57.302187  4458 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.266960  4431 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:57.291754  1881 rpcz_store.cc:276] 0709 12:22:55.626927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.232986 (+1606059us) service_pool.cc:224] Handling call
0709 12:22:57.233382 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.268395 (+ 35013us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.268644 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.312155  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:22:57.312402  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 93098.4us
W20260709 12:22:57.326478  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 164253us
I20260709 12:22:57.327481  1845 leader_election.cc:400] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:57.327817  1845 leader_election.cc:403] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:57.288970  4371 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.290584  1873 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:57.198515  4489 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:57.266225  4419 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:57.329339  2383 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.678s (SENT)
I20260709 12:22:57.235328  4239 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20260709 12:22:57.281394  4234 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.308578  1880 rpcz_store.cc:276] 0709 12:22:55.054272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.054447 (+   175us) service_pool.cc:224] Handling call
0709 12:22:55.054872 (+   425us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:55.055358 (+   486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.055385 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:55.075831 (+ 20446us) log.cc:844] Serialized 52 byte log entry
0709 12:22:56.545900 (+1470069us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:56.546216 (+   316us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.239421 (+693205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.239547 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.270184 (+ 30637us) raft_consensus.cc:1685] finished
0709 12:22:57.270761 (+   577us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.276438 (+  5677us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30581,"prepare.queue_time_us":204373,"prepare.run_cpu_time_us":1741,"prepare.run_wall_time_us":5473,"raft.queue_time_us":53070,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":34251,"thread_start_us":2089458,"threads_started":2,"wal-append.queue_time_us":1664653}
I20260709 12:22:57.277297  4499 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.290306  4429 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:57.334517  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.837s (SENT)
I20260709 12:22:57.336473  4388 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.338302  3922 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5.
I20260709 12:22:57.287632  4266 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:57.277693  4378 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
I20260709 12:22:57.284811  4461 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.321867  4500 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:57.340320  4413 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:57.291363  4290 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:57.356820  4290 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:57.333644  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:57.342694  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:57.343011  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1461) took 3788 ms. Trace:
I20260709 12:22:57.359346  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:57.359177  1801 rpcz_store.cc:276] 0709 12:22:53.554388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.187502 (+633114us) service_pool.cc:224] Handling call
0709 12:22:55.986328 (+1798826us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.548245 (+561917us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.342733 (+794488us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54003b74d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:57.342942 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4136124160}
W20260709 12:22:57.270951  4393 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 2.735s	user 0.033s	sys 0.034s
W20260709 12:22:57.362825  4393 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 2.911s	user 0.034s	sys 0.034s
I20260709 12:22:57.364017  4471 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:57.328567  4480 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:22:57.323091  4459 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.320186  4395 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.347846  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:57.383546  1880 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:22:57.334192  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1637) took 1707 ms. Trace:
I20260709 12:22:57.266516  4292 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.346662  4419 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.328542  4215 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:22:57.352378  2383 leader_election.cc:336] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.033s (SENT)
I20260709 12:22:57.331101  4368 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:57.389058  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:57.390625  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.328935  4458 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.366300  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:57.354841  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2271) took 1813 ms. Trace:
I20260709 12:22:57.357463  4500 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.384495  1873 rpcz_store.cc:276] 0709 12:22:55.627374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.289935 (+1662561us) service_pool.cc:224] Handling call
0709 12:22:57.290375 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.333914 (+ 43539us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.334136 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.385905  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:57.344416  4422 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260709 12:22:57.387518  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:57.389762  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.042s (SENT)
I20260709 12:22:57.392262  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:57.393909  1729 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:57.367636  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:57.405120  1726 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:57.394356  1874 rpcz_store.cc:276] 0709 12:22:55.541310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.186341 (+1645031us) service_pool.cc:224] Handling call
0709 12:22:57.193480 (+  7139us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.193747 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:57.230509 (+ 36762us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.353719 (+123210us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.353778 (+    59us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:57.353844 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.354649 (+   805us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.354760 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13834,"prepare.queue_time_us":44163,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":4407,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":27566,"thread_start_us":119116,"threads_started":1}
I20260709 12:22:57.373104  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:57.366888  4388 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:57.408023  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:57.409103  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.409756  1726 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:57.410497  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.411468  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:57.383579  4371 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:57.414295  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:57.448508  1881 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:22:57.472682  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 143761us
I20260709 12:22:57.450582  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.474172  1726 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:57.392910  4368 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:57.475266  1726 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:57.475657  1726 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:57.473127  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.448841  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:22:57.369611  4429 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:57.386026  4239 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:57.480070  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:57.351722  4499 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.476409  1726 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.413576  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:57.473564  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59357.3us
W20260709 12:22:57.433722  2383 leader_election.cc:336] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.142s (SENT)
W20260709 12:22:57.483980  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50311.3us
I20260709 12:22:57.484150  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:57.486162  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:57.486928  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:57.480789  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:57.494043  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:57.357645  4461 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.497910  1874 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:57.499549  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:57.404130  4500 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:57.411187  4263 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:57.446453  4215 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:57.483858  4405 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:57.508345  4509 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:22:57.508724  4222 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:57.362180  4290 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.475199  4234 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:57.452639  4266 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.388476  4459 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.392045  4395 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.413653  4471 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20260709 12:22:57.479213  4368 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:57.532220  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.519253  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:57.544989  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 90262.4us
I20260709 12:22:57.511780  4489 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:57.519584  4506 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:57.518726  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.586390  1803 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:57.487949  4429 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:57.599242  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 303) took 3219 ms (client timeout 2033 ms). Trace:
W20260709 12:22:57.599946  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1647) took 1860 ms (client timeout 2031 ms). Trace:
W20260709 12:22:57.600608  1872 rpcz_store.cc:269] 0709 12:22:55.739745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.623853 (+884108us) service_pool.cc:224] Handling call
0709 12:22:57.599712 (+975859us) spinlock_profiling.cc:243] Waited 50.9 ms on lock 0x7b54000f0854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:57.599865 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":117091200}
I20260709 12:22:57.601915  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:57.521831  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.516609  1801 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:22:57.603350  1807 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:57.521425  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.604036  1806 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:57.514961  1874 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:57.518181  1802 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:57.604724  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1639) took 1976 ms. Trace:
I20260709 12:22:57.605204  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.520313  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.606199  1800 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:57.602777  1872 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:22:57.493207  4371 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.498648  4385 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
W20260709 12:22:57.527377  4505 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.287s	user 0.001s	sys 0.000s
I20260709 12:22:57.598169  4266 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:57.490293  4378 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:57.516098  4499 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.564244  4399 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:57.569842  1804 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:57.619649  4458 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.484786  4422 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.078s
I20260709 12:22:57.522158  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.620254  4429 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.620882  1805 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:57.584437  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 552) took 1002 ms. Trace:
I20260709 12:22:57.618619  1872 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:57.523622  4512 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.046s
I20260709 12:22:57.605134  1874 rpcz_store.cc:276] 0709 12:22:55.628146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.512535 (+1884389us) service_pool.cc:224] Handling call
0709 12:22:57.514661 (+  2126us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.604470 (+ 89809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.604664 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:57.599728  1877 rpcz_store.cc:269] 0709 12:22:54.379592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.392586 (+ 12994us) service_pool.cc:224] Handling call
0709 12:22:57.599019 (+3206433us) spinlock_profiling.cc:243] Waited 906 ms on lock 0x7b54003525d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:57.599172 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2084888320}
W20260709 12:22:57.608016  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 125516us
I20260709 12:22:57.517294  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.413914  4388 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:22:57.528481  4507 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 12:22:57.615545  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.642933  1800 raft_consensus.cc:2410] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:22:57.616986  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.601416  4290 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:57.585033  4368 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.624732  4461 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.628657  4292 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.559456  4222 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:57.631819  1806 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:22:57.633026  1804 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:57.633839  1803 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:22:57.618582  1807 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:22:57.655575  1881 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:22:57.634227  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1640) took 1943 ms. Trace:
I20260709 12:22:57.625145  4126 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.633435  2418 rpcz_store.cc:276] 0709 12:22:56.582338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.582599 (+   261us) service_pool.cc:224] Handling call
0709 12:22:56.583060 (+   461us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.207689 (+624629us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.207735 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.224977 (+ 17242us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.371889 (+146912us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:57.371922 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.371979 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.584207 (+212228us) raft_consensus.cc:1685] finished
0709 12:22:57.584226 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.584372 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":339198,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":8158,"thread_start_us":761335,"threads_started":2,"wal-append.queue_time_us":222724}
I20260709 12:22:57.656589  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1646) took 1937 ms. Trace:
I20260709 12:22:57.657143  1881 rpcz_store.cc:276] 0709 12:22:55.719238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.654918 (+1935680us) service_pool.cc:224] Handling call
0709 12:22:57.655254 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.656246 (+   992us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.656517 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.658974  4395 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.643911  4422 consensus_queue.cc:1243] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:22:57.656605  1872 rpcz_store.cc:276] 0709 12:22:55.691322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.607812 (+1916490us) service_pool.cc:224] Handling call
0709 12:22:57.618490 (+ 10678us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.633956 (+ 15466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.634173 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.660660  4487 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.661044  1872 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:57.662946  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1649) took 1104 ms. Trace:
I20260709 12:22:57.636265  1874 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:22:57.619693  4399 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.667630  4399 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.668264  4399 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:57.640622  4466 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.636359  4272 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.640455  4371 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.648998  4368 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.651592  4222 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.673496  4222 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:57.673790  4272 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.668790  4399 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.664340  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1641) took 1972 ms. Trace:
I20260709 12:22:57.676388  1874 rpcz_store.cc:276] 0709 12:22:55.691732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.635697 (+1943965us) service_pool.cc:224] Handling call
0709 12:22:57.636033 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.663942 (+ 27909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.664267 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.677292  4500 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.677940  1874 raft_consensus.cc:1217] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Deduplicated request from leader. Original: 2.5->[4.6-6.7]   Dedup: 4.6->[6.7-6.7]
I20260709 12:22:57.649878  1880 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:57.679296  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1643) took 1987 ms. Trace:
I20260709 12:22:57.679649  1880 rpcz_store.cc:276] 0709 12:22:55.692365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.649349 (+1956984us) service_pool.cc:224] Handling call
0709 12:22:57.649656 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.679021 (+ 29365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.679236 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.678584  1874 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 12:22:57.670985  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1638) took 2043 ms. Trace:
W20260709 12:22:57.674058  4511 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.195s	user 0.000s	sys 0.002s
I20260709 12:22:57.682551  1880 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:57.684934  4399 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.691951  4388 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.695282  4500 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.695521  4426 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:57.696105  4388 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:57.696105  4426 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:57.697750  4426 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.698398  4466 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.699030  4426 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:57.701769  4419 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.702474  4228 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:57.702939  4228 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:57.703835  4228 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.705076  4228 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:57.708423  4432 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:57.709052  4471 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:57.709568  4471 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:57.709650  4382 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.710655  4471 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.711391  4297 raft_consensus.cc:1027] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: attempt to promote peer 123095127f334445889c21a42ba1d41e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260709 12:22:57.712314  4471 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:57.714303  4429 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:57.714737  4429 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:57.715786  4429 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.717203  4429 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:57.561743  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 113876us
I20260709 12:22:57.759500  1723 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:57.609467  1729 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:57.764107  1732 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:57.775460  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1545) took 2972 ms. Trace:
I20260709 12:22:57.776034  1724 rpcz_store.cc:276] 0709 12:22:54.802599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.378471 (+575872us) service_pool.cc:224] Handling call
0709 12:22:55.397384 (+ 18913us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.531502 (+1134118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.774948 (+1243446us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54002b66d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:57.775257 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1118472960}
I20260709 12:22:57.779546  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2388) took 3248 ms. Trace:
I20260709 12:22:57.780144  1727 rpcz_store.cc:276] 0709 12:22:54.530580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.647966 (+117386us) service_pool.cc:224] Handling call
0709 12:22:55.034229 (+386263us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:55.040285 (+  6056us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:55.040316 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.040873 (+   557us) log.cc:844] Serialized 26 byte log entry
0709 12:22:56.541974 (+1501101us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:56.542034 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.542129 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.239917 (+697788us) raft_consensus.cc:1685] finished
0709 12:22:57.239938 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.778912 (+538974us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b540044aa50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:57.779307 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5639,"raft.queue_time_us":58765,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":3517,"spinlock_wait_cycles":805660288,"thread_start_us":1445332,"threads_started":1,"wal-append.queue_time_us":1760144}
W20260709 12:22:57.785935  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.788399  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:57.790210  1726 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:57.792760  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.795729  1725 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:57.796101  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.796751  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.797439  1725 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:57.797816  1727 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
W20260709 12:22:57.798187  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:57.799922  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:57.800585  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.800832  1725 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:22:57.800909  1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2383) took 3452 ms (client timeout 1766 ms). Trace:
W20260709 12:22:57.802106  1728 rpcz_store.cc:269] 0709 12:22:54.348874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.350715 (+  1841us) service_pool.cc:224] Handling call
0709 12:22:57.800451 (+3449736us) spinlock_profiling.cc:243] Waited 1.47 s on lock 0x7b54002aa154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:57.800712 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3374216064}
I20260709 12:22:57.802768  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1529) took 4243 ms. Trace:
I20260709 12:22:57.803277  1731 rpcz_store.cc:276] 0709 12:22:53.558704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.794466 (+235762us) service_pool.cc:224] Handling call
0709 12:22:55.986147 (+2191681us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.307991 (+1321844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.801839 (+493848us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540030b454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:57.802395 (+   556us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b540030b450. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:57.802590 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7928221312}
I20260709 12:22:57.807556  1730 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:57.808565  1729 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:57.791133  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.819634  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.821159  1724 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:57.823364  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.824404  1724 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:57.631304  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:57.834726  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:57.851855  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:57.852941  1729 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:57.853408  1770 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:57.663275  1872 rpcz_store.cc:276] 0709 12:22:56.558926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.660484 (+1101558us) service_pool.cc:224] Handling call
0709 12:22:57.660863 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.662595 (+  1732us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.662876 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.681257  1873 rpcz_store.cc:276] 0709 12:22:55.627831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.496283 (+1868452us) service_pool.cc:224] Handling call
0709 12:22:57.496577 (+   294us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.496797 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.496821 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.513972 (+ 17151us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.614016 (+100044us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:57.662507 (+ 48491us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.670468 (+  7961us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.670556 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.670699 (+   143us) raft_consensus.cc:1685] finished
0709 12:22:57.670707 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.670915 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":48826,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":66,"raft.queue_time_us":7750,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":101,"thread_start_us":40966,"threads_started":1,"wal-append.queue_time_us":90498}
I20260709 12:22:57.857918  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1650) took 1298 ms. Trace:
I20260709 12:22:57.838434  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.860368  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 314) took 3419 ms. Trace:
I20260709 12:22:57.861937  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.860908  1879 rpcz_store.cc:276] 0709 12:22:54.441222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.467309 (+ 26087us) service_pool.cc:224] Handling call
0709 12:22:54.495673 (+ 28364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.899120 (+403447us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:54.899155 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:54.899733 (+   578us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.683239 (+783506us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:55.746021 (+ 62782us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.749034 (+  3013us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.859470 (+2110436us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b5400222254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:57.859563 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.859673 (+   110us) raft_consensus.cc:1685] finished
0709 12:22:57.859821 (+   148us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.860092 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49786,"prepare.queue_time_us":101988,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":92349,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":10511,"spinlock_wait_cycles":109579008,"thread_start_us":1058344,"threads_started":2,"wal-append.queue_time_us":1082528,"wal-append.run_cpu_time_us":13469,"wal-append.run_wall_time_us":1549268}
I20260709 12:22:57.686275  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1652) took 1111 ms. Trace:
I20260709 12:22:57.865144  1881 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:57.905752  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2276) took 1374 ms. Trace:
I20260709 12:22:57.867555  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.871824  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2266) took 2807 ms. Trace:
I20260709 12:22:57.870407  1724 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:22:57.840353  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.876690  3922 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:57.877127  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.839412  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:22:57.881753  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.884552  1799 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:57.885783  4368 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.885614  4290 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.858444  1874 rpcz_store.cc:276] 0709 12:22:56.559250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.677301 (+1118051us) service_pool.cc:224] Handling call
0709 12:22:57.677720 (+   419us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:57.857059 (+179339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.857652 (+   593us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.912346  1804 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:57.912169  4419 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.911844  3922 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.913028  4419 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.913609  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.907346  1878 rpcz_store.cc:276] 0709 12:22:55.064523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.968542 (+1904019us) service_pool.cc:224] Handling call
0709 12:22:56.994674 (+ 26132us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:56.995137 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:56.995164 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:56.995814 (+   650us) log.cc:844] Serialized 52 byte log entry
0709 12:22:57.208634 (+212820us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:57.265740 (+ 57106us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.265996 (+   256us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.266072 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.871133 (+605061us) raft_consensus.cc:1685] finished
0709 12:22:57.871185 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.871554 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56636,"prepare.queue_time_us":57160,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":216,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":41016,"thread_start_us":121042,"threads_started":1,"wal-append.queue_time_us":316097}
W20260709 12:22:57.840063  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 208677us
I20260709 12:22:57.918848  4409 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:57.919413  4413 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:57.920131  4409 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.889511  4459 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:57.889870  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.891690  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:57.892988  4226 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:57.894187  4227 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:57.895216  4297 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:22:57.846195  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.836268  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1549) took 3067 ms. Trace:
W20260709 12:22:57.847213  1844 connection.cc:256] RPC call timeout handler was delayed by -1.5810384700002942s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17320000000000002s for a response
I20260709 12:22:57.845403  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.906131  1881 rpcz_store.cc:276] 0709 12:22:56.530916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.658132 (+1127216us) service_pool.cc:224] Handling call
0709 12:22:57.864364 (+206232us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.905493 (+ 41129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.905694 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.912019  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:57.844661  1803 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:22:57.924715  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.925371  4223 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:57.925848  4263 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:22:57.944308  4263 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:57.840255  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.947986  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.949290  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:57.949954  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:57.950228  2419 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:57.924768  4395 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:22:57.951678  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:57.952709  2412 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:57.953661  4223 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.949338  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.955883  1807 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:57.956933  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:57.927731  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:57.957276  4290 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:57.958007  1807 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 3: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:57.958470  1731 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:57.932932  1798 rpcz_store.cc:276] 0709 12:22:54.768246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.336636 (+568390us) service_pool.cc:224] Handling call
0709 12:22:55.337092 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.647385 (+310293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.835620 (+2188235us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b5400132754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:22:57.836009 (+   389us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":710328192}
I20260709 12:22:57.950560  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:57.960157  4290 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.936009  1805 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:57.937238  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:57.933929  1803 pending_rounds.cc:85] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 2
I20260709 12:22:57.933696  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.932567  4227 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.961802  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:57.970237  1804 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:57.942385  1802 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:57.928859  1880 rpcz_store.cc:276] 0709 12:22:56.575212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.680429 (+1105217us) service_pool.cc:224] Handling call
0709 12:22:57.682363 (+  1934us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.686075 (+  3712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.686230 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:57.952582  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:57.955618  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:57.939450  1806 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:22:57.951709  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.926465  3922 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:57.922170  4413 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.008s
I20260709 12:22:57.942497  1801 raft_consensus.cc:2376] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6. Re-sending same reply.
I20260709 12:22:57.964128  1803 pending_rounds.cc:107] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:22:57.956260  4223 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:57.986825  1803 raft_consensus.cc:2889] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:57.988771  2410 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:57.991727  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:57.988440  4368 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.004163  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:58.001626  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.007211  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.008074  1806 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:57.972503  4227 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:58.011739  1804 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:58.014688  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:58.016211  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:22:58.016778  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89872.6us
I20260709 12:22:58.018006  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:22:58.018074  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.019012  4432 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:58.019061  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.024430  1876 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:58.026109  2418 raft_consensus.cc:2410] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:22:58.032922  4409 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:58.034271  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.034998  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.035231  1805 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:58.036079  1802 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:58.036592  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.036875  4266 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:22:58.037323  4266 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:58.037511  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:58.038261  4266 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:58.038417  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86038.2us
I20260709 12:22:58.039163  4371 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:58.039669  4371 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:58.040105  4419 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:58.036104  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 99220.6us
I20260709 12:22:58.035481  1725 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:58.042879  4223 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:58.045112  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.049834  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.051091  4413 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:58.051576  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.052377  1803 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:58.053594  4488 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:58.053818  4438 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:58.055229  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:58.056640  2419 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:22:58.050563  2410 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:22:58.059549  2410 pending_rounds.cc:85] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 5
I20260709 12:22:58.059795  2410 pending_rounds.cc:107] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:22:58.060052  2410 raft_consensus.cc:2889] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:58.061020  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:58.037258  4290 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:58.065335  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.061044  4471 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.049875  4222 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:58.067267  2412 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:58.040139  4371 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:58.063988  2411 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:58.048357  4223 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:58.058334  4248 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:58.077093  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.085780  4223 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:58.087129  1801 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:22:58.087512  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75562.6us
W20260709 12:22:58.088541  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44740.2us
I20260709 12:22:58.085892  4248 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:58.068758  4368 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:58.105337  4222 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:58.107605  4471 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.108251  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.110183  1807 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:58.111407  2412 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:58.107630  1732 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:58.115677  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:58.116309  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:58.116716  1725 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:22:58.119451  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2265) took 3055 ms. Trace:
I20260709 12:22:58.120167  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.120940  1802 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:22:58.118379  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:58.123270  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63961us
I20260709 12:22:58.124305  1730 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 10 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:22:58.118448  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.125514  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.070173  4432 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:22:58.055301  4438 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:22:58.133121  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:58.143222  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:58.143721  4354 raft_consensus.cc:670] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:58.144538  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:58.145275  2412 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:58.147531  4431 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:58.148068  4431 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:58.149011  4473 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.149132  4431 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.150024  4473 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.151394  1725 pending_rounds.cc:85] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 2
I20260709 12:22:58.151952  1725 pending_rounds.cc:107] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:22:58.152305  1725 raft_consensus.cc:2889] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:58.110435  4471 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:58.119858  1875 rpcz_store.cc:276] 0709 12:22:55.063820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.962233 (+1898413us) service_pool.cc:224] Handling call
0709 12:22:56.962744 (+   511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.963090 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:56.981642 (+ 18552us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.899565 (+917923us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b18003690a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:22:58.118803 (+219238us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:58.118896 (+    93us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:58.119012 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.119123 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.119240 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10895,"prepare.queue_time_us":12304,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":66,"raft.queue_time_us":23033,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"spinlock_wait_cycles":28705664,"thread_start_us":154964,"threads_started":1,"wal-append.queue_time_us":1112683}
I20260709 12:22:58.147639  1732 raft_consensus.cc:2410] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:22:58.165143  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.128328  4263 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:58.173313  1806 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:58.134440  4432 consensus_queue.cc:1243] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:22:58.203361  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 43969.7us
I20260709 12:22:58.205686  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.173302  4431 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:58.186628  2412 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:58.237092  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.238266  1806 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:58.237522  1804 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:58.242470  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:58.243536  2417 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:58.244356  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:58.246027  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:58.186333  4488 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:22:58.248395  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.249277  1801 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:58.254079  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.254870  4386 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:58.255517  4354 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:58.255633  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.255970  4212 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.028s
W20260709 12:22:58.257668  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:58.258214  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.258441  4226 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:58.259594  1802 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:58.230401  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:58.255223  4228 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:58.264448  1798 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:58.265977  1798 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:58.266044  1807 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:22:58.268150  1803 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:22:58.269004  1802 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:58.270889  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.277335  1803 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:58.281667  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:22:58.282842  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47411.1us
I20260709 12:22:58.258937  4226 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:58.285435  4429 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:58.284936  4226 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.285960  4429 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.286466  4429 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:58.287735  4473 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:58.289444  1805 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:22:58.292338  1806 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:58.297791  1804 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:58.301333  1730 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:58.256004  4354 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:58.255286  4386 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.305008  1798 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:22:58.304556  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 124619us
I20260709 12:22:58.303462  1803 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:22:58.319928  1801 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:22:58.316378  1807 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:58.320850  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:58.330106  1802 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: [4.6-7.7]
I20260709 12:22:58.313584  4354 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:58.333705  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.339715  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:58.316357  4386 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:58.329948  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53462.1us
I20260709 12:22:58.282644  4266 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.316236  4290 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:58.300362  4223 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.304329  4263 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:22:58.388178  4263 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:58.390341  4263 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:58.391038  4466 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:58.392581  4290 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:58.394104  4395 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:58.363226  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:58.364445  1881 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:58.395363  4458 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:58.396021  4458 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:58.397544  4458 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:58.321822  4368 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.080s
I20260709 12:22:58.365494  4422 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:58.366297  4266 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:58.366990  4413 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 12:22:58.374019  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.377560  4272 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.057s
I20260709 12:22:58.380664  4248 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.392675  4466 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:58.399361  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:58.399426  4368 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:58.415001  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:22:58.400324  1845 connection.cc:256] RPC call timeout handler was delayed by -1.3721471040003053s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
I20260709 12:22:58.400822  4458 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:58.296743  4471 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:22:58.403050  4432 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:22:58.386917  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.321010  4227 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.422705  1728 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:58.403841  4422 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.394054  4290 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.390717  4263 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:58.430820  1845 connection.cc:256] RPC call timeout handler was delayed by -1.160647104000418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2069s for a response
I20260709 12:22:58.453042  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:58.453449  1845 connection.cc:256] RPC call timeout handler was delayed by -1.1237471040003584s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.211s for a response
W20260709 12:22:58.453949  1845 connection.cc:256] RPC call timeout handler was delayed by -1.0661471040002652s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2174s for a response
I20260709 12:22:58.453979  1875 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
W20260709 12:22:58.454309  1845 connection.cc:256] RPC call timeout handler was delayed by -1.0526471040002434s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
I20260709 12:22:58.455873  1874 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:58.457140  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.458058  1874 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:22:58.459468  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.460412  1877 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:58.460846  4471 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:58.461440  4471 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:22:58.461540  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.461900  4471 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:58.434142  4413 consensus_queue.cc:1243] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:22:58.422863  4272 consensus_queue.cc:1243] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:22:58.435428  4500 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 12:22:58.415546  4368 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:58.439039  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.440301  4422 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:58.439040  4248 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:58.441247  4473 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:58.442881  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2268) took 3377 ms (client timeout 1747 ms). Trace:
I20260709 12:22:58.444235  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.474040  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:58.447527  4226 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:58.447558  1880 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:58.471812  1727 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:58.464845  1872 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:22:58.415257  4395 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.473591  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:58.463377  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.462632  1877 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:58.489347  1876 rpcz_store.cc:269] 0709 12:22:55.068607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.542374 (+473767us) service_pool.cc:224] Handling call
0709 12:22:58.442569 (+2900195us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b54002df5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:22:58.442752 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5251247360}
I20260709 12:22:58.488792  1879 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:22:58.416183  4466 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:58.435674  1772 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:58.475412  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:22:58.502732  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70446.9us
I20260709 12:22:58.510739  4515 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:58.511617  4515 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:58.513003  4515 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.515249  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.515877  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.516103  1878 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:22:58.510923  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:58.520071  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 84526.4us
I20260709 12:22:58.514393  4222 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:58.468387  4223 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:58.533604  1876 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:58.534955  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.535745  1876 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:58.536640  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.537233  1876 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:22:58.538153  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.543188  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.544595  1877 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:58.546609  1877 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:22:58.546973  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.547837  4413 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:22:58.543498  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 117835us
I20260709 12:22:58.548316  4354 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.550999  4354 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:58.551832  4266 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:58.551895  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:58.556823  1725 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:58.557211  4488 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:58.557787  4488 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.555114  4371 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.558459  4488 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:58.554086  3922 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:58.559567  3922 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:58.560045  3922 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:58.560793  4371 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:58.561615  1723 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:58.561195  3922 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.563570  3922 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:58.557832  4354 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:58.576171  1872 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:22:58.587736  4466 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.589699  4466 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:58.593312  4227 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:58.605068  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.605922  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.607641  1876 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:22:58.609975  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 85835.8us
I20260709 12:22:58.524397  4500 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.093s
I20260709 12:22:58.613056  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:58.614018  1727 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:58.615976  1876 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:22:58.617128  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 96082.8us
I20260709 12:22:58.606061  1873 raft_consensus.cc:2376] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7. Re-sending same reply.
I20260709 12:22:58.621622  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.531010  4515 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:22:58.651991  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.508806  4432 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.665148  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.668534  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.666267  1872 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:22:58.596017  4429 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:58.678448  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.648772  4473 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:22:58.689364  1873 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:58.691066  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45509.2us
W20260709 12:22:58.700405  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 363190us
I20260709 12:22:58.630505  4227 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:22:58.700405  1725 pending_rounds.cc:85] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 6
I20260709 12:22:58.703289  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:58.706697  1725 pending_rounds.cc:107] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:22:58.635591  4395 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:22:58.704730  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 408071us
I20260709 12:22:58.707158  1725 raft_consensus.cc:2889] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:58.710355  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 347314us
I20260709 12:22:58.719405  1881 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
W20260709 12:22:58.720219  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.708869  4500 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:58.737666  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.746104  1879 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:58.731930  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:58.749465  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.764541  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.774369  1874 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:22:58.778108  4426 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260709 12:22:58.773142  4228 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:22:58.779167  4386 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.781692  4386 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:58.779214  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:58.785204  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:58.744270  4500 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:58.763559  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:22:58.792161  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62288.1us
I20260709 12:22:58.795480  1881 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:58.802729  4386 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:58.760816  3922 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:58.808661  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 303428us
I20260709 12:22:58.815357  4228 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:58.816015  4382 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:58.817240  4382 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:58.817617  4382 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:22:58.823117  1880 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:58.824558  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.832887  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.833436  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:58.834568  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:22:58.797861  4500 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:58.808310  4386 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:58.832113  4488 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.805717  4248 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.841187  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.832252  4409 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:58.860065  1881 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 2.
I20260709 12:22:58.843803  1874 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:22:58.893206  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:58.842763  4228 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:58.818639  4382 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.903930  1801 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:22:58.904207  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.852047  4312 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:22:58.842180  3922 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.909185  1729 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:58.865358  4409 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:58.854717  4248 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:58.922305  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:58.924599  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 218734us
I20260709 12:22:58.924853  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.909937  4546 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 12:22:58.932206  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.933425  4263 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.934280  4368 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:58.935031  4395 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:58.935760  1874 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:22:58.935549  4506 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 12:22:58.937469  1874 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260709 12:22:58.940624  4542 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:22:58.940893  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:58.941820  1723 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:22:58.941931  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:58.942680  4549 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:58.904958  4371 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:58.927416  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:22:58.942683  1875 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:22:58.974130  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 59386.6us
I20260709 12:22:58.931474  1878 raft_consensus.cc:2376] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6. Re-sending same reply.
I20260709 12:22:58.917995  4429 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:58.983847  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:58.864872  4386 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.954967  4432 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:58.997963  4228 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.000028  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:59.001817  4409 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:58.982057  4382 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:59.003273  4409 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:59.004058  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:22:59.005280  4263 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:59.007726  1877 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:58.998276  4419 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:22:59.009032  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.009020  1873 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
W20260709 12:22:59.013893  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.985206  4395 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.011994  4290 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:22:59.010583  4419 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:22:59.049207  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 71171.6us
W20260709 12:22:59.049106  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.033739  4226 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.507s	user 0.009s	sys 0.005s
I20260709 12:22:59.070554  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:59.076671  1877 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:59.075399  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:59.074692  1627 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 3 to 4, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:59.080837  1627 catalog_manager.cc:5697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:59.084700  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:22:59.082743  1627 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:59.118814  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 209182us
W20260709 12:22:59.122906  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 141994us
I20260709 12:22:59.126461  1728 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
W20260709 12:22:59.128779  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.129251  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.129805  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.130558  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 220801us
I20260709 12:22:59.041783  4395 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.142743  1727 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
W20260709 12:22:59.156927  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.157670  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.158692  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:22:59.162024  1880 raft_consensus.cc:2410] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
W20260709 12:22:59.168198  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 309355us
I20260709 12:22:59.125821  4561 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.122355  4212 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:59.175654  1771 leader_election.cc:336] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.747s (SENT)
I20260709 12:22:59.162712  4266 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:59.186996  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.168344  4272 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:59.194250  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:22:59.219502  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.218037  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 133471us
I20260709 12:22:59.217509  4248 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:22:59.248962  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.251703  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 128971us
I20260709 12:22:59.264760  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:59.266430  1727 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:59.197564  4266 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:59.271977  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:59.272305  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:59.194712  4212 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:59.277639  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 119013us
I20260709 12:22:59.243124  4227 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:22:59.246757  4368 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:59.195207  4561 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:59.243734  4459 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:22:59.291690  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:59.299693  2412 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:59.295595  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.247948  3922 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:59.307444  1732 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:59.310511  1729 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:22:59.280215  4223 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:59.280778  4228 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.296304  4272 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:59.335070  4272 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.335706  4272 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:59.337621  4557 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:22:59.339033  4422 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.817s	user 0.009s	sys 0.004s
I20260709 12:22:59.339534  4422 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.334693  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:22:59.341779  4422 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:59.344261  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.344560  4266 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:22:59.349176  1872 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.577s	user 0.009s	sys 0.000s
I20260709 12:22:59.304992  4549 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:22:59.320780  4388 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:59.351257  4459 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.346648  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:59.358989  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 195661us
I20260709 12:22:59.333392  4574 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:22:59.359875   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:59.402194  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:59.376439  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:59.404322  2411 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:22:59.380975  2412 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:59.382301  1877 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:22:59.364974  1872 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
W20260709 12:22:59.400187  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.245812  4419 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:59.335101  4556 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:22:59.388734  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:59.415944  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 107516us
I20260709 12:22:59.412928  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:59.419152  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 98779.5us
W20260709 12:22:59.408941  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 68765.7us
I20260709 12:22:59.410851  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:59.420912  1729 raft_consensus.cc:2368] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7.
I20260709 12:22:59.425707  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:59.430403  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:22:59.431684  2412 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:59.434643  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.353899  4272 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.462698  1727 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
W20260709 12:22:59.463254  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.464597  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:22:59.465657  3922 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:59.424147  4422 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.473526  2412 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:22:59.375414  4223 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.377502  4228 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:59.487318  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 65431.1us
W20260709 12:22:59.488235  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74110.2us
I20260709 12:22:59.496042  1728 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:22:59.503306  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:59.456619  4290 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:22:59.520048  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:22:59.532721  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.456024  4395 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:59.462653  4432 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:22:59.574939  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66044.6us
I20260709 12:22:59.577289  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:22:59.480752  4419 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:59.474648  4312 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.009s	sys 0.044s
W20260709 12:22:59.588045  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 98370.5us
I20260709 12:22:59.439220  4266 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
W20260709 12:22:59.598563  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.496887  4471 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.034s	user 0.011s	sys 0.006s
W20260709 12:22:59.505791  4515 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.867s	user 0.012s	sys 0.002s
W20260709 12:22:59.629644  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.500494  4549 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.145s
W20260709 12:22:59.638492  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 65361us
I20260709 12:22:59.488191  4409 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:59.644443  1727 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20260709 12:22:59.575215  3922 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:59.659855  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63365.7us
I20260709 12:22:59.668371  1730 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 3. Ops: []
I20260709 12:22:59.588449  4371 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:22:59.671643  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 213619us
I20260709 12:22:59.673187  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:59.653364  4500 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.801s	user 0.013s	sys 0.000s
I20260709 12:22:59.502532  4422 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.685549  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:22:59.686919  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 90645.4us
W20260709 12:22:59.661163  4466 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.071s	user 0.012s	sys 0.006s
I20260709 12:22:59.652256  4409 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:59.482975  4388 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.667083  2413 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
W20260709 12:22:59.748373  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.749325  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 92144.5us
I20260709 12:22:59.743307  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:22:59.740154  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:22:59.666311  4263 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.660s	user 0.013s	sys 0.000s
I20260709 12:22:59.753489  4592 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:59.753855  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.755897  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 70478.1us
I20260709 12:22:59.745793  2418 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
W20260709 12:22:59.766727  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.645967  4471 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.683245  4212 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:59.699898  4266 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:22:59.785315  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:59.789696  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:22:59.781688  4422 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:59.792140  2410 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:22:59.665665  4432 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:22:59.760596  4548 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:22:59.772699  4581 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.360s	user 0.002s	sys 0.000s
I20260709 12:22:59.675787  4573 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.451s
I20260709 12:22:59.683668  4545 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:22:59.761439  4515 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:59.807552  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.797299  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54122.6us
W20260709 12:22:59.807617  4578 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260709 12:22:59.808624  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:22:59.764030  4371 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:59.810303  4371 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.798648  2419 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:59.799485  4582 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
I20260709 12:22:59.814881  4388 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:59.808713  4272 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:22:59.788661  4248 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.817366  4248 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:59.819398  4248 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:59.783644  4266 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:22:59.820251  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:59.821105  4266 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:59.800312  4592 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:59.801393  4212 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:59.804040  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1567) took 1197 ms. Trace:
W20260709 12:22:59.804291  4591 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:22:59.807390  4545 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:22:59.719074  3922 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.842733  4594 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.727697  4500 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.843384  4594 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:22:59.843937  4583 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:22:59.845438  1731 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:59.795607  4548 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:22:59.872454  4590 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
W20260709 12:22:59.874193  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.873775  4548 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:22:59.875108  4548 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:22:59.791752  4538 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:22:59.826598  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:59.879343  4565 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:59.826715  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51110.3us
I20260709 12:22:59.879987  4565 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:59.837394  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.883275  4248 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.841958  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:22:59.883891  4248 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:22:59.885080  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49648.3us
I20260709 12:22:59.886130  4248 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:59.842119  4588 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:59.867457  4572 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.392s	user 0.001s	sys 0.000s
I20260709 12:22:59.850832  4266 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:22:59.867117  4545 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:59.835693  4212 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.869412  4471 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:59.839977  4272 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 12:22:59.791496  4413 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.080s
I20260709 12:22:59.835533  4466 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:59.871587  4459 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:59.872040  4589 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:22:59.886794  4248 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 12:22:59.882483  4565 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:59.889353  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:22:59.958724  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69551.8us
I20260709 12:22:59.889554  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:22:59.927873  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.959641  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76511us
I20260709 12:22:59.835906  1725 rpcz_store.cc:276] 0709 12:22:58.607038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.607199 (+   161us) service_pool.cc:224] Handling call
0709 12:22:58.664524 (+ 57325us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:59.099970 (+435446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:59.100000 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:59.115925 (+ 15925us) log.cc:844] Serialized 52 byte log entry
0709 12:22:59.277344 (+161419us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:59.277381 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.277457 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.803745 (+526288us) raft_consensus.cc:1685] finished
0709 12:22:59.803766 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.803951 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":92592,"raft.queue_time_us":151433,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":24510,"thread_start_us":369746,"threads_started":2,"wal-append.queue_time_us":243066}
W20260709 12:22:59.936604  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 127747us
I20260709 12:22:59.932188  4395 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.933429  4212 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:22:59.947408  4382 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.944s	user 0.014s	sys 0.000s
I20260709 12:22:59.949139  4432 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:22:59.951515  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:22:59.952345  3922 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:22:59.953588  4368 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.651s	user 0.013s	sys 0.006s
I20260709 12:22:59.890678  4561 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:59.955647  4466 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:22:59.886960  4515 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:59.970758  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:22:59.970952  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:00.008163  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47668.4us
I20260709 12:23:00.008402  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:00.009126  1725 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:23:00.009886  4226 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:00.010507  4226 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:00.012882  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:00.013840  2411 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:22:59.971813  4545 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:00.015770  4223 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:00.016429  4223 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.017081  4223 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:22:59.889082  4588 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:00.019065  4588 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:59.974736  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:22:59.972283  4596 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.981993  4473 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.973621  4459 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:59.989184  4605 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.983750  4595 raft_consensus.cc:2955] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Committing config change with OpId 5.6: config changed from index 3 to 6, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:22:59.975957  4395 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:22:59.990841  4562 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:22:59.997277  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.068661  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.998597  2382 leader_election.cc:400] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7.
I20260709 12:23:00.069267  2382 leader_election.cc:403] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:23:00.009081  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:22:59.971534  4577 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:22:59.973801  1732 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.649s	user 0.009s	sys 0.001s
I20260709 12:23:00.011606  4226 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.034763  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:00.032481  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:00.027390  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.031747  1723 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 12:23:00.043426  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.032689  4466 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:00.048873  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:00.049685  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:00.050390  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 76124us
I20260709 12:23:00.059686  4409 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:00.061190  4488 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:00.021390  4223 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:00.062559  4228 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:00.063136  4473 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:00.053264  4368 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:00.099316  4547 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7.
I20260709 12:23:00.099757  4368 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:00.065696  4585 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.310s	user 0.000s	sys 0.005s
I20260709 12:23:00.065496  4605 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.104521  4564 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:23:00.104676  4605 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:22:59.973136  4565 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:23:00.105149  4564 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.105278  4473 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:00.105760  4564 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:00.069802  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:00.106633  2411 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:23:00.107362  4459 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:00.107877  4459 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.108278  4459 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:00.108876  4395 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:00.115077  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46458.3us
W20260709 12:23:00.072769  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74419.7us
I20260709 12:23:00.071364  1729 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:00.083884  4382 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.084010  4426 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.091192  2412 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:00.092324  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:00.093480  4409 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:00.093683  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84732.5us
I20260709 12:23:00.095256  4212 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.098222  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:00.098462  2415 raft_consensus.cc:2955] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:00.100847  4368 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.053493  4596 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.074621  4577 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:00.070242  1732 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:00.146076  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:00.146441  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:00.146989  2412 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:00.118600  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2423) took 1082 ms. Trace:
W20260709 12:23:00.127265  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.143235  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.146690  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:00.110772  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.109514  4592 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:00.162751  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:00.147858  1726 rpcz_store.cc:276] 0709 12:22:59.035697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.035820 (+   123us) service_pool.cc:224] Handling call
0709 12:22:59.069894 (+ 34074us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:59.164939 (+ 95045us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:59.164986 (+    47us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:59.173358 (+  8372us) log.cc:844] Serialized 78 byte log entry
0709 12:22:59.395803 (+222445us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:59.395859 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.395947 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.064739 (+668792us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:00.110422 (+ 45683us) raft_consensus.cc:1685] finished
0709 12:23:00.110444 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.118517 (+  8073us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58691,"raft.queue_time_us":53086,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":5663,"spinlock_wait_cycles":13409536,"thread_start_us":160726,"threads_started":1,"wal-append.queue_time_us":448964}
I20260709 12:23:00.167469  1728 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:00.114832  4500 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:00.174401  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.170948  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:00.114854  4223 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.145658  4272 raft_consensus.cc:2764] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:23:00.192865  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:00.192291  1732 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:00.210125  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:00.164414  3922 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:00.216054  1728 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
W20260709 12:23:00.227362  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 135088us
I20260709 12:23:00.233105  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:00.234423  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:00.236105  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.239548  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:00.246153  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:00.132023  4547 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
I20260709 12:23:00.158816  4426 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.158615  4382 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.204614  4458 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:00.147439  4395 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.066s
I20260709 12:23:00.177050  4248 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.071s
I20260709 12:23:00.257658  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:00.258263  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:00.191704  4223 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.190912  4605 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260709 12:23:00.260519  4272 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:00.217540  4592 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.043s
I20260709 12:23:00.273406  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:00.274196  1728 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:00.268502  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:00.281335  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:00.318462  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:00.330037  1877 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:23:00.320744  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:00.257123  4312 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:00.241500  3922 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.325819  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:00.340257  1878 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
W20260709 12:23:00.277700  4601 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:23:00.292503  1732 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260709 12:23:00.331306  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1572) took 1545 ms. Trace:
I20260709 12:23:00.346963  1731 rpcz_store.cc:276] 0709 12:22:58.785679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.785866 (+   187us) service_pool.cc:224] Handling call
0709 12:22:58.786303 (+   437us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.845966 (+1059663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.330987 (+485021us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540035d754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:00.331235 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2435260288}
I20260709 12:23:00.332129  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:00.332948  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:00.333874  1876 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:00.266638  4226 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:00.319271  4549 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:00.364795  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:00.346918  1732 pending_rounds.cc:85] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 4
I20260709 12:23:00.321405  4432 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.366127  1732 pending_rounds.cc:107] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 12:23:00.419871  1732 raft_consensus.cc:2889] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:00.351158  1880 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:00.364717  4422 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:00.370126  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.371192  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 136914us
W20260709 12:23:00.371896  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 103470us
I20260709 12:23:00.375085  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:00.375532  1731 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:23:00.328732  4272 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:00.391803  4604 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:23:00.360114  4488 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.399791  1730 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 3.
I20260709 12:23:00.401714  4223 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:00.462693  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:00.407251  4228 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:23:00.336458  4565 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.338220  4458 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:23:00.437738  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 159484us
I20260709 12:23:00.462647  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:00.361577  4500 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:00.366551  4549 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.503111  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:00.463279  4570 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.464267  4592 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:00.487599  4546 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:00.523067  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:00.523308  1876 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:00.519508  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 353) took 1225 ms. Trace:
I20260709 12:23:00.496825  4549 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:00.447769  4422 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.490630  4226 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:00.446749  4432 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:00.538401  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.524464  1724 rpcz_store.cc:276] 0709 12:22:59.294374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.574147 (+279773us) service_pool.cc:224] Handling call
0709 12:22:59.652893 (+ 78746us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:59.761351 (+108458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:59.761397 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:59.774973 (+ 13576us) log.cc:844] Serialized 26 byte log entry
0709 12:22:59.972770 (+197797us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:59.972838 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.972913 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.519234 (+546321us) raft_consensus.cc:1685] finished
0709 12:23:00.519262 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.519432 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1420,"raft.queue_time_us":78609,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":12362,"thread_start_us":150253,"threads_started":1,"wal-append.queue_time_us":272664}
W20260709 12:23:00.549165  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56462.8us
I20260709 12:23:00.516655  4596 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:00.490685  4488 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.527029  4547 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:00.501258  4466 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.561055  1878 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:00.527558  4546 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:00.516228  4615 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:00.497169  4565 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:00.555950  4223 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:00.565476  4623 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.566387  4422 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:00.570091  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.571225  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:00.563834  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:00.597415  4395 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:00.598313  4616 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.599648  4272 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:00.529776  4592 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:00.605897  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.937278  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.741760  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:00.939461  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 197767us
W20260709 12:23:00.935807  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 364683us
I20260709 12:23:00.563251  4473 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.567901  4547 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:00.746414  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.962620  4616 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:00.963757  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2436) took 1074 ms. Trace:
I20260709 12:23:00.875275  4466 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:00.966686  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:00.967352  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:00.992205  1732 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:00.966336  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 708) took 2177 ms. Trace:
I20260709 12:23:00.963873  4395 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:00.993718  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.002s	sys 0.050s
W20260709 12:23:00.972039  4584 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.157s	user 0.000s	sys 0.006s
I20260709 12:23:00.865872  4633 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:00.747838  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:00.576609  4618 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:00.645365  4564 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.539s	user 0.012s	sys 0.000s
I20260709 12:23:00.989326  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:00.980371  4515 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:00.587577  4312 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.987370  1723 rpcz_store.cc:276] 0709 12:22:59.889126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.889294 (+   168us) service_pool.cc:224] Handling call
0709 12:23:00.963288 (+1073994us) spinlock_profiling.cc:243] Waited 71 ms on lock 0x7b54002a9754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:00.963587 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":163265408}
W20260709 12:23:00.994246  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.002s	sys 0.050s
I20260709 12:23:00.993357  2419 rpcz_store.cc:276] 0709 12:22:58.789341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.789956 (+   615us) service_pool.cc:224] Handling call
0709 12:22:58.790507 (+   551us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.812251 (+1021744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.965865 (+1153614us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54004993d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:00.966175 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2313370880}
W20260709 12:23:00.874848  4611 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.350s	user 0.001s	sys 0.000s
W20260709 12:23:00.979787  4607 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.309s	user 0.000s	sys 0.002s
I20260709 12:23:00.602460  4426 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:00.797991  4459 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.689s	user 0.008s	sys 0.006s
I20260709 12:23:00.876681  4458 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:23:00.990396  4614 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:23:00.640563  4623 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:00.602018  4615 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:01.011062  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 450800us
W20260709 12:23:01.014323  4588 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.962s	user 0.010s	sys 0.007s
I20260709 12:23:00.991873  4473 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.001175  4598 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:23:01.060698  4473 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.061987  4598 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.016014  4565 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.063102  4395 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.063375  4629 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.063699  4395 raft_consensus.cc:2764] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:23:00.747316  4617 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:00.744246  4488 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:00.979566  4223 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:00.983002  3922 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:01.003747  4272 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:00.989632  4594 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.031978  1627 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 4 to 6, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 6 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:01.031591  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 569005us
I20260709 12:23:01.019601  4432 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:01.022336  4515 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.866838  4546 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.057366  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:01.058276  4619 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.384s	user 0.000s	sys 0.001s
I20260709 12:23:01.058003  4588 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:00.987761  4547 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:00.525972  4570 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.066641  1732 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:23:01.033654  4228 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.069868  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.035858  4458 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.075886  2418 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 12:23:01.076496  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.085712  4617 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.120844  2410 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
W20260709 12:23:01.121405  2410 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 669) took 1233 ms (client timeout 1592 ms). Trace:
W20260709 12:23:01.121829  2410 rpcz_store.cc:269] 0709 12:22:59.887734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.064737 (+1177003us) service_pool.cc:224] Handling call
0709 12:23:01.121337 (+ 56600us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:01.127988  4616 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:01.022373  4633 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.132107  4633 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.133116  4546 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:01.135509  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.117120  4592 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.028363  4312 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.097119  4515 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:01.139525  4312 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:01.055943  4621 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260709 12:23:01.140471  4382 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:01.077898  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.029458  4564 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.064188  4629 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.143895  4564 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.043471  4459 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.144867  4629 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:01.096011  4395 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.117146  4272 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:01.145510  4395 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.146391  4272 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.146971  4459 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:01.147763  4395 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.108366  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:23:01.149397  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53907.1us
I20260709 12:23:01.150806  1877 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:01.152268  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.126605  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.118808  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:01.137069  4588 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:00.847674  4226 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.141759  4548 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:01.148399  4432 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:01.150264  4409 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.001s	user 0.009s	sys 0.004s
I20260709 12:23:01.152456  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.153285  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.062633  4598 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.117455  4595 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.180748  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:01.187397  2419 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:01.128233  4623 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.152400  4458 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:01.208561  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:01.210067  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:01.211871  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 93437.1us
I20260709 12:23:01.212831  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:01.211750  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.213722  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:01.182083  4634 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:23:01.217935  1732 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:23:01.159204  4466 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:01.109604  4223 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:01.224936  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 120279us
I20260709 12:23:01.173441  4622 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.168442  4548 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:01.219946  1732 pending_rounds.cc:85] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 4
I20260709 12:23:01.247215  1732 pending_rounds.cc:107] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:23:01.244292  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:01.245389  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:01.249662  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.251060  1732 raft_consensus.cc:2889] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:01.160115  4228 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.172245  4426 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 12:23:01.159672  4473 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 12:23:01.181059  4432 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.215471  4595 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:01.363931  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 374701us
I20260709 12:23:01.379196  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.226753  4639 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:01.386435  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.230455  4588 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:23:01.269579  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.420814  1732 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:01.419811  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:01.369342  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69552.3us
W20260709 12:23:01.363327  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 119155us
I20260709 12:23:01.430909  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:01.425742  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.269191  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 118938us
I20260709 12:23:01.224298  4630 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:23:01.432832  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:01.429525  1724 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
W20260709 12:23:01.255213  4626 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
I20260709 12:23:01.434427  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.436211  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:01.402437  4549 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.855s	user 0.013s	sys 0.000s
I20260709 12:23:01.472326  1874 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:01.405982  4644 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.486413  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:01.363672  4598 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:01.361879  4633 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.481662  3922 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:23:01.435745  4506 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:01.488240  4562 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:23:01.489583  4593 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.678s	user 0.004s	sys 0.003s
I20260709 12:23:01.259063  4622 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.435907  4630 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.493412  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:01.377265  4422 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.479712  4459 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:23:01.470950  4629 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:01.494756  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.495204  4615 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:01.496328  4587 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.697s	user 0.006s	sys 0.005s
I20260709 12:23:01.498057  4643 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:01.499891  4641 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.270s	user 0.001s	sys 0.000s
I20260709 12:23:01.499998  4570 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:01.509303  4577 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:01.511621  4586 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.666s	user 0.001s	sys 0.004s
I20260709 12:23:01.512501  4618 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:01.516119  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:01.516898  4647 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:01.517685  4644 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:01.520617  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.609315  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.521085  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:01.611292  1874 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:01.613867  2411 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:01.523046  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2435) took 1742 ms. Trace:
I20260709 12:23:01.615002  3922 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:01.586297  4422 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.383255  4272 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:01.617585  4422 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:01.617398  4546 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:01.617249  4272 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.587013  4547 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.619256  4595 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:01.620016  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 140488us
I20260709 12:23:01.528250  2419 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:23:01.363669  4312 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:01.532851  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.597044  4570 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:23:01.477465  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44753.4us
I20260709 12:23:01.621436  2412 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:23:01.621960  4622 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:01.622182  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:01.287940  4617 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:01.622460  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.623193  2412 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:01.623282  4636 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.623828  4636 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.356823  4637 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:01.625347  4637 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.625768  4637 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:01.626150  4636 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:01.226168  4395 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.255s
I20260709 12:23:01.627060  4640 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:01.549306  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:01.608618  4577 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:01.441360  4588 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:01.514657  4500 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.017s	user 0.008s	sys 0.008s
I20260709 12:23:01.568296  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2313) took 1115 ms. Trace:
I20260709 12:23:01.629405  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:01.567225  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:01.583863  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:01.584309  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.355312  4548 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.559609  4506 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:01.567832  4562 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:01.569612  4466 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:01.573210  4630 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:01.606945  2413 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:01.441800  4639 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.620712  4547 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:01.626829  4644 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:01.627733  2416 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:01.627977  1724 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:01.497808  4228 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:01.365278  4223 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:01.628300  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2308) took 1330 ms. Trace:
I20260709 12:23:01.649230  4636 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:23:01.614542  1730 rpcz_store.cc:276] 0709 12:22:59.780656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.793731 (+ 13075us) service_pool.cc:224] Handling call
0709 12:23:01.522768 (+1729037us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b540030be54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:01.522957 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1195873664}
I20260709 12:23:01.486985  4549 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.581516  4649 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:01.560016  4633 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:01.633766  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 117738us
I20260709 12:23:01.639045  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:01.647641  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:01.655313  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 161999us
W20260709 12:23:01.659372  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 92304.9us
I20260709 12:23:01.631678  4272 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.637404  4588 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:23:01.706882  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 67907.9us
I20260709 12:23:01.636761  1880 rpcz_store.cc:276] 0709 12:23:00.453094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.479083 (+ 25989us) service_pool.cc:224] Handling call
0709 12:23:00.500041 (+ 20958us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.519693 (+ 19652us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.519741 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.527733 (+  7992us) log.cc:844] Serialized 26 byte log entry
0709 12:23:01.143348 (+615615us) spinlock_profiling.cc:243] Waited 9.65 ms on lock 0x7b18003dab00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:23:01.290263 (+146915us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:01.555272 (+265009us) log.cc:844] Serialized 14 byte log entry
0709 12:23:01.555590 (+   318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.555681 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.555732 (+    51us) raft_consensus.cc:1685] finished
0709 12:23:01.559774 (+  4042us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.568218 (+  8444us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":84375,"prepare.queue_time_us":113720,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":57892,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":8896,"spinlock_wait_cycles":22201216,"thread_start_us":318710,"threads_started":2,"wal-append.queue_time_us":866267}
I20260709 12:23:01.637666  4629 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:01.644344  4422 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:01.670670  1872 rpcz_store.cc:276] 0709 12:23:00.297838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.328768 (+ 30930us) service_pool.cc:224] Handling call
0709 12:23:00.344972 (+ 16204us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.345391 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.345432 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.359904 (+ 14472us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.788098 (+428194us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:00.837140 (+ 49042us) log.cc:844] Serialized 14 byte log entry
0709 12:23:00.839394 (+  2254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.839520 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.628107 (+788587us) raft_consensus.cc:1685] finished
0709 12:23:01.628125 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.628236 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11134,"prepare.queue_time_us":54370,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":18232,"raft.queue_time_us":202806,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":18838,"thread_start_us":308046,"threads_started":1,"wal-append.queue_time_us":662129}
W20260709 12:23:01.629446  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.655855  4500 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:01.681073  4466 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:01.682933  4597 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:01.683285  4432 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.669142  4458 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.659024  4632 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:23:01.686355  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 663) took 2357 ms. Trace:
I20260709 12:23:01.727821  4639 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:01.704550  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:01.712911  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.894395  1726 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:01.895838  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:01.896915  1726 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:01.897980  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.894403  4630 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:01.717420  4649 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:01.637838  4577 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:01.721500  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 108675us
I20260709 12:23:01.903257  4649 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:01.722114  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.904986  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.905440  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.724680  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.879988  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:01.878624  4515 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.740s	user 0.012s	sys 0.001s
W20260709 12:23:01.716845  4600 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.875s	user 0.002s	sys 0.016s
W20260709 12:23:01.714962  2383 connection.cc:256] RPC call timeout handler was delayed by -2.2540844240002116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
W20260709 12:23:01.908164  2383 connection.cc:256] RPC call timeout handler was delayed by -2.1351388400003088s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.215s for a response
I20260709 12:23:01.900555  1726 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:23:01.908339  1723 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:01.765069  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.728976  1731 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
W20260709 12:23:01.757682  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2312) took 1325 ms (client timeout 1656 ms). Trace:
I20260709 12:23:01.910180  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:01.760552  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1583) took 1646 ms. Trace:
I20260709 12:23:01.764842  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.911237  1728 rpcz_store.cc:276] 0709 12:23:00.114159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.326128 (+211969us) service_pool.cc:224] Handling call
0709 12:23:00.340548 (+ 14420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.483352 (+142804us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.483399 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:00.483451 (+    52us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:00.483493 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.760314 (+1276821us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54002b7ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:01.760360 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.760490 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":327895808}
I20260709 12:23:01.728251  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.748354  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.915827  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:01.919627  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:01.919836  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:01.694490  1724 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:23:01.921167  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:02.108448  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:01.967307  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 358) took 2616 ms. Trace:
W20260709 12:23:02.060927  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.003994  1696 connection.cc:256] RPC call timeout handler was delayed by -2.4289881679997052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
W20260709 12:23:02.151507  1696 connection.cc:256] RPC call timeout handler was delayed by -2.3317881680000028s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
W20260709 12:23:02.151925  1696 connection.cc:256] RPC call timeout handler was delayed by -2.2705881679999038s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
I20260709 12:23:01.926020  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:02.152405  1696 connection.cc:256] RPC call timeout handler was delayed by -2.2300881679998383s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1885s for a response
W20260709 12:23:02.149437  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 444909us
I20260709 12:23:01.966320  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:01.956099  2383 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.900s (SENT)
I20260709 12:23:02.154702  1732 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:23:02.155071  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:01.727504  4595 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:02.037848  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.157097  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:02.157635  1872 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:02.158082  1726 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:01.910531  1878 rpcz_store.cc:269] 0709 12:23:00.432108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.432246 (+   138us) service_pool.cc:224] Handling call
0709 12:23:01.757493 (+1325247us) spinlock_profiling.cc:243] Waited 96.8 ms on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:01.757596 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":222731520}
W20260709 12:23:02.155120  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.161521  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 513997us
I20260709 12:23:01.916576  2414 rpcz_store.cc:276] 0709 12:22:59.329118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.329403 (+   285us) service_pool.cc:224] Handling call
0709 12:22:59.388686 (+ 59283us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:59.409231 (+ 20545us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:59.409272 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:59.412177 (+  2905us) log.cc:844] Serialized 52 byte log entry
0709 12:22:59.738411 (+326234us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:59.738439 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.738495 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.488871 (+1750376us) spinlock_profiling.cc:243] Waited 58.5 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:01.671686 (+182815us) raft_consensus.cc:1685] finished
0709 12:23:01.671707 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.686280 (+ 14573us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18901,"raft.queue_time_us":34979,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":16206,"spinlock_wait_cycles":134586112,"thread_start_us":325786,"threads_started":1,"wal-append.queue_time_us":398820}
W20260709 12:23:02.156113  1696 connection.cc:256] RPC call timeout handler was delayed by -1.9042881679997663s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2247s for a response
W20260709 12:23:02.159071  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 123388us
I20260709 12:23:01.878865  4227 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.159893  2413 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:01.909209  4577 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:01.728427  4506 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.149412  1727 rpcz_store.cc:276] 0709 12:22:59.350408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.657231 (+306823us) service_pool.cc:224] Handling call
0709 12:22:59.710191 (+ 52960us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:59.710445 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:59.710473 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:59.763534 (+ 53061us) log.cc:844] Serialized 26 byte log entry
0709 12:22:59.850364 (+ 86830us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:59.850423 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.850498 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.685571 (+1835073us) spinlock_profiling.cc:243] Waited 53.2 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:01.966911 (+281340us) raft_consensus.cc:1685] finished
0709 12:23:01.966942 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.967236 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122263296,"thread_start_us":45512,"threads_started":1,"wal-append.queue_time_us":53077}
I20260709 12:23:02.150521  4636 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:02.199128  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 243060us
I20260709 12:23:02.209408  4636 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:02.210038  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2310) took 1862 ms (client timeout 1769 ms). Trace:
W20260709 12:23:02.210517  1877 rpcz_store.cc:269] 0709 12:23:00.347927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.348043 (+   116us) service_pool.cc:224] Handling call
0709 12:23:02.209836 (+1861793us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b54000f0854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:02.209969 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1650585856}
I20260709 12:23:02.161234  4432 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:01.727824  4597 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.215384  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:02.251699  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.972934  4643 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:02.004797  4312 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:01.761760  4266 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:02.099871  4650 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.058637  4458 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:01.914182  4515 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:02.205986  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:02.216020  2412 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:01.947981  4422 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:02.100689  4263 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:02.002296  4212 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:02.290069  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:01.971227  4562 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.216138  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:02.190824  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.185978  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:02.309448  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.309952  1872 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:02.196478  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.211591  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.192787  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:02.210083  2383 connection.cc:256] RPC call timeout handler was delayed by -1.717051972000263s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25520000000000004s for a response
I20260709 12:23:02.169549  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:02.315843  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 146356us
W20260709 12:23:02.169341  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.319001  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.204499  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.201896  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.153349  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:02.248781  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.219951  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:02.322741  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 102900us
I20260709 12:23:02.324165  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:01.948240  4426 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.051106  4594 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.255190  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:02.334015  1732 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:01.907912  4629 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:01.750535  4549 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:02.252197  1880 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:02.259596  1726 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:02.257489  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:02.338825  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:02.061179  4388 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.272289  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:02.288031  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.205173  4637 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.579s	user 0.010s	sys 0.000s
I20260709 12:23:01.918452  4500 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.295444  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.349159  1875 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.187266  4630 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:02.161965  4595 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:02.216138  1772 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:02.186765  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:02.298333  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 384486us
I20260709 12:23:02.258919  4597 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:02.298121  1724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:02.361168  1724 raft_consensus.cc:606] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:23:02.307170  4515 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:02.191536  4660 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:23:02.362213  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 146167us
I20260709 12:23:02.309222  1881 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:02.321593  1876 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:23:02.321540  1731 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:23:02.295410  4459 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:02.331729  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46645.6us
I20260709 12:23:02.286414  4266 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:02.329639  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.568s (SENT) [suppressed 37 similar messages]
I20260709 12:23:02.208314  4588 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.342334  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.378185  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.378880  1880 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.250038  4227 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:02.345687  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.308032  4548 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.339702  1728 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:02.391234  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:02.359439  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:02.392174  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:02.361701  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:02.368026  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.409210  1877 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:02.295691  4226 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:23:02.372363  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.372406  4638 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:02.331133  4635 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:02.340337  4432 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:02.388314  1872 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:02.424556  1728 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20260709 12:23:02.364064  4616 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:02.403240  1801 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:02.399765  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.362347  4595 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:02.404049  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44771.8us
I20260709 12:23:02.352669  4668 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:02.403349  1727 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.367233  4623 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:02.311628  4263 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:02.343073  4643 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
W20260709 12:23:02.412575  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50975.9us
I20260709 12:23:02.412559  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:02.412175  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:02.381173  4395 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:02.471306  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 59227.8us
I20260709 12:23:02.422189  4638 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:02.422770  4266 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.425812  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.479195  1876 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:02.480242  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.308539  4650 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.363072  4637 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:02.424990  4666 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:02.353262  4594 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:02.344143  4426 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:02.448231  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.500469  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.454417  1881 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
W20260709 12:23:02.501039  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 176985us
I20260709 12:23:02.373692  4500 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.503018  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:02.464148  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:02.506807  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 177178us
I20260709 12:23:02.466768  1874 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:02.507720  1872 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:02.508333  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.423489  4227 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:23:02.424513  4635 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.473403  1805 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:02.482892  1728 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:02.472893  4466 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:02.384261  4459 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.455829  4623 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.412002  4515 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:23:02.364568  4388 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:02.466574  4616 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.455107  4432 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:02.386946  4422 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.495208  1876 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:02.530082  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.480156  4630 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:02.503564  1730 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:23:02.465355  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:02.470971  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58495.9us
I20260709 12:23:02.533663  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:02.534091  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.535297  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:02.493372  4666 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.517393  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:02.520826  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.150s (SENT)
I20260709 12:23:02.518774  1729 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:23:02.525987  1878 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:02.478390  4395 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.531165  1876 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:23:02.579872  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:02.512148  4466 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:02.535450  1872 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:23:02.483671  4413 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:02.490767  4488 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260709 12:23:02.511893  4635 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:02.582540  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:02.608160  1876 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:23:02.613842  1876 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260709 12:23:02.517547  4623 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.531383  4616 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:02.621860  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101106us
I20260709 12:23:02.622185  1881 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 3. Current term is 4. Ops: []
W20260709 12:23:02.623270  4667 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
I20260709 12:23:02.623325  4590 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:23:02.627934  4672 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:23:02.628085  4549 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.622668  4660 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:23:02.627583  4432 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:02.628726  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46285us
I20260709 12:23:02.675845  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:02.628552  4638 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.629202  4643 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:02.677821  1872 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: [2.2-6.4]
I20260709 12:23:02.632030  4594 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:02.633344  4637 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:02.634020  4665 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260709 12:23:02.485945  4629 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.634539  4635 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:02.640967  1876 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: [5.4-6.5]
I20260709 12:23:02.642458  1880 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:02.695439  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.696424  1880 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:02.697396  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.649783  1877 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:23:02.547243  4630 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.698500  4630 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:02.698616  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1671) took 1001 ms. Trace:
I20260709 12:23:02.699033  1877 rpcz_store.cc:276] 0709 12:23:01.697393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.580394 (+883001us) service_pool.cc:224] Handling call
0709 12:23:02.649427 (+ 69033us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:02.698366 (+ 48939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.698562 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:02.700018  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.701112  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.702174  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.703379  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.611159  4466 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:02.704641  1877 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.630889  4650 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:02.681825  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.682121  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:02.683037  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 213922us
W20260709 12:23:02.684381  1798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.005s	sys 0.009s
I20260709 12:23:02.697942  1880 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:02.709566  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:02.710621  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.579386  4647 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:02.618086  4666 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.711800  1881 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:02.704059  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.732568  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:02.733407  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.692948  4488 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:23:02.737033  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 201831us
I20260709 12:23:02.692394  4588 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:02.724378  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.674485  4590 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:02.727777  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:02.729549  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:02.684012  4643 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.731027  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:02.731608  1874 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:02.684754  4676 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:23:02.720551  1798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.006s	sys 0.009s
I20260709 12:23:02.739676  1872 raft_consensus.cc:2955] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:02.734251  4637 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:02.736482  4488 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.747666  4500 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:02.748247  4594 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:02.750535  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:02.728392  4647 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:02.758625  1877 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:02.762428  4643 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:02.776764  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.777557  1877 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:02.778352  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.779560  1877 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:02.751462  4676 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:02.782881  4577 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:02.783455  4577 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:02.783885  4577 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:02.783939  4676 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:02.785581  4676 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:02.769122  4488 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:02.786971  1877 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:23:02.787714  4678 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.171s	user 0.001s	sys 0.000s
I20260709 12:23:02.788403  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.789228  1877 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:02.790057  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.790611  1877 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:02.765120  2411 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 726) took 1694 ms (client timeout 1787 ms). Trace:
W20260709 12:23:02.793210  2411 rpcz_store.cc:269] 0709 12:23:01.056832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.057029 (+   197us) service_pool.cc:224] Handling call
0709 12:23:02.751101 (+1694072us) spinlock_profiling.cc:243] Waited 520 ms on lock 0x7b5400489754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:02.751247 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1195379584}
I20260709 12:23:02.794294  4419 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:23:02.797889  1772 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33790 dropped due to backpressure: service queue is full with 50 items [suppressed 8 similar messages]
W20260709 12:23:02.800930  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.806170  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 443477us
I20260709 12:23:02.769045  4596 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
I20260709 12:23:02.808735  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:02.810228  4647 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:02.810683  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 362) took 1103 ms. Trace:
I20260709 12:23:02.811199  1798 rpcz_store.cc:276] 0709 12:23:01.706757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.721258 (+ 14501us) service_pool.cc:224] Handling call
0709 12:23:01.917726 (+196468us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:02.004332 (+ 86606us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:02.004378 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:02.124144 (+119766us) log.cc:844] Serialized 52 byte log entry
0709 12:23:02.754683 (+630539us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:02.772469 (+ 17786us) log.cc:844] Serialized 14 byte log entry
0709 12:23:02.809587 (+ 37118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.809661 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:02.809704 (+    43us) raft_consensus.cc:1685] finished
0709 12:23:02.809716 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:02.809843 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":127361,"prepare.queue_time_us":17579,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":115737,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":52248,"thread_start_us":630087,"threads_started":1,"wal-append.queue_time_us":649034}
I20260709 12:23:02.814198  4388 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:02.816507  4630 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.818650  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:02.822566  1877 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:02.814641  1805 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:02.710682  4680 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:23:02.771713  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:02.687608  4549 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.757884  1875 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:02.772485  1627 catalog_manager.cc:5697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:02.830235  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1685) took 1079 ms. Trace:
I20260709 12:23:02.830617  1875 rpcz_store.cc:276] 0709 12:23:01.751071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.262271 (+511200us) service_pool.cc:224] Handling call
0709 12:23:02.830158 (+567887us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:02.831561  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:02.832132  1875 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:02.766605  4263 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:02.833700  1880 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:02.823755  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:02.825001  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53375.8us
I20260709 12:23:02.836539  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:02.829558  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:02.826946  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 364) took 1112 ms. Trace:
I20260709 12:23:02.838078  1804 rpcz_store.cc:276] 0709 12:23:01.707322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.720549 (+ 13227us) service_pool.cc:224] Handling call
0709 12:23:02.818480 (+1097931us) spinlock_profiling.cc:243] Waited 8.36 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:23:02.819496 (+  1016us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19221760}
I20260709 12:23:02.848747  1877 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:23:02.826700  4639 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.870137  1875 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:02.826249  4680 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:02.873005  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:02.834540  4630 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:02.834214  4632 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:02.874014  4687 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:02.884922  4506 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:02.889400  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.893711  1872 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:02.840664  4388 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:02.821745  4616 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:02.908233  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:02.837752  4649 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.057s
I20260709 12:23:02.875753  4594 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.876345  4426 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:02.798333  1772 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33790 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1596)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1605)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1598)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1595)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1597)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1599)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 369)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1493)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1587)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1591)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1592)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1603)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1606)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1503)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1515)
W20260709 12:23:02.976052  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.916712  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:02.949712  1727 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:02.886252  4212 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:03.011294  1875 raft_consensus.cc:2368] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7.
I20260709 12:23:02.880417  4500 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.877681  4680 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:03.017267  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:02.880131  4429 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:03.022266  2412 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:03.026572  4515 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.027376  4515 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.029943  4515 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:02.897329  4640 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:02.897125  4473 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:02.987854  4488 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:02.933277  4637 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:23:03.041899  4488 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.022393  4429 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.031065  4500 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:03.017828  4691 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.085s	user 0.002s	sys 0.000s
W20260709 12:23:03.032464  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 115862us
I20260709 12:23:03.035532  4473 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.947530  4687 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:03.051565  4549 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.080278  4687 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.081053  4633 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.077752  4616 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.024842  4594 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.083117  4687 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.085006  4594 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:03.088685  4548 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.683s	user 0.008s	sys 0.003s
I20260709 12:23:03.089672  2415 raft_consensus.cc:2410] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:23:03.090998  4632 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.091573  4632 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.017200  4459 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.035115  4640 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.093227  4666 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.091917  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 679) took 1166 ms. Trace:
I20260709 12:23:02.887302  4639 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.095091  4640 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:03.068485  4649 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:23:03.096527  4649 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:03.096782  4639 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.062742  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.097715  2415 rpcz_store.cc:276] 0709 12:23:01.925528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.925718 (+   190us) service_pool.cc:224] Handling call
0709 12:23:03.091667 (+1165949us) spinlock_profiling.cc:243] Waited 936 ms on lock 0x7b5400580f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:03.091848 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2153555200}
W20260709 12:23:03.064816  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.108606  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.109062  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:03.109486  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.113858  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.114984  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.115607  1877 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:03.033059  4700 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:03.119485  1726 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:03.125926  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.129788  1726 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 6 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
W20260709 12:23:03.082718  4595 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.614s	user 0.010s	sys 0.000s
I20260709 12:23:03.138186  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.139961  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.142272  1731 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:03.140610  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.143962  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:03.044723  1773 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:33790 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1598)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1595)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1597)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1599)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 369)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1493)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1587)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1591)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1592)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1603)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1606)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1503)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1607)
I20260709 12:23:03.149411  1773 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:33802 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1598)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1595)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1597)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1599)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 369)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1493)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1587)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1591)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1592)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1603)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1606)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1503)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1607)
I20260709 12:23:03.155260  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:03.156361  1800 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 2. Current term is 4. Ops: []
I20260709 12:23:03.096872  4649 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:03.054684  4429 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.064857  4426 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.112594  4473 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:03.159363  4633 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:23:03.172380  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1692) took 1201 ms. Trace:
I20260709 12:23:03.172919  1880 rpcz_store.cc:276] 0709 12:23:01.970943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.741215 (+770272us) service_pool.cc:224] Handling call
0709 12:23:02.833372 (+ 92157us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:02.834234 (+   862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.172143 (+337909us) spinlock_profiling.cc:243] Waited 91.8 ms on lock 0x7b5400223b50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:03.172311 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211194240}
I20260709 12:23:03.165226  4500 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:03.175817  4661 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.762s	user 0.000s	sys 0.003s
I20260709 12:23:03.165318  4647 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.166359  4649 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.098688  4459 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:03.183518  4459 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.184092  4459 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:03.162552  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.159636  4515 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:23:03.185725  4669 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.665s	user 0.000s	sys 0.001s
I20260709 12:23:03.160248  1726 raft_consensus.cc:606] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:23:03.190968  1807 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:03.163843  1773 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:33802 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1598)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1595)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1597)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1599)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 369)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1493)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1587)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1591)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1592)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1603)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1606)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1503)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1607)
I20260709 12:23:03.183949  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.195199  1806 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:03.119874  4548 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:03.195791  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 365) took 1488 ms. Trace:
I20260709 12:23:03.180425  4395 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.200698  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:03.210723  1877 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:03.207626  1773 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33802 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1598)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1595)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1597)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1599)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 369)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1493)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1587)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1591)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1592)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1603)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1606)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1503)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1607)
I20260709 12:23:03.203279  1880 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:03.213843  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.214735  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.140527  4676 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:03.211794  1773 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:33802 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1598)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1595)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1593)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1602)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1585)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1597)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1586)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1589)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 367)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1588)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1599)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1584)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1590)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1600)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 369)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 368)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1504)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1495)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1492)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1506)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1508)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1493)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1587)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1591)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1592)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1603)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1606)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1503)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1607)
I20260709 12:23:03.218740  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 359) took 1512 ms. Trace:
I20260709 12:23:03.208215  1806 rpcz_store.cc:276] 0709 12:23:01.707605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.721027 (+ 13422us) service_pool.cc:224] Handling call
0709 12:23:03.183755 (+1462728us) spinlock_profiling.cc:243] Waited 4.71 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:23:03.195727 (+ 11972us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10838784}
I20260709 12:23:03.102216  4466 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.224985  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.231554  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.232218  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.102456  4623 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.236917  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.237605  1727 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:03.233803  1728 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
W20260709 12:23:03.240953  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.141063  4700 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:03.167133  4426 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.244725  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:03.227303  4597 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.847s	user 0.013s	sys 0.000s
I20260709 12:23:03.254491  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.230587  1802 rpcz_store.cc:276] 0709 12:23:01.705948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.720242 (+ 14294us) service_pool.cc:224] Handling call
0709 12:23:01.720611 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:01.771095 (+ 50484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:01.771144 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:01.975489 (+204345us) log.cc:844] Serialized 26 byte log entry
0709 12:23:02.404944 (+429455us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:02.404985 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.405054 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:03.184950 (+779896us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:03.193810 (+  8860us) raft_consensus.cc:1685] finished
0709 12:23:03.193846 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:03.218670 (+ 24824us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77564,"raft.queue_time_us":144222,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":39912,"spinlock_wait_cycles":7232256,"thread_start_us":296943,"threads_started":1,"wal-append.queue_time_us":435407}
I20260709 12:23:03.256949  1806 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:03.257349  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:03.257500  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 186101us
I20260709 12:23:03.216559  4647 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.259682  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:03.260891  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.261546  1877 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:03.262125  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.263651  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:03.264325  1872 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:03.160115  4595 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:03.266229  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1598) took 1540 ms (client timeout 1738 ms). Trace:
W20260709 12:23:03.267498  1806 rpcz_store.cc:269] 0709 12:23:01.726055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.236074 (+1510019us) service_pool.cc:224] Handling call
0709 12:23:03.266156 (+ 30082us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.270054  1801 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:03.270768  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1595) took 1535 ms (client timeout 1809 ms). Trace:
I20260709 12:23:03.271238  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:03.271384  1802 rpcz_store.cc:269] 0709 12:23:01.725359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.256987 (+1531628us) service_pool.cc:224] Handling call
0709 12:23:03.260786 (+  3799us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.272917  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.273393  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.195439  4615 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:23:03.239799  4515 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:03.274132  1731 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:03.282465  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:03.277887  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.230175  4549 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:23:03.278748  4702 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:03.284757  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.265743  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.287161  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:03.293578  2412 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 2.
I20260709 12:23:03.245529  4676 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:03.298080  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:03.283097  2415 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:03.278065  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1602) took 1551 ms (client timeout 1969 ms). Trace:
I20260709 12:23:03.284718  1727 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:03.196772  4643 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.277087  4395 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:03.274053  4506 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:03.294935  1799 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:03.295382  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:03.295436  1872 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
W20260709 12:23:03.296785  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.281309  4638 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:03.277127  4666 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.308702  4683 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:03.281757  4515 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.309602  4705 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.347719  1627 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 4, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:03.348764  4705 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.294032  4630 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.044s
I20260709 12:23:03.352497  4705 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.294553  1877 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:03.256240  4577 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:03.299808  1802 rpcz_store.cc:269] 0709 12:23:01.727024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.272540 (+1545516us) service_pool.cc:224] Handling call
0709 12:23:03.278005 (+  5465us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.264394  4597 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.324550  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.323400  4702 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.327056  4432 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:23:03.327399  4549 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:03.332320  4459 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.334132  4473 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.301648  4615 consensus_queue.cc:1243] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:23:03.339190  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.341089  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:03.342376  4638 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:03.301589  4605 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:03.301397  4458 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.298430  4632 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.310951  4395 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:03.346417  4515 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:03.336555  4506 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:03.346566  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1698) took 1374 ms. Trace:
I20260709 12:23:03.349867  1627 catalog_manager.cc:5697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 1 to 3, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:03.376411  1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 366) took 1479 ms (client timeout 1971 ms). Trace:
W20260709 12:23:03.381490  1626 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 254) took 19270 ms (client timeout 15000 ms). Trace:
I20260709 12:23:03.387264  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.410413  1731 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:03.408628  1726 raft_consensus.cc:606] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:03.354516  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:03.356997  2415 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:03.363183  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.377619  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1676) took 1674 ms. Trace:
W20260709 12:23:03.433189  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1585) took 1715 ms (client timeout 1980 ms). Trace:
I20260709 12:23:03.334916  4426 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.433211  1873 rpcz_store.cc:276] 0709 12:23:01.703264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.582514 (+879250us) service_pool.cc:224] Handling call
0709 12:23:02.635774 (+ 53260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:02.670059 (+ 34285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:02.670110 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:02.670175 (+    65us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:02.670222 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.377381 (+707159us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5400025b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:03.377427 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:03.377563 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34636544}
I20260709 12:23:03.433537  1877 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:03.387768  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.454094  1724 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:03.457682  1872 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:03.459306  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.460242  1728 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:03.369416  4459 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:03.467923  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.469621  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.471390  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.471750  4702 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.436470  4615 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:03.473323  4702 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:03.473714  4615 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.473820  4702 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.476622  4473 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.411535  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.481297  1731 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:03.482147  1732 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:03.483161  4643 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:03.433686  1802 rpcz_store.cc:269] 0709 12:23:01.717623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.362864 (+1645241us) service_pool.cc:224] Handling call
0709 12:23:03.433127 (+ 70263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.484346  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 357) took 1779 ms. Trace:
W20260709 12:23:03.409848  1626 rpcz_store.cc:269] 0709 12:22:44.133380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.133766 (+   386us) service_pool.cc:224] Handling call
0709 12:22:45.401840 (+1268074us) spinlock_profiling.cc:243] Waited 981 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd42090 00007f16abde0f3a 00007f16b1b96e9c 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:23:03.381134 (+17979294us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:52.989696 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:52.990581 (+   885us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:52.990642 (+    61us) write_op.cc:435] Acquired schema lock
0709 12:22:52.990668 (+    26us) tablet.cc:662] Decoding operations
0709 12:22:53.037904 (+ 47236us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:53.038003 (+    99us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:53.038036 (+    33us) tablet.cc:685] Acquiring locks for 22 operations
0709 12:22:53.040075 (+  2039us) tablet.cc:701] Row locks acquired
0709 12:22:53.040144 (+    69us) write_op.cc:260] PREPARE: finished
0709 12:22:53.040594 (+   450us) write_op.cc:270] Start()
0709 12:22:53.040796 (+   202us) write_op.cc:276] Timestamp: P: 1783599773040482 usec, L: 0
0709 12:22:53.040820 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 12:22:53.129764 (+ 88944us) log.cc:844] Serialized 8704 byte log entry
0709 12:22:54.292835 (+1163071us) op_driver.cc:464] REPLICATION: finished
0709 12:22:57.141107 (+2848272us) spinlock_profiling.cc:243] Waited 6.71 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16a8cce0ac 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16af80eef0
0709 12:22:57.345892 (+204785us) write_op.cc:301] APPLY: starting
0709 12:22:57.346002 (+   110us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:57.346414 (+   412us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:57.346453 (+    39us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:57.647572 (+301119us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:57.648019 (+   447us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=22
0709 12:22:57.648068 (+    49us) write_op.cc:312] APPLY: finished
0709 12:22:57.648559 (+   491us) log.cc:844] Serialized 152 byte log entry
0709 12:22:57.690520 (+ 41961us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:57.693538 (+  3018us) write_op.cc:454] Released schema lock
0709 12:22:57.693676 (+   138us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:57.694192 (+   516us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:57.694213 (+    21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":39014,"reported_tablets":36,"spinlock_wait_cycles":2257341824,"child_traces":[["op",{"apply.queue_time_us":3046948,"apply.run_cpu_time_us":23167,"apply.run_wall_time_us":348284,"num_ops":22,"prepare.queue_time_us":128457,"prepare.run_cpu_time_us":25076,"prepare.run_wall_time_us":541470,"raft.queue_time_us":383154,"raft.run_cpu_time_us":69789,"raft.run_wall_time_us":3138758,"replication_time_us":1194052,"spinlock_wait_cycles":15431552,"thread_start_us":3547279,"threads_started":5,"wal-append.queue_time_us":482274,"wal-append.run_cpu_time_us":51912,"wal-append.run_wall_time_us":2573147}]]}
I20260709 12:23:03.408185  1874 rpcz_store.cc:276] 0709 12:23:01.972577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.507972 (+535395us) service_pool.cc:224] Handling call
0709 12:23:03.346317 (+838345us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b5400026554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:03.346498 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":506369792}
I20260709 12:23:03.484753  1800 rpcz_store.cc:276] 0709 12:23:01.705371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.718078 (+ 12707us) service_pool.cc:224] Handling call
0709 12:23:02.871190 (+1153112us) spinlock_profiling.cc:243] Waited 2.09 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:23:02.871424 (+   234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:03.159884 (+288460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.484096 (+324212us) spinlock_profiling.cc:243] Waited 269 ms on lock 0x7b5400067054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:03.484286 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":622824576}
I20260709 12:23:03.485839  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:03.486109  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:03.408740  1729 rpcz_store.cc:269] 0709 12:23:01.897070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.920925 (+ 23855us) service_pool.cc:224] Handling call
0709 12:23:03.376229 (+1455304us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b54000d20d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:03.376350 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":851483392}
W20260709 12:23:03.388772  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1686) took 1497 ms (client timeout 1809 ms). Trace:
I20260709 12:23:03.490126  1803 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 8. Ops: []
I20260709 12:23:03.484524  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:03.489661  1878 rpcz_store.cc:269] 0709 12:23:01.891363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.162138 (+270775us) service_pool.cc:224] Handling call
0709 12:23:03.388470 (+1226332us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:03.388673 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":493582720}
I20260709 12:23:03.483265  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:03.455681  1627 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 4, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:03.494102  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.497346  1724 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:03.428092  1726 raft_consensus.cc:614] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:03.495631  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1586) took 1777 ms (client timeout 2000 ms). Trace:
I20260709 12:23:03.358147  4466 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:03.499402  1800 rpcz_store.cc:269] 0709 12:23:01.717975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.485814 (+1767839us) service_pool.cc:224] Handling call
0709 12:23:03.495570 (+  9756us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.355803  4683 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:03.499948  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.507726  1726 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:03.509517  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.383373  4595 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.370111  4697 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:03.379951  4597 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.521072  1726 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:03.409113  4549 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.376729  4623 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.527465  4506 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.369863  4432 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:03.528807  4432 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:03.496423  4605 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:03.531098  4549 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:03.531766  4605 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.361187  4700 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.539745  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.540807  1732 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:03.541352  4638 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.541955  4697 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:03.505012  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.429152  4395 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:03.510053  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.544939  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.514546  1877 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:03.507891  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.338652  4680 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.517087  4577 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:03.552088  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47148.8us
I20260709 12:23:03.517422  4683 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:03.518612  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.497236  4409 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:03.527842  4706 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
W20260709 12:23:03.550505  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1589) took 1827 ms (client timeout 2024 ms). Trace:
W20260709 12:23:03.568542  1800 rpcz_store.cc:269] 0709 12:23:01.723451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.507532 (+1784081us) service_pool.cc:224] Handling call
0709 12:23:03.550427 (+ 42895us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.371850  4676 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.543118  4697 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.573170  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.569271  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.573195  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 680) took 1042 ms. Trace:
I20260709 12:23:03.570029  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:03.576606  1873 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:03.583192  1877 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:23:03.590876  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.591292  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.591619  1726 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:03.591877  1732 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:03.592741  1731 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:23:03.568614  4675 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:03.609932  4602 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:03.610709  4602 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.615424  4602 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.608227  1731 raft_consensus.cc:2955] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Committing config change with OpId 5.6: config changed from index 3 to 6, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:03.577620  4660 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:03.576062  2416 rpcz_store.cc:276] 0709 12:23:02.530512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.530669 (+   157us) service_pool.cc:224] Handling call
0709 12:23:02.530966 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:02.531178 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:02.821836 (+290658us) log.cc:844] Serialized 14 byte log entry
0709 12:23:03.013040 (+191204us) log.cc:844] Serialized 14 byte log entry
0709 12:23:03.390914 (+377874us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b180034bca0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:23:03.572755 (+181841us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:03.572837 (+    82us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:03.572893 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.572972 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:03.573113 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":53834,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":85,"raft.queue_time_us":19591,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":19134,"spinlock_wait_cycles":7736704,"thread_start_us":609677,"threads_started":3,"wal-append.queue_time_us":343470}
I20260709 12:23:03.394526  4590 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.597790  4562 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:03.634426  1726 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:03.632032  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.641950  1800 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
W20260709 12:23:03.642647  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 367) took 1926 ms (client timeout 2043 ms). Trace:
W20260709 12:23:03.643179  1800 rpcz_store.cc:269] 0709 12:23:01.716043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.603922 (+1887879us) service_pool.cc:224] Handling call
0709 12:23:03.642569 (+ 38647us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.599512  4639 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:23:03.627593  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.622306  1873 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:03.652501  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.636935  1724 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:23:03.584736  4409 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.624652  4615 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:03.675341  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:03.675482  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.633210  4675 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:03.637144  4702 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.693447  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.674906  4248 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:03.707981  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.711794  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.673564  4636 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:03.712352  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.723011  1880 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:03.676016  4639 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:03.726713  4639 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:03.730150  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.726308  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.737905  4248 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:03.738976  4248 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.742677  4702 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.743417  4702 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.746953  1875 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:03.697048  1873 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:03.767251  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:03.768246  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:03.745225  4590 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:03.771982  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.773669  1724 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:03.775637  1875 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:03.776028  1875 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:03.765030  4548 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.544s	user 0.015s	sys 0.000s
I20260709 12:23:03.780802  1724 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:03.781420  1726 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:23:03.776683  1875 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.787724  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.788044  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.788517  1732 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:03.792070  4702 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:03.797914  1875 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:03.799304  2416 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260709 12:23:03.801303  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.802073  1723 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
W20260709 12:23:03.807457  4709 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:23:03.813220  4635 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:03.822321  4680 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:03.822723  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1605) took 2076 ms (client timeout 1714 ms). Trace:
W20260709 12:23:03.823172  1798 rpcz_store.cc:269] 0709 12:23:01.746184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.822449 (+2076265us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:03.822613 (+   164us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:03.823860  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.824292  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.825099  1798 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
W20260709 12:23:03.831207  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1599) took 2104 ms (client timeout 2184 ms). Trace:
W20260709 12:23:03.832032  1798 rpcz_store.cc:269] 0709 12:23:01.726290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.823972 (+2097682us) service_pool.cc:224] Handling call
0709 12:23:03.831143 (+  7171us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.833806  1728 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:03.827324  2383 leader_election.cc:400] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7.
I20260709 12:23:03.836040  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:03.781526  4675 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:03.829759  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:03.838140  2416 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:03.785916  4548 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:23:03.746148  4636 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:03.853781  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.882555  1728 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:03.854591  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:03.854693  1723 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:23:03.857918  1724 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:03.884527  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.883198  1726 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:03.860759  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:03.861531  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:03.866519  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:03.892128  2417 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:03.868378  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:03.871901  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:03.873030  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:03.877338  2419 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:03.897482  1726 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:03.891535  2412 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:23:03.901751  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1584) took 2184 ms (client timeout 2220 ms). Trace:
W20260709 12:23:03.902305  1798 rpcz_store.cc:269] 0709 12:23:01.717232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.832949 (+2115717us) service_pool.cc:224] Handling call
0709 12:23:03.901662 (+ 68713us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.905196  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:03.879895  1875 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:23:03.822428  4409 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:03.909179  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:03.918642  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:03.922519  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 92832.5us
I20260709 12:23:03.920015  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:03.931932  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1590) took 2208 ms (client timeout 2222 ms). Trace:
W20260709 12:23:03.932384  1798 rpcz_store.cc:269] 0709 12:23:01.723686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.903216 (+2179530us) service_pool.cc:224] Handling call
0709 12:23:03.931838 (+ 28622us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.935468  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:03.959827  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:03.911818  2411 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:23:03.851223  4702 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:03.909509  4506 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:03.912405  4605 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:03.937915  4683 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:03.972200  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:03.972708  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.909099  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:03.995832  1875 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:03.978149  1732 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:23:03.980381  1723 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:23:03.921206  4548 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:03.983094  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 138361us
I20260709 12:23:03.995853  1798 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:03.957290  4643 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:04.006157  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1600) took 2279 ms (client timeout 2230 ms). Trace:
W20260709 12:23:04.006596  1798 rpcz_store.cc:269] 0709 12:23:01.726540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.933264 (+2206724us) service_pool.cc:224] Handling call
0709 12:23:04.006073 (+ 72809us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:03.973603  4637 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:04.010094  1875 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:03.975135  4702 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.012849  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:03.984622  4506 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.013783  1875 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:23:04.026938  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.027295  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:03.971117  4629 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 12:23:04.032528  4395 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:04.056300  1875 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:23:04.035593  4717 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
W20260709 12:23:04.036501  4713 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
W20260709 12:23:04.037112  4681 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.371s	user 0.007s	sys 0.000s
I20260709 12:23:04.040076  4266 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:04.042052  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 360) took 2335 ms. Trace:
I20260709 12:23:04.044549  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:04.046398  1874 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:04.048398  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.073577  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 365) took 2176 ms. Trace:
W20260709 12:23:04.055174  4719 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260709 12:23:04.058743  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 369) took 1577 ms (client timeout 1629 ms). Trace:
W20260709 12:23:04.077355  1798 rpcz_store.cc:269] 0709 12:23:02.481129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.007429 (+1526300us) service_pool.cc:224] Handling call
0709 12:23:04.058655 (+ 51226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:04.059126  1875 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:04.016883  4702 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:04.021354  4506 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:04.099499  4718 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:23:04.100952  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:04.101266  4721 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:23:04.074138  1730 rpcz_store.cc:276] 0709 12:23:01.896817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.154048 (+257231us) service_pool.cc:224] Handling call
0709 12:23:04.073254 (+1919206us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54002aa154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:04.073490 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":357203712}
I20260709 12:23:04.072216  1799 rpcz_store.cc:276] 0709 12:23:01.706206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.716864 (+ 10658us) service_pool.cc:224] Handling call
0709 12:23:01.747680 (+ 30816us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:03.314579 (+1566899us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.041749 (+727170us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b5400166cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:04.041977 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3557826432}
I20260709 12:23:04.103933  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:04.088964  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:04.105901  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 368) took 1625 ms (client timeout 1715 ms). Trace:
W20260709 12:23:04.106474  1798 rpcz_store.cc:269] 0709 12:23:02.480829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.088528 (+1607699us) service_pool.cc:224] Handling call
0709 12:23:04.105797 (+ 17269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:04.095373  4616 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.107889  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:04.111195  4616 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:04.115036  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1495) took 1622 ms (client timeout 1834 ms). Trace:
I20260709 12:23:04.014464  4548 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:04.116396  1798 rpcz_store.cc:269] 0709 12:23:02.492454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.107470 (+1615016us) service_pool.cc:224] Handling call
0709 12:23:04.114940 (+  7470us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:04.127655  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:04.128976  1877 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:04.135291  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:04.136482  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1492) took 1654 ms (client timeout 1867 ms). Trace:
W20260709 12:23:04.136957  1798 rpcz_store.cc:269] 0709 12:23:02.481660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.121393 (+1639733us) service_pool.cc:224] Handling call
0709 12:23:04.136400 (+ 15007us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:04.149202  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 177256us
I20260709 12:23:04.150048  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:04.150395  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:04.152931  1877 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:23:04.154357  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:04.161391  1877 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:04.138564  4660 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:04.186504  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:04.186671  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1506) took 1496 ms (client timeout 1671 ms). Trace:
W20260709 12:23:04.191627  1798 rpcz_store.cc:269] 0709 12:23:02.690186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.138929 (+1448743us) service_pool.cc:224] Handling call
0709 12:23:04.186591 (+ 47662us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:04.153237  4728 raft_consensus.cc:993] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: : Instructing follower 123095127f334445889c21a42ba1d41e to start an election
W20260709 12:23:04.196079  1845 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.834s (SENT)
I20260709 12:23:04.146593  4700 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.203864  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:04.208003  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:04.208877  1729 raft_consensus.cc:606] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:04.209300  1729 raft_consensus.cc:614] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:04.212049  4432 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.682s	user 0.014s	sys 0.000s
I20260709 12:23:04.212548  4432 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:04.215201  4674 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:04.152287  4409 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:23:04.216849  4409 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:04.217221  4409 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:04.217458  4616 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:04.195593  4660 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.219372  4636 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:04.219749  4660 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:04.203377  4707 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:04.221922  4707 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.222519  4707 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:04.223052  4629 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:04.222088  4432 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:04.225116  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:04.225847  1729 raft_consensus.cc:606] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:23:04.226163  1729 raft_consensus.cc:614] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:04.227010  4683 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:04.228696  4683 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:23:04.229266   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:04.231441   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:04.232807   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:04.233429  4697 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.234019  4395 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:04.233906  4605 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:04.218086  4409 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:04.237948  4515 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.780s	user 0.012s	sys 0.001s
I20260709 12:23:04.240399  4730 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:04.279641  4730 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:04.280210  4730 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:04.231109  4643 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:04.233947  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:04.243877  2416 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:23:04.277590  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:04.299206  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 149212us
I20260709 12:23:04.316118  1772 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1519) took 1267 ms. Trace:
I20260709 12:23:04.556084  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:04.502686  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.349457  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:04.506904  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 356611us
W20260709 12:23:04.363524  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 451325us
W20260709 12:23:04.546149  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 361) took 2839 ms (client timeout 1566 ms). Trace:
I20260709 12:23:04.554807  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:04.437695  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:04.324424  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.284238  1877 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
W20260709 12:23:04.575387  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:04.575878  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.349001  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:04.576372  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.570508  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:04.578617  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:04.389322  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1594) took 2664 ms (client timeout 1584 ms). Trace:
I20260709 12:23:04.285218  1799 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:04.489565  1876 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:23:04.571095  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:04.241808  4565 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:04.586338  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:04.583119  1731 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:23:04.556203  1772 rpcz_store.cc:276] 0709 12:23:03.048315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.316064 (+1267749us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:04.588089  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:04.588472  1772 service_pool.cc:136] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:33790 dropped due to backpressure: service queue is full with 50 items [suppressed 5 similar messages]
W20260709 12:23:04.585887  1845 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.069s (SENT)
W20260709 12:23:04.586097  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:04.559469  1807 rpcz_store.cc:269] 0709 12:23:01.706492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.708086 (+  1594us) service_pool.cc:224] Handling call
0709 12:23:02.182668 (+474582us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:23:04.545835 (+2363167us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b5400167bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:04.546015 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1881163392}
W20260709 12:23:04.583950  1801 rpcz_store.cc:269] 0709 12:23:01.725115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.484500 (+759385us) service_pool.cc:224] Handling call
0709 12:23:04.388991 (+1904491us) spinlock_profiling.cc:243] Waited 761 ms on lock 0x7b5400066654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:04.389209 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1750317952}
I20260709 12:23:04.589712  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:04.284682  4549 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.753s	user 0.008s	sys 0.007s
I20260709 12:23:04.587751  1731 pending_rounds.cc:85] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 3
I20260709 12:23:04.594102  1727 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260709 12:23:04.594368  1731 pending_rounds.cc:107] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:23:04.594498  1727 pending_rounds.cc:85] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 3
I20260709 12:23:04.594686  1731 raft_consensus.cc:2889] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:04.594558  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:04.555107  4590 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.773s	user 0.009s	sys 0.005s
I20260709 12:23:04.594769  1727 pending_rounds.cc:107] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260709 12:23:04.596895  1727 raft_consensus.cc:2889] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:04.597631  1727 pending_rounds.cc:107] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:23:04.597833  1727 raft_consensus.cc:2889] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:04.452986  4637 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:04.587527  4605 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:04.621865  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1601) took 2895 ms (client timeout 1504 ms). Trace:
I20260709 12:23:04.406092  4562 raft_consensus.cc:1081] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Signalling peer 123095127f334445889c21a42ba1d41e to start an election
I20260709 12:23:04.403815  4728 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:04.588811  1772 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:33790 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1511)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1608)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1496)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1497)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1604)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1491)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1513)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1498)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1517)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1499)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1507)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1509)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1514)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1494)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1512)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1493)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1616)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1510)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1614)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1615)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1502)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1500)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1501)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1587)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1591)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1592)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1603)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1606)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1503)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1505)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1515)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1607)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1613)
I20260709 12:23:04.623713  4426 raft_consensus.cc:2764] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:23:04.613329  4738 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:04.624768  4737 raft_consensus.cc:2764] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:23:04.616923  4638 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.009s
W20260709 12:23:04.617126  4506 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.521s	user 0.012s	sys 0.002s
I20260709 12:23:04.654187  4506 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:04.655232  4588 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 12:23:04.655992  4588 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.656636  4588 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:04.657255  4506 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:04.625970  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 363) took 2918 ms. Trace:
I20260709 12:23:04.660495  1803 rpcz_store.cc:276] 0709 12:23:01.707047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.721111 (+ 14064us) service_pool.cc:224] Handling call
0709 12:23:03.378810 (+1657699us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:23:03.379010 (+   200us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:03.490572 (+111562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.625625 (+1135053us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54001e4454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:04.625895 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263255808}
W20260709 12:23:04.662118  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1496) took 2169 ms (client timeout 2069 ms). Trace:
W20260709 12:23:04.626827  4702 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.536s	user 0.005s	sys 0.007s
I20260709 12:23:04.663208  4702 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:04.664677  2418 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:04.665565  4738 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:04.574623  4694 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:04.666819  4738 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:04.315451  4515 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:04.667688  4694 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:04.670275  4694 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:04.672675  1873 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:04.666146  4702 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:04.234090  4697 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:04.633098  1805 rpcz_store.cc:269] 0709 12:23:01.726759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.363389 (+636630us) service_pool.cc:224] Handling call
0709 12:23:04.621549 (+2258160us) spinlock_profiling.cc:243] Waited 81.4 ms on lock 0x7b5400066154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:04.621760 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187287808}
W20260709 12:23:04.688295  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1497) took 2194 ms (client timeout 2110 ms). Trace:
W20260709 12:23:04.688787  1805 rpcz_store.cc:269] 0709 12:23:02.493484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.688028 (+2194544us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:04.688212 (+   184us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:04.583559  4666 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:23:04.641637  1798 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:04.649076  4488 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:04.697585  4488 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:04.617580  4629 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:23:04.652235  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 349) took 1958 ms. Trace:
I20260709 12:23:04.699108  1876 rpcz_store.cc:276] 0709 12:23:02.694222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.733160 (+ 38938us) service_pool.cc:224] Handling call
0709 12:23:04.651950 (+1918790us) spinlock_profiling.cc:243] Waited 1.75 s on lock 0x7b5400223654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:04.652149 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4036366080}
I20260709 12:23:04.698907  4488 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:04.583278  4643 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:23:04.662622  1803 rpcz_store.cc:269] 0709 12:23:02.492815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.661827 (+2169012us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:04.662003 (+   176us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:04.700956  4488 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:23:04.701562  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1491) took 2220 ms (client timeout 2168 ms). Trace:
W20260709 12:23:04.702057  1803 rpcz_store.cc:269] 0709 12:23:02.481412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.701335 (+2219923us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:04.701487 (+   152us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:23:04.703051  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1513) took 1906 ms (client timeout 1869 ms). Trace:
W20260709 12:23:04.703420  1803 rpcz_store.cc:269] 0709 12:23:02.796162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.702835 (+1906673us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:04.702987 (+   152us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:23:04.704195  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1498) took 2210 ms (client timeout 2174 ms). Trace:
W20260709 12:23:04.704501  1803 rpcz_store.cc:269] 0709 12:23:02.493813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.704064 (+2210251us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:04.704150 (+    86us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:23:04.705379  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1517) took 1908 ms (client timeout 1872 ms). Trace:
W20260709 12:23:04.711603  1803 rpcz_store.cc:269] 0709 12:23:02.797223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.705197 (+1907974us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:04.705324 (+   127us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:23:04.713133  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1499) took 2219 ms (client timeout 2189 ms). Trace:
W20260709 12:23:04.713698  1803 rpcz_store.cc:269] 0709 12:23:02.494098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.712874 (+2218776us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:04.713055 (+   181us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:04.596131  4409 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:04.685729  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:04.587234  4565 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:04.257618  4700 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:04.684067  4694 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:04.692736  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1604) took 2946 ms (client timeout 2865 ms). Trace:
W20260709 12:23:04.732227  1805 rpcz_store.cc:269] 0709 12:23:01.745951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.692532 (+2946581us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:04.692668 (+   136us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:04.617751  4602 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:04.695880  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1596) took 2970 ms (client timeout 1638 ms). Trace:
W20260709 12:23:04.734159  1804 rpcz_store.cc:269] 0709 12:23:01.725598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.695668 (+2970070us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:04.695805 (+   137us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:04.696059  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:04.735472  1724 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:04.726162  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:04.590938  4734 raft_consensus.cc:2764] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:23:04.613615  4549 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:04.684249  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 98377.1us
I20260709 12:23:04.404708  4707 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:04.719631  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1596) took 2824 ms. Trace:
I20260709 12:23:04.640583  4605 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:05.015659  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1597) took 3289 ms (client timeout 1991 ms). Trace:
W20260709 12:23:05.270483  1845 leader_election.cc:336] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.110s (SENT)
I20260709 12:23:05.270895  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:04.959919  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:04.818049  1724 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:04.781191  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.015393  1880 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 12:23:05.206385  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 480325us
I20260709 12:23:04.737098  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:05.270551  1802 rpcz_store.cc:269] 0709 12:23:01.725827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.484236 (+1758409us) service_pool.cc:224] Handling call
0709 12:23:05.015362 (+1531126us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54001eaad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:05.015552 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":527093120}
W20260709 12:23:05.276508  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1509) took 2481 ms (client timeout 2011 ms). Trace:
I20260709 12:23:04.960602  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:05.088229  1697 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:05.276980  1805 rpcz_store.cc:269] 0709 12:23:02.794870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.733246 (+1938376us) service_pool.cc:224] Handling call
0709 12:23:05.276426 (+543180us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:05.004209  1799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1504) took 2386 ms (client timeout 1585 ms). Trace:
W20260709 12:23:05.274103  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1507) took 2548 ms (client timeout 2037 ms). Trace:
I20260709 12:23:05.277741  2416 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:05.272325  2412 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:04.845110  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 250597us
I20260709 12:23:05.280510  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:04.959858  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:05.270560  1881 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:23:05.271394  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:05.274958  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.219619  2385 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.003s	sys 0.020s
W20260709 12:23:05.284972  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.082917  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1588) took 3359 ms (client timeout 2046 ms). Trace:
W20260709 12:23:05.279093  1799 rpcz_store.cc:269] 0709 12:23:02.618124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.103475 (+1485351us) service_pool.cc:224] Handling call
0709 12:23:05.004018 (+900543us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54001d4554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:05.004135 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":414363392}
W20260709 12:23:05.284246  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 196151us
W20260709 12:23:05.070304  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47928.8us
I20260709 12:23:05.287611  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:04.916347  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 363) took 1621 ms (client timeout 2153 ms). Trace:
I20260709 12:23:05.288507  2412 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
W20260709 12:23:05.288487  1872 rpcz_store.cc:269] 0709 12:23:03.295299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.336971 (+ 41672us) service_pool.cc:224] Handling call
0709 12:23:04.916055 (+1579084us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b54003525d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:04.916244 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106620800}
W20260709 12:23:05.279318  1803 rpcz_store.cc:269] 0709 12:23:02.726086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.728444 (+2002358us) service_pool.cc:224] Handling call
0709 12:23:05.274033 (+545589us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:05.283279  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 323472us
I20260709 12:23:05.291497  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:05.292713  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:05.293334  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.295034  2416 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
W20260709 12:23:05.296937  2385 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.720s	user 0.003s	sys 0.021s
I20260709 12:23:05.298038  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:05.299052  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1494) took 2816 ms (client timeout 2385 ms). Trace:
I20260709 12:23:05.298508  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:05.299961  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.276821  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:05.286093  1800 rpcz_store.cc:269] 0709 12:23:01.723201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.644042 (+1920841us) service_pool.cc:224] Handling call
0709 12:23:05.082563 (+1438521us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:05.082775 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920981888}
I20260709 12:23:05.273802  1725 rpcz_store.cc:276] 0709 12:23:01.895411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:02.186168 (+290757us) service_pool.cc:224] Handling call
0709 12:23:03.377792 (+1191624us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:03.378157 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:03.378184 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:03.453473 (+ 75289us) log.cc:844] Serialized 26 byte log entry
0709 12:23:03.551413 (+ 97940us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:03.757788 (+206375us) log.cc:844] Serialized 14 byte log entry
0709 12:23:03.758025 (+   237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.758106 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:03.810229 (+ 52123us) raft_consensus.cc:1685] finished
0709 12:23:03.810266 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:04.719237 (+908971us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54000d20d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:04.719501 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":155601,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":239,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":4839,"spinlock_wait_cycles":2739639040,"thread_start_us":272615,"threads_started":2,"wal-append.queue_time_us":182445,"wal-append.run_cpu_time_us":4667,"wal-append.run_wall_time_us":1169961}
W20260709 12:23:05.299474  1802 rpcz_store.cc:269] 0709 12:23:02.482366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.298872 (+2816506us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:05.298990 (+   118us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:23:05.301184  1799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1493) took 2819 ms (client timeout 2449 ms). Trace:
W20260709 12:23:05.310163  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1512) took 2514 ms (client timeout 2095 ms). Trace:
W20260709 12:23:05.340512  1805 rpcz_store.cc:269] 0709 12:23:02.795871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.309992 (+2514121us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:05.310078 (+    86us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:04.705973  4739 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:23:04.938697  4712 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.232s	user 0.000s	sys 0.004s
W20260709 12:23:05.357556  4741 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.779s	user 0.000s	sys 0.002s
I20260709 12:23:04.797853  4602 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:05.310153  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1616) took 2030 ms (client timeout 1789 ms). Trace:
W20260709 12:23:04.856848  4660 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.637s	user 0.003s	sys 0.012s
I20260709 12:23:05.341912  1879 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:05.345723  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:05.340150  1799 rpcz_store.cc:269] 0709 12:23:02.481896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.300903 (+2819007us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:05.301086 (+   183us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:23:05.368781  1803 rpcz_store.cc:269] 0709 12:23:03.279910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.309908 (+2029998us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:05.310078 (+   170us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:04.728184  4395 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:05.297683  4426 raft_consensus.cc:2764] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260709 12:23:05.350705  4549 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:05.352507  4605 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:04.723685  4548 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:05.062461  4636 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:05.300877  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:05.360909  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1615) took 2081 ms (client timeout 2047 ms). Trace:
W20260709 12:23:05.407823  1802 rpcz_store.cc:269] 0709 12:23:03.279666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.360722 (+2081056us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:05.360837 (+   115us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:23:04.915112  4730 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.634s	user 0.008s	sys 0.006s
W20260709 12:23:05.347565  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1614) took 2068 ms (client timeout 1940 ms). Trace:
I20260709 12:23:04.829533  4675 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:04.841145  4736 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.366506  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:05.422132  2416 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260709 12:23:04.794799  4649 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:05.369871  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1510) took 2574 ms (client timeout 2356 ms). Trace:
I20260709 12:23:05.361064  4739 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:04.945955  4666 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.883621  4515 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:05.272424  4700 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:05.399624  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.437410  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:05.399624  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:05.371829  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2354) took 2010 ms. Trace:
I20260709 12:23:05.297752  4708 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:23:04.877372  4588 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:05.325119  4647 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:05.406752  2419 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:04.797552  4694 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.408928  1724 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:05.457384  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:05.412283  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:05.458079  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1511) took 2662 ms (client timeout 1695 ms). Trace:
I20260709 12:23:05.413259  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:05.458745  1807 rpcz_store.cc:269] 0709 12:23:02.795588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.457774 (+2662186us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:05.457970 (+   196us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:05.459707  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:05.413177  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.461071  1803 raft_consensus.cc:2410] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 12:23:05.461514  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1500) took 2952 ms (client timeout 2954 ms). Trace:
W20260709 12:23:05.461903  1803 rpcz_store.cc:269] 0709 12:23:02.509091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.369799 (+2860708us) service_pool.cc:224] Handling call
0709 12:23:05.461454 (+ 91655us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1940864}
W20260709 12:23:05.412307  1800 rpcz_store.cc:269] 0709 12:23:03.279356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.347287 (+2067931us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:05.347471 (+   184us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:05.343430  4432 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:05.421128  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 591) took 1544 ms. Trace:
I20260709 12:23:05.468392  2411 rpcz_store.cc:276] 0709 12:23:03.865088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.865221 (+   133us) service_pool.cc:224] Handling call
0709 12:23:03.865481 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:03.970974 (+105493us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.409541 (+1438567us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b54005f05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:05.409773 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102017792}
I20260709 12:23:05.420785  4426 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260709 12:23:05.470297  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:05.468436  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.400259  4660 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:05.428943  1805 rpcz_store.cc:269] 0709 12:23:02.795292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.369681 (+2574389us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:05.369788 (+   107us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:04.904220  4697 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:05.371337  4734 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:05.436266  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:05.477524  1732 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:05.297349  4413 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
W20260709 12:23:05.412783  4743 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.019s	sys 0.034s
I20260709 12:23:05.442471  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:05.473428  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:05.442179  1878 rpcz_store.cc:276] 0709 12:23:03.361243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.492040 (+130797us) service_pool.cc:224] Handling call
0709 12:23:03.508541 (+ 16501us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:03.517084 (+  8543us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 12:23:03.517122 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:03.517913 (+   791us) log.cc:844] Serialized 52 byte log entry
0709 12:23:03.635067 (+117154us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:23:03.635107 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:03.635176 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:04.937709 (+1302533us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:05.359804 (+422095us) raft_consensus.cc:1685] finished
0709 12:23:05.359828 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.371755 (+ 11927us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22392,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"spinlock_wait_cycles":36958464,"thread_start_us":116737,"threads_started":1,"wal-append.queue_time_us":187945}
I20260709 12:23:05.481841  2412 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:05.441782  1799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1502) took 2912 ms (client timeout 2897 ms). Trace:
W20260709 12:23:05.482355  1799 rpcz_store.cc:269] 0709 12:23:02.529331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.369603 (+2840272us) service_pool.cc:224] Handling call
0709 12:23:05.441717 (+ 72114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.449294  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:05.483424  2416 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:04.877132  4565 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:05.425956  4675 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:05.463601  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1501) took 2948 ms (client timeout 3262 ms). Trace:
I20260709 12:23:05.491747  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:05.491999  1802 rpcz_store.cc:269] 0709 12:23:02.516809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.411991 (+2895182us) service_pool.cc:224] Handling call
0709 12:23:05.463543 (+ 51552us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.492455  1807 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:05.493040  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.465929  1724 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:05.467576  1731 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:23:05.427299  1772 connection.cc:256] RPC call timeout handler was delayed by -2.2420520340001531s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1787s for a response
I20260709 12:23:05.471232  1725 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:05.427863  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 157408us
I20260709 12:23:05.438911  4739 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:05.424510  4636 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.449949  4588 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:05.456240  4647 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.489640  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.450431  4466 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
W20260709 12:23:05.499969  1845 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.174s (SENT)
I20260709 12:23:05.516757  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:05.440706  4736 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.444633  4666 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.504994  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.505445  1772 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:05.507829  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:05.481060  4743 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.020s	sys 0.035s
I20260709 12:23:05.425172  4730 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.479863  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:05.512255  4750 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.512769  2417 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:05.513221  4426 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:05.510715  1800 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:05.500828  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.481354  4734 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.519578  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:05.531149  1799 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:05.533859  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:05.500276  4675 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:05.566092  2413 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:05.560767  4565 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:05.560223  4488 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.859s	user 0.013s	sys 0.002s
I20260709 12:23:05.558832  4708 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:05.515823  4647 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:05.515213  4432 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:05.581791  2416 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:05.582175  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 308948us
W20260709 12:23:05.582917  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.584092  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 84140.1us
I20260709 12:23:05.504762  4629 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:05.488935  4747 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:05.507000  4588 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.505621  4700 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 12:23:05.645398  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.644474  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:05.646067  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 356537us
I20260709 12:23:05.649293  1807 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:05.650883  2412 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:05.652570  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:23:05.594527  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 303736us
W20260709 12:23:05.645324  1845 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.189s (SENT)
W20260709 12:23:05.652875  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 119169us
I20260709 12:23:05.664613  1804 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:23:05.662915  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:05.517992  4745 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.083s
I20260709 12:23:05.583859  4605 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:05.658177  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:05.662222  4748 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:05.679409  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:05.643865  4750 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.662652  4432 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:05.649838  4588 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:05.593135  4426 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:05.653877  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:05.677003  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 594) took 1807 ms. Trace:
I20260709 12:23:05.657027  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:05.677417  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.676651  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:05.701493  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.653465  1732 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:05.668795  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.681933  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:05.703034  1807 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:05.682271  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 394613us
I20260709 12:23:05.680784  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:05.699421  2418 rpcz_store.cc:276] 0709 12:23:03.869666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.869803 (+   137us) service_pool.cc:224] Handling call
0709 12:23:03.870098 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:04.665103 (+795005us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.676732 (+1011629us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b54005230d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:05.676940 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1826865920}
I20260709 12:23:05.664556  4747 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:05.709044  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1522) took 1023 ms. Trace:
I20260709 12:23:05.716233  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.717286  1805 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:05.717868  1800 rpcz_store.cc:276] 0709 12:23:04.686217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.665734 (+979517us) service_pool.cc:224] Handling call
0709 12:23:05.708980 (+ 43246us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.651765  4734 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:05.657449  4697 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:23:05.726300  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 69352.4us
I20260709 12:23:05.732595  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.662416  4639 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:23:05.733156  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:05.730046  1803 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:23:05.653256  4629 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:05.752264  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 106833us
I20260709 12:23:05.753237  1800 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 8. Ops: [4.4-6.5]
I20260709 12:23:05.746341  1807 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:05.748005  1803 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:23:05.708827  4765 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:05.760396  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1591) took 4036 ms. Trace:
I20260709 12:23:05.760952  1800 rpcz_store.cc:276] 0709 12:23:01.723915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.727874 (+4003959us) service_pool.cc:224] Handling call
0709 12:23:05.752866 (+ 24992us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:05.760084 (+  7218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.760338 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.755177  1803 pending_rounds.cc:85] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 6
I20260709 12:23:05.653015  4708 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.756155  1807 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:23:05.714083  4707 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:05.763005  1803 pending_rounds.cc:107] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:23:05.767930  1803 raft_consensus.cc:2889] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:05.768958  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1592) took 4044 ms. Trace:
I20260709 12:23:05.769245  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:05.769635  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:05.769714  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1603) took 4022 ms. Trace:
I20260709 12:23:05.770066  1872 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:05.770335  1807 rpcz_store.cc:276] 0709 12:23:01.745636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.754718 (+4009082us) service_pool.cc:224] Handling call
0709 12:23:05.755815 (+  1097us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.767290 (+ 11475us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.767589 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.769660  1803 rpcz_store.cc:276] 0709 12:23:01.724173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.747072 (+4022899us) service_pool.cc:224] Handling call
0709 12:23:05.747506 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.768699 (+ 21193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.768901 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.770269  1880 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
W20260709 12:23:05.776155  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 100504us
I20260709 12:23:05.781829  1880 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:05.781536  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:05.783154  1878 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:23:05.785422  1803 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:05.786214  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1505) took 3135 ms. Trace:
I20260709 12:23:05.786620  1803 rpcz_store.cc:276] 0709 12:23:02.650391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.772669 (+3122278us) service_pool.cc:224] Handling call
0709 12:23:05.785188 (+ 12519us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.785937 (+   749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.786152 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:05.787774  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 133889us
I20260709 12:23:05.788765  1872 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:05.789309  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:05.790303  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 513530us
I20260709 12:23:05.791412  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:05.739529  4679 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:05.730610  4395 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:05.678459  4748 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.798928  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:05.799590  1807 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: [3.3-5.4]
I20260709 12:23:05.799856  1874 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:05.800438  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1503) took 3252 ms. Trace:
I20260709 12:23:05.800736  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:05.800812  1807 rpcz_store.cc:276] 0709 12:23:02.547749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.794444 (+3246695us) service_pool.cc:224] Handling call
0709 12:23:05.799301 (+  4857us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:05.800086 (+   785us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.800299 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.798950  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:05.804049  1879 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 4: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:05.806027  1807 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:05.806964  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1607) took 2769 ms. Trace:
I20260709 12:23:05.807817  1807 rpcz_store.cc:276] 0709 12:23:03.037992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.805230 (+2767238us) service_pool.cc:224] Handling call
0709 12:23:05.805626 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.806597 (+   971us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.806896 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.741057  4758 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:23:05.824651  2383 leader_election.cc:336] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.566s (SENT)
I20260709 12:23:05.759910  4765 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:05.822033  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.810346  1800 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:23:05.841934  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:05.822970  4757 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
W20260709 12:23:05.851091  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61029.6us
W20260709 12:23:05.851555  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 196167us
I20260709 12:23:05.855192  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:05.856339  1725 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
W20260709 12:23:05.854125  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 120868us
I20260709 12:23:05.768839  4697 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 12:23:05.855233  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:05.781450  4756 raft_consensus.cc:2764] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260709 12:23:05.868600  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1606) took 3902 ms. Trace:
I20260709 12:23:05.872289  1800 rpcz_store.cc:276] 0709 12:23:01.947969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.774336 (+3826367us) service_pool.cc:224] Handling call
0709 12:23:05.810099 (+ 35763us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.850317 (+ 40218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.850559 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:05.815382  4660 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:05.810276  4694 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:05.873430  2416 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:05.818483  4639 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:05.874130  4639 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:05.873425  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:05.818771  4707 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:23:05.875591  4639 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.877638  4639 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:05.875569  1800 raft_consensus.cc:2376] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7. Re-sending same reply.
I20260709 12:23:05.879989  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.881404  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:05.870517  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:05.882772  2418 raft_consensus.cc:2410] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:23:05.849326  4488 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:05.860807  4762 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.935165  1874 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:05.925566  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:05.845351  4745 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.079s
W20260709 12:23:05.961006  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.911264  4413 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:05.942523  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 120074us
I20260709 12:23:05.922894  4636 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:05.988256  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:06.004240  4773 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:05.989473  4765 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.005393  4730 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.007504  4629 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:06.054242  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66336.9us
I20260709 12:23:05.950093  4764 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.010046  1802 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
W20260709 12:23:06.012120  4409 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.284s	user 0.016s	sys 0.000s
W20260709 12:23:06.013634  4649 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.573s	user 0.011s	sys 0.000s
I20260709 12:23:05.849684  4747 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.015002  4432 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:05.862418  4426 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.015417  4748 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:06.018733  4708 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:06.019506  4679 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:06.006616  4750 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:06.025297  1876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.021s	user 0.022s	sys 0.014s
I20260709 12:23:05.943204  4395 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:06.027554  4754 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.301s	user 0.000s	sys 0.001s
I20260709 12:23:06.027971  4756 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:23:06.029174  4761 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:23:06.028743  4647 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.942909  4700 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:23:06.029850  4753 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.309s	user 0.001s	sys 0.000s
I20260709 12:23:06.030865  1800 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:05.972375  4666 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:05.950855  4762 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:05.924284  4694 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:06.035847  4413 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.036515  2418 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62"
dest_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:60950
I20260709 12:23:06.036701  4734 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.033291  4675 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.038975  4588 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.043529  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2362) took 2426 ms. Trace:
W20260709 12:23:06.045943  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 120465us
I20260709 12:23:06.046486  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:06.047304  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:06.171597  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:06.170939  1873 rpcz_store.cc:276] 0709 12:23:03.616693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.767826 (+151133us) service_pool.cc:224] Handling call
0709 12:23:03.783522 (+ 15696us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:04.673368 (+889846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.043173 (+1369805us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:06.043417 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2044145024}
I20260709 12:23:06.061872  1807 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:06.062448  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1520) took 1420 ms. Trace:
I20260709 12:23:06.055606  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:06.053869  4765 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:23:06.026580  4739 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.511s	user 0.006s	sys 0.000s
W20260709 12:23:06.107918  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.175745  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:06.176383  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:06.181020  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 125560us
I20260709 12:23:06.189316  1881 raft_consensus.cc:2368] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8.
I20260709 12:23:06.132838  2415 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
W20260709 12:23:06.134833  1876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.130s	user 0.023s	sys 0.014s
I20260709 12:23:06.055811  4773 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:06.066805  4546 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:06.160461  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1593) took 4436 ms (client timeout 1948 ms). Trace:
W20260709 12:23:06.196312  1806 rpcz_store.cc:269] 0709 12:23:01.724419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.268285 (+1543866us) service_pool.cc:224] Handling call
0709 12:23:06.160211 (+2891926us) spinlock_profiling.cc:243] Waited 841 ms on lock 0x7b54001671d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:06.160356 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1934190336}
I20260709 12:23:06.166183  2418 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:06.197932  2418 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:06.169050  1724 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:06.056003  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:06.173054  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:06.199045  2419 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:06.200677  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:06.201220  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2375) took 1186 ms. Trace:
I20260709 12:23:06.172778  1802 rpcz_store.cc:276] 0709 12:23:04.642115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.519181 (+877066us) service_pool.cc:224] Handling call
0709 12:23:06.062346 (+543165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.202613  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:06.201749  1875 rpcz_store.cc:276] 0709 12:23:05.014928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.015034 (+   106us) service_pool.cc:224] Handling call
0709 12:23:05.015288 (+   254us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.015715 (+   427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.015742 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.016307 (+   565us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.403317 (+387010us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:05.403359 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.403432 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.200904 (+797472us) raft_consensus.cc:1685] finished
0709 12:23:06.200932 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.201121 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":140,"raft.run_cpu_time_us":67,"raft.run_wall_time_us":69,"thread_start_us":386571,"threads_started":1,"wal-append.queue_time_us":641006}
I20260709 12:23:06.204252  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.205693  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.157620  4775 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:06.155741  4757 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 12:23:06.056365  4266 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:06.176772  2410 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:06.207099  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.211959  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.174230  4772 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.217415  1802 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:06.215816  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2371) took 1257 ms. Trace:
I20260709 12:23:06.217973  1876 rpcz_store.cc:276] 0709 12:23:04.983503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.003103 (+ 19600us) service_pool.cc:224] Handling call
0709 12:23:05.003477 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.003908 (+   431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.003937 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.004427 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:23:06.191050 (+1186623us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:23:06.191114 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.191221 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.215532 (+ 24311us) raft_consensus.cc:1685] finished
0709 12:23:06.215554 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.215738 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1173,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":4065,"thread_start_us":1186165,"threads_started":1,"wal-append.queue_time_us":640521}
I20260709 12:23:06.216269  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2499) took 1338 ms. Trace:
I20260709 12:23:06.220252  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:06.222518  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 367325us
I20260709 12:23:06.224139  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:06.225466  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:06.192186  4432 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:06.221323  1730 rpcz_store.cc:276] 0709 12:23:04.878098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.888902 (+ 10804us) service_pool.cc:224] Handling call
0709 12:23:05.027173 (+138271us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.027736 (+   563us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.027784 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.203052 (+175268us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.667163 (+464111us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:05.667207 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.667273 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.215989 (+548716us) raft_consensus.cc:1685] finished
0709 12:23:06.216018 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.216192 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":149,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":50640,"thread_start_us":354389,"threads_started":1,"wal-append.queue_time_us":556482}
I20260709 12:23:06.197008  4548 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.196033  4515 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:23:06.233906  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:06.298481  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 473852us
W20260709 12:23:06.299044  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 65220.7us
I20260709 12:23:06.210336  4458 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:06.303050  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.303334  1725 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
W20260709 12:23:06.304178  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 69620us
I20260709 12:23:06.173823  4506 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:23:06.212320  4763 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:23:06.168555  4694 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.235947  4750 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:06.217001  4266 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.312409  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:06.323201  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:06.321650  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 121049us
I20260709 12:23:06.311347  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:06.352345  1727 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:06.298537  4636 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:06.345346  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 112832us
I20260709 12:23:06.239141  4772 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.227766  4728 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:23:06.211054  4546 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:06.354055  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.310407  4730 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:06.353529  4266 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:06.089805  4764 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.197157  4739 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.348403  4750 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:06.542292  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1508) took 3788 ms (client timeout 1621 ms). Trace:
W20260709 12:23:06.542809  1798 rpcz_store.cc:269] 0709 12:23:02.753482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.201829 (+1448347us) service_pool.cc:224] Handling call
0709 12:23:06.542032 (+2340203us) spinlock_profiling.cc:243] Waited 325 ms on lock 0x7b54001d4f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:06.542178 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":748577024}
I20260709 12:23:06.372515  4730 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.355892  4782 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.310442  4548 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.227454  4775 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:06.418556  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 193138us
W20260709 12:23:06.495965  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.551225  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:06.552314  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:06.515137  4763 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:06.515450  4762 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.362273  4697 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
W20260709 12:23:06.549710  4787 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.163s	user 0.007s	sys 0.003s
I20260709 12:23:06.518127  4772 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.596693  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1613) took 3333 ms. Trace:
I20260709 12:23:06.597599  1807 rpcz_store.cc:276] 0709 12:23:03.265121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.842026 (+2576905us) service_pool.cc:224] Handling call
0709 12:23:06.061607 (+219581us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.172743 (+111136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.594481 (+421738us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b54001d4550. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:06.594723 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":503820544}
I20260709 12:23:06.388980  4773 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.520702  4675 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:06.521512  4565 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.856s	user 0.012s	sys 0.011s
W20260709 12:23:06.523952  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 165890us
I20260709 12:23:06.535208  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.535938  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:06.348568  4708 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.536576  2418 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:06.535374  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 180682us
I20260709 12:23:06.608920  1799 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:06.609300  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:06.537252  4759 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:23:06.557502  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.559787  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:06.567437  4777 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:06.546516  4750 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:06.516887  4739 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:06.545392  4783 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:06.582274  4764 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:06.585659  4775 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.572647  4763 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.547128  4730 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:06.587034  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.553097  4782 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.591164  4736 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:06.554399  4786 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.327s	user 0.006s	sys 0.006s
I20260709 12:23:06.601408  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1515) took 3804 ms. Trace:
I20260709 12:23:06.602012  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:06.604496  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 246484us
I20260709 12:23:06.607962  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:06.608083  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 295810us
I20260709 12:23:06.561841  4647 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.636761  1807 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:06.614115  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:06.573338  4765 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:06.617784  2418 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:06.578555  4588 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.651402  4765 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:06.651997  4765 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:06.652289  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.653119  1807 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
W20260709 12:23:06.623274  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63587.4us
I20260709 12:23:06.654510  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:23:06.655719  2415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 588) took 2776 ms (client timeout 1504 ms). Trace:
W20260709 12:23:06.656296  2415 rpcz_store.cc:269] 0709 12:23:03.860341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.860488 (+   147us) service_pool.cc:224] Handling call
0709 12:23:06.636738 (+2776250us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54001c3ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:06.636884 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5058910080}
W20260709 12:23:06.658046  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:06.624105  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 88238.8us
W20260709 12:23:06.604018  4738 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.104s	user 0.009s	sys 0.009s
I20260709 12:23:06.625473  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.628216  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:06.618284  4675 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:06.629274  4705 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:06.663888  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56018.2us
I20260709 12:23:06.629030  4757 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:06.619370  4702 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.617636  4548 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:06.617993  4777 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.637153  1803 rpcz_store.cc:276] 0709 12:23:02.796672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.787571 (+2990899us) service_pool.cc:224] Handling call
0709 12:23:05.810843 (+ 23272us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.811133 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.811160 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.819929 (+  8769us) log.cc:844] Serialized 26 byte log entry
0709 12:23:06.181620 (+361691us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:06.518661 (+337041us) log.cc:844] Serialized 14 byte log entry
0709 12:23:06.538175 (+ 19514us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.538246 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.601101 (+ 62855us) raft_consensus.cc:1685] finished
0709 12:23:06.601126 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.601333 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58132,"prepare.queue_time_us":418972,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":5106,"raft.queue_time_us":28652,"spinlock_wait_cycles":9422592,"thread_start_us":657534,"threads_started":2,"wal-append.queue_time_us":502703}
I20260709 12:23:06.618450  4734 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:06.647751  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:06.643891  4565 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.615059  4639 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.664763  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.665225  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:06.665792  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:06.580351  4728 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:23:06.667074  4675 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:06.667529  4647 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:23:06.668200  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1608) took 3624 ms (client timeout 1504 ms). Trace:
W20260709 12:23:06.702636  1801 rpcz_store.cc:269] 0709 12:23:03.043712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.667934 (+3624222us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:06.668109 (+   175us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:06.668612  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.669732  4759 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:23:06.663812  4738 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.182s	user 0.009s	sys 0.009s
I20260709 12:23:06.629426  4783 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.635140  4413 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.622720  4708 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.689626  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:06.691233  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.692178  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:06.712980  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.629s (SENT)
I20260709 12:23:06.666149  4762 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:23:06.685531  4777 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:06.700591  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.710086  1803 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:06.682983  4757 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:06.693621  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1587) took 4970 ms. Trace:
I20260709 12:23:06.702669  4675 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:06.675511  4737 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 12:23:06.696187  4705 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:23:06.699577  4588 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:06.722836  1805 rpcz_store.cc:276] 0709 12:23:01.722930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.728052 (+4005122us) service_pool.cc:224] Handling call
0709 12:23:05.762956 (+ 34904us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.763434 (+   478us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.763479 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:05.764160 (+   681us) log.cc:844] Serialized 26 byte log entry
0709 12:23:06.180376 (+416216us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:06.345518 (+165142us) log.cc:844] Serialized 14 byte log entry
0709 12:23:06.537868 (+192350us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.537959 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.693212 (+155253us) raft_consensus.cc:1685] finished
0709 12:23:06.693248 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.693527 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":191655,"prepare.queue_time_us":393815,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":106,"raft.queue_time_us":323521,"raft.run_cpu_time_us":459,"raft.run_wall_time_us":34589,"thread_start_us":467881,"threads_started":2,"wal-append.queue_time_us":403835}
I20260709 12:23:06.728168  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.718937  1801 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:06.680896  4266 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.671902  4739 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:23:06.681646  4702 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.707420  4767 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:23:06.730165  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.733080  1807 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:06.709028  4783 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:06.744956  1803 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:06.718902  4647 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:06.749766  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84615.3us
W20260709 12:23:06.735072  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 77582.6us
I20260709 12:23:06.751165  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:06.738096  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:06.740571  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:06.753319  1805 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:06.740644  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:06.741528  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:06.741654  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:06.753895  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.694660  4565 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.694684  4639 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.734508  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:06.721752  4413 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:23:06.733110  4679 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.594s	user 0.012s	sys 0.004s
I20260709 12:23:06.754344  1875 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 3: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:06.750412  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:06.718734  4548 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:06.756181  4759 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:06.730268  4588 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.743966  4736 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 12:23:06.759788  4602 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.760615  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2496) took 2068 ms. Trace:
I20260709 12:23:06.742522  4770 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:06.762571  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:06.789813  2383 leader_election.cc:400] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8.
I20260709 12:23:06.762758  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.758354  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:06.750430  4739 consensus_queue.cc:1243] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:23:06.763556  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:06.779539  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.771590  1726 rpcz_store.cc:276] 0709 12:23:04.692630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.693456 (+   826us) service_pool.cc:224] Handling call
0709 12:23:04.693931 (+   475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.694615 (+   684us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:04.694658 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.697534 (+  2876us) log.cc:844] Serialized 26 byte log entry
0709 12:23:06.643585 (+1946051us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b18002a46c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:23:06.699859 (+ 56274us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:06.699926 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.700019 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.760254 (+ 60235us) raft_consensus.cc:1685] finished
0709 12:23:06.760288 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.760547 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75,"raft.queue_time_us":540,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":19258,"spinlock_wait_cycles":33233280,"thread_start_us":43965,"threads_started":1,"wal-append.queue_time_us":2020940}
I20260709 12:23:06.779024  4757 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.780333  4647 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.766053  4679 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:06.783639  4588 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:06.790303  2383 leader_election.cc:403] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:23:06.790918  1803 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:06.792762  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:06.795328  1805 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:23:06.796509  4548 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:06.792547  4737 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:06.839126  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:06.799768  4702 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:06.805048  4739 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:06.767478  4705 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:06.807425  4565 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:06.847672  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.808055  4763 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:06.792727  4798 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:23:06.806479  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:06.809557  4708 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:06.852679  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46327.4us
I20260709 12:23:06.854601  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:06.813445  4562 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:06.788105  4413 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:06.817757  4694 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:06.821441  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.852514  4757 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:06.824061  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.849645  4769 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:06.834924  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:06.837182  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:06.839820  1872 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:23:06.838651  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:06.818706  4730 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:06.811452  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1514) took 4015 ms (client timeout 2065 ms). Trace:
W20260709 12:23:06.853016  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 102076us
I20260709 12:23:06.860859  4739 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:06.863790  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:06.889698  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.892122  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.892436  1806 raft_consensus.cc:2376] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8. Re-sending same reply.
I20260709 12:23:06.892895  1801 raft_consensus.cc:2376] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6. Re-sending same reply.
I20260709 12:23:06.893075  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.878278  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:06.896287  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:06.894649  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.899212  1806 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:23:06.894097  1804 rpcz_store.cc:269] 0709 12:23:02.796412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.735095 (+1938683us) service_pool.cc:224] Handling call
0709 12:23:06.811261 (+2076166us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b540026f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:06.811371 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":902761728}
I20260709 12:23:06.900974  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.901816  1805 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260709 12:23:06.901288  4413 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.902676  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:06.889164  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.899864  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:06.894009  1876 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:23:06.853770  4548 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:06.851646  4782 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:06.912129  4694 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:23:06.912528  4776 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:06.912657  4694 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:06.913002  4776 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:06.913120  4694 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:06.913439  4776 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:06.902194  4777 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.906344  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.915887  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.916388  4707 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:06.916982  4707 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:06.920267  4804 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:06.920496  4739 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:06.894887  4675 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.859596  4705 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:06.990708  4730 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.861917  4759 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:06.991205  4705 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.939381  4777 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:06.908316  4757 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:06.860154  4562 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:06.941123  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.926110  4734 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:06.946580  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:06.923195  2383 leader_election.cc:336] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.043s (SENT)
I20260709 12:23:07.017326  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:06.918129  4707 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.933378  4548 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:06.939173  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.930917  4546 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:06.941012  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.018741  4707 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:07.022266  1803 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:06.943300  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 688) took 2819 ms. Trace:
W20260709 12:23:07.024928  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 101780us
I20260709 12:23:07.025568  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:07.026379  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:06.921639  4737 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:06.921234  4728 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:06.915905  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:06.928280  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 744) took 1610 ms. Trace:
I20260709 12:23:06.921175  4804 consensus_queue.cc:1243] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:23:06.982663  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:06.985822  1805 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:06.920346  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:06.989607  4783 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:07.040324  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 120050us
I20260709 12:23:07.040539  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.041643  1803 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:07.017542  1801 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:06.939828  4765 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.026998  1873 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:07.028369  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:07.028903  1726 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:23:06.934201  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1537) took 2064 ms. Trace:
I20260709 12:23:07.024977  2417 rpcz_store.cc:276] 0709 12:23:04.123698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.124330 (+   632us) service_pool.cc:224] Handling call
0709 12:23:04.145915 (+ 21585us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.594039 (+1448124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.942990 (+1348951us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400573ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:06.943222 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3134272256}
W20260709 12:23:07.047931  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.037313  1804 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:07.047746  1799 rpcz_store.cc:276] 0709 12:23:04.870229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.708804 (+838575us) service_pool.cc:224] Handling call
0709 12:23:06.933963 (+1225159us) spinlock_profiling.cc:243] Waited 875 ms on lock 0x7b540013a6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:06.934121 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2012798208}
I20260709 12:23:07.034381  2413 rpcz_store.cc:276] 0709 12:23:05.317580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.341721 (+ 24141us) service_pool.cc:224] Handling call
0709 12:23:05.366397 (+ 24676us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.651236 (+284839us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.928020 (+1276784us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54005f28d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:06.928218 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":458633984}
I20260709 12:23:07.047747  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:07.050194  1874 raft_consensus.cc:2410] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:23:07.044387  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.051364  1877 raft_consensus.cc:2368] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7.
I20260709 12:23:07.053889  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:07.055083  1880 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:07.060374  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.061583  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.064915  1874 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:07.016917  4734 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:07.047114  1876 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:07.067651  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.071089  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:07.069290  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.071743  1880 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:07.069558  1872 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:23:07.072252  1729 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:07.071763  1726 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:07.031203  4736 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:07.033874  4702 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:07.073486  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.078449  1872 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:07.079061  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:07.079777  1803 raft_consensus.cc:2376] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 6. Re-sending same reply.
I20260709 12:23:07.080721  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.072006  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.081758  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.082095  1876 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.077073  1798 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:07.084234  1726 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:23:07.077241  4777 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:07.043254  4546 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:23:07.085446  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.093742  1878 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:07.089426  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:07.040071  4708 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:23:07.092765  1880 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:07.096711  1726 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:23:07.066191  4730 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:07.077507  4562 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:07.097359  4815 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:23:07.068933  4804 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.104677  1880 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:23:07.168469  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:07.101951  1874 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:07.115743  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:07.116110  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:07.116434  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:07.126644  2418 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:23:07.132673  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:07.133296  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.155568  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.172703  1731 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:07.174006  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.175872  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.176204  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.177055  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.151544  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.091661  4771 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:07.116947  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:07.141054  1878 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:07.158113  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.182264  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.156701  1876 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:07.184497  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:07.187075  1726 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:23:07.187464  1726 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:07.183756  1803 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:07.190356  1876 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:07.193950  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:07.157199  4750 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.527s	user 0.012s	sys 0.000s
W20260709 12:23:07.167061  4816 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:23:07.092700  4702 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:07.200038  1876 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:07.206285  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:07.216306  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:07.208896  1878 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:07.103881  4546 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.115902  4747 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:07.113371  4675 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:07.114931  4395 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:07.114434  4728 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.263729  4728 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:07.125610  4762 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.243614  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.244074  4736 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:07.246443  1878 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:07.267570  4736 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.268034  4736 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:07.269809  1878 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:07.225342  1726 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:07.271564  1726 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:07.273592  4728 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:07.247988  4702 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.227138  4772 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.277006  4772 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.248669  4705 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:07.192939  4804 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.168286  4759 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260709 12:23:07.251776  4765 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.220069  4750 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.281286  4804 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.281018  4776 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.283495  4750 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.255923  4783 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:07.256407  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.111609  4643 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
I20260709 12:23:07.113965  4588 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.267292  4762 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.293411  4588 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:07.294513  1881 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:23:07.295194  4762 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:07.247135  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:07.114750  4734 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:07.297676  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.185441  4771 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:07.247900  1803 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:07.302551  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:07.303370  1803 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:07.228293  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:07.306325  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78135.3us
I20260709 12:23:07.308111  4705 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:23:07.104408  4777 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.178185  4597 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.131s
I20260709 12:23:07.356463  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:07.325045  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.323052  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.311575  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:07.358201  1723 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:07.360764  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:07.366343  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54850us
W20260709 12:23:07.366943  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.360603  1723 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:07.374955  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:07.229203  4546 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:07.378531  1726 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:23:07.379782  1729 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:23:07.378070  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:07.381737  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:07.319398  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:07.385792  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.350056  4728 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.387573  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:07.410355  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.415633  1878 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:07.415550  1696 leader_election.cc:400] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7.
I20260709 12:23:07.412210  1803 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:07.385851  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.313395  4747 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.411535  1723 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.308894  4395 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.417935  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:07.434875  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:07.379666  4730 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:07.371897  4765 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.372503  4705 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:07.427932  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:07.485450  1772 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 2.606s (SENT) [suppressed 54 similar messages]
I20260709 12:23:07.431505  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:07.496310  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:07.377377  4739 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:07.427227  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:07.460352  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 141067us
I20260709 12:23:07.423285  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.434818  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:07.417613  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.538554  1729 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:23:07.387251  4750 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:07.506052  2412 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 740) took 2214 ms (client timeout 2606 ms). Trace:
W20260709 12:23:07.493636  4769 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.599s	user 0.010s	sys 0.003s
I20260709 12:23:07.430652  4757 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:07.547144  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 218765us
I20260709 12:23:07.531292  2419 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:23:07.531522  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 46131.2us
I20260709 12:23:07.479614  4605 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.433266  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1527) took 2603 ms. Trace:
I20260709 12:23:07.564998  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:07.541478  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.409257  4702 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:23:07.566243  4702 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:07.567287  4702 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:07.565594  1798 rpcz_store.cc:276] 0709 12:23:04.829703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.662934 (+1833231us) service_pool.cc:224] Handling call
0709 12:23:06.669425 (+  6491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.088984 (+419559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.432973 (+343989us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:07.433198 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":936956672}
I20260709 12:23:07.570674  1803 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:23:07.571125  4605 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:07.543187  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:07.572410  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:07.574905  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.483700  4823 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:07.544132  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:07.578425  1732 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.578336  4823 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:07.579321  1800 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:07.427518  4822 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:23:07.581183  4823 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:07.581857  4823 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:07.582365  4823 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:07.540678  2412 rpcz_store.cc:269] 0709 12:23:05.292134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.292330 (+   196us) service_pool.cc:224] Handling call
0709 12:23:07.505838 (+2213508us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b54000e6d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:07.505987 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432566144}
I20260709 12:23:07.432379  4777 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.504534  4776 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.325645  4643 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.219s
I20260709 12:23:07.561394  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:07.502163  4765 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:07.587793  4814 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.337s	user 0.004s	sys 0.000s
I20260709 12:23:07.589296  4765 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.590725  1732 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:07.592209  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:07.595789  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:07.596848  1807 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.597716  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.598479  1798 raft_consensus.cc:2376] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 6. Re-sending same reply.
I20260709 12:23:07.599972  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.600533  1803 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:23:07.601588  1803 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:23:07.603535  1798 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:23:07.619952  4762 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:07.621047  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 759) took 1388 ms. Trace:
I20260709 12:23:07.622056  4762 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:07.600729  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:07.623685  1802 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:07.626260  4773 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:07.627130  4773 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:07.627792  4639 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.628428  4773 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.630052  4773 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:07.621600  2419 rpcz_store.cc:276] 0709 12:23:06.232565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.232739 (+   174us) service_pool.cc:224] Handling call
0709 12:23:07.530876 (+1298137us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.550123 (+ 19247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.620767 (+ 70644us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54000e6d50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:07.620965 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2751641472}
I20260709 12:23:07.635262  4823 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.638459  1879 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:07.672008  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:07.673300  2419 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:23:07.673986  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:07.674818  1800 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:07.675985  2412 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 12:23:07.676192  2419 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:07.677292  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:07.678212  2417 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:07.680390  4562 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.683781  1806 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:23:07.698738  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:07.699891  2418 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:07.703977  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:07.705163  2419 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:07.743099  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:07.743251  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:07.744133  2418 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:07.744792  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:07.746034  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:07.746843  2412 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:23:07.745978  1697 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:07.751483  4562 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:07.753451  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:07.753925  2418 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:07.754943  4562 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:23:07.769785  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 465424us
I20260709 12:23:07.436592  4728 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.548406  4597 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.241s
W20260709 12:23:07.790838  4828 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.254s	user 0.001s	sys 0.000s
I20260709 12:23:07.548379  4739 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:07.793061  4739 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:07.795584  4694 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.882s	user 0.009s	sys 0.005s
I20260709 12:23:07.796197  4694 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:07.502833  4647 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.060s
I20260709 12:23:07.547333  4730 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.807253  4730 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:07.809880  4730 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.817505  4546 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:07.818810  2414 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:07.821453  4772 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:07.826152  4736 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.558s	user 0.006s	sys 0.000s
I20260709 12:23:07.829339  4730 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.830241  4730 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:07.831176  1726 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:07.834302  2413 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:07.835188  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1556) took 1096 ms. Trace:
I20260709 12:23:07.836117  4730 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:07.836498  1729 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:07.838481  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:07.839149  1875 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:07.841140  4643 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:07.841643  4643 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:07.843552  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:07.846837  1876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.035s	sys 0.100s
W20260709 12:23:07.847268  1876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.035s	sys 0.100s
W20260709 12:23:07.845805  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 245205us
I20260709 12:23:07.548255  4769 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8.
I20260709 12:23:07.847038  4639 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:07.835664  1805 rpcz_store.cc:276] 0709 12:23:06.738985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.038320 (+299335us) service_pool.cc:224] Handling call
0709 12:23:07.038819 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.052566 (+ 13747us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.052598 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.053152 (+   554us) log.cc:844] Serialized 26 byte log entry
0709 12:23:07.365666 (+312514us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:07.416138 (+ 50472us) log.cc:844] Serialized 14 byte log entry
0709 12:23:07.494652 (+ 78514us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.494749 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.834919 (+340170us) raft_consensus.cc:1685] finished
0709 12:23:07.834946 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.835119 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91363,"prepare.queue_time_us":42928,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":93206,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":7725,"thread_start_us":246514,"threads_started":1,"wal-append.queue_time_us":469327}
I20260709 12:23:07.886973  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:07.873625  1807 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:07.901767  2419 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:07.908241  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.897444  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:07.778604  4728 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:07.930769  2416 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:07.852489  4738 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.826618  4736 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:07.984292  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:07.995711  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 347953us
W20260709 12:23:08.005303  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.004s	sys 0.008s
W20260709 12:23:08.006043  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.005s	sys 0.008s
W20260709 12:23:08.006657  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 571835us
W20260709 12:23:07.955883  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.956396  4639 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:08.044271  4762 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:07.842698  4643 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:07.967816  4413 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260709 12:23:07.991827  4738 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:08.069906  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61811.9us
I20260709 12:23:08.071542  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:08.072628  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.086458  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.067528  1698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.614s	user 0.074s	sys 0.180s
W20260709 12:23:08.088156  1698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.074s	sys 0.180s
W20260709 12:23:08.089534  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.089397  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:08.074519  4728 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:08.158300  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 86919.8us
I20260709 12:23:08.160425  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:08.076802  4760 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.165673  4745 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:08.167470  4466 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:23:08.168514  4707 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.138s	user 0.011s	sys 0.002s
I20260709 12:23:08.170670  1807 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:08.170849  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:08.183899  1805 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:08.172652  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:08.067024  4757 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260709 12:23:08.174827  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:08.181305  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:08.192770  1875 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:08.175086  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:08.196817  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:08.197728  1875 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
W20260709 12:23:08.175626  4829 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.288s	user 0.000s	sys 0.000s
I20260709 12:23:08.176076  1872 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:08.179492  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:08.202370  1798 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:08.143448  4413 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:08.145049  4769 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:23:08.176919  4830 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
W20260709 12:23:08.187367  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 263543us
I20260709 12:23:08.177102  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:08.191767  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.199860  2411 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:08.202907  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:08.203792  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:08.208789  1805 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:08.156822  4747 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:08.243728  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2381) took 1126 ms. Trace:
I20260709 12:23:08.244184  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:08.246475  1877 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.011s	sys 0.028s
I20260709 12:23:08.247035  1874 rpcz_store.cc:276] 0709 12:23:07.116898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.178720 (+ 61822us) service_pool.cc:224] Handling call
0709 12:23:07.211088 (+ 32368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.211424 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.211462 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.227845 (+ 16383us) log.cc:844] Serialized 26 byte log entry
0709 12:23:07.534384 (+306539us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:07.534973 (+   589us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.535058 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.243421 (+708363us) raft_consensus.cc:1685] finished
0709 12:23:08.243451 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.243655 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":288014,"threads_started":1,"wal-append.queue_time_us":601162}
W20260709 12:23:08.248544  1877 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.012s	sys 0.028s
W20260709 12:23:08.221011  1725 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.044s	sys 0.084s
W20260709 12:23:08.250375  1725 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.045s	sys 0.084s
I20260709 12:23:08.251361  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:08.176174  4728 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:08.205464  4757 consensus_queue.cc:1243] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:23:08.203900  4395 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:08.260706  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:08.152307  4639 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.224321  4769 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:08.225322  4546 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:08.212550  4413 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:08.284504  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:08.285846  1872 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:08.265625  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:08.243059  4426 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:23:08.272367  1729 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:08.193428  4760 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:23:08.273648  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 376333us
I20260709 12:23:08.287175  1874 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:08.298360  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:08.243482  4836 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.040s
I20260709 12:23:08.279260  4769 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:08.341077  1872 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20260709 12:23:08.341228  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:08.298557  4736 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:08.344424  4728 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:08.344988  4738 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:08.261102  4747 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.036s
W20260709 12:23:08.425259  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.514739  4846 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.515751  4846 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.517679  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2397) took 1038 ms. Trace:
I20260709 12:23:08.518399  1880 rpcz_store.cc:276] 0709 12:23:07.479044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.575580 (+ 96536us) service_pool.cc:224] Handling call
0709 12:23:07.576082 (+   502us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.576594 (+   512us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.576638 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.577298 (+   660us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.050589 (+473291us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:08.050648 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.050746 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.517389 (+466643us) raft_consensus.cc:1685] finished
0709 12:23:08.517419 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.517597 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8027,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":472732,"threads_started":1,"wal-append.queue_time_us":836635}
W20260709 12:23:08.413322  4831 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.378s	user 0.002s	sys 0.000s
W20260709 12:23:08.391579  4773 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.716s	user 0.011s	sys 0.000s
I20260709 12:23:08.537101  4849 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:08.540081  4836 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:23:08.546422  4750 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.998s	user 0.010s	sys 0.003s
I20260709 12:23:08.546198  4836 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:08.556767  4760 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:23:08.522949  4413 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.557937  4413 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:08.563459  4266 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:08.567278  4836 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:08.568753  4836 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:08.552418  4750 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:08.571158  4846 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:08.583962  2417 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
W20260709 12:23:08.592374  2413 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 693) took 1503 ms (client timeout 1978 ms). Trace:
I20260709 12:23:08.593050  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 696) took 1481 ms. Trace:
W20260709 12:23:08.592960  2413 rpcz_store.cc:269] 0709 12:23:07.088974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.089175 (+   201us) service_pool.cc:224] Handling call
0709 12:23:08.583550 (+1494375us) spinlock_profiling.cc:243] Waited 739 ms on lock 0x7b54003fc854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:08.592289 (+  8739us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1699940096}
I20260709 12:23:08.593519  2414 rpcz_store.cc:276] 0709 12:23:07.111951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.115460 (+  3509us) service_pool.cc:224] Handling call
0709 12:23:08.592872 (+1477412us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b54005f05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:08.592995 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1476160384}
I20260709 12:23:08.597303  4771 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.603364  4643 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:08.609211  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 697) took 1497 ms. Trace:
I20260709 12:23:08.610486  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 698) took 1498 ms. Trace:
I20260709 12:23:08.610961  4694 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:08.611769  2411 rpcz_store.cc:276] 0709 12:23:07.112562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.116187 (+  3625us) service_pool.cc:224] Handling call
0709 12:23:08.610329 (+1494142us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400561b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:08.610431 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2354931968}
I20260709 12:23:08.610920  2416 rpcz_store.cc:276] 0709 12:23:07.112259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.115848 (+  3589us) service_pool.cc:224] Handling call
0709 12:23:08.609038 (+1493190us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b54005609d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:08.609147 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1546896768}
I20260709 12:23:08.574579  4769 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.617965  4466 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:08.616901  4745 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:08.618103  4823 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.042s	sys 0.064s
W20260709 12:23:08.658515  4840 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.379s	user 0.007s	sys 0.002s
I20260709 12:23:08.661551  4734 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:08.631021  4823 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.043s	sys 0.064s
I20260709 12:23:08.629313  4804 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.672209  4804 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.223002  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1618) took 2923 ms. Trace:
I20260709 12:23:08.727707  1724 rpcz_store.cc:276] 0709 12:23:05.299549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.496070 (+196521us) service_pool.cc:224] Handling call
0709 12:23:05.496410 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.198877 (+702467us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.222571 (+2023694us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b54003cc8d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:08.222893 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1546309504}
I20260709 12:23:08.730767  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2519) took 1365 ms. Trace:
W20260709 12:23:08.731465  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.732475  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.733276  1727 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:08.737025  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.005s	sys 0.005s
W20260709 12:23:08.737486  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.005s	sys 0.005s
W20260709 12:23:08.737896  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 780472us
I20260709 12:23:08.729964  1731 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 3.
I20260709 12:23:08.731475  1732 rpcz_store.cc:276] 0709 12:23:07.364873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.597658 (+232785us) service_pool.cc:224] Handling call
0709 12:23:07.598184 (+   526us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.598471 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.598497 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.599141 (+   644us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.068059 (+468918us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:08.068111 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.068185 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.729546 (+661361us) raft_consensus.cc:1685] finished
0709 12:23:08.729636 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.730416 (+   780us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":468480,"threads_started":1,"wal-append.queue_time_us":681458}
W20260709 12:23:08.743140  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.007s	sys 0.004s
W20260709 12:23:08.743755  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.008s	sys 0.004s
W20260709 12:23:08.744158  1878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.000s	sys 0.013s
W20260709 12:23:08.744803  1878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.000s	sys 0.013s
W20260709 12:23:08.746528  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.747196  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.747615  1804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.761s	user 0.005s	sys 0.009s
W20260709 12:23:08.748265  1804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.005s	sys 0.009s
W20260709 12:23:08.749914  1879 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.003s	sys 0.018s
W20260709 12:23:08.750438  1879 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.003s	sys 0.018s
I20260709 12:23:08.753424  1729 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:08.755905  1805 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:08.742751  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:08.760169  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.761489  2410 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.791s	user 0.000s	sys 0.004s
W20260709 12:23:08.762066  2410 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.792s	user 0.000s	sys 0.004s
I20260709 12:23:08.764588  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:08.765954  1732 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:08.766897  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:08.767732  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:08.770207  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:08.770658  2416 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:23:08.771869  4466 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:08.777856  4730 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:08.779211  4515 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.780405  4515 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.786387  4694 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.787786  4694 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:08.766924  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.791934  4745 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.779266  4730 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.797483  4736 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:08.798570  4736 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.755910  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:08.803412  1729 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
W20260709 12:23:08.829993  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.850s	user 0.002s	sys 0.005s
W20260709 12:23:08.830607  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.002s	sys 0.005s
I20260709 12:23:08.801733  4736 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:08.801821  1723 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:08.801882  4769 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:08.777982  4488 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260709 12:23:08.749908  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:08.825322  4426 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:08.821965  4750 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:08.873515  4862 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:08.865281  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2513) took 1506 ms. Trace:
I20260709 12:23:08.855245  4730 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:08.878794  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:08.877122  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 134603us
I20260709 12:23:08.880144  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:08.866852  4694 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:08.877400  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.000s	sys 0.015s
W20260709 12:23:08.891090  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55023.6us
W20260709 12:23:08.896445  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.713s	user 0.000s	sys 0.016s
I20260709 12:23:08.892256  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:08.901729  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2514) took 1537 ms. Trace:
W20260709 12:23:08.902403  1801 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.002s	sys 0.005s
I20260709 12:23:08.902304  4426 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:08.902346  1728 rpcz_store.cc:276] 0709 12:23:07.360341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.361494 (+  1153us) service_pool.cc:224] Handling call
0709 12:23:07.361847 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.587288 (+225441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.587319 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.587763 (+   444us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.055669 (+467906us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:08.058892 (+  3223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.657737 (+598845us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:08.884604 (+226867us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b5400435dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:08.889398 (+  4794us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.891911 (+  2513us) raft_consensus.cc:1685] finished
0709 12:23:08.891927 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.897608 (+  5681us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":88,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":104,"spinlock_wait_cycles":520230144,"thread_start_us":467498,"threads_started":1,"wal-append.queue_time_us":677166}
W20260709 12:23:08.903360  1801 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.002s	sys 0.006s
I20260709 12:23:08.908939  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1635) took 1844 ms. Trace:
I20260709 12:23:08.909894  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:08.911091  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:08.910705  2416 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:08.920040  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2530) took 1033 ms. Trace:
I20260709 12:23:08.924628  1724 rpcz_store.cc:276] 0709 12:23:07.881950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.869360 (+987410us) service_pool.cc:224] Handling call
0709 12:23:08.915450 (+ 46090us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.906621  4426 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:08.929347  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 76832.2us
I20260709 12:23:08.877295  1730 rpcz_store.cc:276] 0709 12:23:07.359072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.359521 (+   449us) service_pool.cc:224] Handling call
0709 12:23:07.359803 (+   282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.372311 (+ 12508us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.372356 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.669193 (+296837us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b2c0005a338. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:23:07.857333 (+188140us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.831449 (+974116us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:08.831590 (+   141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.831749 (+   159us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.847354 (+ 15605us) raft_consensus.cc:1685] finished
0709 12:23:08.847388 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.865194 (+ 17806us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26318,"raft.queue_time_us":12218,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":9463,"spinlock_wait_cycles":2782464,"thread_start_us":860499,"threads_started":1,"wal-append.queue_time_us":905285}
I20260709 12:23:08.934150  1729 raft_consensus.cc:2410] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260709 12:23:08.937124  4739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.025s	sys 0.011s
W20260709 12:23:08.937654  4739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.026s	sys 0.011s
I20260709 12:23:08.938561  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:08.939097  4788 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:08.939692  1799 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:08.940061  4788 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:08.941663  2415 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 2. Current term is 3. Ops: []
I20260709 12:23:08.942943  4763 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:08.945094  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2400) took 1116 ms. Trace:
W20260709 12:23:08.951377  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 360484us
I20260709 12:23:08.945628  1879 rpcz_store.cc:276] 0709 12:23:07.828282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.864752 (+ 36470us) service_pool.cc:224] Handling call
0709 12:23:07.921570 (+ 56818us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.921773 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:07.957809 (+ 36036us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.921135 (+963326us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.940916 (+ 19781us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.944285 (+  3369us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.944389 (+   104us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:08.944519 (+   130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.944624 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.944922 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3062,"prepare.queue_time_us":49213,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":367,"raft.queue_time_us":28021,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":158,"thread_start_us":956373,"threads_started":2,"wal-append.queue_time_us":717736}
I20260709 12:23:08.955292  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1636) took 1620 ms. Trace:
I20260709 12:23:08.956312  4769 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:08.958019  1725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.012s	sys 0.010s
I20260709 12:23:08.957891  4515 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:08.957523  4769 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:08.958817  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 209531us
I20260709 12:23:08.958983  4852 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:23:08.960469  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:08.961580  4515 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:08.961669  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:08.962059  4515 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:08.962531  4515 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:08.962862  4797 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260709 12:23:08.963621  4675 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:08.964417  4675 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:08.964488  4797 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:08.965073  4797 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:08.909340  1801 rpcz_store.cc:276] 0709 12:23:07.064649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.599247 (+534598us) service_pool.cc:224] Handling call
0709 12:23:07.599625 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.845105 (+245480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.845138 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.857659 (+ 12521us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.188424 (+330765us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:08.907851 (+719427us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.908354 (+   503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.908508 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.908582 (+    74us) raft_consensus.cc:1685] finished
0709 12:23:08.908593 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.908846 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":245087,"raft.queue_time_us":248739,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":938949,"threads_started":2,"wal-append.queue_time_us":412405}
I20260709 12:23:08.967661  4769 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:08.968080  1873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.007s	sys 0.007s
W20260709 12:23:08.968504  1873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.008s	sys 0.007s
W20260709 12:23:08.969427  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.007s	sys 0.003s
W20260709 12:23:08.969769  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.007s	sys 0.003s
W20260709 12:23:08.971107  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 629894us
I20260709 12:23:08.938447  4750 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.972321  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:08.973812  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:08.974079  4782 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.121s
I20260709 12:23:08.974740  4426 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.975495  4782 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:08.975788  4859 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:08.975986  4782 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:08.976236  4859 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:08.976684  4859 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:08.977380  4782 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:08.988626  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:08.990062  4679 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:08.990695  4679 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:08.991281  4679 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:08.991695  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:08.955890  1804 rpcz_store.cc:276] 0709 12:23:07.334590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.600383 (+265793us) service_pool.cc:224] Handling call
0709 12:23:07.600669 (+   286us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:07.845646 (+244977us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 12:23:07.845673 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:07.857364 (+ 11691us) log.cc:844] Serialized 52 byte log entry
0709 12:23:08.749188 (+891824us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:23:08.749320 (+   132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.749497 (+   177us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.954892 (+205395us) raft_consensus.cc:1685] finished
0709 12:23:08.954917 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.955202 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":244575,"raft.queue_time_us":245085,"raft.run_cpu_time_us":60,"raft.run_wall_time_us":62,"thread_start_us":761834,"threads_started":1,"wal-append.queue_time_us":966918}
W20260709 12:23:08.998528  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68290.6us
I20260709 12:23:08.985692  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2521) took 1620 ms. Trace:
I20260709 12:23:09.003368  1728 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
W20260709 12:23:09.004056  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 73703.3us
I20260709 12:23:09.004561  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:09.006067  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:09.008451  4771 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:09.020695  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:09.030732  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 67996.5us
I20260709 12:23:08.975509  4426 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:08.966090  4675 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.016081  1772 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:09.040516  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:09.051254  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:09.052397  1728 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:23:09.052623  4639 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:09.053992  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2523) took 1673 ms. Trace:
I20260709 12:23:09.067790  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:09.069330  4867 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.161s	user 0.000s	sys 0.002s
I20260709 12:23:09.071179  1729 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:23:09.072171  4426 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.074048  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2517) took 1710 ms. Trace:
I20260709 12:23:09.010617  4770 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260709 12:23:09.077303  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2401) took 1248 ms. Trace:
I20260709 12:23:09.077715  1873 rpcz_store.cc:276] 0709 12:23:07.828805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.854478 (+ 25673us) service_pool.cc:224] Handling call
0709 12:23:07.920994 (+ 66516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.256440 (+335446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.256475 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.257186 (+   711us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.969110 (+711924us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:08.969184 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.969300 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.077055 (+107755us) raft_consensus.cc:1685] finished
0709 12:23:09.077073 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.077232 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2336,"raft.queue_time_us":122530,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":140,"thread_start_us":1017881,"threads_started":2,"wal-append.queue_time_us":801156}
I20260709 12:23:09.074560  1726 rpcz_store.cc:276] 0709 12:23:07.363022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.409047 (+ 46025us) service_pool.cc:224] Handling call
0709 12:23:09.073574 (+1664527us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b5400414954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:09.073785 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":674491648}
I20260709 12:23:09.088353  1726 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:09.089159  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2524) took 1703 ms. Trace:
I20260709 12:23:09.089566  1726 rpcz_store.cc:276] 0709 12:23:07.385745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.087645 (+1701900us) service_pool.cc:224] Handling call
0709 12:23:09.088104 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.088859 (+   755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.089073 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.091053  1726 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:09.094363  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.102133  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2399) took 1548 ms. Trace:
I20260709 12:23:09.003530  4866 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.092s
W20260709 12:23:09.096292  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71561.7us
I20260709 12:23:09.097429  4515 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.054435  1728 rpcz_store.cc:276] 0709 12:23:07.380277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.053032 (+1672755us) service_pool.cc:224] Handling call
0709 12:23:09.053415 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.053634 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.053662 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.053720 (+    58us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:09.053755 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.053803 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.053918 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.103508  4846 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:09.105890  2419 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 3. Current term is 4. Ops: []
I20260709 12:23:09.107745  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2522) took 1731 ms. Trace:
W20260709 12:23:09.117364  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.118028  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2526) took 1691 ms. Trace:
I20260709 12:23:09.095361  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:09.110689  1799 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:09.002321  4788 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:09.122272  4846 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.122807  4707 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:09.123190  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:09.123353  4707 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:09.123824  4707 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:09.125754  4757 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.126474  4757 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.128822  4757 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:09.128537  4413 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:09.129938  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 764) took 1454 ms. Trace:
I20260709 12:23:09.130040  4413 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:09.130566  4413 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:09.131953  4413 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.133550  4413 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:09.123868  1728 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:09.116309  1727 rpcz_store.cc:276] 0709 12:23:07.376182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.833237 (+457055us) service_pool.cc:224] Handling call
0709 12:23:07.833622 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.733908 (+900286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.107291 (+373383us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b54002d19d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:09.107650 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2055901056}
W20260709 12:23:09.134961  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.118438  1726 rpcz_store.cc:276] 0709 12:23:07.419754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.090361 (+1670607us) service_pool.cc:224] Handling call
0709 12:23:09.090747 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.110621 (+ 19874us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.110849 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.000152  1725 rpcz_store.cc:276] 0709 12:23:07.365399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.614563 (+249164us) service_pool.cc:224] Handling call
0709 12:23:07.614885 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.615140 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:08.251581 (+636441us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.982182 (+730601us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.982437 (+   255us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:08.982576 (+   139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.982695 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.985613 (+  2918us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":738262,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":7479,"raft.queue_time_us":363,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":174,"thread_start_us":1356191,"threads_started":2}
I20260709 12:23:09.145145  4807 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:09.140498  4466 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:09.146629  4466 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.148883  4466 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:09.130369  2410 rpcz_store.cc:276] 0709 12:23:07.675490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.695092 (+ 19602us) service_pool.cc:224] Handling call
0709 12:23:07.695625 (+   533us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.696006 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.696048 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.857651 (+161603us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.762950 (+905299us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:08.763116 (+   166us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.763279 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.068214 (+304935us) spinlock_profiling.cc:243] Waited 9.95 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:09.112638 (+ 44424us) raft_consensus.cc:1685] finished
0709 12:23:09.112674 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.129803 (+ 17129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22877056,"thread_start_us":792276,"threads_started":1,"wal-append.queue_time_us":982691}
W20260709 12:23:09.152235  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.112711  4782 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:09.154976  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.155876  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.144173  4846 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:09.150774  4804 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.109839  1878 rpcz_store.cc:276] 0709 12:23:07.553524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.585902 (+ 32378us) service_pool.cc:224] Handling call
0709 12:23:07.586167 (+   265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.060982 (+474815us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:23:08.077306 (+ 16324us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.745785 (+668479us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.745924 (+   139us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:08.746073 (+   149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.101682 (+355609us) spinlock_profiling.cc:243] Waited 474 ms on lock 0x7b5400118354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:09.101746 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.102035 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":17180,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":68,"raft.queue_time_us":33119,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":380,"spinlock_wait_cycles":1091223808,"thread_start_us":667601,"threads_started":1,"wal-append.queue_time_us":918399}
I20260709 12:23:09.160444  4515 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.161767  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 399) took 2027 ms. Trace:
I20260709 12:23:09.162254  1731 rpcz_store.cc:276] 0709 12:23:07.134564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.173671 (+ 39107us) service_pool.cc:224] Handling call
0709 12:23:09.161484 (+1987813us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b540030af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:09.161640 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3573626752}
I20260709 12:23:09.164080  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 607) took 1465 ms. Trace:
I20260709 12:23:09.164603  2415 rpcz_store.cc:276] 0709 12:23:07.698477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.741207 (+ 42730us) service_pool.cc:224] Handling call
0709 12:23:07.741731 (+   524us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.942160 (+1200429us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.163772 (+221612us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54005438d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:09.164017 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2730373888}
I20260709 12:23:09.171177  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:09.172173  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2388) took 2002 ms. Trace:
I20260709 12:23:09.172717  1876 rpcz_store.cc:276] 0709 12:23:07.169352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.214752 (+ 45400us) service_pool.cc:224] Handling call
0709 12:23:07.215210 (+   458us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.307581 (+ 92371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.307623 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.308285 (+   662us) log.cc:844] Serialized 26 byte log entry
0709 12:23:07.847679 (+539394us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:07.847722 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.171398 (+1323676us) spinlock_profiling.cc:243] Waited 80.6 ms on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:09.171476 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.171554 (+    78us) raft_consensus.cc:1685] finished
0709 12:23:09.171566 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.171945 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11123,"raft.queue_time_us":16850,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":8095,"spinlock_wait_cycles":185405312,"thread_start_us":538937,"threads_started":1,"wal-append.queue_time_us":735495,"wal-append.run_cpu_time_us":2246,"wal-append.run_wall_time_us":1004567}
I20260709 12:23:09.175098  4515 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:09.176083  4515 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.078557  4679 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.181000  4750 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.182708  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1647) took 1006 ms. Trace:
I20260709 12:23:09.183254  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:09.183177  1802 rpcz_store.cc:276] 0709 12:23:08.176137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.176585 (+   448us) service_pool.cc:224] Handling call
0709 12:23:08.176907 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.732310 (+555403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.732427 (+   117us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.732529 (+   102us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:08.732599 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.182179 (+449580us) spinlock_profiling.cc:243] Waited 555 ms on lock 0x7b54002f2e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:09.182251 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.182506 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1275494656}
I20260709 12:23:09.181644  4679 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:09.185595  4679 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.187252  4679 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.186400  4763 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:23:09.216526  4836 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.647s	user 0.005s	sys 0.010s
I20260709 12:23:09.217404  4679 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:09.218569  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 263759us
W20260709 12:23:09.191937  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.219097  1726 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:23:09.164219  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2532) took 1178 ms. Trace:
W20260709 12:23:09.191154  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.195152  1724 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:09.192626  1629 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 3 to 4, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 4 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:09.219703  1731 rpcz_store.cc:276] 0709 12:23:07.985894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.163166 (+1177272us) service_pool.cc:224] Handling call
0709 12:23:09.163631 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.163876 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.163904 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.163967 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:09.164007 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.164060 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.164164 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.198599  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:09.199239  4769 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:08.966560  4797 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:09.207424  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 191542us
I20260709 12:23:09.211131  4515 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:09.211529  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72171.4us
W20260709 12:23:09.211917  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 99018.9us
I20260709 12:23:09.213191  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:09.214589  1627 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 270) took 23516 ms (client timeout 15000 ms). Trace:
I20260709 12:23:09.217567  1732 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:23:09.183331  4853 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:09.198599  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:09.193955  4807 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.235713  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1634) took 1062 ms. Trace:
I20260709 12:23:09.237257  1724 rpcz_store.cc:276] 0709 12:23:08.166519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.194239 (+1027720us) service_pool.cc:224] Handling call
0709 12:23:09.194859 (+   620us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.228378 (+ 33519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.228629 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.238948  2410 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:09.239305  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:09.241015  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:09.241866  1731 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:09.243017  4769 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.243927  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1635) took 1042 ms. Trace:
I20260709 12:23:09.244544  1732 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:09.245239  4769 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:09.246575  4763 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:09.247385  4763 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:09.244436  1725 rpcz_store.cc:276] 0709 12:23:08.204306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.221072 (+1016766us) service_pool.cc:224] Handling call
0709 12:23:09.243262 (+ 22190us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.243555 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.243583 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.243645 (+    62us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:09.243684 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.243739 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.243857 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.219825  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2531) took 1336 ms. Trace:
I20260709 12:23:09.242687  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:09.235379  4853 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:09.250829  4750 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:09.251487  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.251886  4750 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.252157  4853 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.253604  4853 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:09.248793  2414 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:23:09.233193  1730 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:23:09.250916  2419 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:09.250010  1726 rpcz_store.cc:276] 0709 12:23:07.882961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.144756 (+1261795us) service_pool.cc:224] Handling call
0709 12:23:09.218692 (+ 73936us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.219566 (+   874us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.219738 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.254652  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.254956  4757 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.235641  4804 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:23:09.262470  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.257443  4797 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:09.254626  4750 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:09.283751  2415 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:09.284138  1728 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:09.286224  1725 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:09.231601  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8bb813e4fab14a61b810a24b56899586 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
W20260709 12:23:09.292224  1615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60917.8us
I20260709 12:23:09.295202  4702 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:23:09.233877  1627 rpcz_store.cc:269] 0709 12:22:45.698009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.698347 (+   338us) service_pool.cc:224] Handling call
0709 12:22:46.125280 (+426933us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b5400002490. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb40b2 00007f16b1c2d759 00007f16b1bc5a76 00007f16b1b96d58 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc
0709 12:22:46.453762 (+328482us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd42090 00007f16abde0f3a 00007f16b1b96e9c 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:23:09.214326 (+22760564us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:04.105167 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:04.105894 (+   727us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:04.105956 (+    62us) write_op.cc:435] Acquired schema lock
0709 12:23:04.105970 (+    14us) tablet.cc:662] Decoding operations
0709 12:23:04.124605 (+ 18635us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:04.142962 (+ 18357us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:04.146940 (+  3978us) tablet.cc:685] Acquiring locks for 8 operations
0709 12:23:04.407395 (+260455us) tablet.cc:701] Row locks acquired
0709 12:23:04.436771 (+ 29376us) write_op.cc:260] PREPARE: finished
0709 12:23:04.437109 (+   338us) write_op.cc:270] Start()
0709 12:23:04.437308 (+   199us) write_op.cc:276] Timestamp: P: 1783599784436993 usec, L: 0
0709 12:23:04.437328 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:23:04.450915 (+ 13587us) log.cc:844] Serialized 3281 byte log entry
0709 12:23:05.720681 (+1269766us) op_driver.cc:464] REPLICATION: finished
0709 12:23:06.617718 (+897037us) write_op.cc:301] APPLY: starting
0709 12:23:06.631366 (+ 13648us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:06.690756 (+ 59390us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:06.718692 (+ 27936us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:06.786323 (+ 67631us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:06.786667 (+   344us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0709 12:23:06.786711 (+    44us) write_op.cc:312] APPLY: finished
0709 12:23:06.791754 (+  5043us) log.cc:844] Serialized 65 byte log entry
0709 12:23:07.122876 (+331122us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:07.123051 (+   175us) write_op.cc:454] Released schema lock
0709 12:23:07.123131 (+    80us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:07.123824 (+   693us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:07.123860 (+    36us) write_op.cc:454] Released schema lock
0709 12:23:07.587092 (+463232us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
Metrics: {"mutex_wait_us":5614955,"reported_tablets":28,"spinlock_wait_cycles":307007616,"child_traces":[["op",{"apply.queue_time_us":847628,"apply.run_cpu_time_us":38843,"apply.run_wall_time_us":523256,"num_ops":8,"prepare.queue_time_us":295867,"prepare.run_cpu_time_us":32292,"prepare.run_wall_time_us":545653,"raft.queue_time_us":723949,"raft.run_cpu_time_us":38996,"raft.run_wall_time_us":576075,"replication_time_us":1230892,"spinlock_wait_cycles":3771136,"thread_start_us":1615619,"threads_started":5,"wal-append.queue_time_us":785312,"wal-append.run_cpu_time_us":64177,"wal-append.run_wall_time_us":3112388}]]}
I20260709 12:23:09.296525  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.297014  4643 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.297647  4643 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.295389  4770 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:23:09.305454  4702 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:09.291922  1629 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 6 to 7, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 7 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:09.307514  1724 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:23:09.299175  4780 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:09.320477  1730 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260709 12:23:09.306016  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:09.313063  1725 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:09.307736  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.312095  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:09.313930  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:09.325742  1629 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 6 to 7, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 7 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:09.291076  4757 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.328814  1725 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:09.328227  1629 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 2 to 5, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 5 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:09.330127  4757 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.331112  4679 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:09.331349  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:09.331777  4679 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:09.310663  4750 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:09.332537  1728 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:09.332940  4679 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.335167  4866 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:09.335623  4866 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:09.336017  4866 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:09.322485  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:09.333448  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78957.8us
I20260709 12:23:09.338938  1724 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:23:09.339372  1724 pending_rounds.cc:85] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 7
I20260709 12:23:09.339598  1724 pending_rounds.cc:107] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:23:09.339907  1724 raft_consensus.cc:2889] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:09.308561  4859 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.340221  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:09.320942  1730 pending_rounds.cc:85] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 2
I20260709 12:23:09.343106  1730 pending_rounds.cc:107] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260709 12:23:09.343457  1730 raft_consensus.cc:2889] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:09.342401  4859 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.340500  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.350293  1732 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:09.351533  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:09.342319  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:09.361668  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.306074  4702 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:09.364714  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:09.365087  4771 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:09.366365  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 90521.2us
I20260709 12:23:09.366983  4852 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:09.366842  1629 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 2 to 3, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:09.370280  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.371023  1723 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:23:09.376318  1723 pending_rounds.cc:85] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 3
I20260709 12:23:09.376652  1723 pending_rounds.cc:107] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:23:09.376901  1723 raft_consensus.cc:2889] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:09.312734  4763 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:09.269299  4807 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:09.364646  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 122045us
I20260709 12:23:09.376991  4771 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.288899  4804 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.381718  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:09.377187  4852 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.381007  4771 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:09.387243  1629 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 4 to 5, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 5 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:23:09.371711  4736 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.370980  4859 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:09.390213  4852 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:09.394515  1726 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:09.388808  4745 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:09.395640  4745 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:09.396018  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 44088.5us
I20260709 12:23:09.396025  4745 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:09.400162  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:09.403039  4515 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:09.403306  1629 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 2 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:09.407480  4702 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.408759  4702 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:09.410974  4780 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:09.411451  4780 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.411993  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:09.412506  4769 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:09.413041  4769 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.413371  4769 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:09.386016  4763 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:09.415236  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 106865us
W20260709 12:23:09.416854  4756 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 12:23:09.418386  4804 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.419747  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2512) took 2033 ms. Trace:
I20260709 12:23:09.420313  1723 rpcz_store.cc:276] 0709 12:23:07.357532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.357711 (+   179us) service_pool.cc:224] Handling call
0709 12:23:07.358073 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.790622 (+1432549us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54003627a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4da1d 00007f16abd341a4 00007f16abdb6447 00007f16abdcefe7 00007f16abdd07de 00007f16abddab49 00007f16abddb416 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419
0709 12:23:09.386692 (+596070us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.391005 (+  4313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":682473216}
I20260709 12:23:09.405781  1629 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 2 to 5, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:09.411852  4780 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:09.410483  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:09.402484  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:09.437778  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:09.429302  4675 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:09.436853  1629 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 2 to 6. New cstate: current_term: 6 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:09.422183  4745 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.483392  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:09.422526  4763 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:09.478293  4757 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.489635  4515 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:09.498698  4773 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.524019  4773 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:09.511144  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 108744us
I20260709 12:23:09.480763  4750 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.526486  4773 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:09.523080  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 152943us
I20260709 12:23:09.436115  4639 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:23:09.469944  4736 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:09.551290  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.519886  4694 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:09.574575  1629 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:09.572098  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:09.515426  4643 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.564847  4773 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:09.588649  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 175093us
W20260709 12:23:09.592149  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 197175us
W20260709 12:23:09.605607  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 208746us
I20260709 12:23:09.607015  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:09.518158  4757 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.580638  4836 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:23:09.555997  4750 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:09.610707  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2527) took 2027 ms. Trace:
I20260709 12:23:09.545315  4771 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.621004  1729 rpcz_store.cc:276] 0709 12:23:07.577249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.133281 (+1556032us) service_pool.cc:224] Handling call
0709 12:23:09.133831 (+   550us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.134276 (+   445us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:09.430876 (+296600us) log.cc:844] Serialized 14 byte log entry
0709 12:23:09.603947 (+173071us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:09.604029 (+    82us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:09.604107 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.604209 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.604398 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6006,"prepare.queue_time_us":360321,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"raft.queue_time_us":17802,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":4809,"thread_start_us":356036,"threads_started":2}
W20260709 12:23:09.636966  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 327985us
I20260709 12:23:09.587921  4807 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:09.638769  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 222668us
I20260709 12:23:09.638737  1629 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:23:09.641105  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84080us
I20260709 12:23:09.619673  4797 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:23:09.630936  4865 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:23:09.616827  4694 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:09.627974  4805 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:09.622875  4788 raft_consensus.cc:2764] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260709 12:23:09.628332  4771 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:09.631053  4846 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:09.656078  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.663244  1629 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 3 to 6, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:09.670329  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:09.670145  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:09.666746  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:09.632385  4707 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.656770  4797 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:09.680737  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:09.655086  4805 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:09.664705  4846 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:09.694270  1724 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:23:09.695890  1629 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 5 to 6, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:23:09.698333  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 116948us
I20260709 12:23:09.700093  2415 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
W20260709 12:23:09.700853  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.692325  4771 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.704840  4771 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.705394  4750 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:09.704135  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.712261  1629 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 1 to 2, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:23:09.713085  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 57087.6us
I20260709 12:23:09.715744  4782 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:09.730999  4899 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:09.730686  4805 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:09.731535  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.731885  4736 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:09.734537  4894 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:09.748358  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59876us
I20260709 12:23:09.734772  4745 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.759418  4906 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:09.759943  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.735980  4771 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:09.736141  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 100156us
I20260709 12:23:09.737839  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:09.728091  4694 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.691535  4846 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:09.748961  4413 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.615s	user 0.006s	sys 0.010s
I20260709 12:23:09.749109  1629 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 5 to 6, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 6 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:09.760005  4757 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:09.766404  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.755034  4885 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
W20260709 12:23:09.785820  4889 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
W20260709 12:23:09.786291  4896 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:23:09.760152  4906 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:09.709365  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 235878us
W20260709 12:23:09.794765  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 187171us
I20260709 12:23:09.789697  4884 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:09.758275  4866 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.751731  4780 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.805233  1629 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:09.814045  4900 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:23:09.819315  4707 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:09.800499  4797 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.821394  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:09.822180  2415 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:09.822324  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:09.832610  2414 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:09.825708  4757 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:09.831872  4805 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:09.833907  4757 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:09.821645  4797 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:09.811990  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:09.837088  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:09.812747  4853 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.558s	user 0.012s	sys 0.004s
I20260709 12:23:09.846066  4909 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
W20260709 12:23:09.847651  4907 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:23:09.851397  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:09.863499  4846 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.869011  2415 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:23:09.843075  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44892.4us
W20260709 12:23:09.886932  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 142535us
W20260709 12:23:09.890116  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 152421us
I20260709 12:23:09.890596  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2529) took 2037 ms. Trace:
I20260709 12:23:09.889818  4884 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:09.871343  4757 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:09.808463  4702 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.081s
I20260709 12:23:09.905546  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:09.887210  4745 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:09.884774  4836 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.914098  1875 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 12:23:09.914489  1881 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:23:09.915174  1881 pending_rounds.cc:85] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 3
I20260709 12:23:09.915479  1881 pending_rounds.cc:107] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:23:09.915720  1881 raft_consensus.cc:2889] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:09.918046  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 118592us
W20260709 12:23:09.907620  4852 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.515s	user 0.013s	sys 0.000s
I20260709 12:23:09.923142  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:09.926111  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:09.931365  2415 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:09.927951  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:09.911820  4745 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.936605  4745 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:09.938623  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.940349  4413 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:09.940734  4413 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:09.941138  4413 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:09.941170  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:09.939670  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:09.943634  4780 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.945267  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:09.957777  2414 raft_consensus.cc:2376] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6. Re-sending same reply.
I20260709 12:23:09.947064  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:09.925727  4736 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:09.959141  4736 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:09.959807  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:09.960587  1879 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:09.960206  4736 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.961820  4736 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:09.953368  4866 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:09.965409  4897 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260709 12:23:09.920941  4900 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260709 12:23:09.969347  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:09.970126  2414 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:09.970283  4898 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:09.970875  4898 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:09.971818  4898 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.973192  4898 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:09.976202  4757 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.979780  4866 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:23:09.927210  4769 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.513s	user 0.009s	sys 0.005s
I20260709 12:23:09.980890  4763 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:09.980736  4769 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:09.981297  4763 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:09.947605  4906 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:09.916580  4836 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:09.951043  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:10.008798  2414 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:09.951399  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:09.952433  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:09.953366  1872 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:09.897576  1727 rpcz_store.cc:276] 0709 12:23:07.851131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.173231 (+1322100us) service_pool.cc:224] Handling call
0709 12:23:09.173824 (+   593us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:09.249280 (+ 75456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.249309 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:09.274516 (+ 25207us) log.cc:844] Serialized 26 byte log entry
0709 12:23:09.539947 (+265431us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:09.574599 (+ 34652us) log.cc:844] Serialized 14 byte log entry
0709 12:23:09.574927 (+   328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.575027 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.863910 (+288883us) spinlock_profiling.cc:243] Waited 26.3 ms on lock 0x7b2400186ba8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:23:09.870651 (+  6741us) raft_consensus.cc:1685] finished
0709 12:23:09.887843 (+ 17192us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.887987 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61998,"prepare.queue_time_us":117613,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":79553,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":4822,"spinlock_wait_cycles":60460800,"thread_start_us":164393,"threads_started":1,"wal-append.queue_time_us":322422}
I20260709 12:23:09.953642  1875 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 3. Current term is 4. Ops: []
I20260709 12:23:09.954229  4884 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:09.954977  1878 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:09.955513  4675 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:09.998251  4763 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:10.000456  4756 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:10.000679  4804 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:10.000502  4750 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:10.002553  4757 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:10.003487  4846 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:10.003684  4859 raft_consensus.cc:2764] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:23:10.010550  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:10.010488  1876 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:10.026492  1879 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:10.032516  1881 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:10.035372  1730 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:10.037905  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:10.046315  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 370444us
I20260709 12:23:10.013384  4639 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.014444  4515 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:23:10.079807  4763 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:10.083853  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47637.3us
I20260709 12:23:10.084996  4780 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:10.053064  4675 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:10.064059  2410 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:10.065316  4836 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:09.935035  4852 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:10.070989  4639 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.071678  4769 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:10.072903  4788 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.029589  4756 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.075294  4488 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:23:10.084014  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.047124  4900 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.110409  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:10.097227  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:10.092722  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:10.108093  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:10.106884  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56019.1us
W20260709 12:23:10.112033  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 170962us
I20260709 12:23:10.088716  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:10.111003  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:10.091203  4643 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.096599  4807 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:10.126140   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:10.103240  4782 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:23:10.127185  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94039.2us
I20260709 12:23:10.130152  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:10.126612  4906 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 12:23:10.131027  2419 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:10.131860  4900 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:10.132740  4908 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:23:10.108894  4756 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:10.144702  2414 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:10.108457  4675 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:10.151815  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.152462  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:10.134557  4900 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:10.096341  4780 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.114842  4897 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.155921  1800 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:10.123315  4702 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:10.161836  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:10.174697  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.173923  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:10.151371  4906 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.151918  4853 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.113466  4788 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.154812  4771 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.187206  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 85874.4us
W20260709 12:23:10.192708  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.198511  2414 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:10.163524  4702 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.163292  4515 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.204489  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.194612  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:10.205314  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:10.209748  4913 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.080s	user 0.003s	sys 0.000s
I20260709 12:23:10.163679  4807 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:10.197593  4906 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.153343  4643 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:10.212173  4639 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:10.214095  4643 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:10.215514  4515 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:10.257896  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.233080  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:10.276141  2410 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:10.269870  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:10.230661  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:10.219883  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:10.248502  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.280612  4757 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:10.282756  1881 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.295481  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:10.298681  2414 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:10.233201  4900 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.231043  4771 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:10.319808  4900 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.321748  4771 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:10.221477  4884 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:10.322271  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 125064us
I20260709 12:23:10.195631  4897 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:10.323686  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 103982us
I20260709 12:23:10.322909  4675 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.198458  4853 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.324695  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:10.221216  4852 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:10.321817  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 91399.7us
I20260709 12:23:10.222862  4916 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20260709 12:23:10.218679  4515 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:10.328085  4884 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:23:10.327476  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:10.212291  4807 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:10.328302  4899 raft_consensus.cc:2764] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260709 12:23:10.393720  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.433102  1800 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:10.329165  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:10.323132  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:10.356611  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 165546us
I20260709 12:23:10.472375  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:10.475914  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 148592us
I20260709 12:23:10.498814  1881 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:10.499970  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 326182us
I20260709 12:23:10.363274  4805 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:10.328568  4884 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.207513  4702 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:10.301569  4707 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:23:10.503378  4745 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.566s	user 0.011s	sys 0.001s
W20260709 12:23:10.526927  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 203860us
W20260709 12:23:10.500816  4413 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.559s	user 0.010s	sys 0.001s
I20260709 12:23:10.500871  4694 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:10.336787  4780 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:10.524420  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52207.7us
I20260709 12:23:10.500586  4918 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.500368  4804 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:10.536289  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:10.521451  4884 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:23:10.559670  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.600533  1802 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
W20260709 12:23:10.541430  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 455852us
I20260709 12:23:10.356148  4788 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:10.538915  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 386525us
W20260709 12:23:10.604791  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66750.9us
W20260709 12:23:10.606472  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70300.7us
I20260709 12:23:10.612504  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:10.613950  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:10.620757  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:10.627146  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 300822us
I20260709 12:23:10.538718  4643 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.635108  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:10.532470  4807 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:10.606040  4804 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:10.526273  4852 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:10.615752  4639 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:10.661754  4666 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:10.616156  4918 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.538720  4771 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.109s
I20260709 12:23:10.599988  4763 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.052s
I20260709 12:23:10.629565  4488 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:10.667482  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:10.534489  4789 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.117s
I20260709 12:23:10.578760  4413 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:10.666822  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:10.676213  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:10.678731  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 143008us
I20260709 12:23:10.604028  4866 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:23:10.611511  4805 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 12:23:10.686614  4788 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:10.662397  4921 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260709 12:23:10.672070  4515 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 12:23:10.663648  4804 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:10.656447  4898 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:10.615991  4888 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 12:23:10.671072  4639 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:10.701647  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.708828  4413 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:10.709585  1875 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:10.713668  4756 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.580s	user 0.008s	sys 0.004s
W20260709 12:23:10.716405  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49066.5us
I20260709 12:23:10.532886  4745 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.655368  4643 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:10.732514  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:10.733649  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:10.724179  4917 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
I20260709 12:23:10.730464  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:10.717489  4925 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:10.721249  4771 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:10.773918  4639 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:10.717475  4488 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:10.773423  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:10.770967  4788 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:10.735908  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:10.774453  4836 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:10.814468  1803 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:23:10.820923  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 172648us
I20260709 12:23:10.813387  4898 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.786949  4916 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:10.825688  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:10.825688  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:10.794368  4925 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:10.826058  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:10.835325  1798 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:10.836743  1804 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:10.786619  4756 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:10.837037  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:10.857506  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 147994us
I20260709 12:23:10.817092  4413 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:23:10.874868  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 173340us
I20260709 12:23:10.875801  1727 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:10.881506  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:10.884719  1723 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:10.916195  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:10.917111  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:10.915156  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:10.917258  1728 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:10.955796  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:10.956660  1729 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:10.836913  4925 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:10.821503  4771 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:10.959678  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:10.963004  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 232651us
I20260709 12:23:10.832120  4898 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:10.829161  4899 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:10.836974  4916 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:10.894124  4666 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.822242  4675 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.132s
I20260709 12:23:10.830528  4836 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:10.884328  4906 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:10.992302  1804 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:10.979166  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:10.996150  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:23:10.995182  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78194.1us
I20260709 12:23:11.002638  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.005424  1732 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:11.006065  1723 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:23:10.826162  4488 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.019637  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:11.021174  1799 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:23:11.014528  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1693457139999737s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260709 12:23:11.022630  1696 connection.cc:256] RPC call timeout handler was delayed by -1.162145713999962s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1534s for a response
W20260709 12:23:11.023082  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1495457139999417s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15480000000000002s for a response
W20260709 12:23:11.027660  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0469457139997758s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
I20260709 12:23:11.029613  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:10.828807  4788 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.042299  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:11.058344  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.058545  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:11.055022  4782 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.062937  4782 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.065840  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:11.066926  4932 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:23:11.067085  4745 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.068264  1804 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
W20260709 12:23:11.043557  4931 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:11.019088  4707 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:11.032308  4515 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:11.044426  4756 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:11.009367  4666 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:11.103596  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 107585us
I20260709 12:23:11.023128  4939 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.065s
I20260709 12:23:11.076238  4780 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.107560  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.110569  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:11.112932  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:11.091214  4771 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.123837  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.124260  4899 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:23:11.124655  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82488.3us
I20260709 12:23:11.124657  1876 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:11.125512  4916 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:11.127157  4836 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.101713  4940 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:11.130000  4940 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.124722  4899 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:11.131513  4899 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:11.131904  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.133057  1879 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:11.133678  4941 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.125671  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:11.134374  4941 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:11.136349  4923 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.332s	user 0.002s	sys 0.000s
I20260709 12:23:11.138445  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 417) took 1049 ms. Trace:
I20260709 12:23:11.139041  1730 rpcz_store.cc:276] 0709 12:23:10.089195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.089387 (+   192us) service_pool.cc:224] Handling call
0709 12:23:10.109734 (+ 20347us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:10.120120 (+ 10386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.120156 (+    36us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:10.150513 (+ 30357us) log.cc:844] Serialized 78 byte log entry
0709 12:23:10.466105 (+315592us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:10.466178 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.466295 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.138091 (+671796us) raft_consensus.cc:1685] finished
0709 12:23:11.138121 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.138361 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5546,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":18712,"thread_start_us":272024,"threads_started":1,"wal-append.queue_time_us":476967}
I20260709 12:23:11.142346  1799 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:11.092984  4707 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:11.151057  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:11.151365  4941 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:11.149652  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:11.151876  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:11.157563  4702 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.000s
W20260709 12:23:11.158255  4936 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.160s	user 0.001s	sys 0.001s
I20260709 12:23:11.158684  4643 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:11.151412  4938 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:23:11.160993  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.170156  1804 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:11.170122  1798 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:23:11.184943  1730 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:11.175580  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.173409  1801 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
W20260709 12:23:11.176766  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66355us
I20260709 12:23:11.178474  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:11.116608  4413 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:11.181388  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.169091  4675 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:23:11.177562  4702 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.213567  4853 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.215032  4952 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:11.215560  4952 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:11.215925  4952 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:11.211326  4675 consensus_queue.cc:1243] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:23:11.206282  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:11.142866  4780 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.221688  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.218924  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:11.216123  4807 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:11.191462  4941 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:23:11.238298  1806 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:23:11.213625  4702 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:11.224291  4916 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:11.245563  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.249497  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:11.223397  4413 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:11.228981  4853 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:11.246457  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.236073  4807 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:11.258550  4780 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:11.260586  1731 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:11.256000  4940 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:11.272454  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.247404  4757 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:11.211328  4707 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.280560  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.284090  1731 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:11.285158  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.285869  1731 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:11.278088  4949 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:11.293462  1804 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:11.294459  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:11.295017  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:11.295063  4933 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.295672  1807 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:11.296034  4807 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.298461  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2419) took 1130 ms. Trace:
I20260709 12:23:11.298944  1875 rpcz_store.cc:276] 0709 12:23:10.168355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.176637 (+  8282us) service_pool.cc:224] Handling call
0709 12:23:10.192196 (+ 15559us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.779221 (+587025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.298166 (+518945us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:11.298394 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1189227264}
I20260709 12:23:11.300727  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:11.301445  1807 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
W20260709 12:23:11.302139  4884 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.655s	user 0.008s	sys 0.002s
I20260709 12:23:11.301436  4897 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.305250  4950 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:11.306083  4782 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:11.307328  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:11.307987  1805 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:11.309404  4789 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:11.309852  4789 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:11.277190  4940 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.311417  4940 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.315249  4918 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.318933  4769 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:11.320206  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 771) took 1345 ms. Trace:
I20260709 12:23:11.319618  4413 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.321584  4943 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.320652  2415 rpcz_store.cc:276] 0709 12:23:09.982814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.000038 (+ 17224us) service_pool.cc:224] Handling call
0709 12:23:10.002399 (+  2361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.019747 (+ 17348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.019801 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:10.047235 (+ 27434us) log.cc:844] Serialized 26 byte log entry
0709 12:23:10.274701 (+227466us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:11.317931 (+1043230us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b1800112960. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf
0709 12:23:11.318143 (+   212us) log.cc:844] Serialized 14 byte log entry
0709 12:23:11.318418 (+   275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.318535 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.318594 (+    59us) raft_consensus.cc:1685] finished
0709 12:23:11.318707 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.320132 (+  1425us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3134,"prepare.queue_time_us":390483,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":3572,"raft.queue_time_us":1142,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":16312,"spinlock_wait_cycles":11675008,"thread_start_us":521215,"threads_started":2,"wal-append.queue_time_us":261453}
I20260709 12:23:11.322132  4943 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.280191  4853 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.324638  4943 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:11.324694  4853 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.329967  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:11.331343  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.333505  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:23:11.281250  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 88194us
I20260709 12:23:11.330606  1729 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:11.341984  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:11.343374  1798 raft_consensus.cc:2410] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:23:11.344619  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:11.288553  4949 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:11.341246  1728 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:23:11.335647  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:11.347383  4941 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:11.347909  1729 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:23:11.348074  1878 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260709 12:23:11.348455  1806 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:11.332864  4789 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.349160  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.351383  1880 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:11.352192  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.353986  1874 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:11.355284  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.359205  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:11.387985  1807 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:11.360306  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.389076  1876 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:23:11.389343  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:11.365583  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.365844  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.344738  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:11.325327  4943 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.392966  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:11.371280  4916 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:11.373378  4950 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:11.326278  4804 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.544s	user 0.007s	sys 0.007s
I20260709 12:23:11.377684  1805 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:11.378037  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:11.345530  4756 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:11.393865  4639 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.557s	user 0.012s	sys 0.000s
I20260709 12:23:11.416329  2419 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:11.375929  1728 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:11.417855  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.327217  4940 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:11.391470  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:11.296044  4933 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.401736  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:11.402993  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.279616  4757 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.376911  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:11.410701  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.427958  1729 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:23:11.411127  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.415298  4789 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:11.410410  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:11.413058  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:11.383637  4788 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.435031  4955 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:11.435531  4955 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:11.435357  4788 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.435847  1723 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:11.435374  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:11.415230  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:11.438166  1727 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:11.384310  4963 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.376420  4949 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:11.387003  4745 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.421653  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.388118  4941 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:11.422385  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:11.418570  1728 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:11.379180  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:11.302704  4884 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.402571  4916 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:11.425352  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:11.427685  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.432462  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.433480  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:11.429977  4780 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:11.410867  4804 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:11.420307  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.404234  4943 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.784227  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:11.448719  2383 connection.cc:256] RPC call timeout handler was delayed by -1.386600740999711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260709 12:23:11.785627  1727 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:11.557862  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:11.665223  1803 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:11.793697  2415 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:23:11.751354  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:11.427073  4866 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:11.798058  1731 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:23:11.773452  4957 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.488s	user 0.000s	sys 0.001s
I20260709 12:23:11.593400  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:11.825912  1874 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:11.829622  1728 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:11.830611  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:11.833642  2416 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:11.786954  4941 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:11.825479  2413 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
W20260709 12:23:11.776059  4953 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.505s	user 0.001s	sys 0.000s
W20260709 12:23:11.793395  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 773453us
I20260709 12:23:11.822949  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:11.658339  4899 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.526s	user 0.014s	sys 0.000s
I20260709 12:23:11.656912  4933 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.827013  1727 pending_rounds.cc:85] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 4
I20260709 12:23:11.872823  1727 pending_rounds.cc:107] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:23:11.873203  1727 raft_consensus.cc:2889] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:11.774560  4921 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:23:11.772169  4962 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.487s	user 0.002s	sys 0.000s
I20260709 12:23:11.843454  2419 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:11.833480  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:11.426805  4639 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.856386  1731 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:11.435967  4955 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:11.866966  1730 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:11.826647  4780 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:11.646649  4967 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:23:11.870924  4899 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.920945  4899 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:11.794121  4745 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.888979  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:11.921834  4941 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.469209  4963 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:11.923120  4745 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.925046  4963 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:11.440868  4906 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.420s
I20260709 12:23:11.925657  4884 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.848212  4943 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.891744  2416 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:23:11.893793  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2545) took 1032 ms. Trace:
I20260709 12:23:11.441470  4940 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.416s
I20260709 12:23:11.675977  4757 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:11.927906  4941 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.929036  4940 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:11.929617  4940 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:11.922518  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:11.931458  4952 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.710s	user 0.006s	sys 0.008s
I20260709 12:23:11.932026  4952 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:11.934077  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.928143  1729 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:11.900101  4933 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:11.935513  4933 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:11.936048  4933 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:11.907487  4780 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:11.947243  4948 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:11.934697  4952 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:11.949429  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:11.849668  4413 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:11.952338  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:11.915611  4912 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.780s	user 0.000s	sys 0.017s
I20260709 12:23:11.922308  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.947877  4948 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.898310  2415 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:11.976816  1881 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 4.
I20260709 12:23:11.949764  4639 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.895537  4921 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:11.928036  1724 rpcz_store.cc:276] 0709 12:23:10.861047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.861241 (+   194us) service_pool.cc:224] Handling call
0709 12:23:10.880730 (+ 19489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.988177 (+107447us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.988222 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.002297 (+ 14075us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.107535 (+105238us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:11.108064 (+   529us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.108482 (+   418us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.775409 (+666927us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:11.893455 (+118046us) raft_consensus.cc:1685] finished
0709 12:23:11.893477 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.893726 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79709,"raft.queue_time_us":193830,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":29999,"spinlock_wait_cycles":42528128,"thread_start_us":61350,"threads_started":1,"wal-append.queue_time_us":221966}
I20260709 12:23:11.985450  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:11.998858  2412 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:11.990839  4789 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:11.991390  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:11.903445  4788 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:11.996556  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.006500  1877 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:12.003693  1881 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:11.983963  4948 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:12.001504  4925 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.008s	user 0.009s	sys 0.005s
I20260709 12:23:11.977626  4963 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:12.037623  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.044451  1877 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:23:12.003082  4789 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:12.047385  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 188212us
W20260709 12:23:12.051424  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 97328.4us
W20260709 12:23:12.037663  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 658519us
I20260709 12:23:11.977779  4413 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.064293  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.064468  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:12.065032  1881 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:11.930701  4940 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.054697  4789 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:12.053074  4866 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:12.051285  4966 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:12.081636  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.004s	sys 0.012s
W20260709 12:23:12.082250  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.005s	sys 0.012s
I20260709 12:23:12.082545  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:12.082717  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 611270us
I20260709 12:23:12.063791  4956 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:12.091069  4866 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.112287  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:12.111552  2415 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:23:12.073145  4899 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:12.113178  2418 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:23:12.092641  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 398) took 2007 ms (client timeout 1514 ms). Trace:
I20260709 12:23:12.048202  4925 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.094377  4666 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:12.091144  4804 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:12.069564  4921 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.096717  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.099355  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2547) took 1214 ms. Trace:
I20260709 12:23:12.100735  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:12.108428  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.131995  1881 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
W20260709 12:23:12.133318  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.111682  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:12.091578  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.132654  2417 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:12.134575  1872 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:23:12.124907  2419 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:12.137442  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:12.124411  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 62966.5us
W20260709 12:23:12.124825  1802 rpcz_store.cc:269] 0709 12:23:10.085371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.096114 (+ 10743us) service_pool.cc:224] Handling call
0709 12:23:12.092317 (+1996203us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54001676d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:12.092488 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":621798784}
I20260709 12:23:12.128067  4940 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:12.057492  4963 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:12.128506  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 73335.7us
I20260709 12:23:12.079277  4943 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:12.067672  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 642344us
I20260709 12:23:12.187045  4780 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.139982  1879 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:12.131975  1726 rpcz_store.cc:276] 0709 12:23:10.885116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.885284 (+   168us) service_pool.cc:224] Handling call
0709 12:23:10.959474 (+ 74190us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.147506 (+188032us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.147534 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.150495 (+  2961us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.225970 (+ 75475us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:11.245816 (+ 19846us) log.cc:844] Serialized 14 byte log entry
0709 12:23:11.246122 (+   306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.246237 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.769523 (+523286us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b180041cf20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:12.095453 (+325930us) spinlock_profiling.cc:243] Waited 1.56 ms on lock 0x7b180041cf20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:12.099085 (+  3632us) raft_consensus.cc:1685] finished
0709 12:23:12.099111 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.099284 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32727,"prepare.queue_time_us":20212,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":7660,"raft.queue_time_us":179946,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":11027,"spinlock_wait_cycles":12925440,"thread_start_us":231761,"threads_started":2,"wal-append.queue_time_us":125770}
I20260709 12:23:12.131306  4975 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:12.189020  1877 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:12.123590  4956 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.204521  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.208628  4900 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:12.112581  4966 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.137748  4987 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.214071  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:12.152264  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.215718  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:12.220983  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.178519  4906 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.222749  4975 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.260644  4975 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:12.261667  4918 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.262555  4918 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.237054  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:12.266085  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:12.266963  4797 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:12.268172  4797 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:12.269378  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:12.238297  1772 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.213563  4991 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.060s
I20260709 12:23:12.270428  4991 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:12.270774  4991 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:12.271176  4991 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:12.273262  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.274441  4992 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:12.276422  1877 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:12.277091  1803 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:12.278405  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.280158  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.276485  4992 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:12.281656  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1661) took 1322 ms. Trace:
I20260709 12:23:12.282009  4992 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.283412  4992 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.237534  4993 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:12.284785  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.287355  2410 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:12.288393  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.243383  4780 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:12.245256  4963 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:12.220569  4943 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.303496  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:12.304100  2410 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:23:12.305045  4943 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:12.305230  4989 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:12.305292  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.308045  4884 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:12.308801  4884 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:12.248507  4956 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:12.252483  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 843369us
I20260709 12:23:12.311491  2414 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:12.312228  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:12.253175  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 876299us
I20260709 12:23:12.316020  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.317361  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.317029  4757 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:12.318218  4745 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:12.319777  4916 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.253937  4950 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:12.322134  4950 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.322705  4950 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.323029  4757 consensus_queue.cc:1243] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:23:12.256690  4966 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:12.239214  4944 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:12.275949  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:12.282862  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.240386  4807 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.213204  4921 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.240458  4949 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.293644  1872 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:23:12.296289  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.296990  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.343281  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:12.343969  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47747.5us
I20260709 12:23:12.302140  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.302937  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:12.318713  4756 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:23:12.353808  4756 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:12.354936  4756 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.336783  4986 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:12.319975  4745 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:12.357483  4745 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:12.269040  4987 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.322103  4916 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:12.358431  4745 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.253302  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.338197  4918 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:12.360299  1873 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:12.230679  4955 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.362033  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.318243  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.363161  4955 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:12.363510  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45312.9us
I20260709 12:23:12.363890  4955 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:12.364403  4955 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.364895  4925 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:12.364735  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.365912  4413 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:12.367046  4934 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.311s	user 0.004s	sys 0.001s
I20260709 12:23:12.348299  2416 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:12.340503  4898 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:12.369577  4898 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.369994  4898 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.310241  4884 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.373317  1807 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:12.373449  1875 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:12.282058  1732 rpcz_store.cc:276] 0709 12:23:10.959218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.326419 (+367201us) service_pool.cc:224] Handling call
0709 12:23:11.329156 (+  2737us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.329536 (+   380us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:11.377129 (+ 47593us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.252176 (+875047us) spinlock_profiling.cc:243] Waited 6.01 ms on lock 0x7b1800383320. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16afcac58f 00007f16a8ceb571 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd
0709 12:23:12.270321 (+ 18145us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.271122 (+   801us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.271363 (+   241us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:12.271414 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.271512 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.281320 (+  9808us) raft_consensus.cc:1685] finished
0709 12:23:12.281434 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.281585 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":644987,"prepare.queue_time_us":55009,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":41,"raft.queue_time_us":648316,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":11193,"spinlock_wait_cycles":13831424,"thread_start_us":213641,"threads_started":1,"wal-append.queue_time_us":509195}
I20260709 12:23:12.338634  1874 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:23:12.301719  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.349831  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.350757  4780 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:12.351219  4966 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:12.351686  1804 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:12.352093  4956 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:12.353066  4943 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:12.390836  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.391036  4964 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:12.392935  4964 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.393373  4964 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:12.394469  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 118581us
I20260709 12:23:12.395807  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.391662  4918 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 12:23:12.388000  4963 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:12.400213  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 712) took 2173 ms. Trace:
I20260709 12:23:12.400667  2411 rpcz_store.cc:276] 0709 12:23:10.204444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.231135 (+ 26691us) service_pool.cc:224] Handling call
0709 12:23:10.231606 (+   471us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:10.232179 (+   573us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:10.601608 (+369429us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.833535 (+231927us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:10.915072 (+ 81537us) log.cc:844] Serialized 78 byte log entry
0709 12:23:10.960219 (+ 45147us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:10.960537 (+   318us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.960926 (+   389us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.961118 (+   192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.961210 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.254240 (+1293030us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b18003cec20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:12.366435 (+112195us) spinlock_profiling.cc:243] Waited 8.61 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:12.377127 (+ 10692us) raft_consensus.cc:1685] finished
0709 12:23:12.377157 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.377339 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":479926,"prepare.run_cpu_time_us":168,"prepare.run_wall_time_us":4232,"raft.queue_time_us":1820,"raft.run_cpu_time_us":534,"raft.run_wall_time_us":43289,"spinlock_wait_cycles":47682304,"thread_start_us":489069,"threads_started":2,"wal-append.queue_time_us":359883}
I20260709 12:23:12.402287  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.403606  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.387712  4989 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:12.407441  4989 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.407836  4989 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:12.385759  4745 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:12.411286  4986 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:12.411892  4986 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:12.412849  4986 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.415072  1877 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:12.292423  4975 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.097s
I20260709 12:23:12.416530  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.337967  4797 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:12.417663  4797 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.418320  4797 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.390682  4780 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.422070  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:12.293426  4906 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.417127  1877 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:12.418972  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:12.429988  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.392234  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.430716  1876 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:12.383496  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:12.423019  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.433139  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.433917  1802 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:12.435467  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.437304  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:12.432255  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48937.3us
I20260709 12:23:12.435261  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:12.435307  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.440255  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.439241  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1753) took 1072 ms. Trace:
I20260709 12:23:12.441062  1876 rpcz_store.cc:276] 0709 12:23:11.358561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.401483 (+ 42922us) service_pool.cc:224] Handling call
0709 12:23:12.429840 (+1028357us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:23:12.431279 (+  1439us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2947840}
I20260709 12:23:12.364765  4955 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.363231  4949 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:12.459998  1873 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:12.442962  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:12.467057  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 140682us
I20260709 12:23:12.436787  1806 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:12.439740  4966 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:12.496054  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.494657  4966 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:12.497484  4966 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.498473  4906 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:12.470288  1732 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:12.470942  4989 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.499277  1732 raft_consensus.cc:606] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:23:12.499653  1732 raft_consensus.cc:614] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:12.500661  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:12.501754  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:12.502230  1732 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:12.426466  4780 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.473062  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:12.473886  4925 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:12.473129  1724 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:23:12.488426  2416 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:12.528173  1726 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 3.
I20260709 12:23:12.489601  4958 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:12.480196  4707 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:12.439651  4918 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:12.498687  4966 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.451045  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:12.471763  1731 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:12.504691  1874 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:23:12.329010  4948 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:12.530567  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 95393.1us
I20260709 12:23:12.429291  4975 raft_consensus.cc:2764] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260709 12:23:12.570757  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 133555us
I20260709 12:23:12.574448  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:12.570628  1875 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:12.567427  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.580394  4981 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:12.581049  4981 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.581553  4981 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.584049  4985 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:12.595044  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.529006  4757 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:12.542716  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.429097  4940 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.550468  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:12.531208  4933 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.595s	user 0.013s	sys 0.007s
I20260709 12:23:12.534694  4925 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:12.542543  4963 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:12.534318  4941 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:12.560587  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:12.623276  1798 raft_consensus.cc:2368] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7.
I20260709 12:23:12.561277  4707 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:12.562822  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.598274  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.600582  1872 raft_consensus.cc:2368] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5.
I20260709 12:23:12.547519  4943 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:12.632849  1876 raft_consensus.cc:2376] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4. Re-sending same reply.
W20260709 12:23:12.635911  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.636874  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.637702  1874 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:12.538121  1801 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
W20260709 12:23:12.612643  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2552) took 1715 ms (client timeout 2278 ms). Trace:
W20260709 12:23:12.639436  1730 rpcz_store.cc:269] 0709 12:23:10.896777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.188267 (+291490us) service_pool.cc:224] Handling call
0709 12:23:12.612341 (+1424074us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b5400414954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:12.612562 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1475693824}
I20260709 12:23:12.616653  1723 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:12.617794  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 131358us
I20260709 12:23:12.641669  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.621007  1877 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:23:12.614157  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2546) took 1741 ms. Trace:
I20260709 12:23:12.626448  1881 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
W20260709 12:23:12.631423  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.647408  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:12.632754  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70022.8us
I20260709 12:23:12.648862  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:12.649597  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2435) took 1267 ms. Trace:
I20260709 12:23:12.650405  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:12.634241  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:12.650630  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:12.651098  1728 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:12.651250  1724 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:12.651955  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:12.652539  1728 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:12.599766  4757 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:12.643201  1725 rpcz_store.cc:276] 0709 12:23:10.873217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.878502 (+  5285us) service_pool.cc:224] Handling call
0709 12:23:10.895696 (+ 17194us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.171176 (+275480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.171305 (+   129us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.182390 (+ 11085us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.261732 (+ 79342us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:11.771380 (+509648us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:12.323146 (+551766us) spinlock_profiling.cc:243] Waited 4.85 ms on lock 0x7b18000316c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf 00007f16abdf2038
0709 12:23:12.613463 (+290317us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.613796 (+   333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.613881 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.613928 (+    47us) raft_consensus.cc:1685] finished
0709 12:23:12.613940 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.614084 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31420,"prepare.queue_time_us":12359,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":73,"raft.queue_time_us":196067,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":8772,"spinlock_wait_cycles":17901824,"thread_start_us":561099,"threads_started":3,"wal-append.queue_time_us":77891}
I20260709 12:23:12.650038  1879 rpcz_store.cc:276] 0709 12:23:11.363784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.402885 (+ 39101us) service_pool.cc:224] Handling call
0709 12:23:12.631273 (+1228388us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b5400117454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:12.631431 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":818341120}
I20260709 12:23:12.620184  4925 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:12.548091  4898 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.659400  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:12.661629  2410 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
W20260709 12:23:12.662712  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.662633  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:12.628324  4975 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:12.663622  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.624471  4736 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.665036  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:12.666368  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:12.667152  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:12.616147  4940 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.669001  1873 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:12.669493  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:12.669665  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1744) took 1313 ms. Trace:
I20260709 12:23:12.669721  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:12.670143  1873 rpcz_store.cc:276] 0709 12:23:11.356042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.653026 (+1296984us) service_pool.cc:224] Handling call
0709 12:23:12.668813 (+ 15787us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.669443 (+   630us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.669613 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:12.670325  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:12.675750  2415 raft_consensus.cc:1217] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Deduplicated request from leader. Original: 2.3->[2.4-6.6]   Dedup: 2.4->[4.5-6.6]
I20260709 12:23:12.649976  4986 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:12.676473  2415 pending_rounds.cc:85] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 4
I20260709 12:23:12.681581  2415 pending_rounds.cc:107] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:23:12.681978  2415 raft_consensus.cc:2889] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:12.677875  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.652102  4949 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:12.659022  4757 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:12.694550  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:12.678241  4789 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.562s	user 0.011s	sys 0.001s
I20260709 12:23:12.684840  1872 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:12.601447  4993 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:23:12.703943  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1754) took 1345 ms. Trace:
I20260709 12:23:12.704847  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:12.687894  1802 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:12.704299  1872 rpcz_store.cc:276] 0709 12:23:11.376348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.656432 (+1280084us) service_pool.cc:224] Handling call
0709 12:23:12.674103 (+ 17671us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.703647 (+ 29544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.703884 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:12.713083  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:12.714753  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:12.697875  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:12.717887  1873 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:23:12.717926  1730 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:12.718858  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1757) took 1235 ms. Trace:
I20260709 12:23:12.719412  1873 rpcz_store.cc:276] 0709 12:23:11.483705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.690370 (+1206665us) service_pool.cc:224] Handling call
0709 12:23:12.717425 (+ 27055us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.718487 (+  1062us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.718740 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:12.721451  1873 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:23:12.722208  1728 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:23:12.688426  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.689208  1806 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:44522
I20260709 12:23:12.619535  4933 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.689718  1880 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:12.723490  1806 raft_consensus.cc:606] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Received request to transfer leadership
I20260709 12:23:12.723783  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2437) took 1316 ms. Trace:
I20260709 12:23:12.724005  1806 raft_consensus.cc:614] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:12.724262  1880 rpcz_store.cc:276] 0709 12:23:11.407645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.664548 (+1256903us) service_pool.cc:224] Handling call
0709 12:23:12.689555 (+ 25007us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.723543 (+ 33988us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.723730 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:12.724989  4933 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:12.635669  4943 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:12.693490  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:12.687135  4986 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.660315  4769 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:12.635332  4963 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:12.697106  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:12.657687  4987 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:12.722329  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1764) took 1046 ms. Trace:
I20260709 12:23:12.688342  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:12.750317  2412 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
I20260709 12:23:12.737764  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:12.739532  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70122.1us
I20260709 12:23:12.683701  4763 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:12.680264  4736 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.760514  4763 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.691275  4949 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.763029  4949 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:12.673352  4788 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:23:12.765233  4898 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:12.765340  4707 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.716387  4789 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.767875  4789 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:12.768520  1803 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 7. Ops: []
I20260709 12:23:12.725801  4943 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:12.775548  4963 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:12.776192  5000 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:23:12.780277  2383 leader_election.cc:400] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5.
W20260709 12:23:12.781354  4996 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 12:23:12.783471  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:12.752175  4934 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:23:12.788262  4940 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:12.752244  1726 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:12.683099  4756 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:23:12.757264  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2439) took 1129 ms. Trace:
I20260709 12:23:12.795162  1801 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:12.774323  1732 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:12.795300  1872 rpcz_store.cc:276] 0709 12:23:11.628175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.732149 (+1103974us) service_pool.cc:224] Handling call
0709 12:23:12.756456 (+ 24307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.756816 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.756855 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:12.756918 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:12.756967 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.757044 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.757186 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:12.781250  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:12.753808  1873 rpcz_store.cc:276] 0709 12:23:11.675598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.720619 (+1045021us) service_pool.cc:224] Handling call
0709 12:23:12.721242 (+   623us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.721990 (+   748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.722233 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:12.806885  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 118713us
I20260709 12:23:12.743429  4986 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:12.762598  4763 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:12.809199  1724 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:12.819458  1873 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:12.826499  1873 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:12.823205  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1667) took 1444 ms. Trace:
I20260709 12:23:12.827406  1723 rpcz_store.cc:276] 0709 12:23:11.378673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.792179 (+413506us) service_pool.cc:224] Handling call
0709 12:23:12.616363 (+824184us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.647431 (+ 31068us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.822614 (+175183us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b5400414950. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:12.823095 (+   481us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1894458624}
I20260709 12:23:12.824575  4788 consensus_queue.cc:1243] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 12:23:12.835681  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:12.837440  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.839905  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 418) took 1465 ms. Trace:
I20260709 12:23:12.846652  1729 rpcz_store.cc:276] 0709 12:23:11.377688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.468985 (+ 91297us) service_pool.cc:224] Handling call
0709 12:23:11.592053 (+123068us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.935409 (+343356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.839646 (+904237us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b5400327dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:12.839838 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":772335360}
I20260709 12:23:12.840828  1724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:12.852021  1724 raft_consensus.cc:606] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:12.852572  1724 raft_consensus.cc:614] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:12.841861  1723 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:12.831982  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:12.874223  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.891844  1802 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:12.917093  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.918406  1726 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:12.919703  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:12.922050  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.903892  1880 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:12.926028  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:12.927104  1732 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 5
W20260709 12:23:12.931792  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 150603us
I20260709 12:23:12.933488  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:12.844357  4991 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.573s	user 0.001s	sys 0.010s
I20260709 12:23:12.916571  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.949752  1873 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:12.949858  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:12.827162  4797 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.951745  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:12.952517  1873 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:12.953236  4797 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:12.946384  1798 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:23:12.947644  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:12.935793  4769 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:12.959457  4675 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:12.961257  1875 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:12.960532  4675 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:12.966079  4675 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:12.959447  4979 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:12.973337  1806 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:12.973592  1799 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:12.974460  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.974802  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.975123  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.975492  1802 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:12.975662  1799 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:12.976576  1802 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:12.967278  4675 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:12.978541  4675 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:12.979820  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:12.980538  4991 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.976599  1800 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:12.974524  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:12.990969  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:12.947458  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.514s (SENT) [suppressed 4 similar messages]
I20260709 12:23:12.991832  1726 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:23:12.994540  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47126.2us
W20260709 12:23:12.995277  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 163361us
W20260709 12:23:12.996137  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.844034  4789 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:12.920015  4788 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:12.844214  4940 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:13.015736  4789 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.016117  4940 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.018397  4413 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:13.019147  4413 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:12.844224  4770 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.984726  4763 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:23:12.981886  4866 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:13.015177  4788 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:12.945847  4906 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:13.004338  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:13.075717  1729 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:12.839247  4804 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:12.996006  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:13.055490  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.059295  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:12.993029  4979 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:13.020191  4413 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.097469  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:13.110556  1799 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:13.101459  1806 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:13.091796  4763 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:12.982312  4970 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.092s
I20260709 12:23:13.050374  4952 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:13.057456  4955 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.527s	user 0.003s	sys 0.007s
I20260709 12:23:13.047890  4797 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:13.058287  4948 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:13.059702  4805 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.074478  4906 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.136791  4906 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:13.140200  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 136426us
I20260709 12:23:13.094707  4804 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.147591  1732 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:23:13.148254  1873 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:13.056783  4770 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.159466  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:13.160300  2417 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:23:13.160516  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:13.164261  1806 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:13.137727  4956 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.168030  1799 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:13.151520  4804 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:13.177760  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:13.175654  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:13.179838  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:13.186205  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:13.134598  4955 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.156904  4756 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:13.148212  4805 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.134965  4952 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.189548  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:13.187405  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 239959us
I20260709 12:23:13.156782  4993 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:23:13.202973  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:13.193563  4997 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.445s	user 0.001s	sys 0.001s
I20260709 12:23:13.204468  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:13.206254  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:13.211468  1800 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:13.211481  1732 raft_consensus.cc:2368] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8.
W20260709 12:23:13.203487  5004 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.376s	user 0.001s	sys 0.001s
I20260709 12:23:13.215201  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:13.216714  1728 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:13.216979  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:13.217954  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:13.180665  4956 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.201184  4958 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.203212  4771 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.240149  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:13.244053  1799 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:23:13.246054  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.161566  4763 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.199415  4866 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:13.229689  4955 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:13.231276  4999 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.415s	user 0.001s	sys 0.001s
W20260709 12:23:13.236068  5003 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.303s	user 0.000s	sys 0.001s
I20260709 12:23:13.253800  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:13.218911  1728 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:13.255527  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:13.281947  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.282198  1726 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:13.256006  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:13.238518  4770 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:13.255041  4925 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.572s	user 0.007s	sys 0.004s
I20260709 12:23:13.253396  4771 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.239118  4756 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:13.231897  4757 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.515s	user 0.006s	sys 0.005s
I20260709 12:23:13.200035  4769 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:13.198522  4918 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:13.270953  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.208899  4789 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:13.213680  4940 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:13.259871  5009 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
I20260709 12:23:13.211015  4991 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:13.313683  4943 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.543s	user 0.006s	sys 0.004s
I20260709 12:23:13.286238  1728 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.238701  4805 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:13.314168  4943 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.198467  5016 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20260709 12:23:13.263563  4898 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:13.311498  1770 leader_election.cc:400] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8.
I20260709 12:23:13.331351  1770 leader_election.cc:403] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:23:13.316248  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1669) took 1918 ms. Trace:
I20260709 12:23:13.293015  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:13.315503  1678 auto_rebalancer.cc:893] Removing replace marker failed: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:13.253621  4970 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:23:13.325253  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:13.331959  1727 rpcz_store.cc:276] 0709 12:23:11.397675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.826260 (+428585us) service_pool.cc:224] Handling call
0709 12:23:11.826739 (+   479us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:12.265985 (+439246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.266036 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:12.274815 (+  8779us) log.cc:844] Serialized 52 byte log entry
0709 12:23:12.505319 (+230504us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:12.505371 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.505457 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.316064 (+810607us) raft_consensus.cc:1685] finished
0709 12:23:13.316081 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.316199 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":375303,"raft.queue_time_us":376467,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":135,"thread_start_us":229973,"threads_started":1,"wal-append.queue_time_us":400644}
I20260709 12:23:13.333144  1799 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
W20260709 12:23:13.338761  1678 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
I20260709 12:23:13.339185  1873 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:13.311359  4807 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.341406  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:13.339237  1678 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:23:13.351663  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:13.353148  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.360682  5023 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:23:13.361465  5023 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.361932  5023 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:23:13.345340  4745 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.934s	user 0.023s	sys 0.009s
I20260709 12:23:13.363154  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:13.363698  4956 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:13.363929  4745 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:13.364110  1732 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:23:13.366961  4926 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 12:23:13.367256  4866 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.368144  4926 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:13.368598  4926 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:13.369630  4926 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:13.370921  5007 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.280s	user 0.001s	sys 0.000s
I20260709 12:23:13.371114  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:13.371853  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:13.373072  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:13.373521  1726 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:13.375872  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.289505  4925 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.254009  4958 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.384241  1799 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
W20260709 12:23:13.377249  4780 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.806s	user 0.025s	sys 0.000s
I20260709 12:23:13.335996  4805 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:13.385931  1730 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.652s	user 0.015s	sys 0.007s
I20260709 12:23:13.399230  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 723) took 1050 ms. Trace:
I20260709 12:23:13.404654  1730 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:13.316666  4955 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 12:23:13.404749  2414 rpcz_store.cc:276] 0709 12:23:12.349059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.349319 (+   260us) service_pool.cc:224] Handling call
0709 12:23:12.361564 (+ 12245us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.419878 (+ 58314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.419912 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.427937 (+  8025us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.658463 (+230526us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:12.658648 (+   185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.658724 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.398854 (+740130us) raft_consensus.cc:1685] finished
0709 12:23:13.398890 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.399132 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":35293,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":8013,"thread_start_us":168090,"threads_started":1,"wal-append.queue_time_us":263185}
I20260709 12:23:13.413847  1879 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:13.208690  4979 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.8, Last appended by leader: 8, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.410643  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:13.309777  4757 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.403174  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:13.423102  1799 raft_consensus.cc:2376] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4. Re-sending same reply.
I20260709 12:23:13.404966  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:13.426034  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:13.391180  4639 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:13.419201  1730 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:23:13.420133  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:13.420915  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:13.332875  4918 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.427819  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:13.434114  2418 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:13.437407  2413 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
W20260709 12:23:13.438482  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 182594us
I20260709 12:23:13.428802  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:13.440405  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:13.424139  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:13.403707  4926 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:13.427016  2419 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:13.442852  2414 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:13.443485  2414 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:13.431824  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:13.432096  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:13.387830  4921 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.021s	user 0.016s	sys 0.000s
I20260709 12:23:13.375746  4807 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:13.434701  4950 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.111s	user 0.008s	sys 0.007s
I20260709 12:23:13.456439  1727 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:13.445068  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:13.450909  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:13.448519  1725 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:13.448374  1772 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:13.466179  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.435897  4948 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:13.437094  4639 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.451669  1799 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:23:13.449618  4745 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:13.517817  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:13.451617  4952 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:13.524523  1732 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:13.455654  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:13.494745  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49746.7us
I20260709 12:23:13.496523  4918 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:13.497009  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46222.2us
W20260709 12:23:13.529568  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 73989.8us
I20260709 12:23:13.498898  4933 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:13.499428  4985 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:13.416193  4780 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.466637  4899 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:13.504276  4771 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:13.452885  1878 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:13.510722  4639 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:13.490480  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.527096  1799 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:13.520686  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:13.527944  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:13.571950  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44154.3us
I20260709 12:23:13.515658  4948 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:13.531199  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:13.534832  1727 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:13.534971  5028 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:13.535521  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:13.534075  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:13.535910  5014 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:13.477041  4950 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.578681  4989 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:13.579684  4950 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:13.580137  5029 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:23:13.581511  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:13.538959  4757 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:13.540851  4807 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:13.586230  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:13.543303  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 393) took 1442 ms (client timeout 1514 ms). Trace:
W20260709 12:23:13.586949  1881 rpcz_store.cc:269] 0709 12:23:12.100476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.133042 (+ 32566us) service_pool.cc:224] Handling call
0709 12:23:13.543121 (+1410079us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b5400073854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:13.543238 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":607101696}
I20260709 12:23:13.588424  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:13.589196  1881 raft_consensus.cc:2376] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4. Re-sending same reply.
I20260709 12:23:13.544775  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:13.545483  4966 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.981s	user 0.010s	sys 0.014s
I20260709 12:23:13.532301  4933 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:13.546013  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:13.458498  2414 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:13.576897  4992 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.244s	user 0.009s	sys 0.006s
I20260709 12:23:13.573151  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:13.497866  4981 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.916s	user 0.005s	sys 0.009s
I20260709 12:23:13.492444  4769 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.574949  4918 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:13.536674  5025 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:13.583978  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63361.8us
I20260709 12:23:13.513685  4993 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:23:13.586258  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:13.578536  5031 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:23:13.547485  4970 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:13.596026  1804 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:13.598680  2414 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:13.560036  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:13.604935  1800 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:13.529963  4763 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.056s
I20260709 12:23:13.559481  4943 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:13.576138  5007 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:23:13.623405  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:13.623622  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50618.2us
I20260709 12:23:13.600313  4921 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:23:13.576227  4745 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.624207  4939 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:13.583096  4952 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:23:13.622478  4985 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.585774  4898 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:13.630911  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:13.547711  4975 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.873s	user 0.011s	sys 0.001s
I20260709 12:23:13.624647  4899 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:23:13.632011  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 100876us
I20260709 12:23:13.648002  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1579) took 1219 ms. Trace:
I20260709 12:23:13.649055  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:13.662103  1881 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:13.662222  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:13.653252  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:13.664397  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:13.627775  4958 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:13.668550  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:13.650684  4966 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:13.640025  4970 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:13.635492  5036 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:13.661209  1807 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:13.654314  4921 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:13.661849  1805 rpcz_store.cc:276] 0709 12:23:12.428321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.428461 (+   140us) service_pool.cc:224] Handling call
0709 12:23:12.428840 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.429220 (+   380us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:12.620645 (+191425us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.697378 (+ 76733us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.704328 (+  6950us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.704486 (+   158us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:12.704738 (+   252us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.737085 (+ 32347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.737213 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.647389 (+910176us) raft_consensus.cc:1685] finished
0709 12:23:13.647413 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.647924 (+   511us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":111541,"prepare.run_cpu_time_us":126,"prepare.run_wall_time_us":7982,"raft.queue_time_us":50393,"raft.run_cpu_time_us":406,"raft.run_wall_time_us":27922,"thread_start_us":216162,"threads_started":3,"wal-append.queue_time_us":147676}
I20260709 12:23:13.641211  5032 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:13.671006  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:13.659348  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 125347us
I20260709 12:23:13.688910  5024 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:23:13.690651  5035 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:23:13.666309  4975 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:13.659854  4986 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.766s	user 0.011s	sys 0.002s
I20260709 12:23:13.674161  4943 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:13.675087  4941 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:23:13.666831  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.657109  4925 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:13.629671  4981 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.665072  4955 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:13.677520  4899 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:13.716653  1730 raft_consensus.cc:606] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:13.717293  1730 raft_consensus.cc:614] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:13.650988  4745 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.672293  4918 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:13.697274  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:13.671340  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:13.706027  1881 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:13.677757  5016 raft_consensus.cc:2764] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:23:13.738142  5032 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:13.738633  5032 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:13.738916  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:13.740744  4804 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.550s	user 0.011s	sys 0.000s
I20260709 12:23:13.741104  4940 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:13.741209  4804 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.742799  1731 raft_consensus.cc:2368] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7.
I20260709 12:23:13.640154  5025 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.711457  2416 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:13.682577  4413 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:13.743602  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:13.736852  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 65559.3us
I20260709 12:23:13.737360  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:13.754081  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:13.754287  1881 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:13.739065  4952 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:13.697572  4763 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:13.678663  4966 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:13.680745  4970 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:13.659920  4985 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.680022  4989 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.042s
I20260709 12:23:13.660643  4948 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.710423  4943 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.711707  4941 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:13.780067  2414 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:23:13.714895  4955 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:23:13.797827  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.804709  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:13.734921  4918 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:13.739068  5032 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:13.818121  2414 pending_rounds.cc:85] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 6
I20260709 12:23:13.820840  2414 pending_rounds.cc:107] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:23:13.818806  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:13.805877  1876 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:13.821883  2414 raft_consensus.cc:2889] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:13.822954  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:13.824210  1881 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:13.713728  4986 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.782702  4958 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:13.729202  4975 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20260709 12:23:13.821246  4955 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:13.844097  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:13.821743  4943 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:13.825562  4918 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:13.827158  4950 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:23:13.840315  4949 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.077s	user 0.008s	sys 0.006s
I20260709 12:23:13.785490  5036 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:23:13.816201  4757 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 12:23:13.851562  1806 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:13.858806  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.783649  5023 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.864159  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:13.865068  1877 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:13.860994  4940 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:13.870085  4940 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.861397  4949 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.874017  4949 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:13.861225  4950 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:13.876317  4950 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.845925  4805 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8.
I20260709 12:23:13.881567  1876 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:13.890115  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1685) took 1047 ms. Trace:
I20260709 12:23:13.860105  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:13.900424  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.918210  1881 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:23:13.914752  1729 rpcz_store.cc:276] 0709 12:23:12.842191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.113414 (+271223us) service_pool.cc:224] Handling call
0709 12:23:13.132968 (+ 19554us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:13.146571 (+ 13603us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:13.202239 (+ 55668us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.317751 (+115512us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.566615 (+248864us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:13.567545 (+   930us) log.cc:844] Serialized 26 byte log entry
0709 12:23:13.567750 (+   205us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:13.568083 (+   333us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.568287 (+   204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.568369 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.889770 (+321401us) raft_consensus.cc:1685] finished
0709 12:23:13.889806 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.890014 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":23882,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":128,"raft.queue_time_us":96987,"raft.run_cpu_time_us":423,"raft.run_wall_time_us":7700,"thread_start_us":339677,"threads_started":3,"wal-append.queue_time_us":146851}
I20260709 12:23:13.770087  4925 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
I20260709 12:23:13.818202  4921 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:13.863443  4757 consensus_queue.cc:1243] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:23:13.856061  4958 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:13.832963  5040 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.068s
W20260709 12:23:13.939898  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 227554us
W20260709 12:23:13.942806  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 202085us
W20260709 12:23:13.946619  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.947469  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:13.948503  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 204024us
I20260709 12:23:13.951244  1845 leader_election.cc:400] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7.
I20260709 12:23:13.952603  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:13.890834  5006 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.995093  4958 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:13.996407  4958 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:13.957484  4964 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.530s	user 0.013s	sys 0.017s
I20260709 12:23:13.999457  4964 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.001748  4964 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:14.003278  1877 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:14.004897  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 52426.6us
I20260709 12:23:14.006606  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:14.010661  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:14.010571  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:14.017056  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1766) took 1180 ms. Trace:
W20260709 12:23:14.018071  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.017445  1877 rpcz_store.cc:276] 0709 12:23:12.836507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.004152 (+1167645us) service_pool.cc:224] Handling call
0709 12:23:14.016977 (+ 12825us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.036804  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:13.853113  4966 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:13.954528  4805 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:14.013125  4981 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:13.856808  4899 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.010589  4780 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:14.031376  4952 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:14.034817  5039 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:14.077215  5039 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:14.077456  4926 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.626s	user 0.010s	sys 0.000s
I20260709 12:23:14.078390  5039 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:14.044104  4805 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.058475  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:14.062542  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:14.052487  5053 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 12:23:14.052402  4940 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
W20260709 12:23:14.073334  5041 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.304s	user 0.001s	sys 0.000s
I20260709 12:23:14.056332  4804 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:14.104573  2416 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
W20260709 12:23:14.104537  4639 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.518s	user 0.010s	sys 0.000s
I20260709 12:23:14.111717  4780 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:14.079005  4952 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.080277  4866 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:14.122913  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:14.099529  1877 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:14.094972  4966 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:14.119650  1728 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:14.143683  4639 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.137818  4952 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:14.107623  4805 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:23:14.151161  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.012291  5006 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.175506  1877 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:14.176838  5050 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:14.149782  4780 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.188427  4639 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:14.189863  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1573) took 1805 ms. Trace:
I20260709 12:23:14.191109  1802 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:14.191169  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 395) took 2058 ms (client timeout 1762 ms). Trace:
I20260709 12:23:14.098368  4981 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:23:14.193084  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2461) took 1347 ms. Trace:
I20260709 12:23:14.193377  4940 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:14.193166  4906 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.053s	user 0.011s	sys 0.005s
I20260709 12:23:14.195315  4736 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:14.199126  4985 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:14.203817  5006 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:14.206840  5044 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:23:14.208081  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 724) took 1751 ms. Trace:
I20260709 12:23:14.208199  4788 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:14.211802  5054 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:23:14.213512  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1583) took 1565 ms (client timeout 1659 ms). Trace:
I20260709 12:23:14.218636  4933 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:14.327113  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.219228  4943 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.220580  5023 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:14.329932  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.004s	sys 0.020s
W20260709 12:23:14.334388  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.004s	sys 0.020s
W20260709 12:23:14.335016  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 516277us
I20260709 12:23:14.335764  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:14.220463  4952 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:14.222142  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1684) took 1380 ms. Trace:
I20260709 12:23:14.225500  4986 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:14.227067  4866 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:14.228721  5024 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:14.231629  4941 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:14.234171  1881 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:14.211805  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1601) took 1025 ms. Trace:
I20260709 12:23:14.240360  1728 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
W20260709 12:23:14.299079  1878 rpcz_store.cc:269] 0709 12:23:12.133238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.204241 (+ 71003us) service_pool.cc:224] Handling call
0709 12:23:14.190985 (+1986744us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400051754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.191096 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2835701120}
I20260709 12:23:14.299041  1876 rpcz_store.cc:276] 0709 12:23:12.845733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.544462 (+698729us) service_pool.cc:224] Handling call
0709 12:23:14.192862 (+648400us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54001f8454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.193008 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":495495040}
I20260709 12:23:14.299189  1803 rpcz_store.cc:276] 0709 12:23:12.384810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.385090 (+   280us) service_pool.cc:224] Handling call
0709 12:23:12.385404 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.768934 (+383530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.189565 (+1420631us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b540013a954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:14.189791 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880580352}
I20260709 12:23:14.318856  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2436) took 2954 ms. Trace:
I20260709 12:23:14.349479  1806 rpcz_store.cc:276] 0709 12:23:13.186751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.189263 (+  2512us) service_pool.cc:224] Handling call
0709 12:23:14.211554 (+1022291us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b540026fa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.211723 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1490908544}
I20260709 12:23:14.310693  5046 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:14.354712  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 231953us
W20260709 12:23:14.322181  1801 rpcz_store.cc:269] 0709 12:23:12.648405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.648569 (+   164us) service_pool.cc:224] Handling call
0709 12:23:14.213341 (+1564772us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b5400132754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.213449 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":309118848}
I20260709 12:23:14.243096  4955 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.329332  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:14.359187  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:14.359704  1877 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:14.243472  5042 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.342362  1725 rpcz_store.cc:276] 0709 12:23:12.842029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.034444 (+192415us) service_pool.cc:224] Handling call
0709 12:23:13.059422 (+ 24978us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.479740 (+420318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.221856 (+742116us) spinlock_profiling.cc:243] Waited 389 ms on lock 0x7b54002b7fd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:14.222072 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":894335232}
I20260709 12:23:14.361182  5042 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.362608  1877 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: []
W20260709 12:23:14.346909  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.351683  1879 rpcz_store.cc:276] 0709 12:23:11.364183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.657576 (+1293393us) service_pool.cc:224] Handling call
0709 12:23:12.679123 (+ 21547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.414343 (+735220us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.318494 (+904151us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b54002df5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:14.318759 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1689164672}
I20260709 12:23:14.365607  1879 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:14.365665  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:14.366341  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 401) took 2138 ms. Trace:
I20260709 12:23:14.366703  1879 rpcz_store.cc:276] 0709 12:23:12.228038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.365140 (+2137102us) service_pool.cc:224] Handling call
0709 12:23:14.365445 (+   305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.366106 (+   661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.366288 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.367380  4986 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:14.367982  4986 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:14.369014  4986 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:14.369520  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2441) took 2692 ms. Trace:
I20260709 12:23:14.369936  1880 rpcz_store.cc:276] 0709 12:23:11.677436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.725145 (+1047709us) service_pool.cc:224] Handling call
0709 12:23:12.725487 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.923809 (+198322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.369278 (+1445469us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b5400026054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:14.369460 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":409694592}
I20260709 12:23:14.346153  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:14.371836  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2449) took 2072 ms. Trace:
I20260709 12:23:14.374696  1880 rpcz_store.cc:276] 0709 12:23:12.299844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.370964 (+2071120us) service_pool.cc:224] Handling call
0709 12:23:14.371318 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.371508 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.371539 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.371581 (+    42us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:14.371613 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.371666 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.371782 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.350642  1728 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 5
I20260709 12:23:14.314043  2411 rpcz_store.cc:276] 0709 12:23:12.456934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.457167 (+   233us) service_pool.cc:224] Handling call
0709 12:23:12.470755 (+ 13588us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.505991 (+ 35236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.506043 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.592407 (+ 86364us) log.cc:844] Serialized 26 byte log entry
0709 12:23:13.546860 (+954453us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b18000b00e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:23:13.654677 (+107817us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:13.654719 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.654833 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.207720 (+552887us) raft_consensus.cc:1685] finished
0709 12:23:14.207749 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.208006 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63328,"raft.queue_time_us":56245,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":29812,"spinlock_wait_cycles":38897408,"thread_start_us":107624,"threads_started":1,"wal-append.queue_time_us":1092628}
W20260709 12:23:14.362466  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1767) took 1525 ms (client timeout 1868 ms). Trace:
I20260709 12:23:14.376916  1728 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:23:14.377321  1728 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:14.377097  1875 rpcz_store.cc:269] 0709 12:23:12.836752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.979519 (+142767us) service_pool.cc:224] Handling call
0709 12:23:14.362251 (+1382732us) spinlock_profiling.cc:243] Waited 528 ms on lock 0x7b5400025b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.362408 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1214149376}
I20260709 12:23:14.378705  1875 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:14.379369  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2451) took 2079 ms. Trace:
W20260709 12:23:14.379519  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1586) took 1624 ms (client timeout 1983 ms). Trace:
I20260709 12:23:14.379719  1875 rpcz_store.cc:276] 0709 12:23:12.300317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.378195 (+2077878us) service_pool.cc:224] Handling call
0709 12:23:14.378516 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.379129 (+   613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.379314 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:14.379951  1807 rpcz_store.cc:269] 0709 12:23:12.755202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.782948 (+ 27746us) service_pool.cc:224] Handling call
0709 12:23:14.379363 (+1596415us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.379462 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1939486976}
I20260709 12:23:14.381146  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:14.381309  1875 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:14.381775  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2574) took 1166 ms. Trace:
I20260709 12:23:14.357734  1801 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:14.382381  1728 rpcz_store.cc:276] 0709 12:23:13.214949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.216289 (+  1340us) service_pool.cc:224] Handling call
0709 12:23:13.216603 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.119192 (+902589us) spinlock_profiling.cc:243] Waited 803 ms on lock 0x7b58008234d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd340aa 00007f16abdb6447 00007f16abdcefe7 00007f16abdd07de 00007f16abddab49 00007f16abddb416 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4
0709 12:23:14.381496 (+262304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.381718 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1846333696}
I20260709 12:23:14.356277  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:14.373574  4757 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:23:14.383615  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:14.385784  4745 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:14.386153  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:14.387316  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:14.387962  1801 raft_consensus.cc:2368] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6.
I20260709 12:23:14.388756  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:14.389396  1801 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:14.373512  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:14.393123  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:14.394227  4757 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:14.394814  4757 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:14.395802  4757 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:14.396955  2416 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 793) took 1719 ms (client timeout 2149 ms). Trace:
W20260709 12:23:14.397415  2416 rpcz_store.cc:269] 0709 12:23:12.677670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.690128 (+ 12458us) service_pool.cc:224] Handling call
0709 12:23:14.396723 (+1706595us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b54008c0354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.396855 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2290043648}
I20260709 12:23:14.397985  5055 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:23:14.401722  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1681) took 1560 ms (client timeout 1750 ms). Trace:
I20260709 12:23:14.402868  5055 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:14.404585  4804 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:14.404974  4804 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:14.407596  5055 raft_consensus.cc:2955] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 6 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:14.405934  4804 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:14.410320  4804 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:14.412098  4899 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:14.412740  4899 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:14.402230  1731 rpcz_store.cc:269] 0709 12:23:12.841254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.841401 (+   147us) service_pool.cc:224] Handling call
0709 12:23:14.401460 (+1560059us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b54003cc3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.401635 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1821014272}
I20260709 12:23:14.418191  4988 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:14.419325  4884 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:14.420284  4979 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:14.420737  4979 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:14.423389  5050 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:14.423854  5050 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:14.425863  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 400) took 2198 ms. Trace:
W20260709 12:23:14.428843  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1768) took 1591 ms (client timeout 1924 ms). Trace:
W20260709 12:23:14.429291  1873 rpcz_store.cc:269] 0709 12:23:12.837006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.177459 (+340453us) service_pool.cc:224] Handling call
0709 12:23:14.428653 (+1251194us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54000f0854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.428777 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":307310976}
W20260709 12:23:14.436892  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1584) took 1762 ms (client timeout 1924 ms). Trace:
W20260709 12:23:14.437402  1804 rpcz_store.cc:269] 0709 12:23:12.674383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.674575 (+   192us) service_pool.cc:224] Handling call
0709 12:23:14.436652 (+1762077us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b540009fbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.436816 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2070929280}
W20260709 12:23:14.446431  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43834.2us
I20260709 12:23:14.449601  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:14.381901  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2454) took 2081 ms. Trace:
I20260709 12:23:14.450598  1875 rpcz_store.cc:276] 0709 12:23:12.300973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.380646 (+2079673us) service_pool.cc:224] Handling call
0709 12:23:14.380969 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.381658 (+   689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.381850 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.452986  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2456) took 1994 ms. Trace:
I20260709 12:23:14.453434  1875 rpcz_store.cc:276] 0709 12:23:12.458847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.451610 (+1992763us) service_pool.cc:224] Handling call
0709 12:23:14.452036 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.452276 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.452313 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.452371 (+    58us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:14.452420 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.452551 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.452912 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.456333  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:14.457244  2415 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:14.359851  1806 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:14.459807  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:14.454399  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:14.461405  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:14.426298  1877 rpcz_store.cc:276] 0709 12:23:12.227806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.360448 (+2132642us) service_pool.cc:224] Handling call
0709 12:23:14.362426 (+  1978us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.363304 (+   878us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.425626 (+ 62322us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b5400025b50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:14.425795 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3655040}
I20260709 12:23:14.464807  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2455) took 2163 ms. Trace:
I20260709 12:23:14.465243  1873 rpcz_store.cc:276] 0709 12:23:12.301194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.430121 (+2128927us) service_pool.cc:224] Handling call
0709 12:23:14.430450 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.430636 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.430662 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.430706 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:14.430737 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.464621 (+ 33884us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.464751 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.464418  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 413) took 1066 ms. Trace:
I20260709 12:23:14.469152  1805 rpcz_store.cc:276] 0709 12:23:13.398343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.684118 (+285775us) service_pool.cc:224] Handling call
0709 12:23:13.688159 (+  4041us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:13.688553 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:13.688872 (+   319us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.804464 (+115592us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:14.433265 (+628801us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b2c0005c178. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:23:14.433804 (+   539us) log.cc:844] Serialized 26 byte log entry
0709 12:23:14.433975 (+   171us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:14.434011 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.434092 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.464109 (+ 30017us) raft_consensus.cc:1685] finished
0709 12:23:14.464215 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.464353 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":995,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":6490,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":11340,"spinlock_wait_cycles":33903360,"thread_start_us":113310,"threads_started":1,"wal-append.queue_time_us":151649}
I20260709 12:23:14.468472  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:14.477079  1723 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:14.481154  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:14.481873  2414 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:14.486944  1873 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:14.487702  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1769) took 1650 ms. Trace:
I20260709 12:23:14.488011  1873 rpcz_store.cc:276] 0709 12:23:12.837290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.486403 (+1649113us) service_pool.cc:224] Handling call
0709 12:23:14.486735 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.487430 (+   695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.487639 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.491101  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:14.491379  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:14.492024  1723 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:14.491954  1730 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:14.493347  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:14.494125  1726 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:14.494004  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:14.494838  1879 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:14.495047  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:14.495762  1724 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:23:14.495862  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 406) took 2266 ms. Trace:
I20260709 12:23:14.496387  1879 rpcz_store.cc:276] 0709 12:23:12.229654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.367435 (+2137781us) service_pool.cc:224] Handling call
0709 12:23:14.494592 (+127157us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.495572 (+   980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.495796 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.498183  5050 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.499606  5050 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:14.499804  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:14.505497  1875 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:14.506175  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2457) took 2047 ms. Trace:
I20260709 12:23:14.348373  5024 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:14.506266  1724 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
W20260709 12:23:14.508054  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 151829us
I20260709 12:23:14.508867  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:14.314446  4906 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.328359  4918 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:14.517367  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.532776  1877 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:14.533445  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.533720  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2458) took 2074 ms. Trace:
I20260709 12:23:14.517261  1803 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:14.536185  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1606) took 1004 ms. Trace:
I20260709 12:23:14.536599  1803 rpcz_store.cc:276] 0709 12:23:13.531986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.358810 (+826824us) service_pool.cc:224] Handling call
0709 12:23:14.479150 (+120340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.535884 (+ 56734us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.536120 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.355966  4933 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:14.518644  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 57355.9us
I20260709 12:23:14.545467  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:14.506536  1875 rpcz_store.cc:276] 0709 12:23:12.459232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.454554 (+1995322us) service_pool.cc:224] Handling call
0709 12:23:14.505316 (+ 50762us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.505955 (+   639us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.506122 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.510924  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:14.546671  2416 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:14.523885  2414 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:14.464315  1807 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:14.551683  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:14.531177  1801 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:14.531581  4987 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:23:14.538242  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.534170  1877 rpcz_store.cc:276] 0709 12:23:12.459578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.463933 (+2004355us) service_pool.cc:224] Handling call
0709 12:23:14.532568 (+ 68635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.533402 (+   834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.533653 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:14.561043  1727 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:14.542390  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:14.564054  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:14.508633  5008 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 12:23:14.574146  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:14.576536  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:14.577893  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:14.575006  1728 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 12:23:14.574048  1845 connection.cc:256] RPC call timeout handler was delayed by -1.0702706299998681s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16590000000000002s for a response
I20260709 12:23:14.581290  5028 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.578589  1730 raft_consensus.cc:606] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:23:14.582161  1730 raft_consensus.cc:614] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:14.581939  5028 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:14.584993  5028 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:14.587006  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:14.587949  1730 raft_consensus.cc:606] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:23:14.588428  1730 raft_consensus.cc:614] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:14.552711  1875 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:23:14.553596  1723 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:14.555361  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:14.508278  4916 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:14.485827  4950 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:14.592007  4950 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:14.594249  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.565569  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:14.469920  4988 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.594604  4950 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:14.615499  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 399) took 2387 ms. Trace:
I20260709 12:23:14.532974  4984 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:14.537786  5039 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:14.540612  5020 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:14.622931  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 88726.4us
W20260709 12:23:14.642056  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 90162.8us
W20260709 12:23:14.646716  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 91461.9us
I20260709 12:23:14.647979  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:14.562649  4763 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:14.627491  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:14.459532  5046 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.559826  4745 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:23:14.618408  1876 rpcz_store.cc:276] 0709 12:23:12.227545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.358629 (+2131084us) service_pool.cc:224] Handling call
0709 12:23:14.531332 (+172703us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.531576 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:14.601051 (+ 69475us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.613426 (+ 12375us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.613714 (+   288us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:14.613962 (+   248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.614338 (+   376us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.615133 (+   795us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":615,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":79142,"threads_started":2}
I20260709 12:23:14.559338  4987 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.547206  4770 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20260709 12:23:14.625633  4981 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.566918  4898 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
I20260709 12:23:14.639762  5042 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:14.676142  5032 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.832s	user 0.006s	sys 0.006s
I20260709 12:23:14.676774  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:14.707077  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.708387  1877 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:14.635073  4736 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:23:14.550314  5006 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.108s
I20260709 12:23:14.711910  5006 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:14.712492  5006 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:14.713639  5006 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:14.678117  5056 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.319s	user 0.001s	sys 0.000s
I20260709 12:23:14.661329  1731 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:23:14.656263  5034 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.160s	user 0.006s	sys 0.009s
I20260709 12:23:14.647085  5039 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:14.659282  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 248249us
I20260709 12:23:14.736493  4955 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:14.541509  4639 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.109s
I20260709 12:23:14.662328  4950 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:14.700598  2416 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
W20260709 12:23:14.704408  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 283737us
I20260709 12:23:14.757334  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:14.758337  1730 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:14.764695  1877 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:14.705883  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2450) took 2405 ms. Trace:
I20260709 12:23:14.672339  4948 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:14.686215  4987 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:14.734239  5046 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:14.683864  4853 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.852255  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:14.661899  4707 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:14.744393  1723 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:23:14.745252  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:14.751282  1874 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 12:23:14.736498  4970 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.909s	user 0.011s	sys 0.000s
I20260709 12:23:14.772054  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:14.772774  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 196904us
I20260709 12:23:14.866948  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:14.867624  2414 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:14.867666  2416 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:23:14.773881  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 358361us
W20260709 12:23:14.869752  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.794433  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:14.848767  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52355.9us
I20260709 12:23:14.682036  5040 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.058s
I20260709 12:23:14.852968  1876 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:14.768335  1880 rpcz_store.cc:276] 0709 12:23:12.300105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.375873 (+2075768us) service_pool.cc:224] Handling call
0709 12:23:14.376215 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.376442 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:14.630299 (+253857us) log.cc:844] Serialized 14 byte log entry
0709 12:23:14.705373 (+ 75074us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.705449 (+    76us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:14.705534 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.705632 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.705803 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11192,"prepare.queue_time_us":238905,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":149,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":168,"thread_start_us":272076,"threads_started":2}
I20260709 12:23:14.742662  5039 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:14.858335  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:14.887290  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:14.748021  5006 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:14.654090  4984 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:14.860631  1728 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:14.707028  4926 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.152s
I20260709 12:23:14.715238  5042 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.140s
I20260709 12:23:14.772924  4948 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:14.883386  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:14.702879  4981 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.873783  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:15.013279  1723 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:14.917691  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 269847us
I20260709 12:23:14.918751  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:15.012848  2414 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:15.015076  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:14.919687  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:15.006246  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:14.893425  4963 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.007s	sys 0.015s
I20260709 12:23:14.771116  4413 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.122s
I20260709 12:23:14.859614  4950 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:14.758675  4866 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.126s
I20260709 12:23:14.870462  4675 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.854355  4921 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.117s
I20260709 12:23:15.021209  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:14.737221  5032 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:14.925290  5055 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:15.060273  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:15.060989  1877 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:15.039559  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:15.047991  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.922492  4756 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.915940  4780 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:15.012826  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:14.916227  4898 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.057263  5073 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:23:14.915882  4639 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:15.063169  1805 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 12:23:15.038916  4963 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.007s	sys 0.015s
I20260709 12:23:15.062700  1723 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:14.920276  4941 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:15.007730  4943 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:15.007771  4952 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:15.013554  4948 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:15.008024  5020 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:15.078814  4984 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:15.086745  1876 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:14.880119  4853 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:15.109917  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.921669  4707 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.147s
I20260709 12:23:14.927461  4884 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.141s
I20260709 12:23:15.121775  1725 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:15.046665  5023 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.870504  4970 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:15.060022  5055 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:15.119959  4780 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:15.215392  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:15.290891  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:15.267858  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:15.122287  4898 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:15.307960  1805 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:15.308672  1877 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 4: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:23:15.300247  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 416945us
I20260709 12:23:15.309916  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:15.313277  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:15.314544  1877 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 8
W20260709 12:23:15.318799  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:15.268879  4804 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.846s	user 0.010s	sys 0.002s
I20260709 12:23:15.065688  4675 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:15.084934  4756 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:15.295910  4918 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:15.321632  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:15.268205  5073 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:23:15.322744  2410 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:15.324766  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:15.337740  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 316598us
I20260709 12:23:15.342376  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:15.320319  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:15.347620  1876 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:15.175984  4639 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:15.217519  4941 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.315430  4805 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.121s	user 0.011s	sys 0.001s
I20260709 12:23:15.313362  4943 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:15.350410  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:15.353770  4981 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:15.311682  4707 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:15.313534  4939 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.352094  1807 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:23:15.300984  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:15.321439  4952 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:23:15.258546  5023 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:15.372756  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:15.179818  5020 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:15.383359  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:15.401680  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2464) took 2194 ms (client timeout 1972 ms). Trace:
W20260709 12:23:15.402350  1881 rpcz_store.cc:269] 0709 12:23:13.207362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.945536 (+738174us) service_pool.cc:224] Handling call
0709 12:23:15.401422 (+1455886us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b5400260a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:15.401596 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":502680448}
I20260709 12:23:15.404083  1877 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:15.406602  1877 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: [2.2-8.5]
I20260709 12:23:15.392084  1880 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:15.393086  1876 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:23:15.408769  1876 raft_consensus.cc:606] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:15.409193  1877 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: [5.7-9.9]
I20260709 12:23:15.410697  1877 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:15.414341  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 104035us
I20260709 12:23:15.358775  4804 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:15.296954  4950 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:15.415553  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:15.396307  1805 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:15.420524  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:15.380903  5061 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.488s	user 0.001s	sys 0.001s
I20260709 12:23:15.422525  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:15.365681  4943 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:15.414471  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:15.426277  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:15.368569  4805 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:15.359469  5055 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:15.367798  4941 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:15.393368  5042 consensus_queue.cc:1243] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:23:15.563818  1728 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:15.564527  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:15.512501  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:15.455721  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:15.446715  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:15.447202  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:15.450523  1881 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:23:15.567019  2412 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:15.565527  1723 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:15.568997  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1776) took 1008 ms. Trace:
I20260709 12:23:15.512984  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:15.725787  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43056 (request call id 85) took 3785 ms. Trace:
I20260709 12:23:15.746157  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2466) took 2430 ms. Trace:
I20260709 12:23:15.392002  5014 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:15.606542  1807 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:15.664836  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:15.685902  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2472) took 1447 ms. Trace:
I20260709 12:23:15.618923  5055 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:15.713551  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:15.686674  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1680) took 2542 ms. Trace:
W20260709 12:23:15.635267  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.539700  1880 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:23:15.752660  1802 rpcz_store.cc:276] 0709 12:23:13.144166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.144314 (+   148us) service_pool.cc:224] Handling call
0709 12:23:14.190913 (+1046599us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.300926 (+110013us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.686389 (+1385463us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540013a950. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:15.686559 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2405811072}
I20260709 12:23:15.575028  1876 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:15.649902  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.746572  1872 rpcz_store.cc:276] 0709 12:23:11.940699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.796930 (+856231us) service_pool.cc:224] Handling call
0709 12:23:15.720801 (+2923871us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b5400116f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdde735 00007f16abdd69fa 00007f16b0d9e4c9 00007f16aa31ac54 00007f16aa31abe2 00007f16aa31ab11 00007f16aa31aa8c 00007f16aa319d42 00007f16aa11348c 00007f16aa1128c6
0709 12:23:15.725660 (+  4859us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4394306432}
I20260709 12:23:15.754325  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1786) took 1129 ms. Trace:
I20260709 12:23:15.753634  1880 pending_rounds.cc:85] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 2
I20260709 12:23:15.755700  1880 pending_rounds.cc:107] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:23:15.576323  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:15.756309  1880 raft_consensus.cc:2889] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:15.747483  1881 rpcz_store.cc:276] 0709 12:23:14.561005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.403481 (+842476us) service_pool.cc:224] Handling call
0709 12:23:15.450185 (+ 46704us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.568691 (+118506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.568935 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:15.484225  4639 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:15.396245  5087 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.750491  1874 rpcz_store.cc:276] 0709 12:23:14.238501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.345765 (+107264us) service_pool.cc:224] Handling call
0709 12:23:15.685554 (+1339789us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b5400116f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:15.685804 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":871368064}
W20260709 12:23:15.565764  4966 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.392s	user 0.013s	sys 0.000s
I20260709 12:23:15.399051  4975 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.483341  4949 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:15.754719  1876 rpcz_store.cc:276] 0709 12:23:14.624614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.507538 (+882924us) service_pool.cc:224] Handling call
0709 12:23:15.574723 (+ 67185us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.754075 (+179352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.754273 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:15.759089  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:15.757110  4866 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.003s	sys 0.016s
W20260709 12:23:15.684795  5075 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.444s	user 0.002s	sys 0.000s
W20260709 12:23:15.536664  5066 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.395s	user 0.001s	sys 0.000s
I20260709 12:23:15.460466  5089 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.682996  5050 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.182s	user 0.011s	sys 0.003s
W20260709 12:23:15.730531  5078 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
W20260709 12:23:15.670742  5064 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.498s	user 0.002s	sys 0.000s
I20260709 12:23:15.763669  1874 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:15.618686  5045 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
I20260709 12:23:15.764616  1799 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:15.765491  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1784) took 1141 ms. Trace:
I20260709 12:23:15.870013  1880 rpcz_store.cc:276] 0709 12:23:14.624024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.464190 (+840166us) service_pool.cc:224] Handling call
0709 12:23:15.516973 (+ 52783us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.765173 (+248200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.765422 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:15.540920  5016 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.746906  1873 rpcz_store.cc:276] 0709 12:23:13.315524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.488809 (+1173285us) service_pool.cc:224] Handling call
0709 12:23:14.489111 (+   302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:14.535767 (+ 46656us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.535801 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:14.554398 (+ 18597us) log.cc:844] Serialized 26 byte log entry
0709 12:23:15.081332 (+526934us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:15.081376 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.081455 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:15.672927 (+591472us) raft_consensus.cc:1685] finished
0709 12:23:15.672960 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:15.746062 (+ 73102us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46231,"raft.queue_time_us":48603,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":18192,"thread_start_us":422245,"threads_started":1,"wal-append.queue_time_us":927757}
I20260709 12:23:15.607458  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:15.779112  5023 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:15.850929  4797 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.483705  4898 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:15.757398  5063 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.420s	user 0.000s	sys 0.001s
W20260709 12:23:15.712744  5068 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.324s	user 0.001s	sys 0.000s
I20260709 12:23:15.745754  4756 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:15.721421  4970 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:15.773917  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:15.397540  4939 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:15.662295  5032 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:15.752202  4853 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:15.756810  5040 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:15.835215  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.773757  4952 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.684952  5088 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.562968  5042 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.210s
W20260709 12:23:15.781672  5083 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
W20260709 12:23:15.782171  5084 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.422s	user 0.000s	sys 0.001s
I20260709 12:23:15.743484  5065 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.328s
W20260709 12:23:15.849437  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.760998  5014 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:15.774670  4866 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.003s	sys 0.016s
I20260709 12:23:15.772650  4949 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:15.874493  1723 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:15.446810  4804 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:15.774116  4975 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:15.684547  5092 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.202s
I20260709 12:23:15.758922  4943 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:15.905457  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 652) took 1454 ms. Trace:
I20260709 12:23:15.881282  4805 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:15.916914  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:15.937003  5025 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:15.937809  5025 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:15.933302  2419 rpcz_store.cc:276] 0709 12:23:14.451347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.462991 (+ 11644us) service_pool.cc:224] Handling call
0709 12:23:14.508692 (+ 45701us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:14.509009 (+   317us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.509038 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:14.509525 (+   487us) log.cc:844] Serialized 26 byte log entry
0709 12:23:14.707309 (+197784us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:14.707366 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.707457 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:15.905067 (+1197610us) raft_consensus.cc:1685] finished
0709 12:23:15.905098 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:15.905362 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":78373,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":52937,"thread_start_us":139694,"threads_started":1,"wal-append.queue_time_us":374405}
I20260709 12:23:15.938228  5025 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.938689  1731 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260709 12:23:15.939142  1731 pending_rounds.cc:85] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 3
I20260709 12:23:15.939361  1731 pending_rounds.cc:107] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260709 12:23:15.939597  1731 raft_consensus.cc:2889] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:15.938090  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:15.941666  5049 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:15.920171  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:15.903470  5020 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:15.919245  5093 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:23:15.945097  4769 raft_consensus.cc:2764] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:23:15.890130  4797 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:15.881567  5101 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:15.878508  5016 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:15.922516  5042 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:15.909725  4952 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:15.777438  5087 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:15.864326  5089 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.007140  4797 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.007647  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1783) took 1384 ms. Trace:
W20260709 12:23:16.007755  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8bb813e4fab14a61b810a24b56899586 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a20e99e125b84513a44e88860667ed61 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:23:16.009434  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 90668.5us
I20260709 12:23:15.909830  5040 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:15.921784  5050 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.076s
I20260709 12:23:15.938705  5025 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:16.012903  1877 rpcz_store.cc:276] 0709 12:23:14.623627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.411648 (+788021us) service_pool.cc:224] Handling call
0709 12:23:15.412308 (+   660us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:15.567302 (+154994us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.567334 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:15.659561 (+ 92227us) log.cc:844] Serialized 608 byte log entry
0709 12:23:15.912344 (+252783us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:15.912396 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.912474 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.007252 (+ 94778us) raft_consensus.cc:1685] finished
0709 12:23:16.007282 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.007538 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16851,"raft.queue_time_us":153542,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":41626,"thread_start_us":163460,"threads_started":1,"wal-append.queue_time_us":317099}
W20260709 12:23:15.998126  4933 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.450s	user 0.016s	sys 0.000s
I20260709 12:23:16.022130  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:16.025420  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:15.997745  5042 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:16.034204  1772 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:16.031903  5040 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.038003  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:16.039050  1799 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:15.991528  5091 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:16.040761  2419 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:23:16.048135  2419 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:16.043229  4933 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:16.043717  1723 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:16.038815  4941 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.044507  4797 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:15.926715  5088 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.028769  5049 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.027905  5080 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:16.038967  4966 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:16.016069  5024 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.462s	user 0.018s	sys 0.008s
I20260709 12:23:16.052098  5014 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.053388  4805 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:16.036629  5097 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:16.053762  1772 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:16.058141  1799 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:16.052130  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 662) took 1513 ms. Trace:
I20260709 12:23:16.062866  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:16.002002  5101 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.081859  1872 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:23:16.050995  5016 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.049398  2419 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.037729  4900 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.094530  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:16.078766  2417 rpcz_store.cc:276] 0709 12:23:14.538859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.540058 (+  1199us) service_pool.cc:224] Handling call
0709 12:23:14.545296 (+  5238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.689643 (+144347us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:23:14.689677 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.689719 (+    42us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:14.689766 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.051834 (+1362068us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b5400840154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:16.051897 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.052044 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":331342208}
I20260709 12:23:16.085767  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:16.014577  4949 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.086666  1877 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:16.087632  4797 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:16.087988  5101 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:16.090407  4948 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.879s	user 0.006s	sys 0.006s
W20260709 12:23:16.092140  5102 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:23:16.100368  2417 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:16.075450  4966 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:16.103431  2417 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:16.149063  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:16.150060  2418 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:16.153723  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:16.078159  5080 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.112902  2417 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.093432  4866 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:16.161846  2417 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:16.164541  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 411) took 2831 ms. Trace:
I20260709 12:23:16.072238  5049 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.165242  1879 rpcz_store.cc:276] 0709 12:23:13.334508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.497193 (+1162685us) service_pool.cc:224] Handling call
0709 12:23:14.497467 (+   274us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:14.536408 (+ 38941us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.536442 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:14.543114 (+  6672us) log.cc:844] Serialized 52 byte log entry
0709 12:23:14.854430 (+311316us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:15.016638 (+162208us) log.cc:844] Serialized 14 byte log entry
0709 12:23:15.290450 (+273812us) log.cc:844] Serialized 14 byte log entry
0709 12:23:15.292956 (+  2506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.293080 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.164154 (+871074us) raft_consensus.cc:1685] finished
0709 12:23:16.164182 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.164454 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":116884,"prepare.queue_time_us":167293,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":4073,"raft.queue_time_us":98843,"raft.run_cpu_time_us":577,"raft.run_wall_time_us":74849,"thread_start_us":255416,"threads_started":2,"wal-append.queue_time_us":486125}
I20260709 12:23:16.106257  4948 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.059051  4804 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:16.159523  4987 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.303s	user 0.012s	sys 0.005s
I20260709 12:23:16.106829  4805 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:16.169787  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:16.180516  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:16.151844  4966 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:16.185786  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.079555  5024 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.186811  5016 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.187376  5016 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.188423  5024 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.063872  5091 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.189824  5016 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.190572  2415 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:16.191217  5091 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:16.159523  4949 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:16.198314  4939 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.148504  5050 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:16.203191  5050 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.204175  4986 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:16.204653  4986 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:16.193953  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:16.196218  4958 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.196731  5040 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.196878  4948 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.197155  4941 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:16.197512  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:16.102615  4900 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.208418  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.210281  4941 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.211242  5040 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.213460  5049 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:16.205849  4987 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.191551  5016 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:16.215247  4941 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.220244  4956 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:16.221686  4987 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.222460  4900 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.212209  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:16.251103  1802 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:16.212687  4958 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.225705  1805 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:23:16.227653  1804 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:16.231702  4988 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.241463  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:16.244554  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:16.247882  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:16.086081  5042 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.264362  2418 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:16.268711  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.247031  4956 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.270023  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.245630  5020 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:16.277236  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:16.272933  1802 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:16.248867  5016 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.279312  5042 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:16.278723  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:16.281033  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.275590  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:16.253522  5050 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.285880  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.284816  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:16.255270  5040 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.286845  1804 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:16.287484  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:16.287920  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.288621  1804 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:16.275930  5013 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.271004  4925 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:16.205564  4986 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.276257  4956 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.290412  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.288261  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:16.292315  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.297638  1805 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:16.298784  5080 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.299144  1805 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:23:16.292303  5013 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:16.300359  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.301568  5013 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.294956  5088 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.302584  1805 raft_consensus.cc:2955] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:23:16.272993  5082 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.902s	user 0.001s	sys 0.009s
I20260709 12:23:16.262995  5089 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.290081  5055 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:23:16.274641  4975 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.276759  4950 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:16.312589  4950 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:16.313576  4950 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:16.316826  5082 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.970s	user 0.001s	sys 0.010s
I20260709 12:23:16.318405  5115 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.318742  5055 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:16.319453  5055 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:16.319708  5016 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.319152  5115 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.285948  4941 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:16.320943  4941 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:16.283061  4987 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:16.320490  5055 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.322582  1802 raft_consensus.cc:2410] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:16.328437  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:16.330334  2417 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:23:16.330382  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:16.332468  1696 leader_election.cc:400] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6.
I20260709 12:23:16.332901  1696 leader_election.cc:403] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:16.330957  2417 pending_rounds.cc:85] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 5
I20260709 12:23:16.338058  2417 pending_rounds.cc:107] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:23:16.339293  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.342752  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:16.347965  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.337534  5032 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:23:16.284581  5020 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.338378  2417 raft_consensus.cc:2889] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.323731  4987 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.356395  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.358685  5115 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.358808  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.361181  5088 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:16.352470  4949 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6.
W20260709 12:23:16.373533  5006 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.366s	user 0.008s	sys 0.006s
W20260709 12:23:16.374452  5111 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:23:16.338416  4986 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:16.331367  5040 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
W20260709 12:23:16.374713  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.379864  4958 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.351114  4941 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.381911  5032 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:16.384501  1873 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:16.385695  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:16.441602  1806 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:16.442153  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1692) took 2016 ms. Trace:
I20260709 12:23:16.386966  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2600) took 1889 ms. Trace:
I20260709 12:23:16.442961  1727 rpcz_store.cc:276] 0709 12:23:14.497079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.498917 (+  1838us) service_pool.cc:224] Handling call
0709 12:23:14.499229 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.561657 (+ 62428us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.386704 (+1825047us) spinlock_profiling.cc:243] Waited 61.5 ms on lock 0x7b540030a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:16.386908 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":141494656}
I20260709 12:23:16.388031  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:16.444743  1879 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:16.388253  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 428) took 1841 ms. Trace:
I20260709 12:23:16.445855  1725 rpcz_store.cc:276] 0709 12:23:14.546526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.554584 (+  8058us) service_pool.cc:224] Handling call
0709 12:23:14.637982 (+ 83398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.227249 (+589267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.387946 (+1160697us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b54003ccdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:16.388198 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1111853952}
I20260709 12:23:16.389086  1877 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:23:16.390458  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:16.391980  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:16.392192  4925 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.392891  1874 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:16.393509  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:16.394723  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.449259  1878 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:16.397681  1802 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:16.398734  1728 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:16.397219  4987 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.355142  4950 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:16.405079  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.407807  5087 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.418926  5115 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.423264  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:16.401911  4966 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.428648  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:16.397133  5023 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:23:16.442549  1806 rpcz_store.cc:276] 0709 12:23:14.425496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.385432 (+1959936us) service_pool.cc:224] Handling call
0709 12:23:16.442082 (+ 56650us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:16.452586  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94964.5us
I20260709 12:23:16.452750  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:16.453621  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:16.427837  5025 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.455646  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.468113  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:16.456717  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.458314  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:16.422709  5020 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.458520  1802 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:23:16.450802  5040 consensus_queue.cc:1243] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260709 12:23:16.453887  5116 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:23:16.459692  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.477581  1725 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:16.466966  1806 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:16.463969  2417 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:16.474673  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.479274  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.480096  1727 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 4.
W20260709 12:23:16.480259  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.453404  4925 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.475848  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:16.490120  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:16.491169  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:16.479290  1728 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:16.495025  1728 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:16.496307  1728 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:16.496395  4925 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:16.500444  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.501237  1728 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:16.485695  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:16.499661  1725 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:23:16.502816  4958 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:16.456204  5110 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.508286  1881 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:16.459391  5006 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 12:23:16.486992  5089 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:16.517200  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:16.486752  5032 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:16.523212  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:16.524091  1873 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:16.525221  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:16.525664  4948 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:16.526081  1873 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:16.526212  4948 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:16.526860  4948 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:16.483587  4666 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:23:16.533836  5121 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:16.533624  4956 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.510027  4958 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.536533  4958 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:16.534360  5121 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:16.509119  4949 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:16.469565  5115 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.511811  5020 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:16.515935  5023 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260709 12:23:16.543952  4949 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:16.560410  5039 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.641s	user 0.014s	sys 0.002s
I20260709 12:23:16.519345  5110 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.548277  5020 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:16.583457  5025 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:16.571054  4949 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:16.591082  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1615) took 1497 ms. Trace:
I20260709 12:23:16.597361  4756 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:16.604807  5115 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.606890  5110 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.609362  4952 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:16.611956  4943 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.620810  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2596) took 2125 ms. Trace:
I20260709 12:23:16.621274  1729 rpcz_store.cc:276] 0709 12:23:14.495120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.495240 (+   120us) service_pool.cc:224] Handling call
0709 12:23:14.495781 (+   541us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.591428 (+ 95647us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.591464 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.591513 (+    49us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:14.591554 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.620503 (+2028949us) spinlock_profiling.cc:243] Waited 95.4 ms on lock 0x7b540030b454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:16.620556 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.620706 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":219423616}
I20260709 12:23:16.624573  4966 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.586447  4956 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.627997  4950 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:16.631102  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:16.631927  1873 raft_consensus.cc:2368] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
I20260709 12:23:16.634536  1873 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:16.636598  5089 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:16.580410  5020 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.575490  5039 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.655830  4979 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.658458  4666 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.659090  4666 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.661432  4666 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.578972  5121 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.666411  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:16.922267  1881 raft_consensus.cc:2410] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
W20260709 12:23:16.700793  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 208348us
I20260709 12:23:16.693192  1799 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:16.594440  1798 rpcz_store.cc:276] 0709 12:23:15.093959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.227846 (+1133887us) service_pool.cc:224] Handling call
0709 12:23:16.228238 (+   392us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:16.243861 (+ 15623us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:16.243912 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:16.248663 (+  4751us) log.cc:844] Serialized 52 byte log entry
0709 12:23:16.429360 (+180697us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:16.521391 (+ 92031us) log.cc:844] Serialized 14 byte log entry
0709 12:23:16.578898 (+ 57507us) log.cc:844] Serialized 14 byte log entry
0709 12:23:16.579169 (+   271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.579255 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.579300 (+    45us) raft_consensus.cc:1685] finished
0709 12:23:16.579315 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.591007 (+ 11692us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64239,"prepare.queue_time_us":169757,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":90,"raft.queue_time_us":71528,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":7870,"spinlock_wait_cycles":17002368,"thread_start_us":222082,"threads_started":2,"wal-append.queue_time_us":163002}
I20260709 12:23:16.925117  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:16.926227  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:16.929311  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:16.644928  4948 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.930881  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.931552  1728 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:16.926738  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:16.701141  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 134188us
W20260709 12:23:16.941569  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.944049  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.947480  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.947480  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:16.948400  1723 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:16.949986  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:16.786433  5057 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.865s	user 0.005s	sys 0.008s
W20260709 12:23:16.923527  4986 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.506s	user 0.010s	sys 0.001s
I20260709 12:23:16.954576  1729 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:23:16.957466  1729 pending_rounds.cc:85] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 4
I20260709 12:23:16.957775  1729 pending_rounds.cc:107] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:23:16.958120  1729 raft_consensus.cc:2889] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.945205  4948 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.953665  5032 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:23:16.963033  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.963863  1725 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:16.968575  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:16.894861  4899 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:16.911381  4941 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:16.972702  1804 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:16.893319  5055 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:16.946082  4916 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:23:16.956974  5057 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.073s	user 0.005s	sys 0.015s
I20260709 12:23:16.946498  4950 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:23:16.983701  5124 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 12:23:16.984946  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:16.666534  4987 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:16.988149  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:16.989118  1725 raft_consensus.cc:2376] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11. Re-sending same reply.
I20260709 12:23:16.991005  4906 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.992439  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:16.995572  4675 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:16.996694   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:23:17.001663  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:17.006482  1727 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:23:17.007280  4949 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.007836  1800 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:17.008049  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:17.008496  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1616) took 1914 ms. Trace:
I20260709 12:23:17.009057  1725 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:16.988668  4987 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:17.010092  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1613) took 1998 ms. Trace:
I20260709 12:23:17.008927  1800 rpcz_store.cc:276] 0709 12:23:15.098646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.007242 (+1908596us) service_pool.cc:224] Handling call
0709 12:23:17.007660 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.008254 (+   594us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.008438 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:17.010480  1801 rpcz_store.cc:276] 0709 12:23:15.091091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.093060 (+  1969us) service_pool.cc:224] Handling call
0709 12:23:15.150542 (+ 57482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.297656 (+147114us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.297709 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:15.307781 (+ 10072us) log.cc:844] Serialized 26 byte log entry
0709 12:23:15.409016 (+101235us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:15.409081 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.009702 (+1600621us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400132254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:17.009749 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.009794 (+    45us) raft_consensus.cc:1685] finished
0709 12:23:17.009805 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.010034 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6596,"raft.queue_time_us":7427,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":10683,"spinlock_wait_cycles":321776896,"thread_start_us":54089,"threads_started":1,"wal-append.queue_time_us":119980,"wal-append.run_cpu_time_us":1948,"wal-append.run_wall_time_us":1003419}
I20260709 12:23:17.012899  4949 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.988780  2419 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:23:17.009325  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2597) took 2513 ms. Trace:
I20260709 12:23:17.026404  1726 rpcz_store.cc:276] 0709 12:23:14.496207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.499435 (+  3228us) service_pool.cc:224] Handling call
0709 12:23:14.499767 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.596073 (+ 96306us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:14.596136 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:14.596207 (+    71us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:14.596270 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.009097 (+2412827us) spinlock_profiling.cc:243] Waited 96 ms on lock 0x7b540030aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:17.009155 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.009267 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":220782464}
I20260709 12:23:16.953290  5036 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260709 12:23:17.030265  1725 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
W20260709 12:23:17.029922  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.035998  1728 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:17.044101  5039 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:16.979586  4941 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.980296  5121 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:17.039795   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:16.980501  5055 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:16.979159  4950 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:23:17.065613  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.068120  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:17.075809  1629 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 170) took 22382 ms (client timeout 15000 ms). Trace:
I20260709 12:23:17.068291  4950 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:17.078297  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2603) took 1663 ms. Trace:
I20260709 12:23:17.078740  1731 rpcz_store.cc:276] 0709 12:23:15.415015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.429959 (+ 14944us) service_pool.cc:224] Handling call
0709 12:23:15.430413 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.940015 (+509602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.077314 (+1137299us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b54002b7ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:17.078231 (+   917us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1167998336}
W20260709 12:23:17.073884  4958 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.537s	user 0.007s	sys 0.006s
I20260709 12:23:17.079996  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.083582  1875 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:17.071350  4899 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:17.081293  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:17.082178  5115 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 12:23:17.092538  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.093559  1876 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:23:17.089908  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.094710  1875 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:17.101939  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.083910  4736 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:23:17.108140  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.108867  5055 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.109086  1875 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:17.109979  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.111263  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.112371  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.113333  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.114289  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.086594  4958 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.119585  5038 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:17.120061  5038 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:17.120834  5038 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.121376  4958 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.121862  5038 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:17.122987  1724 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:17.123765   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:17.124783  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:17.114737  1876 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.110556  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.136821  2419 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260709 12:23:17.143276  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.144524  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.145865  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.146590  1874 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:17.147258  5130 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
W20260709 12:23:17.147799  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.148402  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.148954  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.151495  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.152293  1880 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:17.152326  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.153656  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.154422  1874 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:17.155146  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.155774  1874 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:17.155368  5020 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.156060  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 656) took 2703 ms. Trace:
I20260709 12:23:17.156488  2410 rpcz_store.cc:276] 0709 12:23:14.452640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.458002 (+  5362us) service_pool.cc:224] Handling call
0709 12:23:14.458339 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.355028 (+896689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.155835 (+1800807us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54006b0154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:17.156006 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1987237760}
I20260709 12:23:17.157083  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:17.157893  1731 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 9 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:23:17.157989  5020 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.086458  4899 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:17.156728  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.158993  4899 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:17.159096  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 417) took 2285 ms. Trace:
I20260709 12:23:17.160000  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.160665  1874 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:17.159592  1872 rpcz_store.cc:276] 0709 12:23:14.873885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.754685 (+880800us) service_pool.cc:224] Handling call
0709 12:23:15.768836 (+ 14151us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.082309 (+313473us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.158799 (+1076490us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b54002dfad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:17.159024 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":715722112}
I20260709 12:23:17.161589  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.161962  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.162912  1874 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:17.155781  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.164834  1874 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:17.166669  1880 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:23:17.170089  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:17.175160  4989 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:17.176321  5058 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:17.176741  5058 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:17.177634  5058 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.178999  5058 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:17.183892  5046 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:17.184296  1874 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:23:17.184594  5046 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:17.185746  1874 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:23:17.185442  5046 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.186154  1874 pending_rounds.cc:85] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 5
I20260709 12:23:17.186390  1874 pending_rounds.cc:107] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:23:17.187475  1725 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:23:17.186616  1874 raft_consensus.cc:2889] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:17.160743  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.192255  1873 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
W20260709 12:23:17.077143  1629 rpcz_store.cc:269] 0709 12:22:54.693777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.702997 (+  9220us) service_pool.cc:224] Handling call
0709 12:22:57.792557 (+3089560us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd42090 00007f16abde0f3a 00007f16b1b96e9c 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:23:17.075710 (+19283153us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:10.257129 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:10.368641 (+111512us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:10.368734 (+    93us) write_op.cc:435] Acquired schema lock
0709 12:23:10.368756 (+    22us) tablet.cc:662] Decoding operations
0709 12:23:10.370744 (+  1988us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:10.496485 (+125741us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:10.500315 (+  3830us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:23:11.043744 (+543429us) tablet.cc:701] Row locks acquired
0709 12:23:11.043815 (+    71us) write_op.cc:260] PREPARE: finished
0709 12:23:11.044050 (+   235us) write_op.cc:270] Start()
0709 12:23:11.044236 (+   186us) write_op.cc:276] Timestamp: P: 1783599791043970 usec, L: 0
0709 12:23:11.044263 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:23:11.045304 (+  1041us) log.cc:844] Serialized 6420 byte log entry
0709 12:23:11.651114 (+605810us) op_driver.cc:464] REPLICATION: finished
0709 12:23:11.992155 (+341041us) write_op.cc:301] APPLY: starting
0709 12:23:11.992253 (+    98us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:11.992636 (+   383us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:11.992830 (+   194us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:12.201300 (+208470us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:12.201896 (+   596us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=16
0709 12:23:12.201976 (+    80us) write_op.cc:312] APPLY: finished
0709 12:23:12.323739 (+121763us) log.cc:844] Serialized 113 byte log entry
0709 12:23:12.339820 (+ 16081us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:12.342498 (+  2678us) write_op.cc:454] Released schema lock
0709 12:23:12.359768 (+ 17270us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:12.360223 (+   455us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:12.360252 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24452,"reported_tablets":28,"spinlock_wait_cycles":6092868480,"child_traces":[["op",{"apply.queue_time_us":333444,"apply.run_cpu_time_us":12438,"apply.run_wall_time_us":368030,"num_ops":16,"prepare.queue_time_us":172233,"prepare.run_cpu_time_us":44540,"prepare.run_wall_time_us":1016875,"raft.queue_time_us":113101,"raft.run_cpu_time_us":53622,"raft.run_wall_time_us":270434,"replication_time_us":580542,"thread_start_us":497384,"threads_started":4,"wal-append.queue_time_us":201926,"wal-append.run_cpu_time_us":19685,"wal-append.run_wall_time_us":2121302}]]}
I20260709 12:23:17.194872  2419 pending_rounds.cc:85] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 5
I20260709 12:23:17.195351  2419 pending_rounds.cc:107] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:23:17.195638  2419 raft_consensus.cc:2889] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:17.198832  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 737) took 1157 ms. Trace:
I20260709 12:23:17.202139  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.202386  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.199965  2419 rpcz_store.cc:276] 0709 12:23:16.040081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.040237 (+   156us) service_pool.cc:224] Handling call
0709 12:23:16.040612 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.988589 (+947977us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b580063e6d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd340aa 00007f16abdb6447 00007f16abdcefe7 00007f16abdd07de 00007f16abddab49 00007f16abddb416 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4
0709 12:23:17.197516 (+208927us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.197832 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1240087296}
I20260709 12:23:17.212497  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:17.138996  4987 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.215232  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:17.217555  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.221300  1874 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: [2.2-8.5]
W20260709 12:23:17.211148  4963 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260709 12:23:17.221071  5036 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:17.226451  1873 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:17.222632  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:17.222193  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.227772  1874 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260709 12:23:17.229700  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:17.231594  1876 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:17.225055  4958 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:17.236238  4941 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.236337  4958 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:17.236440  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8bb813e4fab14a61b810a24b56899586 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:17.241674  1875 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:17.232514  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:17.256027  1801 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:17.258234  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:17.259164  1806 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:17.263816  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:17.264655  1801 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:17.272837  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 148100us
W20260709 12:23:17.278075  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.280226  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:17.218890  4788 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260709 12:23:17.237023  4958 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:17.253454  5023 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:17.297770  1798 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:17.274168  5121 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:17.298606  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:17.253866  5055 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:17.302909  1873 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:17.223563  4987 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.314357  1801 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:17.319527  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:17.320894  1801 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.321385  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:17.301538  5020 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 12:23:17.325904  1880 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:17.334007  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:17.297169  4950 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.322760  5028 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:23:17.401808  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.602574  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.602952  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.411115  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.317279  5134 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:17.594606  5057 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.594936  5115 consensus_queue.cc:1243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:17.355104  5138 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:17.595345  5137 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:17.595742  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 315631us
I20260709 12:23:17.309306  4899 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.598122  4958 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:17.599893  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 265965us
I20260709 12:23:17.615968  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:17.617884  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:17.325343  5055 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260709 12:23:17.600363  1801 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:17.604131  1806 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:17.599263  5136 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:17.605887  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:17.630900  2417 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.616084  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.632098  2410 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:17.350620  5121 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.061s
I20260709 12:23:17.620175  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.642249  2414 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:23:17.620733  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:17.621460  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:17.643628  2413 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.610889  5057 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.608793  4943 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.634934  5058 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:17.635517  4921 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:17.691147  4921 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:17.692088  4921 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.693161  4921 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:17.635726  4899 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:17.637466  5079 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 2.269s	user 0.006s	sys 0.000s
I20260709 12:23:17.638638  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:17.696719  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1694) took 3123 ms. Trace:
I20260709 12:23:17.697070  1732 rpcz_store.cc:276] 0709 12:23:14.573218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.576258 (+  3040us) service_pool.cc:224] Handling call
0709 12:23:14.576757 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:14.646445 (+ 69688us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:14.646475 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:14.646989 (+   514us) log.cc:844] Serialized 26 byte log entry
0709 12:23:14.773548 (+126559us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:14.773610 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.773706 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.636797 (+2863091us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:17.696383 (+ 59586us) raft_consensus.cc:1685] finished
0709 12:23:17.696403 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.696639 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72927,"raft.queue_time_us":69405,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":9138,"spinlock_wait_cycles":41397504,"thread_start_us":126219,"threads_started":1,"wal-append.queue_time_us":664487}
I20260709 12:23:17.643656  2411 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.634074  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:17.700449  2415 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:17.607683  5023 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:17.689985  4943 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.680696  5115 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:17.715160  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98290us
I20260709 12:23:17.716205  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:17.611766  5138 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:17.634763  4788 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:17.733635  2417 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:17.733911  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1699) took 2991 ms. Trace:
I20260709 12:23:17.733961  2411 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:17.734392  1724 rpcz_store.cc:276] 0709 12:23:14.742715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.742928 (+   213us) service_pool.cc:224] Handling call
0709 12:23:16.870129 (+2127201us) spinlock_profiling.cc:243] Waited 2.62 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3eee 000000000036eb39 00007f16b0d271e0 00007f16b0df00c4 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:23:16.870336 (+   207us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.123381 (+253045us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.733577 (+610196us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54003cd2d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:17.733833 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":586606208}
I20260709 12:23:17.719558  5055 consensus_queue.cc:1243] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:23:17.739480  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:17.741302  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:17.745613  2413 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:17.746181  2418 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:17.720290  5136 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.748468  4979 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:17.749004  5136 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.750494  4958 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.751175  4958 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:17.751446  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:17.752306  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.752904  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:17.753350  4804 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:17.753796  4804 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:17.754148  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.751194  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.755237  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1618) took 2655 ms. Trace:
I20260709 12:23:17.755512  5141 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:17.756299  5141 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.757844  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2497) took 1121 ms. Trace:
I20260709 12:23:17.754192  2418 raft_consensus.cc:2410] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:23:17.723873  5020 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:17.729396  4941 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:17.758261  1872 rpcz_store.cc:276] 0709 12:23:16.636198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.163063 (+526865us) service_pool.cc:224] Handling call
0709 12:23:17.163417 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:17.171107 (+  7690us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:17.171145 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.171812 (+   667us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.317745 (+145933us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:17.317792 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.317872 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.757583 (+439711us) raft_consensus.cc:1685] finished
0709 12:23:17.757610 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.757778 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":765,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":96,"thread_start_us":145433,"threads_started":1,"wal-append.queue_time_us":584040}
I20260709 12:23:17.763307  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.746212  2411 raft_consensus.cc:2376] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7. Re-sending same reply.
I20260709 12:23:17.764325  2413 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:17.765271  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.765934  2418 raft_consensus.cc:2376] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6. Re-sending same reply.
I20260709 12:23:17.768560  2416 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:23:17.755703  1803 rpcz_store.cc:276] 0709 12:23:15.099323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.570311 (+470988us) service_pool.cc:224] Handling call
0709 12:23:15.621414 (+ 51103us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.748313 (+126899us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:15.868091 (+119778us) log.cc:844] Serialized 14 byte log entry
0709 12:23:16.009037 (+140946us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.227654 (+1218617us) spinlock_profiling.cc:243] Waited 2.59 ms on lock 0x7b2c0005c178. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:23:17.261533 (+ 33879us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.324655 (+ 63122us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:17.324693 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.324765 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.754862 (+430097us) raft_consensus.cc:1685] finished
0709 12:23:17.754996 (+   134us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.755164 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":139579,"prepare.queue_time_us":28527,"prepare.run_cpu_time_us":6827,"prepare.run_wall_time_us":63873,"raft.queue_time_us":130957,"raft.run_cpu_time_us":387,"raft.run_wall_time_us":23026,"spinlock_wait_cycles":5953920,"thread_start_us":257566,"threads_started":3,"wal-append.queue_time_us":650306,"wal-append.run_cpu_time_us":1445,"wal-append.run_wall_time_us":1001470}
I20260709 12:23:17.798478  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:17.758481  5053 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:23:17.800170  4987 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:17.800839  4987 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:17.754611  4804 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.801405  5141 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.801977  4804 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.744877  4788 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.804680  4788 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.801023  5138 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.9, Last appended by leader: 9, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.796497  5136 raft_consensus.cc:2764] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:23:17.801011  4950 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.807189  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.808871  2416 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:17.810123  2416 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:17.798583  5020 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:17.811930  5020 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.815898  5144 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:17.815982  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.816735  5144 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.817415  2411 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:17.819042  5142 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:17.819792  5142 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.822413  5142 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:17.823431  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:17.824244  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.824381  2412 raft_consensus.cc:2368] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
W20260709 12:23:17.826092  5135 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:23:17.827610  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2495) took 1288 ms. Trace:
I20260709 12:23:17.828027  1881 rpcz_store.cc:276] 0709 12:23:16.539475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.936864 (+397389us) service_pool.cc:224] Handling call
0709 12:23:16.965929 (+ 29065us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:16.966372 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:16.966405 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:16.969421 (+  3016us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.298516 (+329095us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:17.298561 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.298641 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.827327 (+528686us) raft_consensus.cc:1685] finished
0709 12:23:17.827353 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.827538 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":621,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":328646,"threads_started":1,"wal-append.queue_time_us":639788}
I20260709 12:23:17.829777  5028 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:17.830384  5028 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:17.822328  1627 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 6, now has a pending config: VOTER 075704438992440697732181e49ccd09 (127.0.140.65), VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66), VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67), VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 6 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:17.871289  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:17.872979  2413 raft_consensus.cc:2410] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260709 12:23:17.873925  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 235363us
I20260709 12:23:17.873596  1627 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 4, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 4 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:17.878583  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:17.878278  1627 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 5, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:23:17.880095  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.881493  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 265579us
I20260709 12:23:17.880187  1627 catalog_manager.cc:5697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 6, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 6 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:17.882629  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:17.883615  1627 catalog_manager.cc:5697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:17.900085  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.900830  1880 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:17.900538  1627 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:17.903599  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:17.871361  5058 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:17.911013  1627 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 8, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 8 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:17.913141  1627 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:17.889007  4987 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:17.914958  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.922240  1881 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:17.925916  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 287317us
I20260709 12:23:17.928638  4736 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:17.910207  5076 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:23:17.907155  5136 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:23:17.929150  4736 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:17.930989  4921 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:17.931731  4921 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.931896  4736 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.933085  4736 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:17.933374  4943 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:17.933923  4943 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:17.934275  4921 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:17.937438  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1702) took 1712 ms. Trace:
I20260709 12:23:17.942046  4797 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:17.935803  5057 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:17.936781  5076 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:17.942890  5094 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.943343  5076 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:17.943629  5094 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.944375  5076 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.944558  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:17.934403  4943 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.944691  5089 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.947052  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:17.950209  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:17.945111  2415 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:17.952190  4805 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:17.952698  4805 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:17.953122  4805 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:17.953431  5089 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.955292  2410 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:17.956404  4933 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.956915  5089 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:17.951330  4943 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:17.942961  4770 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.948773  4926 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:17.960914  2418 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:17.949657  5076 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:17.962019  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:18.010423  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48483.8us
I20260709 12:23:17.964960  1881 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
W20260709 12:23:17.967921  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.967413  1627 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 6, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:17.957003  4933 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.011927  4770 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:17.971302  4921 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:17.971848  5057 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:17.975536  5115 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:23:18.012768  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.923348  5058 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:17.921983  4987 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:17.979281  5144 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:17.966297  4797 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:17.982601  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:17.937914  1805 rpcz_store.cc:276] 0709 12:23:16.224788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.302019 (+ 77231us) service_pool.cc:224] Handling call
0709 12:23:16.302278 (+   259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.302462 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:16.341677 (+ 39215us) log.cc:844] Serialized 14 byte log entry
0709 12:23:16.591644 (+249967us) log.cc:844] Serialized 14 byte log entry
0709 12:23:16.652445 (+ 60801us) log.cc:844] Serialized 14 byte log entry
0709 12:23:17.084075 (+431630us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b18003e9ec0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:17.936840 (+852765us) spinlock_profiling.cc:243] Waited 8.57 ms on lock 0x7b18003e9ec0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:23:17.937018 (+   178us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:17.937089 (+    71us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:17.937151 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.937225 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.937366 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1354896,"prepare.queue_time_us":127421,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":101,"raft.queue_time_us":1346211,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":6477,"spinlock_wait_cycles":57806464,"thread_start_us":235766,"threads_started":2,"wal-append.queue_time_us":169738,"wal-append.run_cpu_time_us":2690,"wal-append.run_wall_time_us":1140116}
I20260709 12:23:17.987717  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:17.994068  1876 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:18.006062  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:18.016183  4900 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:18.016947  2417 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:18.027235  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 309046us
I20260709 12:23:18.060166  1875 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:18.061076  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.019174  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:18.019268  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:18.066444  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 447606us
W20260709 12:23:18.074515  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55441.6us
I20260709 12:23:18.071671  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:18.072486  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45741.4us
W20260709 12:23:18.074805  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 68835.9us
W20260709 12:23:18.079059  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.659s (SENT) [suppressed 19 similar messages]
I20260709 12:23:18.079578  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:18.079630  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:18.009588  4963 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:18.073637  1627 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 5 to 7, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 7 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:18.084456  1627 catalog_manager.cc:5697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:18.086179  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.026432  5023 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:18.015293  4921 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.087471  4921 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:18.087744  5023 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:18.089273  4805 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.006556  4666 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:18.021271  4789 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:18.027129  5150 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.109154  5150 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.111529  5150 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:18.113691  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:18.114346  1876 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:18.015818  4770 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:18.070732  4900 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:18.015455  4948 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.027257  5094 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:18.099334  4918 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:18.104362  5021 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:18.104595  5151 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:18.082748  4950 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:18.073450  5121 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:23:18.105443  5058 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.105837  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.106386  5055 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:18.081410  5030 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:18.015023  4933 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.038846  5152 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.083902  5144 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:18.070886  4987 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.123494  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:18.029362  5017 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20260709 12:23:18.093850  4986 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:18.132350  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:18.188634  1877 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:23:18.102377  4789 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:18.192950  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 226215us
I20260709 12:23:18.151785  1872 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:18.153780  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:18.196414  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:18.155074  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:18.101913  4788 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.126276  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:18.124064  5057 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.204421  5057 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.205269  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:18.206183  1725 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:18.136019  4805 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.207508  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:18.208285  1726 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:18.185250  1730 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:18.086239  1627 catalog_manager.cc:5697] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 5, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 5 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:18.094317  5038 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:18.191430  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 111915us
W20260709 12:23:18.225620  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.225896  1873 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:23:18.146929  1731 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:18.081842  5006 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:18.142601  5039 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.119884  5115 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.199687  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:18.123771  5016 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:18.210040  1627 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 3 to 6, term changed from 5 to 6, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to <none>, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 6 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:18.224314  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:18.185453  5144 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.225102  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:18.226270  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.229576  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.195199  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:18.238114  1877 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:18.239916  1772 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.242400  1772 leader_election.cc:400] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
I20260709 12:23:18.134984  4943 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.184872  5030 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:18.224665  5145 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:18.243906  1726 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:18.245077  1772 leader_election.cc:403] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:18.244551  1627 catalog_manager.cc:5697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 9. New cstate: current_term: 9 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:18.247776  1876 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:18.203342  4789 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.115706  4666 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.200986  5012 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:18.199162  4918 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:18.185633  4770 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.086s
I20260709 12:23:18.184208  4804 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:18.147562  4948 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.223786  5154 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:23:18.273381  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.257128  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 381614us
I20260709 12:23:18.249293  4805 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:18.259498  1627 catalog_manager.cc:5697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 5, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 5 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:18.257193  4958 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.247321  5144 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:18.233145  5016 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:18.296581  4899 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:18.301697  4948 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:18.302021  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 72552.6us
I20260709 12:23:18.301895  4899 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.284096  5058 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.303289  4950 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:18.303699  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.304200  4797 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:18.304414  5134 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:18.305738  5134 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:18.306119  5134 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:18.304231  4950 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:23:18.309618  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:18.309890  4941 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:18.234083  5038 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.303232  5058 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.312645  4958 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.313298  5058 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:18.313861  5038 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.231576  4788 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.314414  5058 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.315341  4958 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:18.289757  5154 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.316110  5154 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.316291  4788 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:18.316550  1725 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:18.250640  4926 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:18.277402  5080 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.257231  5145 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:18.192854  5152 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.244429  5115 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.270179  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:18.298838  5030 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.326228  2382 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:18.299245  5076 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:18.329401  5076 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:18.329958  5076 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.330402  5076 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.300019  4666 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.300213  5156 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:18.236172  5006 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:18.325192  4788 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:23:18.275353  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77947.3us
I20260709 12:23:18.286818  4804 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.317104  5036 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:18.343956  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1720) took 1087 ms. Trace:
I20260709 12:23:18.315165  5058 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.355490  1806 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:18.244129  5039 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.368388  1725 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:23:18.253760  4986 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 12:23:18.353154  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.368611  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:18.369556  1803 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:18.353983  1804 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:23:18.257747  4943 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.333726  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:18.356935  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.373512  1875 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:18.359690  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.365481  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:18.284286  5089 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:23:18.296741  1627 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:18.286657  5163 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
W20260709 12:23:18.370203  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 284303us
W20260709 12:23:18.368026  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 98108.1us
I20260709 12:23:18.371582  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:18.323170  5057 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.415225  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:18.343446  4804 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:18.252110  4900 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.382829  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.419951  1804 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.380604  1627 catalog_manager.cc:5697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.317641  5028 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.356999  4926 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:18.422079  1627 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 5 to 6, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 6 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:18.384363  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:18.419097  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.384076  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:18.430594  1627 catalog_manager.cc:5697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:18.372355  4950 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.432989  1627 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 6, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 6 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:18.435336  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.437131  1627 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 reported cstate change: term changed from 6 to 8, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 8 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:18.433761  4950 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.445669  1627 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 5, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 5 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:18.484839  1627 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 8, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 8 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.428529  5057 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:18.457245  5028 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.486227  1627 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 4, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 4 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:18.487094  5057 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.487577  5057 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.487704  1627 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 1 to 5, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 5 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:18.489216  1627 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 7, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 7 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:18.478139  1804 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:18.480576  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:18.356207  4788 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.491508  1878 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:18.459683  4900 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:18.421864  4921 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.502417  1726 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
W20260709 12:23:18.484752  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 110893us
I20260709 12:23:18.505865  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:18.438989  5055 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:18.477138  5006 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:23:18.478381  5174 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.460022  4958 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260709 12:23:18.530768  1881 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:23:18.377147  4918 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.521292  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:18.465817  5160 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 12:23:18.429795  5145 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.482147  5159 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:23:18.495087  5020 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:23:18.493149  5023 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.040s
I20260709 12:23:18.549257  5145 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:18.535192  5179 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:18.520705  5121 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.023s
I20260709 12:23:18.556309  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:18.555685  5164 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:23:18.522904  5176 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:23:18.552299  5053 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:18.557919  4941 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.571549  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 750) took 1227 ms. Trace:
I20260709 12:23:18.605113  2419 rpcz_store.cc:276] 0709 12:23:17.344701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.592449 (+247748us) service_pool.cc:224] Handling call
0709 12:23:17.592971 (+   522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:17.863747 (+270776us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:17.863786 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.864524 (+   738us) log.cc:844] Serialized 26 byte log entry
0709 12:23:18.296386 (+431862us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:18.296426 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.296528 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.571306 (+274778us) raft_consensus.cc:1685] finished
0709 12:23:18.571330 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.571478 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20539,"raft.queue_time_us":131752,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":6775,"thread_start_us":701660,"threads_started":2,"wal-append.queue_time_us":681021}
I20260709 12:23:18.572418  5058 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:18.572914  5055 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.565366  5159 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:18.574556  4921 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.575393  4943 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:18.578693  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1717) took 1343 ms. Trace:
I20260709 12:23:18.538591  4926 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.601197  4805 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:18.631980  4805 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:18.632530  4805 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:18.601759  4986 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:18.602289  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 184447us
I20260709 12:23:18.533929  5174 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.573037  5057 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:18.617177  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 198181us
I20260709 12:23:18.550328  5144 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.580595  5179 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:18.616410  5152 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:18.732348  5170 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260709 12:23:18.345232  1799 rpcz_store.cc:276] 0709 12:23:17.256902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.257036 (+   134us) service_pool.cc:224] Handling call
0709 12:23:17.260587 (+  3551us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:17.260901 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:17.263340 (+  2439us) log.cc:844] Serialized 14 byte log entry
0709 12:23:18.323553 (+1060213us) log.cc:844] Serialized 14 byte log entry
0709 12:23:18.324262 (+   709us) log.cc:844] Serialized 14 byte log entry
0709 12:23:18.325655 (+  1393us) log.cc:844] Serialized 14 byte log entry
0709 12:23:18.342395 (+ 16740us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:18.342480 (+    85us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:18.342556 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.343729 (+  1173us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.343885 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":757026,"prepare.queue_time_us":10000,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":74,"raft.queue_time_us":783290,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":42242,"spinlock_wait_cycles":27531648,"thread_start_us":323447,"threads_started":1,"wal-append.queue_time_us":30835}
W20260709 12:23:18.736665  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.737668  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.738255  4987 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.539s	user 0.008s	sys 0.003s
I20260709 12:23:18.738812  4987 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:18.741482  5180 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:23:18.743286  4987 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.625034  1807 rpcz_store.cc:276] 0709 12:23:17.254680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.254956 (+   276us) service_pool.cc:224] Handling call
0709 12:23:17.289329 (+ 34373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.012013 (+722684us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.012064 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:18.074205 (+ 62141us) log.cc:844] Serialized 26 byte log entry
0709 12:23:18.373549 (+299344us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:18.373591 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.373655 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.578448 (+204793us) raft_consensus.cc:1685] finished
0709 12:23:18.578474 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.578622 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":722137,"raft.queue_time_us":723410,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":109,"thread_start_us":265991,"threads_started":1,"wal-append.queue_time_us":413678}
I20260709 12:23:18.744668  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.560649  5163 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:23:18.693850  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:18.747357  5158 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.449s	user 0.000s	sys 0.004s
I20260709 12:23:18.620306  4797 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:23:18.622494  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:18.750415  1804 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:18.695659  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.752564  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.754029  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.613869  4941 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.619386  5159 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:18.698558  4789 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.700147  5179 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:18.677209  4986 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:18.701046  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.787309  4986 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.702101  2417 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:18.788913  4986 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.698601  5057 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:18.706672  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.791745  1805 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.791602  5179 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.792914  1881 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:18.703524  4804 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.795262  4804 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:18.721220  5039 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.796272  4789 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:18.723587  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.798394  1803 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:18.705622  4805 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.800701  4805 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.748970  5076 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:18.730129  5168 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
I20260709 12:23:18.802449  5016 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.807602  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.808686  1803 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 9 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:23:18.808543  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.809396  1799 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:18.809653  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.754190  1846 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:18.810738  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:18.811288  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 57207.1us
I20260709 12:23:18.811550  1799 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:18.825167  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:18.825489  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 270130us
I20260709 12:23:18.825984  1880 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:18.827309  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.784722  2418 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:18.754720  1801 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:18.787847  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.829183  1798 raft_consensus.cc:2368] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10.
I20260709 12:23:18.762701  4963 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.804984  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.830562  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.830694  1806 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:18.807698  5016 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:18.812299  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:18.760768  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:18.747535  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53803.7us
I20260709 12:23:18.847913  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.813236  5057 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:18.814268  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 816) took 1073 ms. Trace:
I20260709 12:23:18.849061  2414 rpcz_store.cc:276] 0709 12:23:17.740577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.742932 (+  2355us) service_pool.cc:224] Handling call
0709 12:23:17.743420 (+   488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:17.765733 (+ 22313us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:17.765761 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.766236 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 12:23:18.007555 (+241319us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:18.007619 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.008760 (+  1141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.745479 (+736719us) spinlock_profiling.cc:243] Waited 18.9 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:18.813937 (+ 68458us) raft_consensus.cc:1685] finished
0709 12:23:18.813959 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.814201 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15309,"raft.queue_time_us":79934,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"spinlock_wait_cycles":43373312,"thread_start_us":240919,"threads_started":1,"wal-append.queue_time_us":510390}
I20260709 12:23:18.815901  5021 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:18.816535  5153 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:23:18.819164  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.762007  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1723) took 1504 ms. Trace:
I20260709 12:23:18.784482  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.804008  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1719) took 1548 ms. Trace:
I20260709 12:23:18.758463  2410 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
W20260709 12:23:18.865779  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 81409.4us
I20260709 12:23:18.865329  1800 rpcz_store.cc:276] 0709 12:23:17.255584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.255741 (+   157us) service_pool.cc:224] Handling call
0709 12:23:17.256227 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:17.999577 (+743350us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:17.999622 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:18.005072 (+  5450us) log.cc:844] Serialized 26 byte log entry
0709 12:23:18.378081 (+373009us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:18.378120 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.378184 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.803662 (+425478us) raft_consensus.cc:1685] finished
0709 12:23:18.803696 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.803933 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":749061,"raft.queue_time_us":743591,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":4460,"thread_start_us":363365,"threads_started":1,"wal-append.queue_time_us":523352}
I20260709 12:23:18.847100  4789 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.864380  1802 rpcz_store.cc:276] 0709 12:23:17.258011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.258161 (+   150us) service_pool.cc:224] Handling call
0709 12:23:17.295234 (+ 37073us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.001547 (+706313us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.001618 (+    71us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:18.006200 (+  4582us) log.cc:844] Serialized 26 byte log entry
0709 12:23:18.135427 (+129227us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b48001c0158. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:23:18.236234 (+100807us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:23:18.236279 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.236356 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.761617 (+525261us) raft_consensus.cc:1685] finished
0709 12:23:18.761646 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.761928 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":753902,"raft.queue_time_us":709299,"raft.run_cpu_time_us":2580,"raft.run_wall_time_us":130673,"spinlock_wait_cycles":12728576,"thread_start_us":222079,"threads_started":1,"wal-append.queue_time_us":745726}
I20260709 12:23:18.869318  4789 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.870223  5176 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:18.870880  5176 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:18.871753  5176 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.872210  4789 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:18.873170  5020 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.873847  5020 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.875162  5134 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.876950  5134 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:18.878100  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:18.838217  1845 leader_election.cc:336] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.659s (SENT)
I20260709 12:23:18.840279  1799 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:18.840539  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.881162  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.882726  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.883648  1806 raft_consensus.cc:2368] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9.
I20260709 12:23:18.841226  1805 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:18.848073  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.885480  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.886061  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.854882  1801 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:18.886732  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.859465  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:18.887179  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49015.6us
I20260709 12:23:18.852102  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:18.884380  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.888464  1806 raft_consensus.cc:2368] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8.
I20260709 12:23:18.888543  1771 leader_election.cc:400] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
I20260709 12:23:18.886363  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.856637  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:18.889832  1805 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:18.841441  5121 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.839617  4963 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.919776  1801 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 5. Current term is 7. Ops: []
I20260709 12:23:18.918730  1803 raft_consensus.cc:2368] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9.
I20260709 12:23:18.927513  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.928334  1806 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:18.928984  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72425.7us
I20260709 12:23:18.929183  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.930645  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.900707  1798 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:18.864037  5058 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:23:18.922439  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 70402.4us
I20260709 12:23:18.939116  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.920354  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.939663  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:18.913561  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.864833  5039 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:18.938674  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.944234  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.938235  1801 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:18.945806  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:18.924847  5017 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:18.949200  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 384874us
I20260709 12:23:18.951884  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:18.958316  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.928239  5163 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:18.934003  5152 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:18.961784  1798 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:23:18.936156  5021 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:18.962618  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:18.959326  1806 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:18.935845  5053 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:18.968317  1801 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:18.951190  4958 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.934151  4789 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:23:18.954392  4921 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:18.977712  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.978714  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:18.895614  5153 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:18.977002  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:18.953207  4987 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.990072  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.981348  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.959232  5190 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.952895  5169 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:18.987921  1729 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:18.994939  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:18.948786  4780 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:23:18.995710  1729 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:18.999490  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.971318  4770 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:18.989151  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.951586  5172 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:18.933177  5121 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.990414  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:18.971565  5021 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.932706  5159 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:18.966044  5058 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 12:23:18.971022  5163 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:19.021585  1845 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.924s (SENT)
I20260709 12:23:19.000154  1798 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:18.991748  4805 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:18.989638  5152 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:19.014885  4770 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:19.019053  5080 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.693s	user 0.011s	sys 0.001s
I20260709 12:23:19.055935  4805 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:19.056056  5080 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
I20260709 12:23:19.056448  4770 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.057420  1804 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:19.057318  4805 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.021788  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.022363  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.998766  5186 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:19.021145  5021 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:19.033602  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:19.070055  1873 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:19.028050  5076 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.037518  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:19.039113  4941 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:19.048002  5163 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:18.995698  5190 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.049368  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:19.049665  5189 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:23:18.998189  4921 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:19.130075  4921 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.030251  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:19.135844  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 114275us
W20260709 12:23:19.137468  1845 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.983s (SENT)
I20260709 12:23:19.138059  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:19.039753  5053 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.050691  5020 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:18.987164  4958 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.066463  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:18.992056  1801 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:19.079967  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.101085  1872 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:18.994342  4987 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.113134  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:19.130641  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:19.060330  1802 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:23:19.061234  5058 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:23:19.140319  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:19.068651  5186 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.141362  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:19.151255  1845 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.053s (SENT)
W20260709 12:23:19.200989  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49805us
I20260709 12:23:19.066314  5156 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:19.156250  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:19.141229  2415 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:19.204573  5156 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.194131  2411 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:19.196766  1873 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:19.196869  2412 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:19.196115  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:19.212741  1872 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:19.214854  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:19.217641  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:19.196086  5154 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.879s	user 0.011s	sys 0.000s
I20260709 12:23:19.155711  5191 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:19.157287  5163 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:19.209656  5186 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:19.155052  4941 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
W20260709 12:23:19.244107  5205 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.140s	user 0.000s	sys 0.004s
I20260709 12:23:19.240200  5138 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:19.232746  5152 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.236912  5153 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:19.234889  5154 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.253275  5115 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:19.253890  5154 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:19.254907  5017 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:23:19.345108  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 382254us
I20260709 12:23:19.118494  1729 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
W20260709 12:23:19.127647  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 175825us
I20260709 12:23:19.374928  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:19.536459  1875 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:19.537366  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:19.378384  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:19.539932  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:19.540824  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.541190  1729 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:19.388232  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:19.365526  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.411835  5089 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:19.265585  5144 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.547735  1802 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:23:19.262048  5174 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:19.271461  5190 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.549250  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:19.550539  1802 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:19.420233  5186 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:19.479177  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.275617  5156 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.538342  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.559260  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:19.527941  5057 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:19.486747  5076 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:19.528435  5176 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:19.530284  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:19.531842  4963 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.593s	user 0.019s	sys 0.016s
I20260709 12:23:19.532825  1878 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:19.373800  1872 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:19.190842  1806 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:19.552597  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:19.278306  5172 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:19.554271  5121 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.556823  2416 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 9. Preceding OpId from leader: term: 7 index: 9. (term mismatch)
W20260709 12:23:19.574841  5134 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.658s	user 0.013s	sys 0.012s
I20260709 12:23:19.574527  5154 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:19.575922  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.579258  5153 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.580128  4986 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:19.558204  4797 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:19.569231  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:19.581226  1803 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
W20260709 12:23:19.582175  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46568.7us
I20260709 12:23:19.582918  5021 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.583798  5186 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.584857  1876 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:19.585191  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:19.595187  5156 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.586256  5198 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:19.641601  5021 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:19.642966  4948 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:19.603093  1875 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:19.615983  2416 pending_rounds.cc:85] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 8
I20260709 12:23:19.646063  2416 pending_rounds.cc:107] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 6.9
I20260709 12:23:19.646622  2416 raft_consensus.cc:2889] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:19.621683  4986 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:19.651307  4986 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:19.641614  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.651160  4797 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.653851  5144 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.654744  5144 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.655431  5154 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.657930  5144 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:19.602362  4963 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.566586  5057 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:19.608642  5076 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.634429  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:19.635799  5153 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.636999  1873 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:19.638185  5215 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.105s	user 0.002s	sys 0.000s
I20260709 12:23:19.690382  4987 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:19.692890  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:19.693830  1805 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:19.697163  5076 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:19.698436  4805 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:19.699525  4805 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.700161  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:19.663206  1807 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:19.663544  1802 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:19.665094  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55635.1us
I20260709 12:23:19.662915  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:19.708405  1800 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:19.658449  5154 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:19.701345  1729 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:19.704877  1798 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:19.650607  4921 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:19.658655  5144 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.662499  5193 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.662254  5159 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:19.715260  4797 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.659580  5156 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.721692  5017 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.630147  5176 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.744014  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:19.745294  5153 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:19.661798  4804 consensus_queue.cc:1175] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" responder_term: 5 status { last_received { term: 5 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 5 index: 3 } }
I20260709 12:23:19.750422  5017 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.751348  4963 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.748010  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:19.748062  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 43761.8us
I20260709 12:23:19.764725  5057 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.767756  5138 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.768342  5115 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.750478  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:19.773324  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.722277  5160 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:23:19.823689  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101529us
I20260709 12:23:19.841651  5057 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:19.796586  4921 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.844547  4921 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:19.847854  5160 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.849645  5156 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:19.863168  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:19.864007  5176 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.864194  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:19.827397  5115 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.865360  1880 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:19.857779  1802 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:19.858208  5138 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:19.851158  4926 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:19.874655  4958 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:19.875303  4926 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.875857  4926 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:19.876299  5138 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.877135  5138 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:19.877588  5138 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:19.878046  5138 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:19.857758  5017 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.788017  5144 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:19.880861  5016 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.023s	user 0.015s	sys 0.016s
I20260709 12:23:19.881621  5144 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.883427  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:19.885226  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:19.850773  5160 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.886771  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.795001  5193 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:19.905741  5193 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:19.912428  4986 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:19.913712  4950 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:19.910051  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:19.924432  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.927964  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.928294  4986 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.929246  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.930254  1802 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:19.931093  4805 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:19.931861  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.949756  1802 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:23:19.982650  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.989531  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:19.991662  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:19.993993  4926 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.044966  5138 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.050480  4948 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:20.051151  4948 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.055348  4926 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:20.064728  5138 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:19.851368  1799 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:20.072486  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1637) took 1139 ms. Trace:
I20260709 12:23:20.072911  1799 rpcz_store.cc:276] 0709 12:23:18.932885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.938901 (+  6016us) service_pool.cc:224] Handling call
0709 12:23:20.072224 (+1133323us) spinlock_profiling.cc:243] Waited 422 ms on lock 0x7b5400133154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:20.072399 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":971308416}
I20260709 12:23:20.070338  1802 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:20.079988  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1647) took 1096 ms. Trace:
I20260709 12:23:20.081305  4948 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.080910  1799 rpcz_store.cc:276] 0709 12:23:18.983794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.074047 (+1090253us) service_pool.cc:224] Handling call
0709 12:23:20.074466 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.075750 (+  1284us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:20.076148 (+   398us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.079554 (+  3406us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.079638 (+    84us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:20.079687 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.079763 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.079919 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1615,"raft.queue_time_us":712,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":128,"thread_start_us":2287,"threads_started":1}
I20260709 12:23:20.084224  4948 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:20.108001  4788 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:20.110031  4899 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:20.122367  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 672) took 2316 ms. Trace:
I20260709 12:23:20.123046  2416 rpcz_store.cc:276] 0709 12:23:17.805391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.810728 (+  5337us) service_pool.cc:224] Handling call
0709 12:23:17.834925 (+ 24197us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.647480 (+1812555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.121930 (+474450us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b540049eb54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:20.122250 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3957852032}
I20260709 12:23:20.128767   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:20.152158  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:20.154281  1879 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:20.158033  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:20.166323  1807 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
W20260709 12:23:20.167769  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1620) took 1455 ms (client timeout 1716 ms). Trace:
W20260709 12:23:20.167904  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 465370us
I20260709 12:23:20.170775  4788 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:23:20.172642  4788 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:20.172230  1807 rpcz_store.cc:269] 0709 12:23:18.711847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.745696 (+ 33849us) service_pool.cc:224] Handling call
0709 12:23:19.550567 (+804871us) spinlock_profiling.cc:243] Waited 6.29 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:23:20.167643 (+617076us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14460160}
I20260709 12:23:20.175948  4788 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.186112  1805 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:20.193518  1807 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:20.196754  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.199676  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.199832  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.203011  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.205617  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:20.206430  1799 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:20.211308  4941 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.212265  4941 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.217703  4941 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:20.218497  1805 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:20.221292  1807 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260709 12:23:20.226344  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 399437us
I20260709 12:23:20.230019  4788 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:20.230942  4788 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.235191  4788 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:20.240967  4899 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:20.242249  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:20.252012  4899 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:20.252687  4899 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:20.253193  4899 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.258397  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:20.263973  5138 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:20.264499  5138 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:20.264950  5138 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:20.267447  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:20.268208  2416 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:20.270289  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:20.271392  2416 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:20.281467  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2513) took 1224 ms. Trace:
I20260709 12:23:20.282011  1873 rpcz_store.cc:276] 0709 12:23:19.055249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.208012 (+152763us) service_pool.cc:224] Handling call
0709 12:23:20.278620 (+1070608us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:20.279743 (+  1123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":362648832}
I20260709 12:23:20.260558  1799 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:20.246675  1807 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:20.296738  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.297501  1807 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
W20260709 12:23:20.299782  1806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.011s	sys 0.022s
W20260709 12:23:20.300117  1806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.011s	sys 0.022s
I20260709 12:23:20.300796  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1745) took 1364 ms. Trace:
I20260709 12:23:20.301683  4899 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.289677  5138 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:20.303670  1874 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.020s	sys 0.016s
W20260709 12:23:20.304005  1874 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.020s	sys 0.016s
I20260709 12:23:20.304800  2416 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:20.301126  1806 rpcz_store.cc:276] 0709 12:23:18.936635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.599799 (+663164us) service_pool.cc:224] Handling call
0709 12:23:19.635218 (+ 35419us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:19.714495 (+ 79277us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.714534 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:19.735317 (+ 20783us) log.cc:844] Serialized 52 byte log entry
0709 12:23:20.300452 (+565135us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:23:20.300492 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.300559 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.300607 (+    48us) raft_consensus.cc:1685] finished
0709 12:23:20.300617 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.300746 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33818,"raft.queue_time_us":59166,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":235,"thread_start_us":530833,"threads_started":1,"wal-append.queue_time_us":233851}
I20260709 12:23:20.316011  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.317087  1799 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:20.310488  5138 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:20.330125  1727 raft_consensus.cc:2368] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10.
I20260709 12:23:20.336920  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:20.362928  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.373421  4899 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:20.373526  4788 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:20.378932  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.379086  1807 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:20.379746  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:20.381781  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.382205  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:20.374416  4788 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.384219  1806 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:20.390286  4941 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:20.394340  1805 raft_consensus.cc:2410] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:23:20.403225  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:20.403221  4941 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:20.405745  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:20.405732  4941 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:20.407050  5159 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:20.407781  5159 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.408340  5159 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:20.405823  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.411090  4788 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:20.411648  1731 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:20.413813  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2632) took 1359 ms. Trace:
I20260709 12:23:20.414156  1731 rpcz_store.cc:276] 0709 12:23:19.052652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.405198 (+1352546us) service_pool.cc:224] Handling call
0709 12:23:20.412286 (+  7088us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:20.418277  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:20.423606  4941 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:20.426667  4941 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:20.428632  4941 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.431736  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.435271  5055 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:23:20.438169  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:20.439414  5055 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:20.439853  5055 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:20.440294  5055 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:20.443475  5240 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
W20260709 12:23:20.448480  1772 connection.cc:256] RPC call timeout handler was delayed by -2.0080290170003536s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:23:20.449358  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:20.450608  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:20.458263  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.451783  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.458896  1731 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:20.459455  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.453162  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.460376  1802 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 9 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:23:20.460705  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:20.461164  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.461288  1799 raft_consensus.cc:2376] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8. Re-sending same reply.
I20260709 12:23:20.455281  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:20.456223  1697 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.952s (SENT)
I20260709 12:23:20.459589  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.464299  1805 raft_consensus.cc:2368] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8.
I20260709 12:23:20.465368  5089 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:20.465726  4948 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:20.465982  5089 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:20.466864  4926 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:20.467574  4926 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:20.467491  5089 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.467928  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.468883  1729 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:20.469394  4926 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:20.459614  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1729) took 1495 ms (client timeout 1993 ms). Trace:
W20260709 12:23:20.471112  1731 rpcz_store.cc:269] 0709 12:23:18.964337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.427999 (+1463662us) service_pool.cc:224] Handling call
0709 12:23:20.459489 (+ 31490us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:20.471558  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:20.472323  2412 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:20.472579  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.472500  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.473570  1731 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
W20260709 12:23:20.474058  1723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2633) took 1421 ms (client timeout 1748 ms). Trace:
I20260709 12:23:20.474180  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2629) took 1423 ms. Trace:
I20260709 12:23:20.474766  1731 rpcz_store.cc:276] 0709 12:23:19.051759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.472262 (+1420503us) service_pool.cc:224] Handling call
0709 12:23:20.474103 (+  1841us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:20.474706  1723 rpcz_store.cc:269] 0709 12:23:19.052943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.653929 (+600986us) service_pool.cc:224] Handling call
0709 12:23:20.473978 (+820049us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:20.469486  1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2634) took 1416 ms (client timeout 1851 ms). Trace:
W20260709 12:23:20.484110  1729 rpcz_store.cc:269] 0709 12:23:19.053226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.733381 (+680155us) service_pool.cc:224] Handling call
0709 12:23:20.469422 (+736041us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:20.497192  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1742) took 1626 ms. Trace:
I20260709 12:23:20.497671  1804 rpcz_store.cc:276] 0709 12:23:18.864583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.558366 (+693783us) service_pool.cc:224] Handling call
0709 12:23:19.583292 (+ 24926us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.583904 (+   612us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.583952 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.584843 (+   891us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.627973 (+ 43130us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:19.628023 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.628194 (+   171us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.490223 (+862029us) raft_consensus.cc:1685] finished
0709 12:23:20.490248 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.490435 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1990,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":243,"thread_start_us":42215,"threads_started":1,"wal-append.queue_time_us":335161}
I20260709 12:23:20.498323  4941 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.501421  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:20.502133  1723 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.502444  4941 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:20.517154  1800 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.808s	user 0.010s	sys 0.006s
I20260709 12:23:20.519134  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.520866  1800 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:20.521535  1731 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:20.521648  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1747) took 1575 ms. Trace:
I20260709 12:23:20.522306  1800 rpcz_store.cc:276] 0709 12:23:18.946658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.707518 (+760860us) service_pool.cc:224] Handling call
0709 12:23:19.708106 (+   588us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.521396 (+813290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.521601 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:20.524434  5159 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.522329  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1740) took 1467 ms. Trace:
I20260709 12:23:20.527344  1731 rpcz_store.cc:276] 0709 12:23:19.054938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.475919 (+1420981us) service_pool.cc:224] Handling call
0709 12:23:20.522256 (+ 46337us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:20.535557  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.008s	sys 0.012s
W20260709 12:23:20.537819  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.009s	sys 0.013s
W20260709 12:23:20.538233  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 673065us
W20260709 12:23:20.550256  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 112253us
I20260709 12:23:20.556521  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.557484  1805 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:20.558751  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.560259  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:20.561857  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:20.562961  4899 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:23:20.553610  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66412.4us
I20260709 12:23:20.570088  1723 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
W20260709 12:23:20.570998  1723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2635) took 1517 ms (client timeout 2008 ms). Trace:
W20260709 12:23:20.571502  1723 rpcz_store.cc:269] 0709 12:23:19.053510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.496454 (+1442944us) service_pool.cc:224] Handling call
0709 12:23:20.570896 (+ 74442us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:20.574126  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:20.583714  1876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s	user 0.018s	sys 0.027s
W20260709 12:23:20.584317  1876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.019s	sys 0.028s
I20260709 12:23:20.578261  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:20.589320  1804 raft_consensus.cc:2368] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8.
I20260709 12:23:20.592403  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.599731  5159 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:20.601749  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8bb813e4fab14a61b810a24b56899586 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:23:20.613744  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2636) took 1559 ms (client timeout 2076 ms). Trace:
W20260709 12:23:20.614426  1731 rpcz_store.cc:269] 0709 12:23:19.053822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.537148 (+1483326us) service_pool.cc:224] Handling call
0709 12:23:20.613633 (+ 76485us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:20.631191  4666 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.632059  4666 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.632309  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.633338  1731 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
W20260709 12:23:20.633975  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1736) took 1667 ms (client timeout 2214 ms). Trace:
W20260709 12:23:20.635757  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 907589us
I20260709 12:23:20.636991  4943 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:20.637337  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:20.637673  4943 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:20.638970  4943 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.641070  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1741) took 1784 ms. Trace:
W20260709 12:23:20.634339  1731 rpcz_store.cc:269] 0709 12:23:18.966119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.615411 (+1649292us) service_pool.cc:224] Handling call
0709 12:23:20.633906 (+ 18495us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:20.641471  1801 rpcz_store.cc:276] 0709 12:23:18.856777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.546089 (+689312us) service_pool.cc:224] Handling call
0709 12:23:19.546764 (+   675us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.556394 (+  9630us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 12:23:19.556427 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.558019 (+  1592us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.611037 (+ 53018us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:23:19.611088 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.611203 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.640795 (+1029592us) raft_consensus.cc:1685] finished
0709 12:23:20.640815 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.640986 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1535,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":221,"thread_start_us":40252,"threads_started":1,"wal-append.queue_time_us":324889}
I20260709 12:23:20.629544  5089 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.651155  5250 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.651575  4666 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:20.659595  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82759.7us
I20260709 12:23:20.653301  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.668906  5249 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.674710  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 450) took 1009 ms. Trace:
I20260709 12:23:20.671787  5193 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:20.652293  5089 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:20.675915  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64777.9us
I20260709 12:23:20.677299  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2516) took 1565 ms. Trace:
I20260709 12:23:20.676085  1731 rpcz_store.cc:276] 0709 12:23:19.665628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.652950 (+987322us) service_pool.cc:224] Handling call
0709 12:23:20.674625 (+ 21675us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:20.687103  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.676697  5193 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:20.699851  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.700627  1731 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:20.707310  2416 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:20.686745  4987 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.711585  2383 leader_election.cc:400] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10.
I20260709 12:23:20.712340  2383 leader_election.cc:403] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:20.713044  1872 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.003s	user 0.020s	sys 0.057s
I20260709 12:23:20.680595  1874 rpcz_store.cc:276] 0709 12:23:19.111657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.571372 (+459715us) service_pool.cc:224] Handling call
0709 12:23:19.577215 (+  5843us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.578083 (+   868us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.578135 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.586346 (+  8211us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.723999 (+137653us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:20.304327 (+580328us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.676938 (+372611us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.677032 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.677081 (+    49us) raft_consensus.cc:1685] finished
0709 12:23:20.677096 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.677226 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":372327,"prepare.queue_time_us":191578,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":4271,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":240,"thread_start_us":693846,"threads_started":2,"wal-append.queue_time_us":328065}
W20260709 12:23:20.716522  1872 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.007s	user 0.020s	sys 0.057s
I20260709 12:23:20.661916  5250 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.703815  5193 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:20.726760  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:20.729022  2412 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:20.729729  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1727) took 1765 ms. Trace:
I20260709 12:23:20.730413  1731 rpcz_store.cc:276] 0709 12:23:18.963844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.706848 (+1743004us) service_pool.cc:224] Handling call
0709 12:23:20.729625 (+ 22777us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:20.735147  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47851.7us
I20260709 12:23:20.736456  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:20.738176  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:20.686340  5249 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.745404  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:20.717370  4989 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:20.752698  1731 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:20.747012  5055 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:20.756017  1723 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:20.741411  4943 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.756877  4943 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.759137  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:20.759657  4943 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:20.759907  1881 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:20.761152  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:20.761631  5055 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:20.762449  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:20.773017  1881 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:20.747480  4899 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:20.774294  1874 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:20.785048  1879 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:20.786145  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84056.5us
I20260709 12:23:20.790218  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:20.732438  4987 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.819530  5248 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:23:20.826223  4987 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:20.798059  1873 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:20.831490  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:20.792111  4989 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.832216  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:20.843686  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 206483us
I20260709 12:23:20.843618  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:20.786005  4788 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:23:20.846864  1881 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:20.836258  4989 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:20.850250  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:20.852461  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:20.853178  1881 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:20.849478  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2518) took 1700 ms. Trace:
I20260709 12:23:20.845078  5248 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:20.859764  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:20.855072  5246 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.190s	user 0.000s	sys 0.002s
I20260709 12:23:20.848511  5089 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:20.845477  4941 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.013s
I20260709 12:23:20.864755  1880 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:20.854674  1872 rpcz_store.cc:276] 0709 12:23:19.148686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.610376 (+461690us) service_pool.cc:224] Handling call
0709 12:23:19.680948 (+ 70572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.709502 (+ 28554us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:20.717974 (+1008472us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.845205 (+127231us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.845273 (+    68us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:20.849252 (+  3979us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.849315 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.849430 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":209354,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":8594,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":3737,"thread_start_us":1100091,"threads_started":2}
I20260709 12:23:20.865809  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:20.864337  5263 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:20.868991  5250 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:20.869343  5263 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:20.869901  5250 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:20.870353  5250 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.857183  5257 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:20.871856  5257 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.870687  5263 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.900046  4899 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10.
I20260709 12:23:20.903471  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:20.909374  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 133720us
I20260709 12:23:20.870714  5250 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:20.911173  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:20.912038  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 125118us
I20260709 12:23:20.899977  5257 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:20.916163  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.921175  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:20.922657  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 58636.1us
W20260709 12:23:20.925555  1845 connection.cc:256] RPC call timeout handler was delayed by -1.0456556700000874s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.31980000000000003s for a response
I20260709 12:23:20.863847  5258 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:23:20.919291  1873 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:20.951572  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:20.950681  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:20.926926  5263 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:20.953969  1872 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:20.958197  1724 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:20.958657  5055 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:20.975050  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 124470us
I20260709 12:23:20.975963  5089 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.976325  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:20.976594  5089 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:20.978163  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1635) took 2054 ms (client timeout 1765 ms). Trace:
I20260709 12:23:20.958520  5262 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:20.981736  5262 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:20.982317  5262 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.966248  5055 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:20.984808  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:20.978663  1803 rpcz_store.cc:269] 0709 12:23:18.923849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.928907 (+  5058us) service_pool.cc:224] Handling call
0709 12:23:20.977899 (+2048992us) spinlock_profiling.cc:243] Waited 640 ms on lock 0x7b54001eb4d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:20.978052 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1471487744}
I20260709 12:23:20.986070  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:20.986851  1873 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:20.985008  5055 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:20.992668  1881 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:20.999500  1879 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:20.965473  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:21.001328  1880 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:20.998268  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:21.004165  1724 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:21.004218  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.008008  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:21.015480  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.018080  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.018997  1724 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:21.021564  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.028333  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:21.030758  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54499.4us
I20260709 12:23:21.021564  5258 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:21.017272  5057 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:21.018184  5249 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:21.062306  1724 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:21.064460  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:21.031334  4989 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:21.036814  1872 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:21.064972  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 452) took 1398 ms. Trace:
I20260709 12:23:21.072527  1881 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:21.064009  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.035413  5258 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:21.077209  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.071642  1724 rpcz_store.cc:276] 0709 12:23:19.666278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.019663 (+1353385us) service_pool.cc:224] Handling call
0709 12:23:21.064903 (+ 45240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.078693  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.080113  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.061646  5089 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:21.081600  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 123462us
I20260709 12:23:21.089908  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.071146  4989 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.072105  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:21.099506  4989 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:21.100404  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68710.9us
I20260709 12:23:21.081473  4943 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:21.106171  1875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.306s	user 0.007s	sys 0.015s
I20260709 12:23:21.085354  1724 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:21.110481  1873 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:21.111485  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:21.087039  5053 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.058s
I20260709 12:23:21.087961  2416 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:21.088496  5273 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:21.088380  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.057457  5057 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.123916  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.092245  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.092875  4987 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:21.097289  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:21.036624  5261 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:21.098850  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:21.111654  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.118113  1880 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:21.086604  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:21.129784  2413 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:21.129879  4987 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:21.135913  1872 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:21.133451  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.135272  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:21.132321  4987 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:21.139951  1875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.346s	user 0.008s	sys 0.015s
I20260709 12:23:21.140410  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.065438  5248 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:21.151731  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.134037  5057 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:21.160827  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.123224  5053 consensus_queue.cc:1243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:21.065862  5256 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.113s
I20260709 12:23:21.165544  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1724) took 2202 ms. Trace:
I20260709 12:23:21.166078  1724 rpcz_store.cc:276] 0709 12:23:18.963046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.141471 (+2178425us) service_pool.cc:224] Handling call
0709 12:23:21.165471 (+ 24000us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.170640  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.173019  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:21.150343  5261 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:21.177696  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.181595  5191 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.182207  5191 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.153246  5274 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:21.164994  4921 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:21.189513  5144 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:21.191119  5272 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.192094  5272 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:21.193675  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2507) took 2161 ms (client timeout 1618 ms). Trace:
W20260709 12:23:21.194167  1878 rpcz_store.cc:269] 0709 12:23:19.032524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.048999 (+ 16475us) service_pool.cc:224] Handling call
0709 12:23:21.193383 (+2144384us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b5400073d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:21.193556 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":568602368}
I20260709 12:23:21.203560  5261 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.204499  4943 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:21.210083  2417 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:23:21.212975  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.241322  1724 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:21.246738  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 160211us
I20260709 12:23:21.248004  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:21.243137  1770 connection.cc:256] RPC call timeout handler was delayed by -1.0166841000000204s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
I20260709 12:23:21.249528  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.250412  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:21.251490  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:21.252053  1878 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:21.252323  1727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 444) took 2892 ms (client timeout 1937 ms). Trace:
W20260709 12:23:21.252728  1727 rpcz_store.cc:269] 0709 12:23:18.359688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.360254 (+   566us) service_pool.cc:224] Handling call
0709 12:23:21.252064 (+2891810us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54002d14d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:21.252237 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2842530944}
I20260709 12:23:21.254165  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.260109  1727 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
W20260709 12:23:21.256492  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 396785us
I20260709 12:23:21.262046  1873 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:21.255920  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:21.262392  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.253949  4989 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.264021  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:21.253003  5285 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:21.270678  5285 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.271086  5285 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:21.271003  4943 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:21.271363  4989 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:21.272672  1724 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:21.274173  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:21.274842  5274 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:21.275462  5274 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.275058  1724 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:21.278250  5144 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:21.279307  5144 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:21.279875  5144 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:21.280596  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:21.286686  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:21.148447  5258 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:21.292600  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 159236us
I20260709 12:23:21.311695  1878 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
W20260709 12:23:21.314836  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 150511us
I20260709 12:23:21.260303  4921 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:21.331308  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 216538us
I20260709 12:23:21.332288  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:21.333684  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:21.300349  5274 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:21.335433  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54903.7us
I20260709 12:23:21.339154  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:21.342862  5284 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.329406  5276 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:21.315173  5248 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:21.370206  1731 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.001s
I20260709 12:23:21.385417  1731 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
W20260709 12:23:21.386000  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 451) took 1720 ms (client timeout 2014 ms). Trace:
W20260709 12:23:21.386404  1731 rpcz_store.cc:269] 0709 12:23:19.665985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.737830 (+1071845us) service_pool.cc:224] Handling call
0709 12:23:21.385938 (+648108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.387429  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:21.370574  1728 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:21.313004  5191 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:21.372699  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.373920  1724 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 5.
I20260709 12:23:21.377341  1878 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:21.388859  1878 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:21.388937  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1737) took 2418 ms. Trace:
I20260709 12:23:21.389395  1724 rpcz_store.cc:276] 0709 12:23:18.970146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.273832 (+2303686us) service_pool.cc:224] Handling call
0709 12:23:21.388870 (+115038us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.390450  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:21.391387  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.408965  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2645) took 1152 ms. Trace:
I20260709 12:23:21.409711  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 460) took 1010 ms. Trace:
I20260709 12:23:21.384847  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:21.391201  1801 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:21.410092  1728 rpcz_store.cc:276] 0709 12:23:20.378123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.098375 (+720252us) service_pool.cc:224] Handling call
0709 12:23:21.388300 (+289925us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.412777  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:21.392241  1724 raft_consensus.cc:2368] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10.
I20260709 12:23:21.409472  1731 rpcz_store.cc:276] 0709 12:23:20.256786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.387185 (+1130399us) service_pool.cc:224] Handling call
0709 12:23:21.408887 (+ 21702us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.415594  5280 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:21.371268  5273 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:21.408399  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:21.423218  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:21.371908  5258 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:21.428730  4921 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.430934  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 764) took 1166 ms. Trace:
I20260709 12:23:21.432964  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.378022  5276 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:21.335078  5250 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.408862  5272 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:21.431631  2419 rpcz_store.cc:276] 0709 12:23:20.264084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.321088 (+1057004us) service_pool.cc:224] Handling call
0709 12:23:21.430830 (+109742us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.473811  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.470453  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:21.443177  1878 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.471685  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.468770  1798 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:21.475833  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:21.477773  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.412061  5191 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:21.481105  1728 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:21.477118  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:21.482647  1878 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:21.485759  5281 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20260709 12:23:21.486243  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 154006us
I20260709 12:23:21.486559  5253 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:21.488258  2383 leader_election.cc:400] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8.
I20260709 12:23:21.487890  5191 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.490242  5191 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:21.391932  5284 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.492699  5284 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:21.493350  5284 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:21.493906  5284 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:21.444904  4987 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.495219  5284 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.496611  4987 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:21.497524  1879 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:21.498983  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.499737  1728 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:21.499799  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:21.501842  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:21.504751  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.511297  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.472905  5276 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:21.522931  1723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1733) took 2557 ms (client timeout 2204 ms). Trace:
W20260709 12:23:21.523540  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50911.1us
I20260709 12:23:21.524499  5279 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:21.524893  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:21.526286  4989 raft_consensus.cc:2764] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:23:21.513092  5283 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.103s
I20260709 12:23:21.527715  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:21.513885  5271 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:21.528496  1874 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:21.529814  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:21.515547  5250 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:21.531360  5288 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:23:21.516607  5193 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.777s	user 0.008s	sys 0.003s
I20260709 12:23:21.483553  5285 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.532235  5193 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.534354  1880 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:21.480377  5262 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.534744  5285 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:21.534922  4788 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 12:23:21.547984  5285 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:21.535742  5278 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260709 12:23:21.537140  5270 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:21.549620  5270 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.551095  1876 raft_consensus.cc:1217] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Deduplicated request from leader. Original: 4.7->[5.8-6.9]   Dedup: 5.8->[6.9-6.9]
I20260709 12:23:21.551837  1876 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: [5.8-6.9]
I20260709 12:23:21.552950  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1828) took 1891 ms. Trace:
I20260709 12:23:21.537408  5286 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.536633  5271 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:21.553489  5191 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:23:21.537153  5193 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:21.554080  5271 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.555760  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.557245  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.558393  1724 raft_consensus.cc:2410] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 7 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:23:21.558894  5144 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.560155  4926 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:21.523465  1723 rpcz_store.cc:269] 0709 12:23:18.965335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.572496 (+1607161us) service_pool.cc:224] Handling call
0709 12:23:21.522615 (+950119us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b540030b454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:21.522837 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":373435136}
I20260709 12:23:21.533080  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.577364  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 454) took 1204 ms. Trace:
I20260709 12:23:21.578258  1724 rpcz_store.cc:276] 0709 12:23:20.372661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.556902 (+1184241us) service_pool.cc:224] Handling call
0709 12:23:21.577219 (+ 20317us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:21.586328  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82257.6us
I20260709 12:23:21.587256  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.590430  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:21.600467  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 187731us
I20260709 12:23:21.601115  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:21.602707  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:21.602514  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 72777.1us
I20260709 12:23:21.590440  1726 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
W20260709 12:23:21.604658  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 191958us
I20260709 12:23:21.554230  5191 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:21.553412  1877 rpcz_store.cc:276] 0709 12:23:19.661308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.154388 (+493080us) service_pool.cc:224] Handling call
0709 12:23:20.154939 (+   551us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:20.665152 (+510213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.665206 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:20.666188 (+   982us) log.cc:844] Serialized 52 byte log entry
0709 12:23:21.113021 (+446833us) spinlock_profiling.cc:243] Waited 5.47 ms on lock 0x7b1800482680. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:23:21.160136 (+ 47115us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:21.174849 (+ 14713us) log.cc:844] Serialized 14 byte log entry
0709 12:23:21.308824 (+133975us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.308997 (+   173us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.550493 (+241496us) raft_consensus.cc:1685] finished
0709 12:23:21.550609 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.552733 (+  2124us) inbound_call.cc:177] Queueing success response
0709 12:23:21.562494 (+  9761us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b1800482680. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
Metrics: {"mutex_wait_us":503547,"prepare.queue_time_us":14090,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":14834,"raft.queue_time_us":532124,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":3665,"spinlock_wait_cycles":21812352,"thread_start_us":179825,"threads_started":2,"wal-append.queue_time_us":646543}
I20260709 12:23:21.545028  5262 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:21.614634  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.618455  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:21.618619  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.613945  2382 leader_election.cc:400] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10.
I20260709 12:23:21.594172  5280 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:21.631330  1878 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:21.602169  5250 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:21.554082  5286 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.622815  5144 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:21.610826  5283 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.649595  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:21.613097  5289 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:21.655406  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 44446.1us
I20260709 12:23:21.659036  1723 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:21.632258  5159 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:21.661111  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.663832  1878 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:21.664005  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 448) took 3290 ms. Trace:
I20260709 12:23:21.664384  1723 rpcz_store.cc:276] 0709 12:23:18.369945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.608846 (+3238901us) service_pool.cc:224] Handling call
0709 12:23:21.658760 (+ 49914us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.659650 (+   890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.659903 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.664804  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.638432  5250 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.666123  1877 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:21.669940  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.670715  1877 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:21.671653  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:21.672148  1877 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:21.670040  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63376.1us
I20260709 12:23:21.678411  1726 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:23:21.678865  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:21.679512  1878 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:21.679582  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1725) took 2716 ms. Trace:
I20260709 12:23:21.679957  1726 rpcz_store.cc:276] 0709 12:23:18.963319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.609696 (+2646377us) service_pool.cc:224] Handling call
0709 12:23:21.675046 (+ 65350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:21.679264 (+  4218us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.679499 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.681310  1726 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:21.681911  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1732) took 2716 ms. Trace:
I20260709 12:23:21.678857  5191 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:21.682935  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.665791  5288 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:21.687731  5288 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.688784  4941 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:21.691041  5159 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:21.691759  5280 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.639223  1728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:21.696079  1728 raft_consensus.cc:606] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:21.696698  5274 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:21.696748  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40200 (request call id 28) took 1228 ms. Trace:
I20260709 12:23:21.700012  1728 rpcz_store.cc:276] 0709 12:23:20.468493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.604111 (+1135618us) service_pool.cc:224] Handling call
0709 12:23:21.696659 (+ 92548us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.682250  1726 rpcz_store.cc:276] 0709 12:23:18.965099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.680757 (+2715658us) service_pool.cc:224] Handling call
0709 12:23:21.681120 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.681664 (+   544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.681836 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.700788  1878 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:23:21.704505  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.705915  1878 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:23:21.677109  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.712226  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:21.708678  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.715359  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.668190  5250 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:21.675525  5144 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:21.668150  5271 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:21.722342  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:21.728659  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 122074us
W20260709 12:23:21.735112  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 235413us
I20260709 12:23:21.730571  1878 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:21.738060  2383 leader_election.cc:400] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8.
I20260709 12:23:21.743855  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:21.745195  1878 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:23:21.722038  5270 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:21.752555  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:21.725095  5248 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:21.726078  5089 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.737756  5144 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.757155  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:21.731493  5272 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:21.760915  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:21.660004  5283 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:21.764529  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.767933  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:21.768829  1801 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:21.741717  5271 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:21.765482  1798 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:21.771198  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:21.775262  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:21.725318  5249 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:23:21.779047  1803 raft_consensus.cc:2410] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:21.780757  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:21.758517  5248 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:21.818955  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63295.2us
W20260709 12:23:21.777505  5057 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.591s	user 0.012s	sys 0.004s
I20260709 12:23:21.781366  1804 raft_consensus.cc:2376] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7. Re-sending same reply.
W20260709 12:23:21.863072  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 91931.4us
I20260709 12:23:21.768616  5144 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:21.779167  5257 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.855s	user 0.007s	sys 0.010s
I20260709 12:23:21.874018  1877 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:21.777841  5271 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.768168  5272 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:21.880224  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.882184  1874 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
W20260709 12:23:21.883311  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44113.5us
W20260709 12:23:21.884888  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.877219  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 447) took 3507 ms. Trace:
I20260709 12:23:21.779534  5274 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:21.780452  5298 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:21.891575  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2527) took 1060 ms. Trace:
I20260709 12:23:21.892025  1877 rpcz_store.cc:276] 0709 12:23:20.832094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.876830 (+1044736us) service_pool.cc:224] Handling call
0709 12:23:21.877157 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.891240 (+ 14083us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:21.891268 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:21.891312 (+    44us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:21.891351 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.891407 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.891519 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.768127  5089 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.818313  5270 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:21.846464  5248 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:21.889285  5272 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:21.886097  1730 rpcz_store.cc:276] 0709 12:23:18.369298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.369444 (+   146us) service_pool.cc:224] Handling call
0709 12:23:18.369818 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.210138 (+1840320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.210206 (+    68us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.210253 (+    47us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:20.210304 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.876774 (+1666470us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b540044a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:21.876829 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.877025 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2412,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":11449,"spinlock_wait_cycles":4230933888}
I20260709 12:23:21.911108  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:21.888365  5271 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:21.918411  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2637) took 2319 ms. Trace:
I20260709 12:23:21.923395  1730 rpcz_store.cc:276] 0709 12:23:19.623344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.911396 (+2288052us) service_pool.cc:224] Handling call
0709 12:23:21.913376 (+  1980us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.913561 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:21.913585 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:21.913622 (+    37us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:21.913656 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.913705 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.913818 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.914999  1877 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:21.921806  1874 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:21.922061  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56333.8us
I20260709 12:23:21.929147  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2528) took 1096 ms. Trace:
I20260709 12:23:21.929683  1874 rpcz_store.cc:276] 0709 12:23:20.838766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.896622 (+1057856us) service_pool.cc:224] Handling call
0709 12:23:21.916597 (+ 19975us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.928779 (+ 12182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.929068 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.779059  5276 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.934278  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2529) took 1074 ms. Trace:
I20260709 12:23:21.934875  1877 rpcz_store.cc:276] 0709 12:23:20.851026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.893128 (+1042102us) service_pool.cc:224] Handling call
0709 12:23:21.914764 (+ 21636us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.924908 (+ 10144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.925174 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:21.867676  5057 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:21.944275  5294 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:23:21.944988  5299 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:21.939203  5298 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:21.948652  5276 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:21.949240  5276 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.949745  5276 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.951128  5286 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:21.926700  5270 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:21.952890  5295 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 12:23:21.953891  5270 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:21.954367  5290 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.222s	user 0.001s	sys 0.000s
W20260709 12:23:21.956230  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.973307  1878 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:21.927362  5291 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:21.956416  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:21.879098  5257 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.888150  5191 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
W20260709 12:23:21.987967  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46885.2us
I20260709 12:23:21.996450  1878 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260709 12:23:21.987287  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:21.995615  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.005313  1881 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:21.982864  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 230357us
I20260709 12:23:21.945886  5299 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:22.019694  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:22.022163  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:22.024277  1727 raft_consensus.cc:2368] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10.
I20260709 12:23:22.026886  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:22.028939  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:22.031772  1730 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:22.031909  1804 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:22.032466  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1741) took 1827 ms. Trace:
I20260709 12:23:22.032850  1730 rpcz_store.cc:276] 0709 12:23:20.205136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.028806 (+1823670us) service_pool.cc:224] Handling call
0709 12:23:22.031613 (+  2807us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.032230 (+   617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.032413 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:22.033912  5273 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:21.947959  5298 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:21.997454  5291 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:22.034094  1730 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:23:22.051002  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:22.053192  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.074332  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:22.043355  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.052630  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1835) took 1297 ms. Trace:
I20260709 12:23:22.057070  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:22.083626  1724 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:23:22.084548  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1743) took 1878 ms. Trace:
I20260709 12:23:22.084942  1724 rpcz_store.cc:276] 0709 12:23:20.205794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.030632 (+1824838us) service_pool.cc:224] Handling call
0709 12:23:22.072082 (+ 41450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.084190 (+ 12108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.084442 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:22.054567  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:22.086884  1802 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:22.152024  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2640) took 1901 ms. Trace:
I20260709 12:23:22.081156  1879 rpcz_store.cc:276] 0709 12:23:20.755283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.161883 (+406600us) service_pool.cc:224] Handling call
0709 12:23:22.052362 (+890479us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b5400026f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:22.052512 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":749713792}
I20260709 12:23:22.152946  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:22.146194  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2519) took 2576 ms. Trace:
I20260709 12:23:22.087330  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:22.085201  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63989.3us
I20260709 12:23:22.195351  1724 rpcz_store.cc:276] 0709 12:23:20.251651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.105665 (+1854014us) service_pool.cc:224] Handling call
0709 12:23:22.118872 (+ 13207us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.142535 (+ 23663us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.142561 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:22.151632 (+  9071us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:22.151690 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.151748 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.151959 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:22.199046  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:22.123759  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:22.079981  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2639) took 1829 ms. Trace:
I20260709 12:23:22.129278  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:22.101428  1772 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.140.126:43371, user_credentials={real_user=slave}} blocked reactor thread for 70190.8us
I20260709 12:23:22.211197  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1734) took 3205 ms. Trace:
W20260709 12:23:22.126093  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43818.9us
I20260709 12:23:22.221262  1730 rpcz_store.cc:276] 0709 12:23:20.250468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.033575 (+1783107us) service_pool.cc:224] Handling call
0709 12:23:22.033900 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.079705 (+ 45805us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.079926 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:22.050489  5254 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.208s	user 0.007s	sys 0.011s
I20260709 12:23:22.197259  1875 rpcz_store.cc:276] 0709 12:23:19.553904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.644927 (+ 91023us) service_pool.cc:224] Handling call
0709 12:23:19.645565 (+   638us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.698093 (+ 52528us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:19.732620 (+ 34527us) log.cc:844] Serialized 14 byte log entry
0709 12:23:21.437208 (+1704588us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:21.473376 (+ 36168us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:21.478632 (+  5256us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.126720 (+648088us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b5400222754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:22.126772 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.129528 (+  2756us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12410,"prepare.queue_time_us":220400,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":63,"raft.queue_time_us":56441,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"spinlock_wait_cycles":49249024,"thread_start_us":1661299,"threads_started":3,"wal-append.queue_time_us":241124,"wal-append.run_cpu_time_us":1949,"wal-append.run_wall_time_us":1033525}
W20260709 12:23:22.037607  5268 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.967s	user 0.004s	sys 0.008s
I20260709 12:23:22.230654  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:22.232205  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:22.233611  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:22.035436  5302 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
W20260709 12:23:22.056479  5304 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.127s	user 0.001s	sys 0.001s
I20260709 12:23:22.227679  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:22.154574  4926 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.594s	user 0.015s	sys 0.004s
I20260709 12:23:22.235981  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:22.219475  1798 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:22.069414  5312 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:22.040848  5283 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:22.242100  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:22.220018  1726 rpcz_store.cc:276] 0709 12:23:18.965607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.701215 (+2735608us) service_pool.cc:224] Handling call
0709 12:23:21.752249 (+ 51034us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:21.906259 (+154010us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:21.906301 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:21.918732 (+ 12431us) log.cc:844] Serialized 26 byte log entry
0709 12:23:21.982445 (+ 63713us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:21.982499 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.982567 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:22.167429 (+184862us) raft_consensus.cc:1685] finished
0709 12:23:22.167471 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.171323 (+  3852us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":132287,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":4365,"thread_start_us":198933,"threads_started":2,"wal-append.queue_time_us":147708}
I20260709 12:23:22.057789  5276 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:22.082084  5273 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:22.035596  4770 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:22.167171  5057 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:22.109925  4948 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.128168  5115 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:22.251205  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:22.144024  5089 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:22.263334  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:22.042531  5299 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:22.262918  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.262151  1872 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:22.306104  1801 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:22.269232  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:22.306838  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:22.268813  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:22.308341  1879 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:22.309934  1879 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:22.310343  1879 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:22.311872  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 370669us
I20260709 12:23:22.246201  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:22.314738  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:22.242100  5249 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:22.307202  1725 raft_consensus.cc:2376] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9. Re-sending same reply.
I20260709 12:23:22.324807  1807 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:22.247417  5274 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:22.311286  1879 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:22.316704  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70606.5us
I20260709 12:23:22.243826  5310 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:22.337893  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 769) took 1875 ms. Trace:
I20260709 12:23:22.338459  1875 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:22.317832  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.348886  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:22.318557  5191 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.326606  5283 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:22.351733  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:22.315474  4770 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:22.352643  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:22.356259  1801 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:22.366252  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:22.337245  5249 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:22.318248  4948 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:22.257540  5312 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:22.334195  5274 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:22.325770  5291 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:22.375217  1879 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:22.422029  2419 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
W20260709 12:23:22.533286  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:22.485280  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.423328  2412 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: [8.8-11.9]
I20260709 12:23:22.338394  2415 rpcz_store.cc:276] 0709 12:23:20.368144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.615948 (+247804us) service_pool.cc:224] Handling call
0709 12:23:20.639125 (+ 23177us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.639546 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:20.673270 (+ 33724us) log.cc:844] Serialized 14 byte log entry
0709 12:23:22.242723 (+1569453us) spinlock_profiling.cc:243] Waited 959 us on lock 0x7b18003b0a40. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:23:22.242957 (+   234us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:22.243019 (+    62us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:22.243081 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.243175 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.243304 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3599,"prepare.queue_time_us":86034,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":4639,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":63004,"spinlock_wait_cycles":28973696,"thread_start_us":82235,"threads_started":2,"wal-append.queue_time_us":105936}
W20260709 12:23:22.514485  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.734584  5313 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:22.705209  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:22.528741  1875 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:22.820055  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2535) took 1268 ms. Trace:
I20260709 12:23:22.823504  1875 rpcz_store.cc:276] 0709 12:23:21.551564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.371873 (+820309us) service_pool.cc:224] Handling call
0709 12:23:22.504339 (+132466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.819744 (+315405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.819990 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:22.423089  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 74317.4us
W20260709 12:23:22.822357  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 497006us
W20260709 12:23:22.804602  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 438427us
W20260709 12:23:22.503826  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 77703us
I20260709 12:23:22.839931  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:22.493731  5057 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.379330  5296 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:22.366971  5055 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 6, Time since last communication: 0.477s
W20260709 12:23:22.689180  5271 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.751s	user 0.014s	sys 0.000s
W20260709 12:23:22.568697  5250 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.824s	user 0.015s	sys 0.000s
W20260709 12:23:22.900076  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:22.583643  5272 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.663s	user 0.011s	sys 0.000s
W20260709 12:23:22.911652  1726 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.604s	user 0.012s	sys 0.000s
I20260709 12:23:22.945606  1726 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:22.371978  5306 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.525s
I20260709 12:23:22.944995  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:22.369243  5191 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:22.653182  5283 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:22.966077  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:22.928398  1879 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260709 12:23:22.969019  2414 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:22.969524  1879 pending_rounds.cc:85] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 4
I20260709 12:23:22.970171  1879 pending_rounds.cc:107] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 12:23:22.970608  1879 raft_consensus.cc:2889] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:22.967593  2415 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:23:22.911324  5317 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
W20260709 12:23:22.971675  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 131834us
I20260709 12:23:22.975498  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:22.976907  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:22.938040  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:22.949527  5288 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:22.984663  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:22.982141  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1855) took 1565 ms. Trace:
I20260709 12:23:22.965770  5055 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:22.987250  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:22.991722  1875 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:23:22.991562  1879 rpcz_store.cc:276] 0709 12:23:21.416633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.309266 (+892633us) service_pool.cc:224] Handling call
0709 12:23:22.309678 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.981824 (+672146us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.982072 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:22.901984  5057 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:22.990348  5248 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.069s	user 0.011s	sys 0.002s
I20260709 12:23:23.017897  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1730) took 4053 ms. Trace:
W20260709 12:23:23.026965  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1836) took 2271 ms (client timeout 2955 ms). Trace:
I20260709 12:23:23.027681  1877 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:23.027750  4948 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:23.028203  5305 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:23.028719  5138 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:22.819090  4770 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.029676  5249 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:23.030303  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 447) took 2474 ms (client timeout 1790 ms). Trace:
I20260709 12:23:23.032135  5191 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:23.034515  2414 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:22.903880  5296 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:23:22.980090  5251 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.021s	sys 0.046s
W20260709 12:23:23.107815  5251 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.022s	sys 0.053s
W20260709 12:23:23.048519  1801 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.676s	user 0.010s	sys 0.000s
W20260709 12:23:23.051786  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.109159  1801 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:23.049661  1731 rpcz_store.cc:276] 0709 12:23:18.964588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.611823 (+2647235us) service_pool.cc:224] Handling call
0709 12:23:21.654968 (+ 43145us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.686885 (+ 31917us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:21.686923 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:21.686973 (+    50us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:21.687026 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.017483 (+1330457us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b54004362d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:23.017529 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.017702 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48315776}
W20260709 12:23:23.111074  5320 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.248s	user 0.000s	sys 0.005s
W20260709 12:23:23.111770  5028 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 12:23:23.112615  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2641) took 2860 ms. Trace:
I20260709 12:23:23.060526  5138 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:23:23.113758  1800 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.006s	sys 0.006s
W20260709 12:23:23.114205  1800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.006s	sys 0.006s
W20260709 12:23:23.029973  5308 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.987s	user 0.003s	sys 0.000s
W20260709 12:23:23.050071  1880 rpcz_store.cc:269] 0709 12:23:20.755697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.138746 (+383049us) service_pool.cc:224] Handling call
0709 12:23:23.019335 (+1880589us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b540021b1d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:23.026887 (+  7552us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":876478720}
W20260709 12:23:23.120435  5144 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.226s	user 0.013s	sys 0.000s
I20260709 12:23:23.121076  5144 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.060226  5305 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.122035  5305 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:23.122494  2410 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 774) took 2084 ms (client timeout 2002 ms). Trace:
W20260709 12:23:23.123070  2410 rpcz_store.cc:269] 0709 12:23:21.037943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.122101 (+2084158us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:23.122248 (+   147us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:23:23.049885  5303 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.052s	user 0.000s	sys 0.006s
I20260709 12:23:23.043625  5055 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:23.124432  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.125515  5055 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.127405  5055 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:22.981096  5271 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:23.094216  1807 rpcz_store.cc:269] 0709 12:23:20.555604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.556136 (+   532us) service_pool.cc:224] Handling call
0709 12:23:23.023631 (+2467495us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b540004f6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:23.030172 (+  6541us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3877579520}
I20260709 12:23:22.968314  5272 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.098642  1879 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:23.133055  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 152140us
I20260709 12:23:23.133469  1879 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:23.062896  4948 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.100936  5191 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:23.101675  5333 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:23:23.101794  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.104184  5296 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:23.152091  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.151633  1879 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.153123  1879 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:23.104667  5283 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.106159  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:23.106833  5337 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:23.061357  5248 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.110330  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:23.113049  1724 rpcz_store.cc:276] 0709 12:23:20.252033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.325400 (+2073367us) service_pool.cc:224] Handling call
0709 12:23:22.335931 (+ 10531us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.422457 (+ 86526us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.422487 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:22.844358 (+421871us) log.cc:844] Serialized 26 byte log entry
0709 12:23:23.055380 (+211022us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:23.055443 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.055553 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.112357 (+ 56804us) raft_consensus.cc:1685] finished
0709 12:23:23.112377 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.112525 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18816,"raft.queue_time_us":69894,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":16156,"thread_start_us":107239,"threads_started":1,"wal-append.queue_time_us":200079}
I20260709 12:23:23.111869  1731 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:23.160904  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2644) took 2904 ms. Trace:
I20260709 12:23:23.161362  1731 rpcz_store.cc:276] 0709 12:23:20.256408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.111292 (+2854884us) service_pool.cc:224] Handling call
0709 12:23:23.111671 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.160612 (+ 48941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.160845 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:23.113250  5334 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
W20260709 12:23:23.119148  5323 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:23:23.121343  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2631) took 4069 ms. Trace:
I20260709 12:23:23.126086  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.092528  1872 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:23.054731  5276 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:23.136088  1770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.867s (SENT) [suppressed 57 similar messages]
I20260709 12:23:23.132001  5271 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:22.986075  5250 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.090932  5258 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:23:23.142910  2417 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.006s	sys 0.016s
W20260709 12:23:23.181190  2417 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.006s	sys 0.016s
W20260709 12:23:23.155371  1725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.005s	sys 0.020s
I20260709 12:23:23.112653  5028 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:23.157795  1803 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:23.158471  1806 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
W20260709 12:23:23.104184  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1841) took 2347 ms (client timeout 2506 ms). Trace:
I20260709 12:23:23.166079  1723 rpcz_store.cc:276] 0709 12:23:19.052386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.720780 (+2668394us) service_pool.cc:224] Handling call
0709 12:23:21.761743 (+ 40963us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:21.864845 (+103102us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:21.864877 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:21.869752 (+  4875us) log.cc:844] Serialized 26 byte log entry
0709 12:23:21.914250 (+ 44498us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:21.914282 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.128089 (+213807us) spinlock_profiling.cc:243] Waited 76.2 ms on lock 0x7b54002d19d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:22.128155 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.029257 (+901102us) spinlock_profiling.cc:243] Waited 3.94 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:23.121061 (+ 91804us) raft_consensus.cc:1685] finished
0709 12:23:23.121090 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.121265 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":241,"raft.queue_time_us":55438,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":8378,"spinlock_wait_cycles":184350848,"thread_start_us":37069,"threads_started":1,"wal-append.queue_time_us":126956}
I20260709 12:23:23.176847  1807 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:23.098232  5329 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.074s
W20260709 12:23:23.179417  1878 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.004s	sys 0.023s
W20260709 12:23:23.182036  1725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.006s	sys 0.021s
W20260709 12:23:23.182513  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 197983us
I20260709 12:23:23.224073  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:23.185771  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49710.8us
W20260709 12:23:23.186488  1873 rpcz_store.cc:269] 0709 12:23:20.757229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.111348 (+354119us) service_pool.cc:224] Handling call
0709 12:23:23.104012 (+1992664us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5400223b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:23.104115 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":301859200}
I20260709 12:23:23.178256  1731 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:23:23.223261  1878 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.005s	sys 0.024s
W20260709 12:23:23.225975  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 190770us
W20260709 12:23:23.226692  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 62837.5us
I20260709 12:23:23.227046  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1853) took 1810 ms. Trace:
I20260709 12:23:23.228236  4899 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.227402  1878 rpcz_store.cc:276] 0709 12:23:21.416148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.024058 (+607910us) service_pool.cc:224] Handling call
0709 12:23:22.058475 (+ 34417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.082485 (+ 24010us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:22.327676 (+245191us) log.cc:844] Serialized 14 byte log entry
0709 12:23:23.226562 (+898886us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.226617 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:23.226743 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.226845 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.226989 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55427,"prepare.queue_time_us":321988,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":659871,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":2532,"thread_start_us":964236,"threads_started":3,"wal-append.queue_time_us":479545}
I20260709 12:23:23.229082  4899 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.224339  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2642) took 2971 ms. Trace:
I20260709 12:23:23.240510  1730 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:23.240767  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 459) took 2863 ms. Trace:
I20260709 12:23:23.241405  1731 rpcz_store.cc:276] 0709 12:23:20.377784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.162289 (+2784505us) service_pool.cc:224] Handling call
0709 12:23:23.177929 (+ 15640us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.225849 (+ 47920us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.240697 (+ 14848us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.243800  2414 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:23.239319  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 457) took 2862 ms. Trace:
W20260709 12:23:23.257077  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.194499  5055 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:23.279699  5263 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:23.280350  5263 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.238538  1725 rpcz_store.cc:276] 0709 12:23:20.252523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.325641 (+2073118us) service_pool.cc:224] Handling call
0709 12:23:22.335236 (+  9595us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.351295 (+ 16059us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:22.508477 (+157182us) log.cc:844] Serialized 14 byte log entry
0709 12:23:23.223885 (+715408us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.223945 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:23.223997 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.224075 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.224262 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":126554,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":29075,"thread_start_us":632538,"threads_started":1,"wal-append.queue_time_us":671187}
I20260709 12:23:23.245249  1724 rpcz_store.cc:276] 0709 12:23:20.377178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.160370 (+2783192us) service_pool.cc:224] Handling call
0709 12:23:23.183138 (+ 22768us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.195304 (+ 12166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.195349 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.195418 (+    69us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:23.239054 (+ 43636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.239140 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.239244 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.142160  4948 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.317086  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1775) took 1175 ms. Trace:
I20260709 12:23:23.320180  5354 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.320907  5354 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.322886  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.323925  2417 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
W20260709 12:23:23.324173  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:23.325181  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 96884.8us
I20260709 12:23:23.325125  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:23.155803  5191 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.238652  5306 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:23.330549  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:23.341302  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 499708us
I20260709 12:23:23.344195  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 455) took 1030 ms. Trace:
I20260709 12:23:23.354542  1804 rpcz_store.cc:276] 0709 12:23:22.313887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.665653 (+351766us) service_pool.cc:224] Handling call
0709 12:23:22.666278 (+   625us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.666665 (+   387us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.666704 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:22.667462 (+   758us) log.cc:844] Serialized 26 byte log entry
0709 12:23:22.981607 (+314145us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:22.981668 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.981789 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.343893 (+362104us) raft_consensus.cc:1685] finished
0709 12:23:23.343919 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.344111 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":313540,"threads_started":1,"wal-append.queue_time_us":367996}
I20260709 12:23:23.317646  1805 rpcz_store.cc:276] 0709 12:23:22.142353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.151621 (+  9268us) service_pool.cc:224] Handling call
0709 12:23:22.152010 (+   389us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:22.152477 (+   467us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.152519 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:22.202701 (+ 50182us) log.cc:844] Serialized 52 byte log entry
0709 12:23:22.707572 (+504871us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:22.707657 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.707781 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.316738 (+608957us) raft_consensus.cc:1685] finished
0709 12:23:23.316759 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.317022 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2527,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":285338,"threads_started":1,"wal-append.queue_time_us":760594}
W20260709 12:23:23.367126  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84813.6us
W20260709 12:23:23.352633  2418 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 763) took 3066 ms (client timeout 2028 ms). Trace:
I20260709 12:23:23.206331  5298 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.110s
I20260709 12:23:23.372968  1731 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:23.180408  5276 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.381831  1879 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 12:23:23.371701  2418 rpcz_store.cc:269] 0709 12:23:20.263732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.316066 (+3052334us) service_pool.cc:211] Skipping call since client already timed out
0709 12:23:23.329572 (+ 13506us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:23:23.170104  5337 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.400173  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:23.390806  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.398159  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1860) took 1662 ms. Trace:
I20260709 12:23:23.401546  1879 rpcz_store.cc:276] 0709 12:23:21.735479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.044650 (+1309171us) service_pool.cc:224] Handling call
0709 12:23:23.045069 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.397627 (+352558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.398030 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.403263  1879 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:23.426009  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1862) took 1689 ms. Trace:
I20260709 12:23:23.429368  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2653) took 2145 ms. Trace:
I20260709 12:23:23.429996  1731 rpcz_store.cc:276] 0709 12:23:21.251356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.242556 (+1991200us) service_pool.cc:224] Handling call
0709 12:23:23.310202 (+ 67646us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.396645 (+ 86443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.396881 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.395642  5329 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.432051  5329 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.432418  5353 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:23.434248  5288 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:23.432067  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:23.436775  1803 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:23.438678  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:23.440506  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2542) took 1284 ms. Trace:
I20260709 12:23:23.441497  1879 rpcz_store.cc:276] 0709 12:23:22.119279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.402564 (+1283285us) service_pool.cc:224] Handling call
0709 12:23:23.403024 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.403833 (+   809us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.404075 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.426568  1880 rpcz_store.cc:276] 0709 12:23:21.736213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.119776 (+1383563us) service_pool.cc:224] Handling call
0709 12:23:23.120234 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.120506 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:23.339760 (+219254us) log.cc:844] Serialized 14 byte log entry
0709 12:23:23.425578 (+ 85818us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.425642 (+    64us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:23.425696 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.425776 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.425944 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58618,"thread_start_us":235871,"threads_started":2}
I20260709 12:23:23.391296  4899 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:23.437178  5263 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:23.485719  1798 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:23.451047  5313 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.639s	user 0.006s	sys 0.006s
I20260709 12:23:23.339838  5341 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.146s
W20260709 12:23:23.468623  5273 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.162s	user 0.010s	sys 0.002s
I20260709 12:23:23.352527  5271 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
I20260709 12:23:23.496459  5348 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
I20260709 12:23:23.498447  5253 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:23.503968  5343 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:23:23.504566  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2537) took 1915 ms. Trace:
I20260709 12:23:23.506119  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:23.506322  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 81645.9us
I20260709 12:23:23.506407  5313 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:23.508121  1724 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:23.311955  5138 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.839059  5138 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:23.840937  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1864) took 1829 ms. Trace:
I20260709 12:23:23.516600  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:23.394028  5258 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:23:23.843726  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 327201us
I20260709 12:23:23.844017  5258 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:23.538735  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 295841us
I20260709 12:23:23.848150  5336 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:23.485189  5288 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.848671  5336 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:23.849084  5288 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.849694  5336 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.851117  5368 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:23.851627  5288 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:23.548462  1875 rpcz_store.cc:276] 0709 12:23:21.589211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.881225 (+1292014us) service_pool.cc:224] Handling call
0709 12:23:22.921548 (+ 40323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.043987 (+122439us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.504237 (+460250us) spinlock_profiling.cc:243] Waited 66.4 ms on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:23.504492 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":152636160}
I20260709 12:23:23.557875  5342 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:23.863448  5342 consensus_queue.cc:1243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:23.864096  5342 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:23.864449  5342 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:23.507325  2414 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:23.867362  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.559018  2418 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:23:23.566357  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2656) took 2314 ms. Trace:
I20260709 12:23:23.516685  5274 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:23.547811  5253 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:23.871657  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.871999  5358 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.872648  5358 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.875905  5191 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:23.877580  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.878374  2414 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
W20260709 12:23:23.576427  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 222010us
I20260709 12:23:23.878638  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.879937  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.579285  1723 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:23.555665  5272 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:23.521090  5273 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.880968  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:23.883121  2418 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:23:23.883350  2419 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:23.883119  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:23.883886  5273 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:23.608886  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 91111.4us
I20260709 12:23:23.885027  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:23.885105  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.885262  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:23.885947  2419 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:23.879452  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.555927  5345 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
W20260709 12:23:23.587275  5328 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:23:23.887023  5262 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:23.607793  5349 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:23.888245  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.888622  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.889113  2419 raft_consensus.cc:2410] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:23:23.625648  5356 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.087s	user 0.000s	sys 0.005s
I20260709 12:23:23.890002  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.628563  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1863) took 1624 ms. Trace:
I20260709 12:23:23.629458  5276 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:23.630046  5337 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:23.588490  5329 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:23.641901  5255 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 2.804s	user 0.017s	sys 0.008s
I20260709 12:23:23.618079  5248 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:23.645912  5250 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:23.647095  5365 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:23.892890  5365 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.654471  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1770) took 1581 ms. Trace:
I20260709 12:23:23.890679  1873 rpcz_store.cc:276] 0709 12:23:22.004264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.225470 (+1221206us) service_pool.cc:224] Handling call
0709 12:23:23.310762 (+ 85292us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:23.390755 (+ 79993us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:23.390828 (+    73us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.425802 (+ 34974us) log.cc:844] Serialized 26 byte log entry
0709 12:23:23.564406 (+138604us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:23.564466 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.564585 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.628329 (+ 63744us) raft_consensus.cc:1685] finished
0709 12:23:23.628349 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.628491 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35632,"raft.queue_time_us":107767,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":49795,"thread_start_us":84709,"threads_started":1,"wal-append.queue_time_us":169435}
I20260709 12:23:23.655021  5341 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:23.893839  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.895179  5341 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.897671  5341 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:23.488974  4770 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:23:23.610949  1770 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.564s (SENT)
W20260709 12:23:23.974114  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 363188us
W20260709 12:23:23.974449  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:23.977559  1770 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.641s (SENT)
I20260709 12:23:23.980049  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:23.980841  2410 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:23.664106  5281 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:23.666548  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2536) took 2114 ms. Trace:
I20260709 12:23:23.668367  2411 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:23.668910  5305 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.000s
W20260709 12:23:23.669297  1727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2646) took 3412 ms (client timeout 2175 ms). Trace:
W20260709 12:23:23.983491  1727 rpcz_store.cc:269] 0709 12:23:20.257108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.262042 (+1004934us) service_pool.cc:224] Handling call
0709 12:23:23.669062 (+2407020us) spinlock_profiling.cc:243] Waited 756 ms on lock 0x7b540044af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:23.669207 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1738178816}
W20260709 12:23:23.671537  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2666) took 1554 ms (client timeout 1851 ms). Trace:
W20260709 12:23:23.671927  5296 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.507s	user 0.010s	sys 0.000s
I20260709 12:23:23.672338  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 438) took 2980 ms. Trace:
I20260709 12:23:23.674559  2417 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
W20260709 12:23:23.682207  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 282063us
W20260709 12:23:23.683525  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 96556.2us
W20260709 12:23:23.684082  5310 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.094s	user 0.007s	sys 0.005s
I20260709 12:23:23.701797  5144 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:23.987759  5310 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.405s	user 0.008s	sys 0.005s
I20260709 12:23:23.729599  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1783) took 1363 ms. Trace:
I20260709 12:23:23.988094  5310 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:23.988276  1799 rpcz_store.cc:276] 0709 12:23:22.366353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.733757 (+367404us) service_pool.cc:224] Handling call
0709 12:23:22.836992 (+103235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.877948 (+ 40956us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.877983 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:22.878030 (+    47us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:22.878072 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.729327 (+851255us) spinlock_profiling.cc:243] Waited 40.7 ms on lock 0x7b54002f2e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:23.729376 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.729526 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93637504}
I20260709 12:23:23.733799  4666 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:23.989140  5310 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.744567  1731 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:23.990891  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2657) took 2738 ms. Trace:
I20260709 12:23:23.991227  1731 rpcz_store.cc:276] 0709 12:23:21.252724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.431045 (+2178321us) service_pool.cc:224] Handling call
0709 12:23:23.743928 (+312883us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:23:23.744179 (+   251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.990581 (+246402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.990832 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2678016}
I20260709 12:23:23.986229  5305 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:23.992681  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:23.747247  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:23.994269  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.784572  5291 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.814565  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:23.491633  4899 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:23.832304  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:23.480453  5057 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:23.833525  5354 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:23.868113  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.868798  1724 rpcz_store.cc:276] 0709 12:23:21.252334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.315403 (+2063069us) service_pool.cc:224] Handling call
0709 12:23:23.507913 (+192510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.566019 (+ 58106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.566290 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:23.841544  1878 rpcz_store.cc:276] 0709 12:23:22.011481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.229183 (+1217702us) service_pool.cc:224] Handling call
0709 12:23:23.229607 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.434333 (+204726us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:23.581953 (+147620us) log.cc:844] Serialized 14 byte log entry
0709 12:23:23.681336 (+ 99383us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:23.681396 (+    60us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:23.681454 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.840598 (+159144us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54002df0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:23.840660 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.840852 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":121059,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":3593,"raft.queue_time_us":87,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"spinlock_wait_cycles":470257536,"thread_start_us":246142,"threads_started":2,"wal-append.queue_time_us":91277}
I20260709 12:23:23.865434  5342 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:23.880964  5340 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.551s	user 0.000s	sys 0.004s
I20260709 12:23:23.880722  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:23.556732  5271 consensus_queue.cc:1243] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 8.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:23:23.851989  5368 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
W20260709 12:23:23.881098  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 442) took 3093 ms (client timeout 2565 ms). Trace:
I20260709 12:23:23.893884  1806 rpcz_store.cc:276] 0709 12:23:22.073001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.073179 (+   178us) service_pool.cc:224] Handling call
0709 12:23:22.073517 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.222706 (+1149189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.654150 (+431444us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540004d8d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:23.654396 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2494624512}
I20260709 12:23:23.655565  5298 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:23.898397  4770 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.664044  5357 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.886619  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:23.899228  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53169.2us
W20260709 12:23:23.885285  4943 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 12:23:23.893414  5312 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:23.890664  2412 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:23.900172  5336 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:23.901084  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.900599  5253 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.905022  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.907187  5365 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:23.907876  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:23.425467  5055 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.066s
W20260709 12:23:23.932055  5325 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.862s	user 0.005s	sys 0.000s
I20260709 12:23:23.943641  4948 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:23.947886  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2652) took 2696 ms. Trace:
I20260709 12:23:23.948845  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:23.950562  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1859) took 2215 ms. Trace:
I20260709 12:23:23.952234  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:23.953346  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:23.953775  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.954871  1804 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:23:23.955874  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:23.958134  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.958298  5364 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.958544  4788 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.958794  5313 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:23.961694  5251 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:23.962159  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 113456us
I20260709 12:23:23.963017  5272 raft_consensus.cc:993] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: : Instructing follower 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
W20260709 12:23:23.963611  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 82756.1us
I20260709 12:23:23.964120  4941 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:23.994632  5305 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:24.011788  5196 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:23.997048  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:24.013026  5196 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:23:24.013474  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.013959  5196 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:24.014451  5196 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:23.969473  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:23.982635  1877 rpcz_store.cc:276] 0709 12:23:21.551819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.817358 (+1265539us) service_pool.cc:224] Handling call
0709 12:23:23.027439 (+210081us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.050813 (+ 23374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.666321 (+615508us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540021b1d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:23.666478 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":347855744}
W20260709 12:23:23.984414  1730 rpcz_store.cc:269] 0709 12:23:22.116983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.237995 (+121012us) service_pool.cc:224] Handling call
0709 12:23:23.671294 (+1433299us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b54002a9254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:23.671429 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2126037888}
I20260709 12:23:24.024870  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:24.005070  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:24.028316  1874 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:24.005455  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:24.029057  1873 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:23.986958  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:23.707532  5257 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:23.989673  4666 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:23.829034  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.014628  5270 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:24.015277  2419 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:23:24.045879  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 159316us
I20260709 12:23:24.015678  2418 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:24.013350  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:24.017638  1881 rpcz_store.cc:269] 0709 12:23:20.787269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.119449 (+332180us) service_pool.cc:224] Handling call
0709 12:23:23.880873 (+2761424us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54002df5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:23.881014 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1986800512}
W20260709 12:23:24.037734  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50907.9us
I20260709 12:23:23.994433  2410 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260709 12:23:24.020164  1798 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:23:24.020913  1803 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 9 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:23:24.047708  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 767) took 3752 ms. Trace:
I20260709 12:23:24.021853  1805 raft_consensus.cc:2376] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8. Re-sending same reply.
I20260709 12:23:24.018219  1723 rpcz_store.cc:276] 0709 12:23:21.251022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.187190 (+1936168us) service_pool.cc:224] Handling call
0709 12:23:23.312194 (+125004us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.881830 (+569636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.947603 (+ 65773us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b540044aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:23.947815 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":613594368}
I20260709 12:23:24.017424  1872 rpcz_store.cc:276] 0709 12:23:21.735045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.020043 (+1284998us) service_pool.cc:224] Handling call
0709 12:23:23.026459 (+  6416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.167215 (+140756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.950253 (+783038us) spinlock_profiling.cc:243] Waited 61.8 ms on lock 0x7b540001f4d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:23.950481 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":142087552}
I20260709 12:23:23.964171  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:23.994376  5144 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:23.985351  1876 rpcz_store.cc:276] 0709 12:23:20.697468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.703691 (+  6223us) service_pool.cc:224] Handling call
0709 12:23:21.550881 (+847190us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:21.552217 (+  1336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.672135 (+2119918us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b54002df0d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:23.672264 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1923071872}
I20260709 12:23:24.019243  1807 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:24.103101  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:24.104890  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 113747us
I20260709 12:23:24.103505  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.047081  1732 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:24.045532  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:24.105468  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:24.112967  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 88188.4us
I20260709 12:23:24.113312  2417 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:23:24.107698  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:24.114492  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87632.4us
I20260709 12:23:24.111454  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:24.109613  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:24.111515  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62913.8us
I20260709 12:23:24.116446  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.113775  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:24.048100  2411 rpcz_store.cc:276] 0709 12:23:20.306648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.678310 (+1371662us) service_pool.cc:224] Handling call
0709 12:23:21.678672 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.668064 (+1989392us) spinlock_profiling.cc:243] Waited 8.73 ms on lock 0x7b540047c320. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:23:23.984958 (+316894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.041751 (+ 56793us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b54005609d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:24.047641 (+  5890us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1495467648}
I20260709 12:23:24.117549  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:24.016314  5313 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.037022  5305 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.178306  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.184396  2419 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
W20260709 12:23:24.184957  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67556.3us
I20260709 12:23:24.186142  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:24.132313  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:24.187402  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 306749us
I20260709 12:23:24.187951  2417 raft_consensus.cc:2376] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7. Re-sending same reply.
I20260709 12:23:24.189342  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.124256  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:24.191007  2410 raft_consensus.cc:2376] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10. Re-sending same reply.
I20260709 12:23:24.192222  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:24.180123  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.019491  5251 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:24.003289  5281 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:24.193683  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.032033  4770 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:24.206708  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:24.208779  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:24.210028  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94405us
I20260709 12:23:24.225353  2419 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:24.211330  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.227502  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:24.228127  1729 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:24.228988  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.229792  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.228155  1873 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:24.105065  5312 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:24.271806  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:24.221097  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 256971us
I20260709 12:23:24.221976  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:24.104573  4899 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.045542  5270 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:24.322355  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50661.6us
I20260709 12:23:24.239882  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.231861  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.323938  2419 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
W20260709 12:23:24.241446  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 127755us
I20260709 12:23:24.121652  5249 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:24.013929  4788 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:23.998342  5377 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.225s
I20260709 12:23:24.269538  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.037501  5364 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.031246  5298 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.040035  5357 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.221312  5349 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.040252  5291 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:24.302548  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.231564  5055 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:24.312533  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1739) took 5341 ms. Trace:
I20260709 12:23:24.362910  1728 rpcz_store.cc:276] 0709 12:23:18.970667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.711259 (+2740592us) service_pool.cc:224] Handling call
0709 12:23:21.720727 (+  9468us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.225277 (+504550us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.225306 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:22.300961 (+ 75655us) log.cc:844] Serialized 26 byte log entry
0709 12:23:22.661521 (+360560us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:22.661581 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.661690 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.931202 (+1269512us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:24.312226 (+381024us) raft_consensus.cc:1685] finished
0709 12:23:24.312247 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.312466 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":106404,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":11001,"spinlock_wait_cycles":7110400,"thread_start_us":781482,"threads_started":2,"wal-append.queue_time_us":646836}
I20260709 12:23:24.315654  1731 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:24.223882  5272 raft_consensus.cc:1081] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Signalling peer 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
I20260709 12:23:24.367982  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:24.368870  2410 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:24.246495  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.247597  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:24.119073  5337 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.317322  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:24.317926  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.222437  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:24.383538  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66285us
I20260709 12:23:24.315944  5312 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.355348  5355 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:24.413143  1881 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: [6.4-8.5]
I20260709 12:23:24.413622  1798 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:24.156373  5144 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.103863  5057 raft_consensus.cc:2764] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:23:24.434453  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.439532  1725 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:24.118765  5365 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:24.223145  5281 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:24.205441  5089 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.106s
W20260709 12:23:24.456187  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 252476us
I20260709 12:23:24.466199  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:24.334017  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.336231  1807 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:24.011548  5269 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.172s
I20260709 12:23:24.337368  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:24.302969  5245 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:24.303216  5330 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:24.474352  2418 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:24.329281  2383 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.014s (SENT)
I20260709 12:23:24.350062  2414 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 4.
I20260709 12:23:24.309110  5362 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:24.480554  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:24.481175  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 265561us
W20260709 12:23:24.481858  1844 leader_election.cc:336] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 2.204s (SENT)
I20260709 12:23:24.481947  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:24.482328  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:24.354631  2416 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:24.282291  5276 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260709 12:23:24.376736  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.369500  5271 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260709 12:23:24.379886  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:24.375837  1696 connection.cc:256] RPC call timeout handler was delayed by -1.5111934459996519s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29550000000000004s for a response
I20260709 12:23:24.009940  5296 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.385380  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:24.393837  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.401324  2417 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:24.321272  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:24.494709  2412 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:24.495291  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 841) took 1027 ms. Trace:
W20260709 12:23:24.414613  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 202954us
I20260709 12:23:24.412618  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.496618  1727 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:24.419023  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:24.496657  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:24.423805  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.424842  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:24.497707  1800 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:24.425149  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:24.426245  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:24.428880  1805 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:23:24.495654  2412 rpcz_store.cc:276] 0709 12:23:23.468232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.281601 (+813369us) service_pool.cc:224] Handling call
0709 12:23:24.495230 (+213629us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.429973  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:24.434553  1726 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 7 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:24.435101  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:24.443181  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:24.443908  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:24.444731  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:24.445566  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:24.447067  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:24.446503  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:24.447948  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.498553  2411 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:24.448964  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:24.499753  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80818.4us
I20260709 12:23:24.504843  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.455264  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:24.505666  1723 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:24.468645  2419 raft_consensus.cc:2410] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 11 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:24.223667  5305 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:24.510584  5380 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
W20260709 12:23:24.502008  2440 debug-util.cc:398] Leaking SignalData structure 0x7b08002df8a0 after lost signal to thread 564
I20260709 12:23:24.511718  5138 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:24.512230  5138 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:24.499001  5289 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:24.513317  5138 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.514324  5289 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:24.514340  5305 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:24.512068  2440 debug-util.cc:398] Leaking SignalData structure 0x7b08002dfc20 after lost signal to thread 1664
W20260709 12:23:24.516706  2440 debug-util.cc:398] Leaking SignalData structure 0x7b08002b8ea0 after lost signal to thread 1756
W20260709 12:23:24.517048  2440 debug-util.cc:398] Leaking SignalData structure 0x7b08002b8e80 after lost signal to thread 1757
I20260709 12:23:24.375397  5263 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
I20260709 12:23:24.488953  2410 raft_consensus.cc:2410] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:23:24.377274  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.520010  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.521262  5313 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:24.523128  2410 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:23:24.524235  5296 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:24.525666  5364 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:24.526716  2440 debug-util.cc:398] Leaking SignalData structure 0x7b0800356ec0 after lost signal to thread 1831
I20260709 12:23:24.327802  4666 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.489694  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 834) took 1299 ms. Trace:
W20260709 12:23:24.491178  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 261012us
I20260709 12:23:24.488267  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:24.453547  1875 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:24.505051  1872 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:24.454638  1880 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:24.336501  5251 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:24.506745  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 169425us
I20260709 12:23:24.501602  1877 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:24.501992  1878 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:24.502578  1874 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:24.412557  5312 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:24.503391  1879 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:24.502897  1801 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:24.503738  1728 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:24.534160  2418 rpcz_store.cc:276] 0709 12:23:23.176294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.428625 (+1252331us) service_pool.cc:224] Handling call
0709 12:23:24.459704 (+ 31079us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.475056 (+ 15352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.475339 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:24.508616  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 179363us
I20260709 12:23:24.535280  1881 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:24.442539  5355 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:24.475104  4943 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:23:24.515080  5289 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:24.527227  2415 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:24.528579  2440 debug-util.cc:398] Leaking SignalData structure 0x7b080032e6e0 after lost signal to thread 1905
W20260709 12:23:24.568329  2440 debug-util.cc:398] Leaking SignalData structure 0x7b08003cfb60 after lost signal to thread 1906
I20260709 12:23:24.491930  5358 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:24.489478  2416 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:24.415726  5337 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.532727  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.574626  1727 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:24.536952  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:24.493202  5250 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:24.526899  5377 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:24.482962  5330 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:24.375020  5349 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.477259  5365 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:24.581722  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.532804  5154 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:24.528786  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 143500us
I20260709 12:23:24.599376  1772 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:24.571482  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:24.506356  5368 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:24.526474  5253 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:24.532550  5057 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:24.520236  5383 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:24.580404  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:24.580422  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:24.526189  5272 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.582093  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 835) took 1387 ms. Trace:
I20260709 12:23:24.581749  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:24.590567  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.632720  2416 rpcz_store.cc:276] 0709 12:23:23.194483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.489005 (+1294522us) service_pool.cc:224] Handling call
0709 12:23:24.489311 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.581808 (+ 92497us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.582034 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.627812  1800 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
W20260709 12:23:24.569783  2440 debug-util.cc:398] Leaking SignalData structure 0x7b08003a80c0 after lost signal to thread 2443
W20260709 12:23:24.634305  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 97479us
W20260709 12:23:24.634565  2440 debug-util.cc:398] Leaking SignalData structure 0x7b080010e420 after lost signal to thread 2444
I20260709 12:23:24.635161  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:24.580245  4666 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:24.579778  4943 consensus_queue.cc:1243] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:23:24.563335  5138 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:24.598300  1727 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:24.605978  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:24.640228  1880 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:24.610896  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114313us
I20260709 12:23:24.572167  5263 consensus_queue.cc:1243] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:23:24.573133  5251 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:24.623589  2413 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:23:24.624616  1729 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:24.664361  1772 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:24.664775  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "36bc2f66a73947468d61113c226ed318"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:24.665491  1729 raft_consensus.cc:606] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:24.545758  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:24.666977  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.667400  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:24.667575  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:24.668711  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:24.580253  5341 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:24.675753  1727 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:24.573520  5296 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.678490  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:24.631450  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.621830  5055 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:24.677330  1873 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:24.694304  1801 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:24.696208  1881 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:24.698325  1881 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:23:24.679025  1879 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
W20260709 12:23:24.679888  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.498332  5366 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:24.701699  5366 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:24.702414  5366 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:24.703591  1803 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:24.704942  1879 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:24.705654  1877 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:24.707697  1727 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 9 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:24.635313  5383 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:24.710582  2419 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:24.711880  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 206304us
I20260709 12:23:24.720719  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2658) took 3426 ms. Trace:
I20260709 12:23:24.721407  1727 rpcz_store.cc:276] 0709 12:23:21.281664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.680777 (+3399113us) service_pool.cc:224] Handling call
0709 12:23:24.688012 (+  7235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.708206 (+ 20194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.708392 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.676852  5253 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:24.669322  4943 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:24.681267  5250 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:24.739506  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:24.740108  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 72628.8us
I20260709 12:23:24.678525  5341 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:24.744881  1877 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:24.688342  5057 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:24.749171  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 177729us
I20260709 12:23:24.780608  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:24.784281  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:24.629329  5154 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.736547  5263 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:24.722530  5383 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:24.765069  5393 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:24.767470  5382 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:23:24.754424  5357 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:24.827785  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 292912us
I20260709 12:23:24.769032  5310 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:24.752032  1727 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:24.832118  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1747) took 3526 ms. Trace:
I20260709 12:23:24.789763  1880 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:24.726308  5269 raft_consensus.cc:2764] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:23:24.790547  5253 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:24.832677  1727 rpcz_store.cc:276] 0709 12:23:21.305993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.742986 (+3436993us) service_pool.cc:224] Handling call
0709 12:23:24.746324 (+  3338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.831765 (+ 85441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.832041 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.834437  5253 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.835629  5313 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:24.836191  5313 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:24.795550  1728 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:24.738866  5362 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:24.837685  5362 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:24.838337  1728 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:24.838747  5362 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.839823  5349 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:24.790210  5365 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.663798  5272 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:24.821552  4941 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.781s	user 0.009s	sys 0.002s
W20260709 12:23:24.821995  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 133507us
I20260709 12:23:24.824527  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 837) took 1517 ms. Trace:
I20260709 12:23:24.779315  5341 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:24.749176  5055 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.080s
W20260709 12:23:24.574997  5283 consensus_queue.cc:1175] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" responder_term: 9 status { last_received { term: 9 index: 6 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:24.813567  5144 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:24.836261  2413 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:23:24.839289  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2659) took 3524 ms. Trace:
I20260709 12:23:24.868185  1728 rpcz_store.cc:276] 0709 12:23:21.314492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.837691 (+3523199us) service_pool.cc:224] Handling call
0709 12:23:24.838074 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.838935 (+   861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.839216 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.747064  5306 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260709 12:23:24.710939  5296 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.874862  5296 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:24.875653  5089 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:24.876173  5089 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:24.877192  5089 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.878604  5089 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:24.880434  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.880720  1728 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:23:24.880995  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.797907  5305 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:24.881541  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1761) took 1402 ms. Trace:
I20260709 12:23:24.881969  1728 rpcz_store.cc:276] 0709 12:23:23.479600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.869516 (+1389916us) service_pool.cc:224] Handling call
0709 12:23:24.880306 (+ 10790us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.881189 (+   883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.881430 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.845630  5398 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:24.883188  5398 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.883616  5398 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:24.888549  5154 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:24.888978  2414 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:23:24.889317  2414 pending_rounds.cc:85] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 5
I20260709 12:23:24.889539  2414 pending_rounds.cc:107] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:23:24.889745  2414 raft_consensus.cc:2889] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:24.878597  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:24.893652  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.895021  5288 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.895489  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:24.901820  5196 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.903416  5337 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:24.904009  5337 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:23:24.904284  5330 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.905103  1728 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:23:24.905570  5330 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:24.855046  2412 rpcz_store.cc:276] 0709 12:23:23.307228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.500870 (+1193642us) service_pool.cc:224] Handling call
0709 12:23:24.540358 (+ 39488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.823945 (+283587us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.823991 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:24.824057 (+    66us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:24.824102 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.824232 (+   130us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:24.824429 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":47978,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":8123,"thread_start_us":282071,"threads_started":1}
I20260709 12:23:24.828349  5291 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:24.854269  4941 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:24.910706  5281 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.799955  4899 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:24.752671  4788 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:24.796423  5263 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:23:24.856477  5396 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260709 12:23:24.861008  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.870708  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 787) took 1025 ms. Trace:
I20260709 12:23:24.753270  4943 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.935235  2413 rpcz_store.cc:276] 0709 12:23:23.845069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.835382 (+990313us) service_pool.cc:224] Handling call
0709 12:23:24.835825 (+   443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.870049 (+ 34224us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.870611 (+   562us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.745244  5355 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.878700  5397 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:24.914095  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2643) took 4661 ms. Trace:
I20260709 12:23:24.959564  1726 rpcz_store.cc:276] 0709 12:23:20.255980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.946385 (+2690405us) service_pool.cc:224] Handling call
0709 12:23:22.946712 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.501027 (+1554315us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.913824 (+412797us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b540030b454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:24.914029 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3421190528}
W20260709 12:23:24.894635  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:24.961485  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.846247  1872 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:24.852828  1727 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:24.917724  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:24.924269  1732 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.741s	user 0.012s	sys 0.000s
I20260709 12:23:24.964133  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1755) took 1511 ms. Trace:
I20260709 12:23:24.964444  1732 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:24.965200  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1757) took 1512 ms. Trace:
I20260709 12:23:24.965593  1881 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:24.964167  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46516.2us
I20260709 12:23:24.794404  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:24.964733  1727 rpcz_store.cc:276] 0709 12:23:23.452314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.846985 (+1394671us) service_pool.cc:224] Handling call
0709 12:23:24.852599 (+  5614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.963784 (+111185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.964062 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:24.968048  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 173767us
I20260709 12:23:24.925462  2411 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:24.970969  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 707) took 1148 ms. Trace:
I20260709 12:23:24.971436  2411 rpcz_store.cc:276] 0709 12:23:23.822141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.376370 (+554229us) service_pool.cc:224] Handling call
0709 12:23:24.970880 (+594510us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.933157  5263 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:24.927253  5089 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:23:24.939841  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:24.911580  4770 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.661s	user 0.010s	sys 0.000s
W20260709 12:23:24.945514  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.894366  5312 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.914642  5337 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:24.965613  1732 rpcz_store.cc:276] 0709 12:23:23.453100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.013034 (+559934us) service_pool.cc:224] Handling call
0709 12:23:24.965108 (+952074us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:24.979025  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.983240  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:24.898280  5270 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:24.931727  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 253315us
I20260709 12:23:24.958607  5281 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:25.021306  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.029152  1726 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:24.974210  5196 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:24.895591  5288 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:24.974532  5371 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:25.036912  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:25.040076  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 249768us
I20260709 12:23:25.040884  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:25.042320  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:24.978327  5296 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:23:25.044591  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 55898.6us
W20260709 12:23:25.048213  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.980312  4788 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:24.991848  5057 raft_consensus.cc:2764] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:23:25.021016  5354 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:23:25.058734  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.082059  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:25.064668  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.065518  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:24.978063  5269 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.081s
I20260709 12:23:25.097785  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:25.098570  1879 raft_consensus.cc:2376] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8. Re-sending same reply.
I20260709 12:23:25.099560  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:24.914371  5398 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.981046  4941 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260709 12:23:25.090224  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:25.111788  1726 raft_consensus.cc:2376] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12. Re-sending same reply.
I20260709 12:23:25.091904  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:25.093480  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.089893  1727 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
W20260709 12:23:25.103961  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.105230  1876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.012s	sys 0.019s
W20260709 12:23:25.113191  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.113452  1876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.012s	sys 0.019s
W20260709 12:23:25.113564  1770 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.851s (SENT)
I20260709 12:23:25.028290  5271 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:23:24.989711  4770 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:24.981168  5393 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.103s
I20260709 12:23:25.071432  5270 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:25.123972  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83172us
W20260709 12:23:25.130270  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 72555.2us
I20260709 12:23:25.080657  5354 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:25.134541  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:25.135361  1879 raft_consensus.cc:2376] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11. Re-sending same reply.
I20260709 12:23:25.023236  5400 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.138s
I20260709 12:23:25.138126  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:25.139128  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55648.9us
I20260709 12:23:25.140704  5154 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 12:23:25.144369  5388 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.163s
I20260709 12:23:25.145484  5341 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.130479  5397 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.017s
I20260709 12:23:25.147921  5245 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:25.148397  5245 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:25.073839  5296 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:25.150202  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 850) took 1210 ms. Trace:
I20260709 12:23:25.123118  5299 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:25.151918  1799 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:25.117854  4948 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.155501  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:25.168882  1726 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:25.169961  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.170742  5407 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:25.170992  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:25.165719  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.167024  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.173591  1872 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:25.163523  2417 rpcz_store.cc:276] 0709 12:23:23.940034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.317623 (+377589us) service_pool.cc:224] Handling call
0709 12:23:25.149990 (+832367us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b54005b0254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:25.150137 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41414528}
W20260709 12:23:25.174642  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.137405  5357 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:25.187191  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:25.189536  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 57499.1us
W20260709 12:23:25.202723  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 80127.6us
I20260709 12:23:25.207691  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:25.171420  5296 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.212572  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:25.208685  2383 leader_election.cc:336] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.664s (SENT)
I20260709 12:23:25.172372  5299 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:25.181183  4941 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:25.227393  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.228713  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:25.218433  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:25.209306  5336 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.177s	user 0.009s	sys 0.004s
I20260709 12:23:25.219493  5245 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.164265  5057 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:23:25.224659  5288 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:25.301666  5138 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.635s	user 0.008s	sys 0.003s
W20260709 12:23:25.185907  5289 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.598s	user 0.010s	sys 0.001s
I20260709 12:23:25.183571  5357 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.254925  5335 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:25.255399  5408 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:25.256611  4943 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:25.235033  4941 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:25.325106  5408 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.227092  5276 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:23:25.181449  5306 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:23:25.227025  5402 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:23:25.228925  5354 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.317951  5312 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:25.239128  5362 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:25.272655  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.347959  1879 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:25.238435  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:25.349119  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 110780us
I20260709 12:23:25.235944  5271 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:25.275159  1726 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:25.274493  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:25.304409  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.299896  5089 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:25.307832  5383 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.308790  5337 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:25.206009  5269 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:25.182701  4948 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:25.366407  5263 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.367901  4948 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:25.368795  5263 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:25.370901  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:25.323701  5289 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.371699  1726 raft_consensus.cc:2376] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8. Re-sending same reply.
I20260709 12:23:25.372936  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.373654  5289 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:25.374269  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.374616  1878 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:25.351257  5354 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:25.375463  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.376600  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.377143  1726 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:25.377907  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.362077  4770 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:25.379061  5409 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:23:25.362398  5245 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:25.380154  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:25.380908  5413 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:23:25.381438  5272 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:25.269904  5299 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:25.332942  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 141784us
I20260709 12:23:25.313390  5398 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:25.311147  5405 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.272109  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 705) took 1452 ms. Trace:
I20260709 12:23:25.324404  5357 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:25.333863  4788 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.345846  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.423380  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:25.423920  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.362484  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.352945  1875 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:25.338465  5306 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:23:25.381790  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.319444  1725 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:25.336635  5408 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:25.353344  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.421211  2418 rpcz_store.cc:276] 0709 12:23:23.821357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.540109 (+718752us) service_pool.cc:224] Handling call
0709 12:23:24.574589 (+ 34480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.593930 (+ 19341us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:24.830441 (+236511us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.123970 (+293529us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.128223 (+  4253us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.128467 (+   244us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.128729 (+   262us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.131090 (+  2361us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.131171 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.271859 (+140688us) raft_consensus.cc:1685] finished
0709 12:23:25.271878 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.272049 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41162,"prepare.queue_time_us":287199,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":83553,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":41855,"spinlock_wait_cycles":94628608,"thread_start_us":372117,"threads_started":2,"wal-append.queue_time_us":308658}
I20260709 12:23:25.437690  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 784) took 2866 ms. Trace:
I20260709 12:23:25.438915  4788 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:25.441720  4788 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:25.366568  5249 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.033s	user 0.009s	sys 0.004s
I20260709 12:23:25.442644  5249 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:25.442919  5408 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.444128  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.444707  5269 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:25.445523  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 792) took 1200 ms. Trace:
I20260709 12:23:25.446238  5269 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:25.447132  5408 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:25.448082  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.443011  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:25.449169  1726 raft_consensus.cc:606] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:25.449334  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.430708  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:25.449599  1726 raft_consensus.cc:614] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:25.445429  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.450659  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:25.453192  5386 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:25.453855  5386 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:25.454762  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:25.456578  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.456825  5313 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:25.457453  1875 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:25.446262  2414 rpcz_store.cc:276] 0709 12:23:24.245280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.890519 (+645239us) service_pool.cc:224] Handling call
0709 12:23:25.150649 (+260130us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.170939 (+ 20290us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.170980 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.171594 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.317495 (+145901us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:25.317781 (+   286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.317884 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.318073 (+   189us) raft_consensus.cc:1685] finished
0709 12:23:25.318088 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.445226 (+127138us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54005b0250. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:25.445455 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11996,"spinlock_wait_cycles":597499264,"thread_start_us":145186,"threads_started":1,"wal-append.queue_time_us":132113}
I20260709 12:23:25.459475  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:25.459504  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:25.438212  2415 rpcz_store.cc:276] 0709 12:23:22.571441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.990902 (+419461us) service_pool.cc:224] Handling call
0709 12:23:23.029337 (+ 38435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.568185 (+1538848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.437382 (+869197us) spinlock_profiling.cc:243] Waited 1.5 s on lock 0x7b54004560d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:25.437610 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3444323072}
W20260709 12:23:25.421841  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56997.2us
I20260709 12:23:25.462630  5362 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:23:25.462950  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.465277  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:25.465492  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:25.390906  5329 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.017s	sys 0.030s
W20260709 12:23:25.467159  5329 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.017s	sys 0.031s
I20260709 12:23:25.467293  1875 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:25.469492  1875 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:25.471330  5362 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:25.473192  4770 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:25.473827  4770 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:25.474385  4770 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:25.475373  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:25.476285  5362 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:25.473204  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:25.481302  5383 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:25.481531  5270 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:25.482221  5270 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.482699  5270 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:25.485281  5312 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:25.485756  5312 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:25.474495  5299 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:25.412916  5424 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:25.446717  5269 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:25.489619  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:25.489924  2415 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:23:25.490510  2413 raft_consensus.cc:2410] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:23:25.491080  5386 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:25.492579  5386 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:25.494582  5425 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:25.432130  5405 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:25.500303  5313 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:25.504477  1730 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:25.453238  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:25.507831  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 380468us
W20260709 12:23:25.418674  5330 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.513s	user 0.013s	sys 0.003s
I20260709 12:23:25.453383  5357 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.490324  5269 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:25.520756  1872 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:25.512856  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:25.522428  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:25.524940  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.536505  1801 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:25.525372  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.537959  1800 raft_consensus.cc:2410] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:23:25.524513  2411 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:25.526342  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.535738  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.542275  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:25.543435  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:25.544245  1798 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:25.544344  1804 raft_consensus.cc:2368] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6.
I20260709 12:23:25.545104  1802 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:25.546005  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:25.509094  5383 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:25.550225  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:25.543054  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:25.552475  2411 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:25.554041  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.533602  5271 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.554735  1801 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:25.531672  5362 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.557066  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:25.564730  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:25.567957  5251 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.897s	user 0.007s	sys 0.007s
I20260709 12:23:25.555330  5383 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:25.578344  5383 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:23:25.623463  5357 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:25.626866  5335 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:25.627475  5335 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:25.631099  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:25.634699  1874 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:25.486716  5312 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.583452  5405 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:25.590847  5362 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:25.549615  5330 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:23:25.594128  5422 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260709 12:23:25.600921  5407 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:25.601099  1798 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
W20260709 12:23:25.602919  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.603788  4941 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.568030  5271 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.568243  5371 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:25.608979  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2675) took 1585 ms. Trace:
I20260709 12:23:25.666620  1728 rpcz_store.cc:276] 0709 12:23:24.023197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.906081 (+882884us) service_pool.cc:224] Handling call
0709 12:23:24.906702 (+   621us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.602723 (+696021us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:23:25.602755 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:25.607128 (+  4373us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:25.607181 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.608721 (+  1540us) spinlock_profiling.cc:243] Waited 696 ms on lock 0x7b540030be54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:25.608771 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.608915 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":32316,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":1599795968}
W20260709 12:23:25.664789  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8bb813e4fab14a61b810a24b56899586 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:25.659132  5330 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.670106  5330 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:25.669508  4941 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:25.670728  5271 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:25.684953  5312 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:25.685413  4666 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:25.685847  4666 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:25.651229  5057 raft_consensus.cc:2764] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:23:25.686822  4666 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:25.673506  5424 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:25.687986  4666 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:25.638664  5270 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:25.615185  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.615388  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:25.689144  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.689252  1804 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
W20260709 12:23:25.690440  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.620826  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.692400  1803 raft_consensus.cc:2410] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:23:25.674091  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:25.638067  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.639369  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:25.694579  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.695703  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.696643  1798 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:25.701678  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.559820  5269 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:25.662773  1874 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:25.551074  5355 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
I20260709 12:23:25.671155  1805 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:25.679255  5342 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:25.682080  5298 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:25.684006  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2674) took 1661 ms. Trace:
I20260709 12:23:25.712117  1725 rpcz_store.cc:276] 0709 12:23:24.022688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.540810 (+518122us) service_pool.cc:224] Handling call
0709 12:23:25.683749 (+1142939us) spinlock_profiling.cc:243] Waited 249 ms on lock 0x7b54002aa654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:25.683921 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":573349248}
I20260709 12:23:25.684122  1731 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:25.673239  5362 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:25.702605  1802 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:25.704309  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.717527  1804 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
W20260709 12:23:25.709038  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69785.6us
I20260709 12:23:25.709457  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:25.711092  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.710716  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.718897  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:25.717639  1806 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:25.720104  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.720312  1845 leader_election.cc:400] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6.
I20260709 12:23:25.720722  1845 leader_election.cc:403] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:23:25.608637  4770 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:25.645778  1800 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:25.722570  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.697752  5371 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:25.724117  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.706287  5057 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:25.726610  1807 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:23:25.727458  1807 pending_rounds.cc:85] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 4
I20260709 12:23:25.727732  1807 pending_rounds.cc:107] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:23:25.727985  1807 raft_consensus.cc:2889] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:25.728677  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.729413  1874 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:25.730206  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.738402  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:25.739151  2415 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:25.740546  5406 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.429s	user 0.004s	sys 0.000s
I20260709 12:23:25.741678  1874 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:23:25.743134  1874 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:23:25.751602  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:25.755036  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.755226  5028 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6.
I20260709 12:23:25.755697  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:25.758224  1877 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:23:25.759110  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.761440  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.767051  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:25.768329  1878 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:23:25.770772  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:25.771576  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:25.772017  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:25.774847  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.775501  1877 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:25.778012  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:25.779230  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:25.723269  5281 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:25.785751  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:25.786702  1874 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:25.788837  1878 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:25.777115  5408 raft_consensus.cc:2764] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260709 12:23:25.798189  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.798795  1729 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:25.800402  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:25.762578  5330 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:25.778972  5349 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:25.780000  5289 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:25.803473  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:25.803264  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.777948  5057 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:23:25.804639  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.821803  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:25.822892  1725 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:25.805066  5299 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:25.784317  5398 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:25.815270  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:25.819566  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:25.801468  5355 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.818277  4788 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.011s
I20260709 12:23:25.831671  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.834324  1804 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:25.838721  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:25.805603  5386 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:23:25.827917  5362 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:25.849932  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:25.886915  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.832437  5349 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:25.833714  5298 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:23:25.918619  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 66280.4us
I20260709 12:23:25.934620  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:25.941390  1804 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:25.805294  5335 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:25.944878  1725 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:23:25.914253  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 98179.2us
I20260709 12:23:25.836752  5355 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:25.874605  5296 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.051s	sys 0.129s
I20260709 12:23:25.830909  5424 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.084s
I20260709 12:23:25.950490  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:25.837880  5398 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:25.913385  5271 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:25.842703  5299 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:25.973137  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.951367  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:25.948029  5405 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:25.995901  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45481us
I20260709 12:23:25.996652  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:26.000506  5433 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:26.000726  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 839) took 2629 ms. Trace:
I20260709 12:23:26.001114  2419 rpcz_store.cc:276] 0709 12:23:23.371559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.506636 (+1135077us) service_pool.cc:224] Handling call
0709 12:23:24.512859 (+  6223us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.719648 (+206789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.000476 (+1280828us) spinlock_profiling.cc:243] Waited 197 ms on lock 0x7b54008c0354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:26.000667 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":454240640}
I20260709 12:23:26.002192  5283 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:26.002971  5283 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:26.003162  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.000322  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:25.938969  5395 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.041s
I20260709 12:23:26.002429  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:26.018519  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:25.831946  5371 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.018085  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:26.021289  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 477) took 1099 ms. Trace:
I20260709 12:23:26.021898  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.022907  1727 rpcz_store.cc:276] 0709 12:23:24.922035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.975538 (+ 53503us) service_pool.cc:224] Handling call
0709 12:23:26.021068 (+1045530us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b54001905d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:26.021209 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":238123776}
I20260709 12:23:26.023521  1801 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:23:26.025194  5407 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:26.025012  5398 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.004194  5283 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:26.002197  5437 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 12:23:26.028903  4788 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:25.970935  5298 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:26.035698  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:25.953820  5417 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260709 12:23:25.979290  5271 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:26.006763  5362 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:23:26.046584  1724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2673) took 2024 ms (client timeout 2175 ms). Trace:
W20260709 12:23:26.047706  1724 rpcz_store.cc:269] 0709 12:23:24.022438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.402462 (+380024us) service_pool.cc:224] Handling call
0709 12:23:26.041033 (+1638571us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540044af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:26.046497 (+  5464us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2358982912}
I20260709 12:23:25.990557  5405 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:26.037168  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:26.024420  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:26.045370  5330 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:26.005820  5299 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:26.054355  5298 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.055768  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.056763  1725 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:26.058326  1803 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:26.059353  1880 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:23:26.037194  5407 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:26.038450  5432 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.187s	user 0.001s	sys 0.000s
I20260709 12:23:26.063413  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:26.064388  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:26.073053  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1768) took 1661 ms. Trace:
I20260709 12:23:26.080302  1725 rpcz_store.cc:276] 0709 12:23:24.411438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.040799 (+1629361us) service_pool.cc:224] Handling call
0709 12:23:26.056565 (+ 15766us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.072705 (+ 16140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.072984 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:26.074276  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:26.060696  1727 raft_consensus.cc:606] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:26.046094  5271 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:26.058056  5313 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:26.058061  5365 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.193s	user 0.009s	sys 0.010s
I20260709 12:23:26.086495  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:26.091996  1724 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:26.093545  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:26.062891  5341 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:26.094183  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.067083  5347 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:26.098397  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1770) took 1179 ms. Trace:
I20260709 12:23:26.069847  5362 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:26.102980  1725 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:26.104023  1724 rpcz_store.cc:276] 0709 12:23:24.918659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.073389 (+1154730us) service_pool.cc:224] Handling call
0709 12:23:26.091752 (+ 18363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.098090 (+  6338us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.098322 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:26.096074  5365 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.231s	user 0.010s	sys 0.011s
I20260709 12:23:26.110566  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 474) took 1700 ms. Trace:
I20260709 12:23:26.109127  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.112051  1801 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:26.110404  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 475) took 1189 ms. Trace:
I20260709 12:23:26.112691  1725 rpcz_store.cc:276] 0709 12:23:24.921384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.081156 (+1159772us) service_pool.cc:224] Handling call
0709 12:23:26.102744 (+ 21588us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.107561 (+  4817us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.110328 (+  2767us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:26.100251  5249 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:26.095670  5269 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:26.100039  4941 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:23:26.124939  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.128472  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.103919  5347 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.131429  1873 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:23:26.062769  5349 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:26.139334  5438 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 12:23:26.118007  5342 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.140317  5342 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.116984  4788 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:26.100665  5313 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.114471  5337 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:26.143702  5337 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.144824  5313 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:26.146149  5431 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:23:26.146605  5347 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:26.114423  5289 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.147742  5289 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.150177  5289 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:26.130353  5405 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.153872  5405 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:26.114759  5330 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:26.090606  5310 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:23:26.155067  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1868) took 1783 ms. Trace:
I20260709 12:23:26.146072  5337 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:26.146371  5435 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:23:26.132438  1725 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 12:23:26.198990  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 990267us
I20260709 12:23:26.159586  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:26.194775  1875 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:26.188812  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 704) took 2370 ms. Trace:
I20260709 12:23:26.193629  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 457) took 1107 ms. Trace:
I20260709 12:23:26.209165  1872 rpcz_store.cc:276] 0709 12:23:25.085737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.174399 (+ 88662us) service_pool.cc:224] Handling call
0709 12:23:26.193314 (+1018915us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b5400223654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:26.193527 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":677084416}
I20260709 12:23:26.210698  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:26.211517  1872 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:23:26.214821  1799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1793) took 1835 ms (client timeout 1888 ms). Trace:
I20260709 12:23:26.215775  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 795) took 1709 ms. Trace:
W20260709 12:23:26.215396  1799 rpcz_store.cc:269] 0709 12:23:24.379710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.379959 (+   249us) service_pool.cc:224] Handling call
0709 12:23:26.214500 (+1834541us) spinlock_profiling.cc:243] Waited 642 ms on lock 0x7b5400132c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:26.214713 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1477055744}
I20260709 12:23:26.216332  2416 rpcz_store.cc:276] 0709 12:23:24.506850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.937597 (+430747us) service_pool.cc:224] Handling call
0709 12:23:24.937983 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.938245 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:25.302550 (+364305us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.215321 (+912771us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.215391 (+    70us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:26.215450 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.215538 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.215697 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":871710,"prepare.queue_time_us":86628,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":2637,"thread_start_us":370732,"threads_started":2,"wal-append.queue_time_us":60619}
I20260709 12:23:26.204630  2410 rpcz_store.cc:276] 0709 12:23:23.819011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.524093 (+705082us) service_pool.cc:224] Handling call
0709 12:23:24.524428 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.525099 (+   671us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.525133 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.525682 (+   549us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.950144 (+424462us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.162284 (+212140us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.739787 (+577503us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:26.188220 (+448433us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.188344 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.188398 (+    54us) raft_consensus.cc:1685] finished
0709 12:23:26.188521 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.188722 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1050409,"prepare.queue_time_us":136887,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":1030317,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":13646,"spinlock_wait_cycles":6486400,"thread_start_us":593763,"threads_started":2,"wal-append.queue_time_us":702800}
I20260709 12:23:26.195605  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:26.194000  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.02309e+06us
W20260709 12:23:26.224678  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.226899  2416 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:26.227066  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.233449  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:26.240324  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.241619  1799 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:26.186664  5433 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:23:26.245532  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:26.246752  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:26.142688  5342 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:26.271572  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.252542  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:26.276247  1872 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:26.277621  2416 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:26.286621  2411 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:23:26.160168  4941 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:26.199359  1876 rpcz_store.cc:276] 0709 12:23:24.371263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.392386 (+ 21123us) service_pool.cc:224] Handling call
0709 12:23:24.445663 (+ 53277us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.456718 (+ 11055us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.456755 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.473734 (+ 16979us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.129792 (+656058us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:25.129924 (+   132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.130018 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.152220 (+1022202us) spinlock_profiling.cc:243] Waited 34.4 ms on lock 0x7b240023a9f8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:23:26.154651 (+  2431us) raft_consensus.cc:1685] finished
0709 12:23:26.154678 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.154966 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":127,"raft.queue_time_us":5311,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":36911,"spinlock_wait_cycles":79202176,"thread_start_us":573981,"threads_started":1,"wal-append.queue_time_us":682036}
I20260709 12:23:26.198621  5335 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:26.214092  5288 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:26.298635  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:26.215934  5144 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:26.311247  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.311338  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:26.312170  1876 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:26.312563  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.313594  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:26.313988  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.314863  1806 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 5: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:26.315438  4941 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.308090  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:26.316713  1732 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:26.310295  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.290959  5355 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:26.319105  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:26.290959  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:26.319952  1874 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:26.320333  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:26.321018  1801 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:26.321345  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:26.322163  1874 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:26.155828  5299 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.323093  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:26.324170  1725 raft_consensus.cc:2410] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:26.325604  1725 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:26.319941  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:26.331012  1798 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:26.332538  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.333737  1798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:33802
I20260709 12:23:26.334350  1798 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:26.315863  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:26.111104  1726 rpcz_store.cc:276] 0709 12:23:24.410412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.452226 (+1041814us) service_pool.cc:224] Handling call
0709 12:23:25.452695 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.453008 (+   313us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.453040 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.453642 (+   602us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.771990 (+318348us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:26.109913 (+337923us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.110239 (+   326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.110316 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.110358 (+    42us) raft_consensus.cc:1685] finished
0709 12:23:26.110368 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.110508 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":23718,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5637,"raft.queue_time_us":77,"thread_start_us":638071,"threads_started":2,"wal-append.queue_time_us":357051}
I20260709 12:23:26.336002  1806 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:23:26.334693  1798 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:26.339484  2413 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:26.305104  1872 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:26.345499  5393 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.346681  5393 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.290426  5362 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.288055  5407 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:26.370000  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.392127  1732 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:26.376744  2410 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:26.379742  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:26.385727  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.388116  1725 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:26.386620  1804 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:26.399662  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.402158  1804 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:26.402321  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:26.392874  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.395586  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:26.391366  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 191644us
I20260709 12:23:26.404456  2411 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:26.405931  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:26.406311  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1786133450000307s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
W20260709 12:23:26.406917  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.790s (SENT)
I20260709 12:23:26.407342  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:26.408946  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.403633  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:26.411160  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.411976  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.413290  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:26.414398  1872 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:26.415972  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:26.417158  1801 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:26.447515  2416 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:26.390986  5335 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.421311  5365 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:26.423106  1876 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:23:26.427139  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:26.427510  5289 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:23:26.429592  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.422377  4770 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:26.435685  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:26.467403  1798 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.469165  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:26.457369  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1774) took 1399 ms. Trace:
I20260709 12:23:26.417500  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:26.459517  1725 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:26.459964  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46746.8us
I20260709 12:23:26.460929  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.465504  2411 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:26.475265  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:26.476042  1872 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:26.478960  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:26.479786  1725 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:26.480224  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:26.482585  1723 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:26.483800  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:26.485787  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 215992us
I20260709 12:23:26.488389  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:26.488974  1804 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:26.487052  1798 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:26.490842  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:26.468497  5365 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.472204  1727 rpcz_store.cc:276] 0709 12:23:25.058000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.090266 (+1032266us) service_pool.cc:224] Handling call
0709 12:23:26.097262 (+  6996us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.097521 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:26.121692 (+ 24171us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.457005 (+335313us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.457065 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:26.457117 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.457186 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.457307 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18435,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":67,"raft.queue_time_us":660,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":1888,"thread_start_us":334201,"threads_started":1}
I20260709 12:23:26.495317  5329 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:26.496227  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:26.493182  1872 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:26.498512  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:26.502084  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:26.502944  1727 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:26.503535  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:26.504789  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:26.479882  5289 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:26.505685  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.507258  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.512778  5313 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:26.513180  1798 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:26.514003  5271 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.490697  5159 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.516891  5159 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.519522  5159 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:26.520967  5271 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:26.524070  1804 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:26.526105  1804 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:26.527861  1804 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:26.532744  5393 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:26.494633  5365 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:26.470419  5335 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.547010  4788 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:26.551837  5347 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.554212  5347 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:26.555069  1724 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:26.557297  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2555) took 2107 ms. Trace:
I20260709 12:23:26.557900  1881 rpcz_store.cc:276] 0709 12:23:24.450247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.699725 (+249478us) service_pool.cc:224] Handling call
0709 12:23:24.965330 (+265605us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.966471 (+  1141us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.557006 (+1590535us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b5400072e50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:26.557210 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":610034560}
I20260709 12:23:26.541973  5289 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.571748  1881 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:23:26.501933  5329 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:26.580941  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.582749  1881 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:26.584671  1881 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:23:26.589087  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:26.589290  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.590111  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:26.595347  1881 pending_rounds.cc:85] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 4
I20260709 12:23:26.595775  1881 pending_rounds.cc:107] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:23:26.602195  1881 raft_consensus.cc:2889] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:26.573686  5335 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:26.627008  1878 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:26.649631  1878 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:26.626273  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:26.652297  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:26.616003  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:26.628373  1879 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:26.632274  1801 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:26.587087  5329 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:26.655539  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1701) took 1080 ms. Trace:
I20260709 12:23:26.657861  1879 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:26.658265  1801 rpcz_store.cc:276] 0709 12:23:25.574708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.581677 (+1006969us) service_pool.cc:224] Handling call
0709 12:23:26.632023 (+ 50346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.655291 (+ 23268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.655481 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:26.659449  5269 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:26.660187  1879 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:26.661715  5299 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:23:26.619129  5393 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:26.619299  4770 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:23:26.592046  5271 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.074s
W20260709 12:23:26.685431  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 448) took 1633 ms (client timeout 2021 ms). Trace:
W20260709 12:23:26.685957  1880 rpcz_store.cc:269] 0709 12:23:25.052549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.065250 (+ 12701us) service_pool.cc:224] Handling call
0709 12:23:26.685169 (+1619919us) spinlock_profiling.cc:243] Waited 974 ms on lock 0x7b5400073354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:26.685359 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2239604992}
I20260709 12:23:26.670964  5393 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.689831  1880 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:23:26.696738  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 80858.7us
I20260709 12:23:26.691027  5386 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:26.693555  5393 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:26.702366  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.709214  1801 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:26.710577  1801 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:26.711192  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1805) took 1109 ms. Trace:
I20260709 12:23:26.711498  1801 rpcz_store.cc:276] 0709 12:23:25.602217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.710089 (+1107872us) service_pool.cc:224] Handling call
0709 12:23:26.710400 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.710965 (+   565us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.711135 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:26.703333  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:26.715113  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:26.721930  5347 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:26.722977  5159 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:26.724390  5159 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:26.724519  4788 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:26.725014  5159 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:26.726023  4788 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:26.726502  4788 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:26.721673  5348 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:23:26.727496  4788 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.729144  4788 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:26.727275  5159 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.728379  5348 consensus_queue.cc:1243] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:23:26.732216  5159 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:26.736315  5348 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:26.748292  1880 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:26.749971  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.756780  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:26.757771  1876 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:26.702368  5386 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:26.719463  5355 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:26.773752  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:26.777413  1801 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:26.759334  5405 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.605s	user 0.000s	sys 0.013s
W20260709 12:23:26.781411  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 117606us
I20260709 12:23:26.770747  5348 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.783835  5312 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:26.784399  5312 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:26.785342  5312 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.786409  5312 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:26.791994  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:26.783718  5348 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.792595  1801 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:26.776420  5249 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.639s	user 0.011s	sys 0.000s
W20260709 12:23:26.796882  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 142342us
I20260709 12:23:26.761582  5407 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:23:26.805691  1798 raft_consensus.cc:1217] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Deduplicated request from leader. Original: 7.5->[8.6-8.6]   Dedup: 8.6->[]
I20260709 12:23:26.791370  5355 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:26.806300  1798 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:23:26.810391  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1700) took 1236 ms. Trace:
I20260709 12:23:26.810879  1804 rpcz_store.cc:276] 0709 12:23:25.574458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.528645 (+954187us) service_pool.cc:224] Handling call
0709 12:23:26.529017 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:26.531552 (+  2535us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:26.531605 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:26.561184 (+ 29579us) log.cc:844] Serialized 26 byte log entry
0709 12:23:26.690490 (+129306us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:26.697893 (+  7403us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.698250 (+   357us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.698328 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.810172 (+111844us) raft_consensus.cc:1685] finished
0709 12:23:26.810193 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.810329 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16636,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":2027,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":7082,"spinlock_wait_cycles":25077120,"thread_start_us":114259,"threads_started":1,"wal-append.queue_time_us":187993}
I20260709 12:23:26.812947  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.814399  1804 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:26.816679  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1806) took 1214 ms. Trace:
W20260709 12:23:26.817082  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.797025  5249 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:26.817018  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1799) took 1247 ms. Trace:
W20260709 12:23:26.820248  5453 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:23:26.826267  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:26.816068  5405 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.854436  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1798) took 1285 ms. Trace:
I20260709 12:23:26.840761  4941 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:26.806396  5386 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.843845  5462 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.843245  5355 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.856968  5462 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.859236  5386 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:26.859364  5462 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:26.844404  5349 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:23:26.845590  1801 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:26.847431  1804 rpcz_store.cc:276] 0709 12:23:25.602511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.813826 (+1211315us) service_pool.cc:224] Handling call
0709 12:23:26.814107 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.814676 (+   569us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.816624 (+  1948us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:26.861845  1804 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:26.862551  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1807) took 1259 ms. Trace:
I20260709 12:23:26.862908  1804 rpcz_store.cc:276] 0709 12:23:25.602760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.861280 (+1258520us) service_pool.cc:224] Handling call
0709 12:23:26.861647 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.862290 (+   643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.862492 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:26.864408  1804 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:23:26.865018  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1705) took 1237 ms. Trace:
I20260709 12:23:26.865646  5348 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:26.852478  1802 rpcz_store.cc:276] 0709 12:23:25.569089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.358020 (+788931us) service_pool.cc:224] Handling call
0709 12:23:26.358453 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.358758 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:26.619626 (+260868us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.804537 (+184911us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.805312 (+   775us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.805361 (+    49us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:26.805400 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.805474 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.816904 (+ 11430us) inbound_call.cc:177] Queueing success response
0709 12:23:26.832704 (+ 15800us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b18003ea7c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16b0d01328 00007f16b0cf8901 00007f16b0cf88a7 00007f16b0df0288 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508
Metrics: {"mutex_wait_us":49292,"prepare.queue_time_us":81432,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":2278,"raft.queue_time_us":10442,"raft.run_cpu_time_us":2354,"raft.run_wall_time_us":28820,"spinlock_wait_cycles":34236800,"thread_start_us":364274,"threads_started":2}
I20260709 12:23:26.867810  4788 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:23:26.868273  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:26.869135  1802 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:26.869343  4788 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:26.870397  4788 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:26.863874  5405 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.875000  4788 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:26.876052  4788 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:26.877269  4788 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:26.877969  5405 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:26.880652  5445 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.332s	user 0.000s	sys 0.003s
I20260709 12:23:26.881296  1727 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:26.881542  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1883) took 1178 ms. Trace:
I20260709 12:23:26.881814  5313 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:26.865382  1804 rpcz_store.cc:276] 0709 12:23:25.628036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.863739 (+1235703us) service_pool.cc:224] Handling call
0709 12:23:26.864069 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.864760 (+   691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.864965 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:26.870476  1802 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:26.884418  1804 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:23:26.885305  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 463) took 1256 ms. Trace:
I20260709 12:23:26.885842  1802 rpcz_store.cc:276] 0709 12:23:25.628695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.869838 (+1241143us) service_pool.cc:224] Handling call
0709 12:23:26.870149 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.884932 (+ 14783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.885230 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:26.881922  1874 rpcz_store.cc:276] 0709 12:23:25.708317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.323024 (+614707us) service_pool.cc:224] Handling call
0709 12:23:26.323588 (+   564us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:26.324667 (+  1079us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:26.324699 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:26.325263 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:23:26.501734 (+176471us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:26.644095 (+142361us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.644390 (+   295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.644487 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.876317 (+231830us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b1800480520. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:26.881231 (+  4914us) raft_consensus.cc:1685] finished
0709 12:23:26.881339 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.881484 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":155349,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":616,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":109,"spinlock_wait_cycles":28981120,"thread_start_us":176068,"threads_started":1,"wal-append.queue_time_us":333749}
I20260709 12:23:26.891573  5365 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.892669  5393 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.893980  5365 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:26.884709  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:26.894749  5393 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:26.895260  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:26.895991  5159 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:26.896414  5159 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.896876  5159 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:26.901201  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 470) took 1270 ms. Trace:
I20260709 12:23:26.901891  1723 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:26.901774  1804 rpcz_store.cc:276] 0709 12:23:25.630428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.883865 (+1253437us) service_pool.cc:224] Handling call
0709 12:23:26.884189 (+   324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:26.900864 (+ 16675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.901126 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:26.856477  1806 rpcz_store.cc:276] 0709 12:23:25.568823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.337846 (+769023us) service_pool.cc:224] Handling call
0709 12:23:26.338109 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.338343 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:26.406748 (+ 68405us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.812531 (+405783us) spinlock_profiling.cc:243] Waited 7.43 ms on lock 0x7b180026cc20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:23:26.854051 (+ 41520us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.854103 (+    52us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:26.854182 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.854248 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.854368 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5911,"prepare.queue_time_us":68281,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":70,"raft.queue_time_us":2116,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":93,"spinlock_wait_cycles":17090304,"thread_start_us":41266,"threads_started":1}
I20260709 12:23:26.913655  1804 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:26.915719  1804 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:23:26.916460  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.918093  1806 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:26.919727  1806 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 5. Current term is 6. Ops: []
I20260709 12:23:26.922389  5270 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:26.917451  5329 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.865099  4941 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:26.924898  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:26.925586  1727 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:26.927436  1806 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:26.927505  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 467) took 2967 ms. Trace:
I20260709 12:23:26.928402  1731 rpcz_store.cc:276] 0709 12:23:23.959865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.891304 (+931439us) service_pool.cc:224] Handling call
0709 12:23:25.683928 (+792624us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.713509 (+ 29581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.927311 (+1213802us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b54002aa650. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:26.927443 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1822068480}
I20260709 12:23:26.925094  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1808) took 1258 ms. Trace:
I20260709 12:23:26.933681  1802 rpcz_store.cc:276] 0709 12:23:25.666932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.886741 (+1219809us) service_pool.cc:224] Handling call
0709 12:23:26.924535 (+ 37794us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.924754 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:26.924780 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.924825 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:26.924864 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.924924 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.925037 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:26.934270  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:26.975988  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.949815  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 465) took 1320 ms. Trace:
I20260709 12:23:26.911948  5348 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:26.926503  5248 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.007s
I20260709 12:23:26.971323  5283 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.936285  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:26.935757  5310 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:26.937814  5329 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:26.933424  5355 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.007s
I20260709 12:23:26.943420  4941 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:26.922554  4666 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.936084  5330 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 12:23:26.953086  1799 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:26.955251  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:26.957342  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:26.957574  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1679) took 1431 ms. Trace:
I20260709 12:23:27.002905  1805 rpcz_store.cc:276] 0709 12:23:25.525736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.525960 (+   224us) service_pool.cc:224] Handling call
0709 12:23:26.957345 (+1431385us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b540004f6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:26.957507 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":301529856}
I20260709 12:23:26.963912  2416 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:23:27.004601  2416 pending_rounds.cc:85] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 7
I20260709 12:23:27.004990  2416 pending_rounds.cc:107] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:23:27.005307  2416 raft_consensus.cc:2889] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:26.968266  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:27.007165  2410 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
W20260709 12:23:26.973096  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.974167  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:26.976488  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.009234  2419 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:26.976485  1801 rpcz_store.cc:276] 0709 12:23:25.629206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.872069 (+1242863us) service_pool.cc:224] Handling call
0709 12:23:26.872497 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.872730 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:26.873404 (+   674us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.949397 (+ 75993us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.949467 (+    70us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:26.949519 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.949598 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.949745 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2663,"prepare.queue_time_us":182,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":14920,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":61803,"threads_started":1,"wal-append.queue_time_us":63502}
I20260709 12:23:26.983731  1727 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:26.988325  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2688) took 1470 ms. Trace:
W20260709 12:23:26.992861  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 463) took 3034 ms (client timeout 1944 ms). Trace:
I20260709 12:23:26.993076  1732 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:26.979231  5249 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:26.983094  5347 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:27.037297  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.039753  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 462) took 1411 ms. Trace:
W20260709 12:23:27.025249  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.040419  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
W20260709 12:23:27.014871  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.041244  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:27.040252  1800 rpcz_store.cc:276] 0709 12:23:25.628310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.994212 (+365902us) service_pool.cc:224] Handling call
0709 12:23:27.039528 (+1045316us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b54001e3f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:27.039690 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1554901888}
I20260709 12:23:27.042448  5248 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.041098  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:27.013408  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1681) took 1485 ms. Trace:
I20260709 12:23:27.043779  2410 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:27.044297  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.043957  1807 rpcz_store.cc:276] 0709 12:23:25.527845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.527984 (+   139us) service_pool.cc:224] Handling call
0709 12:23:25.528300 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.755956 (+227656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.013069 (+1257113us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54001ea5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:27.013325 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":455210752}
W20260709 12:23:27.042284  1730 rpcz_store.cc:269] 0709 12:23:23.958852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.115215 (+156363us) service_pool.cc:224] Handling call
0709 12:23:26.992599 (+2877384us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b54000d1bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:26.992756 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2809101056}
I20260709 12:23:27.047945  5347 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.048586  5310 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:27.047956  5249 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.051770  5249 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:27.053182  5347 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:27.053843  5347 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:27.054204  5347 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:27.054713  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 794) took 2548 ms. Trace:
I20260709 12:23:27.055243  2412 rpcz_store.cc:276] 0709 12:23:24.506523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.917379 (+410856us) service_pool.cc:224] Handling call
0709 12:23:24.917640 (+   261us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:24.917861 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:24.941256 (+ 23395us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.441953 (+500697us) spinlock_profiling.cc:243] Waited 8.88 ms on lock 0x7b180041d520. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:23:26.274752 (+832799us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.875094 (+600342us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b180041d520. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:27.054107 (+179013us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b180041d520. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:23:27.054274 (+   167us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:27.054334 (+    60us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:27.054396 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.054474 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.054628 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":603125,"prepare.queue_time_us":23392,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":15673,"raft.queue_time_us":579409,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":9702,"spinlock_wait_cycles":108816128,"thread_start_us":250814,"threads_started":1,"wal-append.queue_time_us":852548}
I20260709 12:23:27.057021  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1748) took 5750 ms. Trace:
I20260709 12:23:27.057535  1729 rpcz_store.cc:276] 0709 12:23:21.306276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.779124 (+3472848us) service_pool.cc:224] Handling call
0709 12:23:24.817693 (+ 38569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.802312 (+984619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.056751 (+1254439us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b54002b6bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:27.056956 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2226345856}
I20260709 12:23:27.059136  1729 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 8. Ops: []
I20260709 12:23:27.059741  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2699) took 1361 ms. Trace:
I20260709 12:23:27.059765  5144 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:27.060091  1729 rpcz_store.cc:276] 0709 12:23:25.698392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.058565 (+1360173us) service_pool.cc:224] Handling call
0709 12:23:27.058957 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:27.059499 (+   542us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.059686 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:27.061866  1729 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:23:27.063059  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 659450us
I20260709 12:23:27.061405  5144 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.039199  1726 rpcz_store.cc:276] 0709 12:23:25.518055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.335342 (+817287us) service_pool.cc:224] Handling call
0709 12:23:26.898415 (+563073us) spinlock_profiling.cc:243] Waited 46.5 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:23:26.898661 (+   246us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:26.899161 (+   500us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:26.899191 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:26.899893 (+   702us) log.cc:844] Serialized 52 byte log entry
0709 12:23:26.987781 (+ 87888us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:26.987830 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.987901 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.987971 (+    70us) raft_consensus.cc:1685] finished
0709 12:23:26.987985 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.988253 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8366,"raft.queue_time_us":6128,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":12051,"spinlock_wait_cycles":106956544,"thread_start_us":87396,"threads_started":1,"wal-append.queue_time_us":84990}
I20260709 12:23:27.071429  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1775) took 1990 ms. Trace:
W20260709 12:23:27.071798  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1695) took 1498 ms (client timeout 1675 ms). Trace:
I20260709 12:23:27.072487  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 453) took 2002 ms. Trace:
W20260709 12:23:27.072535  1803 rpcz_store.cc:269] 0709 12:23:25.573049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.701151 (+128102us) service_pool.cc:224] Handling call
0709 12:23:27.071554 (+1370403us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54001e3a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:27.071718 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779286272}
I20260709 12:23:27.073284  1875 rpcz_store.cc:276] 0709 12:23:25.069702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.530843 (+461141us) service_pool.cc:224] Handling call
0709 12:23:26.194255 (+663412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.203733 (+  9478us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.072284 (+868551us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b5400223650. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:27.072425 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1482122112}
I20260709 12:23:27.071925  1724 rpcz_store.cc:276] 0709 12:23:25.080663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.108164 (+1027501us) service_pool.cc:224] Handling call
0709 12:23:26.108500 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.562923 (+454423us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.071168 (+508245us) spinlock_profiling.cc:243] Waited 446 ms on lock 0x7b54002b75d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:27.071360 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1026535168}
W20260709 12:23:27.064589  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 238394us
I20260709 12:23:27.065138  1805 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:23:27.066666  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.041167  1727 raft_consensus.cc:606] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:23:27.066987  2410 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:27.067293  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2700) took 1361 ms. Trace:
I20260709 12:23:27.080761  1729 rpcz_store.cc:276] 0709 12:23:25.701045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.061232 (+1360187us) service_pool.cc:224] Handling call
0709 12:23:27.061541 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:27.062280 (+   739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.062458 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:27.082847  1727 raft_consensus.cc:614] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:27.084443  1801 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:27.091215  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:27.093134  1727 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:23:27.094491  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 84054.1us
I20260709 12:23:27.096887  5393 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:27.097739  5393 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.100770  5348 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:27.102069  5348 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.103063  5393 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:27.108922  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 153788us
I20260709 12:23:27.111519  1730 pending_rounds.cc:85] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 3
I20260709 12:23:27.112128  1730 pending_rounds.cc:107] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:23:27.112569  1730 raft_consensus.cc:2889] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:27.043351  5248 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.115070  1724 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 8 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:27.117025  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.064267  5312 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.995826  5383 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
W20260709 12:23:27.120738  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 50421.9us
I20260709 12:23:27.120947  1729 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:26.994521  5283 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.141337  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:27.108520  5028 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:27.114837   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:27.145179  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2690) took 1626 ms. Trace:
I20260709 12:23:27.118695  4941 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:27.165313  1732 rpcz_store.cc:276] 0709 12:23:25.518833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.397034 (+878201us) service_pool.cc:224] Handling call
0709 12:23:26.992803 (+595769us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:27.042712 (+ 49909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.144860 (+102148us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b54000d1bd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:27.145079 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1369012608}
I20260709 12:23:27.155495  5296 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.156196  1725 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:23:27.156848  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1691) took 1585 ms. Trace:
I20260709 12:23:27.168097  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:27.169759  1880 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:27.121430  1881 pending_rounds.cc:85] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 1
I20260709 12:23:27.108315  5398 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:27.082223  4899 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:27.157516  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1874) took 2732 ms. Trace:
W20260709 12:23:27.159264  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 752299us
I20260709 12:23:27.175431  1724 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260709 12:23:27.176172  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.176784  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:27.140585  5313 raft_consensus.cc:2764] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260709 12:23:27.161597  1802 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:27.181891  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2694) took 1661 ms. Trace:
I20260709 12:23:27.182300  1725 rpcz_store.cc:276] 0709 12:23:25.520186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.503661 (+983475us) service_pool.cc:224] Handling call
0709 12:23:27.155963 (+652302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.171266 (+ 15303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.181697 (+ 10431us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b54003cd7d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:27.181828 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1418353024}
I20260709 12:23:27.162463  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:27.159255  1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2692) took 1639 ms (client timeout 1845 ms). Trace:
I20260709 12:23:27.120807  5347 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:27.195418  1728 rpcz_store.cc:269] 0709 12:23:25.519503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.707954 (+188451us) service_pool.cc:224] Handling call
0709 12:23:27.147115 (+1439161us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54002b70d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:27.159177 (+ 12062us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3102018816}
I20260709 12:23:27.172888   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:27.172729  2419 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:27.010900  4666 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.121438  5349 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.201859  1732 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:23:27.202404  1732 pending_rounds.cc:85] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 8
I20260709 12:23:27.202694  1732 pending_rounds.cc:107] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:23:27.203011  1732 raft_consensus.cc:2889] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:27.199018  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:27.114452  5289 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:23:27.162642  5447 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.388s	user 0.005s	sys 0.000s
W20260709 12:23:27.255944  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 57067.9us
I20260709 12:23:27.174175  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:27.162343  5306 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:27.261780  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 99421.3us
I20260709 12:23:27.175504  1873 rpcz_store.cc:276] 0709 12:23:24.424771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.691754 (+266983us) service_pool.cc:224] Handling call
0709 12:23:24.743579 (+ 51825us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:26.138667 (+1395088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.148271 (+1009604us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400222c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:27.157454 (+  9183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3190814080}
I20260709 12:23:27.168295  1798 rpcz_store.cc:276] 0709 12:23:25.571871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.514634 (+942763us) service_pool.cc:224] Handling call
0709 12:23:26.805586 (+290952us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:26.806696 (+  1110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.147479 (+340783us) spinlock_profiling.cc:243] Waited 291 ms on lock 0x7b5400132750. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:27.156780 (+  9301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":668310784}
I20260709 12:23:27.257843  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:27.173188  1881 pending_rounds.cc:107] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 5.2
I20260709 12:23:27.263634  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:27.264118  1881 raft_consensus.cc:2889] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:27.256645  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.164052  5028 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:27.260006  1872 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:27.174849  5398 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:27.175093  4899 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:27.263263  1880 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:27.276180  1806 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:27.288476  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.201643  5288 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:27.165450  5312 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.173273  5159 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.295646  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:27.419446  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 471) took 1014 ms. Trace:
I20260709 12:23:27.420115  1723 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:27.419946  1806 rpcz_store.cc:276] 0709 12:23:26.405447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.931317 (+525870us) service_pool.cc:224] Handling call
0709 12:23:26.931747 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:27.296613 (+364866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.419221 (+122608us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54003b7ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:27.419387 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":785296640}
I20260709 12:23:27.379604  4666 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:27.301707  5462 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.381846  5283 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:27.262174  5306 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:27.298136  4899 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:27.286898  5398 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:27.258391  5349 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.301096  5347 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:27.194859  5269 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.101s
I20260709 12:23:27.183393  5296 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.193334  5365 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.107s
W20260709 12:23:27.399886  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:27.399736  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 195315us
I20260709 12:23:27.401328  5329 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:23:27.402381  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 138849us
I20260709 12:23:27.459549  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:27.403239  5355 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:27.409909  5028 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.413940  5312 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:27.266183  5405 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:27.263103  5191 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.153s
I20260709 12:23:27.361703  1626 catalog_manager.cc:5697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:27.481547  1728 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:23:27.421615  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.377017   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:27.418620  5159 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:27.446197  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:27.448436  2410 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260709 12:23:27.485980  2410 pending_rounds.cc:85] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 6
I20260709 12:23:27.486358  2410 pending_rounds.cc:107] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260709 12:23:27.486644  2410 raft_consensus.cc:2889] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:27.451731  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.430001  5291 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:27.455725  1872 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:27.475693  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:27.472343  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:27.492671  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:27.494887  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 416483us
I20260709 12:23:27.440662  5462 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.520226  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:27.461129  5248 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:27.504184  1723 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:27.520990  1725 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:27.504465  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:27.507025  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 397025us
I20260709 12:23:27.511965  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:27.516741  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:27.518994  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:27.524223  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:27.524788  1724 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:27.524925  1798 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:27.519524  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:27.522209  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:27.526054  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:27.498171  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:27.527987  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 359922us
I20260709 12:23:27.494040  5312 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.608733  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 804) took 1215 ms. Trace:
I20260709 12:23:27.613659  5342 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:27.612931  2413 rpcz_store.cc:276] 0709 12:23:26.393526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.393688 (+   162us) service_pool.cc:224] Handling call
0709 12:23:26.437381 (+ 43693us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:27.608283 (+1170902us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.608327 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:27.608375 (+    48us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:27.608423 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.608503 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.608654 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":724336,"raft.queue_time_us":757063,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":55712,"thread_start_us":377633,"threads_started":1}
I20260709 12:23:27.614110  5342 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:27.497747  5476 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:27.609520  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 111213us
I20260709 12:23:27.623723  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:27.624488  1725 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:27.629844  1798 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:27.629762  2413 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:27.646451  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 115709us
I20260709 12:23:27.532470  5159 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.477288  5283 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:27.532339  5475 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.659408  1802 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:27.688488  1800 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:23:27.688679  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:27.628450  5476 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.519855  5028 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:23:27.670485  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:27.677448  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.016s	sys 0.020s
I20260709 12:23:27.679257  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:27.701117  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.016s	sys 0.020s
I20260709 12:23:27.518515  5291 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:27.484064  5306 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.500624  5405 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:27.689441  1805 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:27.497010  5383 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.646399  5269 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.646051  5355 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:27.658960  5329 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.622807  5312 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:27.733705  1875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.016s	sys 0.017s
W20260709 12:23:27.752337  1875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.016s	sys 0.017s
W20260709 12:23:27.752398  1731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.000s	sys 0.014s
I20260709 12:23:27.626914  4770 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
W20260709 12:23:27.738552  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.016s	sys 0.019s
I20260709 12:23:27.699074  5476 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:27.718998  5283 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:27.757651  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.016s	sys 0.019s
W20260709 12:23:27.762450  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 603666us
W20260709 12:23:27.759464  1799 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.011s	sys 0.085s
W20260709 12:23:27.764796  1799 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.011s	sys 0.085s
I20260709 12:23:27.735580  5306 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:27.689026  5477 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.096s
I20260709 12:23:27.743431  5480 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.829308  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:27.831451  1725 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:27.832127  1805 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:27.615162  5342 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.772500  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2697) took 2103 ms. Trace:
I20260709 12:23:27.695290  5159 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.696465  5475 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.732659  5383 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.886829  5383 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:27.757439  5491 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:27.887985  5491 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.891559  5335 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:27.891985  5491 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:27.893055  5491 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:27.770150  5407 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:23:27.793398  5482 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.355s	user 0.002s	sys 0.000s
I20260709 12:23:27.836362  1731 rpcz_store.cc:276] 0709 12:23:25.668757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.052023 (+1383266us) service_pool.cc:224] Handling call
0709 12:23:27.052445 (+   422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:27.052691 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:27.209486 (+156795us) log.cc:844] Serialized 14 byte log entry
0709 12:23:27.766235 (+556749us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:27.766305 (+    70us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:27.766364 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.772186 (+  5822us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.772406 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42788,"prepare.queue_time_us":575349,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":5664,"raft.queue_time_us":9070,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"thread_start_us":704761,"threads_started":2}
I20260709 12:23:27.835212  5488 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:23:27.895617  5488 consensus_queue.cc:1243] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 8.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:27.896940  5488 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:27.897351  5488 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:27.835788  5480 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.900512  5480 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:27.900653  5365 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:27.901650  5365 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:23:27.902007  5365 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:27.751700  5355 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.851053  5487 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.852324  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 481) took 1629 ms. Trace:
I20260709 12:23:27.904841  1726 rpcz_store.cc:276] 0709 12:23:26.223253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.070525 (+847272us) service_pool.cc:224] Handling call
0709 12:23:27.116467 (+ 45942us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.116688 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.116713 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.117209 (+   496us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.677070 (+559861us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:27.677138 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.677263 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.851991 (+174728us) raft_consensus.cc:1685] finished
0709 12:23:27.852020 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.852246 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":496608,"threads_started":1,"wal-append.queue_time_us":600651}
I20260709 12:23:27.765288  5270 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.738037  5269 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.913435  5487 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.915899  5269 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:27.755172  5329 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.860234  5312 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:27.767064  5490 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.861320  5462 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:27.861289  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:27.924507  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63415.1us
I20260709 12:23:27.872896  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2702) took 2085 ms. Trace:
I20260709 12:23:27.926117  1730 rpcz_store.cc:276] 0709 12:23:25.790424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.067323 (+1276899us) service_pool.cc:224] Handling call
0709 12:23:27.111247 (+ 43924us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.113400 (+  2153us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.113450 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.115438 (+  1988us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.482992 (+367554us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:27.483046 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.483136 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.872549 (+389413us) raft_consensus.cc:1685] finished
0709 12:23:27.872583 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.872807 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":367084,"threads_started":1,"wal-append.queue_time_us":643529}
W20260709 12:23:27.881093  5486 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 12:23:27.909498  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:27.888001  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:27.929275  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.929620  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:27.912206  2413 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:27.903426  1626 catalog_manager.cc:5697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 1 to 5, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 5 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:27.917420  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 800) took 1688 ms. Trace:
W20260709 12:23:27.918880  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:27.870424  4941 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.704s	user 0.011s	sys 0.000s
I20260709 12:23:27.932126  2415 rpcz_store.cc:276] 0709 12:23:26.228527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.244890 (+ 16363us) service_pool.cc:224] Handling call
0709 12:23:26.295910 (+ 51020us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.062520 (+766610us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:27.062556 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.063177 (+   621us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.534462 (+471285us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:27.534516 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.534603 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.917170 (+382567us) raft_consensus.cc:1685] finished
0709 12:23:27.917193 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.917355 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":766085,"raft.queue_time_us":766290,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":470816,"threads_started":1,"wal-append.queue_time_us":852088}
W20260709 12:23:27.934198  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.828663  5489 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:23:27.944437  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.834151  5349 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:27.886552  5475 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:27.913558  5270 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.920367  5335 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:27.957556  1727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.016s	sys 0.040s
I20260709 12:23:27.959914  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:27.961027  1727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.016s	sys 0.040s
I20260709 12:23:27.961387  2413 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
W20260709 12:23:27.960381  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 434395us
I20260709 12:23:27.921912  5249 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:27.969659  2412 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:27.972276  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:27.973800  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:27.973850  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:27.974704  2410 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:27.973678  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.978286  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:27.979940  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:27.981011  2410 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:27.981281  2412 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:27.981947  2410 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:27.983328  1872 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:23:27.987914  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:27.990049  1872 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:23:27.988230  2410 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:27.998675  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.937839  5296 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:27.944850  1771 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.010s	sys 0.037s
I20260709 12:23:28.001382  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:28.007849  1771 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.011s	sys 0.038s
I20260709 12:23:28.009174  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:27.898367  5488 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:27.976760  5349 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:27.980960  5475 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.008378  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 843898us
I20260709 12:23:27.976468  5249 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:28.003171  1876 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:28.002096  5342 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:27.995894  5337 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:27.998689  5312 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:27.932317  5490 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:27.998548  4941 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:23:27.964974  5335 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:28.003953  5383 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:28.003314  5487 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:28.047119  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:27.913575  5365 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.047969  2416 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
W20260709 12:23:28.035521  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.920663  5291 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.029671  1619 maintenance_manager.cc:419] P 4002daa3304c4fb4be053abd4780c195: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033385
I20260709 12:23:27.993774  4666 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 12:23:27.920547  5405 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.036919  1877 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
W20260709 12:23:28.038937  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:28.043697  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 529595us
W20260709 12:23:28.045601  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 343538us
W20260709 12:23:28.045991  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 334128us
W20260709 12:23:28.116811  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:28.049168  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.047505  5383 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.051039  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:28.027985  5349 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:28.034092  5312 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.135322  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:28.005275  5283 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.045857  5488 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:28.037519  5329 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:28.110414  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.006s	sys 0.021s
I20260709 12:23:28.103539  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:28.025470  5475 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.122645  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:28.141930  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:23:28.147651  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.007s	sys 0.021s
W20260709 12:23:28.152474  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 656134us
I20260709 12:23:28.153349  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:28.157562  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:28.121884  5365 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:28.170202  4941 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:28.173133  5499 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.163929  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:28.192129  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:28.122519  5383 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:28.170723  5296 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:28.140707  5386 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.140703  5159 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:28.170203  5497 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:23:28.148808  5507 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:28.143045  5312 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:28.203066  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:28.162742  5487 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:28.145596  5502 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:28.202440  5386 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:28.190138  4941 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:28.247220  5506 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:23:28.195549  5377 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:28.320762  5377 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.201119  5499 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.141300  5249 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:28.230669  5355 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:28.249614  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1902) took 1129 ms. Trace:
I20260709 12:23:28.327059  1875 rpcz_store.cc:276] 0709 12:23:27.119752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.119900 (+   148us) service_pool.cc:224] Handling call
0709 12:23:27.122444 (+  2544us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.157031 (+ 34587us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.157060 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.157475 (+   415us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.727294 (+569819us) spinlock_profiling.cc:243] Waited 8.33 ms on lock 0x7b48000ce7d8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:23:27.752811 (+ 25517us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:27.752863 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.752933 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.249198 (+496265us) raft_consensus.cc:1685] finished
0709 12:23:28.249228 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.249527 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49848,"raft.queue_time_us":35057,"raft.run_cpu_time_us":4603,"raft.run_wall_time_us":573949,"spinlock_wait_cycles":19163392,"thread_start_us":594991,"threads_started":1,"wal-append.queue_time_us":880367}
W20260709 12:23:28.250772  5493 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
I20260709 12:23:28.233644  4666 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.089s
I20260709 12:23:28.336333  4666 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.337517  5386 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:28.337376  4666 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:28.274958  5507 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:28.341097  5507 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:28.316562  5498 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 12:23:28.225484  5288 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:28.366209  5288 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:28.366411  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1828) took 1481 ms. Trace:
I20260709 12:23:28.367843  5288 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:28.371855  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 168858us
I20260709 12:23:28.373387  5144 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:28.257572  5349 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.148365  5477 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
W20260709 12:23:28.259663  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.264464  4770 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:28.154142  5489 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 12:23:28.367080  1799 rpcz_store.cc:276] 0709 12:23:26.884620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.008032 (+123412us) service_pool.cc:224] Handling call
0709 12:23:27.013228 (+  5196us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.013761 (+   533us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.013811 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.040992 (+ 27181us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.765352 (+724360us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:27.765426 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.765696 (+   270us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.366085 (+600389us) raft_consensus.cc:1685] finished
0709 12:23:28.366117 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.366320 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":25299,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":3928,"thread_start_us":656808,"threads_started":1,"wal-append.queue_time_us":947855}
I20260709 12:23:28.271274  5296 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:28.380795  5296 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:28.381368  5296 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:28.297773  1725 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:28.303112  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:28.313364  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:28.383818  1730 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:28.241812  5480 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:28.385424  5480 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:28.386198  5480 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:28.389969  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:28.393249  5508 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:28.347805  5502 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:28.297588  5501 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:28.317659  4941 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:28.408123  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 409972us
I20260709 12:23:28.409232  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:28.326491  5499 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:28.359664  5490 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:28.419096  1807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.020s	sys 0.094s
W20260709 12:23:28.419598  1807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.020s	sys 0.094s
I20260709 12:23:28.419932  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:28.422082  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:28.256495  5475 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:28.402114  2412 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
W20260709 12:23:28.407096  4899 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.952s	user 0.016s	sys 0.000s
W20260709 12:23:28.439679  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.398280  5270 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:28.406082  5502 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.467528  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:28.483449  2417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 805) took 2081 ms (client timeout 1647 ms). Trace:
W20260709 12:23:28.487396  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:28.487684  2417 rpcz_store.cc:269] 0709 12:23:26.401957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.402078 (+   121us) service_pool.cc:224] Handling call
0709 12:23:28.480259 (+2078181us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b5400573ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:28.483276 (+  3017us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3918041856}
W20260709 12:23:28.458868  5028 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.735s	user 0.010s	sys 0.006s
I20260709 12:23:28.506567  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1780) took 2147 ms. Trace:
I20260709 12:23:28.504573  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:28.408655  5487 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260709 12:23:28.421701  5501 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:28.484416  5490 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:28.524956  5393 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.537271  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.066s	sys 0.089s
W20260709 12:23:28.518579  5347 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.061s	user 0.014s	sys 0.000s
I20260709 12:23:28.497855  5502 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:28.562060  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.066s	sys 0.089s
I20260709 12:23:28.497745  5499 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:28.496259  5270 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:28.467800  4899 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.532306  4666 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:28.559835  5490 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:28.496964  5503 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
W20260709 12:23:28.572397  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 426923us
I20260709 12:23:28.592895  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:28.522362  5028 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
W20260709 12:23:28.575554  5494 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:23:28.591216  5310 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.002s	sys 0.014s
W20260709 12:23:28.585532  5398 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.128s	user 0.012s	sys 0.003s
I20260709 12:23:28.589782  5499 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:28.598299  4666 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.586297  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 425932us
I20260709 12:23:28.596870  5490 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:28.568746  5393 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:28.581161  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 530173us
I20260709 12:23:28.521116  1727 rpcz_store.cc:276] 0709 12:23:26.394248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.095565 (+701317us) service_pool.cc:224] Handling call
0709 12:23:27.095894 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.169892 (+ 73998us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.169930 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.170594 (+   664us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.961617 (+791023us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:28.487132 (+525515us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.506072 (+ 18940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.506187 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.506249 (+    62us) raft_consensus.cc:1685] finished
0709 12:23:28.506268 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.506469 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":190368,"prepare.queue_time_us":53601,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":104806,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":83516,"thread_start_us":1171281,"threads_started":2,"wal-append.queue_time_us":974555}
I20260709 12:23:28.673277  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:28.674117  2417 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:28.594163  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:28.679443  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:28.680775  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86710.3us
I20260709 12:23:28.582306  5270 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:28.689801  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:28.681974  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:28.598656  5405 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.701599  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 304183us
I20260709 12:23:28.657675  5269 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:23:28.647001  5310 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.003s	sys 0.014s
I20260709 12:23:28.691080  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:28.708456  2412 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:28.639294  5514 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
W20260709 12:23:28.715467  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:28.687222  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:28.700574  5476 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.930s	user 0.015s	sys 0.000s
W20260709 12:23:28.719614  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.005s	sys 0.013s
W20260709 12:23:28.720067  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.005s	sys 0.013s
W20260709 12:23:28.720383  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 545586us
I20260709 12:23:28.704536  5521 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:28.654605  5398 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:28.733719  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 341150us
I20260709 12:23:28.758342  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:28.681640  4666 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:28.725394  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 200685us
I20260709 12:23:28.707939  5269 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.745442  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:28.721017  5383 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.517s	user 0.010s	sys 0.003s
I20260709 12:23:28.694326  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:28.760620  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:28.763845  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.771818  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:28.725102  5508 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:28.720263  5515 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:23:28.738078  5335 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.679s	user 0.013s	sys 0.001s
I20260709 12:23:28.781257  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:28.782241  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:28.704872  5405 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.778626  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:28.803382  1728 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:28.801156  1725 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
W20260709 12:23:28.781924  5511 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
W20260709 12:23:28.819103  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.777869  5269 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:28.763864  5524 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:28.818603  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:28.778200  5347 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:28.724212  5499 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260709 12:23:28.791206  5248 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:28.807085  5365 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.604s	user 0.012s	sys 0.002s
I20260709 12:23:28.783092  5521 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:28.845458  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 151747us
I20260709 12:23:28.850160  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:28.819689  5515 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.724491  5476 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:28.821064  5283 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.769140  5519 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.082s
I20260709 12:23:28.828457  5535 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:23:28.872543  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 114318us
I20260709 12:23:28.787583  5523 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.093s
I20260709 12:23:28.833847  5524 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.820876  5509 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:28.777602  5520 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.086s
I20260709 12:23:28.815359  5383 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.795403  5291 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:23:28.849172  5355 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:28.862169  5028 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:28.849912  5501 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:28.820209  5335 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.898214  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:28.906895  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.846268  5407 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:23:28.889333  4899 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:28.920044  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1840) took 1259 ms. Trace:
I20260709 12:23:28.848682  5347 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:28.940419  1807 rpcz_store.cc:276] 0709 12:23:27.660486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.660716 (+   230us) service_pool.cc:224] Handling call
0709 12:23:27.696412 (+ 35696us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.696753 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:23:27.696799 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.723562 (+ 26763us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.420119 (+696557us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:23:28.420181 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.420283 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.919738 (+499455us) raft_consensus.cc:1685] finished
0709 12:23:28.919768 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:28.919974 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":661480,"threads_started":1,"wal-append.queue_time_us":856958}
I20260709 12:23:28.917897  5028 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:28.925302  5405 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:28.926095  5335 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:28.966430  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68352.8us
W20260709 12:23:28.927235  5525 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
W20260709 12:23:28.928500  5516 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:23:28.928829  5509 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.930403  5500 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.897s	user 0.004s	sys 0.005s
I20260709 12:23:28.932524  5386 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.887048  5492 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:23:28.934381  5312 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.620s	user 0.009s	sys 0.000s
I20260709 12:23:28.874477  5515 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.030961  5312 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.032528  5515 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:28.879889  5348 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:29.033177  5312 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.033804  5312 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:29.034175  5312 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.034503  5312 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:28.918958  5508 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.035045  5348 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:29.035524  5348 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.035722  5383 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:28.862275  5248 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.015816  5270 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:28.888528  5535 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:28.926637  5501 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:29.008924  5499 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:28.926664  5529 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:23:28.926055  5528 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:23:29.012979  5335 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.017529  5544 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:29.011194  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 238834us
I20260709 12:23:29.076510  5248 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:29.069435  5028 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:29.066648  5144 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.687s	user 0.013s	sys 0.001s
I20260709 12:23:29.078732  5144 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:23:29.079221  4941 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.626s	user 0.013s	sys 0.000s
I20260709 12:23:28.919540  5540 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:29.015506  1729 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.081810  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 884) took 1055 ms. Trace:
I20260709 12:23:29.082322  2413 rpcz_store.cc:276] 0709 12:23:28.026603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.026764 (+   161us) service_pool.cc:224] Handling call
0709 12:23:28.027222 (+   458us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:28.035392 (+  8170us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:28.035431 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:28.036084 (+   653us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.392037 (+355953us) spinlock_profiling.cc:243] Waited 6.84 ms on lock 0x7b18002fae80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:23:28.674018 (+281981us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:28.674083 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.674163 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:29.081412 (+407249us) raft_consensus.cc:1685] finished
0709 12:23:29.081439 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.081718 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22600,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":49618,"spinlock_wait_cycles":15728000,"thread_start_us":266487,"threads_started":1,"wal-append.queue_time_us":710063}
W20260709 12:23:28.941733  5306 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.144s	user 0.014s	sys 0.003s
I20260709 12:23:28.886313  5283 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.087085  5283 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.087250  4941 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.089406  2410 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:29.090215  4941 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.017659  5509 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.091212  1731 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:29.092693  5509 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.028657  5542 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.093777  5542 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.069999  5499 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.096349  5542 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:28.968372  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:29.097915  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 129633us
I20260709 12:23:29.099777  1729 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:29.100502  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:29.102355  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:29.103348  5492 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:29.103806  5491 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:29.056003  5502 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.065871  5541 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
W20260709 12:23:29.104813  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.010841  5393 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:29.106694  5502 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:29.107100  5398 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:29.015023  5365 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:29.044625  5386 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:29.056061  5480 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.669s	user 0.004s	sys 0.009s
W20260709 12:23:29.062060  5507 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.720s	user 0.007s	sys 0.004s
I20260709 12:23:29.065541  5476 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:29.111109  5480 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.111632  5507 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.113150  5476 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.113581  5480 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.114470  1730 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:29.011639  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 103492us
I20260709 12:23:28.926028  5521 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.066588  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2710) took 1597 ms. Trace:
I20260709 12:23:28.919262  5355 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.895953  5524 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:29.062077  5490 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.077358  5335 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.079002  5028 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.117823  5499 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.122200  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:29.122433  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:29.123020  2414 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:29.124106  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:29.117700  1723 rpcz_store.cc:276] 0709 12:23:27.468901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.521783 (+ 52882us) service_pool.cc:224] Handling call
0709 12:23:27.522156 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.528813 (+  6657us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.528849 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.529450 (+   601us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.782032 (+1252582us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b1800376cc0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:23:28.874708 (+ 92676us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:23:28.874774 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.874887 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:29.066238 (+191351us) raft_consensus.cc:1685] finished
0709 12:23:29.066263 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.066514 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14810,"raft.queue_time_us":6321,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":77449,"spinlock_wait_cycles":8092032,"thread_start_us":92439,"threads_started":1,"wal-append.queue_time_us":1517031}
I20260709 12:23:29.118191  5355 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:29.103997  5296 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.722s	user 0.013s	sys 0.000s
I20260709 12:23:29.104274  5492 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:29.127705  5492 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:29.129415  5492 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.067940  5535 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.110519  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1903) took 1990 ms. Trace:
I20260709 12:23:29.119936  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:29.131289  5535 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:29.131237  1881 rpcz_store.cc:276] 0709 12:23:27.120112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.120735 (+   623us) service_pool.cc:224] Handling call
0709 12:23:27.121149 (+   414us) raft_consensus.cc:1471] Updating replica for 5 ops
0709 12:23:27.298744 (+177595us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.298815 (+    71us) raft_consensus.cc:1539] Triggering prepare for 5 ops
0709 12:23:27.316133 (+ 17318us) log.cc:844] Serialized 130 byte log entry
0709 12:23:27.705491 (+389358us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:23:27.705545 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.705653 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:28.929795 (+1224142us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:29.110137 (+180342us) raft_consensus.cc:1685] finished
0709 12:23:29.110163 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.110445 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12526,"raft.queue_time_us":243958,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":36673,"spinlock_wait_cycles":9415040,"thread_start_us":295746,"threads_started":1,"wal-append.queue_time_us":584928}
I20260709 12:23:29.065214  5312 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.113905  5476 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.135072  5365 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:29.136305  5365 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.136914  5476 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:29.138746  5513 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.556s	user 0.002s	sys 0.000s
I20260709 12:23:29.139202  2412 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.140405  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 871) took 2181 ms. Trace:
I20260709 12:23:29.135205  5312 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:29.141165  5505 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.614s	user 0.001s	sys 0.002s
I20260709 12:23:29.064270  5347 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.068190  5269 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.144630  5544 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:29.144802  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:29.145593  5544 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:29.146023  5544 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:29.147176  5269 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:29.071445  5488 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.913s	user 0.014s	sys 0.000s
I20260709 12:23:29.148020  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 880) took 1523 ms. Trace:
I20260709 12:23:29.149222  2419 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260709 12:23:29.149682  2419 pending_rounds.cc:85] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 6
I20260709 12:23:29.149906  2419 pending_rounds.cc:107] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 12:23:29.121450  5490 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:29.148900  2418 rpcz_store.cc:276] 0709 12:23:27.624372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.624625 (+   253us) service_pool.cc:224] Handling call
0709 12:23:27.625016 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.625426 (+   410us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.625475 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.626123 (+   648us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.056534 (+430411us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:28.056593 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.056687 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:29.140474 (+1083787us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:29.146480 (+  6006us) raft_consensus.cc:1685] finished
0709 12:23:29.146522 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.147922 (+  1400us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":171645,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":9253,"spinlock_wait_cycles":268087168,"thread_start_us":429964,"threads_started":1,"wal-append.queue_time_us":870041}
W20260709 12:23:29.057480  5342 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.027s	user 0.010s	sys 0.004s
I20260709 12:23:29.121865  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:29.149962  5405 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:29.122056  2413 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:29.160190  5405 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.147979  5544 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.160833  5405 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:29.161223  5386 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.161638  5544 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.160143  2416 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:29.147925  5269 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:29.165460  5269 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.159291  5296 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.169041  5296 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:29.170536  5512 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.611s	user 0.003s	sys 0.000s
I20260709 12:23:29.172134  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:29.152384  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:29.140995  2411 rpcz_store.cc:276] 0709 12:23:26.958497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.960656 (+  2159us) service_pool.cc:224] Handling call
0709 12:23:26.961032 (+   376us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:27.661777 (+700745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.661836 (+    59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:27.662722 (+   886us) log.cc:844] Serialized 52 byte log entry
0709 12:23:28.195299 (+532577us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:28.195354 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.198038 (+  2684us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:29.137954 (+939916us) spinlock_profiling.cc:243] Waited 57.5 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:29.139996 (+  2042us) raft_consensus.cc:1685] finished
0709 12:23:29.140031 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.140326 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":200198,"raft.queue_time_us":202264,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":6746,"spinlock_wait_cycles":132251008,"thread_start_us":893690,"threads_started":2,"wal-append.queue_time_us":798842}
I20260709 12:23:29.174444  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:29.155010  1732 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:29.150146  2419 raft_consensus.cc:2889] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:29.121474  5335 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:29.175527  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:29.176832  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:29.184157  1723 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 12:23:29.184381  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 439035us
W20260709 12:23:29.177786  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 327785us
I20260709 12:23:29.175232  1806 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
W20260709 12:23:29.188854  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 72459.9us
I20260709 12:23:29.191263  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:29.191895  2410 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 12:23:29.192812  2410 pending_rounds.cc:85] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 8
I20260709 12:23:29.193183  2410 pending_rounds.cc:107] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:23:29.193460  2410 raft_consensus.cc:2889] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:29.194270  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 808) took 1214 ms. Trace:
I20260709 12:23:29.194612  2410 rpcz_store.cc:276] 0709 12:23:27.980164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.980524 (+   360us) service_pool.cc:224] Handling call
0709 12:23:27.980881 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.088974 (+1108093us) spinlock_profiling.cc:243] Waited 46.5 ms on lock 0x7b58006f62d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd340aa 00007f16abdb6447 00007f16abdcefe7 00007f16abdd07de 00007f16abddab49 00007f16abddb416 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4
0709 12:23:29.193933 (+104959us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.194209 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106841856}
I20260709 12:23:29.180912  2412 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:29.201072  1799 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:29.202977  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:29.204684  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 386156us
I20260709 12:23:29.190701  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:29.204190  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:29.214834  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.005s	sys 0.013s
I20260709 12:23:29.216662  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:29.231112  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.006s	sys 0.013s
I20260709 12:23:29.231449  1802 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:29.218372  2410 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:29.113983  5507 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:29.233278  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:29.233459  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:29.234056  2411 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:29.234731  1807 raft_consensus.cc:2410] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:23:29.218139  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:29.219650  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:29.224077  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:29.224769  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:29.248358  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.253418  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.253511  4941 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:29.255190  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.256064  1881 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:29.256337  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:29.256597  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.257001  1729 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:29.257372  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.257532  1875 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:29.258216  1876 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:29.255088  4941 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.236145  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:29.183851  5296 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:29.183580  5312 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:29.182884  5477 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:29.266271  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.278540  1799 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:29.266793  5555 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:23:29.274623  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:29.274642  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.268280  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:29.282137  1800 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:29.283835  1877 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:29.279886  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:29.284909  2416 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:29.280524  2410 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.285704  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:29.277788  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.110250  5348 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.292425  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:29.298041  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 107402us
I20260709 12:23:29.201637  5144 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20260709 12:23:29.258695  5540 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:29.316176  5552 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:29.316625  5540 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.317185  5551 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:29.317286  5540 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:29.317873  5551 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.318356  5551 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:29.315383  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.321924  4941 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.316082  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.325017  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:29.259840  5349 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.327683  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:29.330941  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45322.1us
I20260709 12:23:29.281561  4899 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.313462  5542 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.314502  1876 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:29.314767  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.315299  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:29.348485  1872 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:29.315614  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:29.349148  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 89212.4us
I20260709 12:23:29.349411  1880 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:29.315923  1874 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:29.351845  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.342545  2410 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:29.312857  5538 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.342916  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:29.350020  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:29.352578  1877 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:29.352670  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:29.354964  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:29.245057  5335 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.364521  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:29.292265  5502 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.315348  5537 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:29.367465  1727 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:29.369117  5552 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.363898  5501 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:23:29.340692  5398 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:29.281541  5514 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260709 12:23:29.404842  1872 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:29.413573  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:29.279274  5524 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.412916  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.375756  1728 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:29.294538  5499 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.298689  5383 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.106s
I20260709 12:23:29.383109  1729 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:29.388121  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.449435  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:29.450099  1728 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:29.391019  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:29.400010  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:29.344964  1846 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.647s (SENT) [suppressed 28 similar messages]
I20260709 12:23:29.402091  1880 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:29.452986  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.404080  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:29.372416  4899 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.405411  5552 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:29.456018  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 106108us
I20260709 12:23:29.282413  5480 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.095s
I20260709 12:23:29.408058  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:29.325927  4666 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:29.317886  5269 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.476987  5555 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:29.477090  5269 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.477699  5555 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.478168  5555 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.477906  5269 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.368755  5520 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:29.410034  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:29.443884  5491 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.365825  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:29.366310  5349 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.388546  5335 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.399343  5502 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.345968  5312 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.280547  5342 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.109s
I20260709 12:23:29.478981  5476 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:29.505115  1728 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:29.466694  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:29.481812  5520 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.316847  5546 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.145s
I20260709 12:23:29.406761  5398 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.475559  1726 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:29.466722  5306 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:23:29.460994  5541 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:29.377476  5542 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:29.444358  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 132951us
I20260709 12:23:29.569157  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:29.570178  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:29.378518  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.293874  5477 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:29.509012  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 289421us
I20260709 12:23:29.459966  5499 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.480037  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:29.501588  5509 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.577697  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.519810  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:29.423528  1872 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:29.579942  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:29.580643  1872 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:29.578397  1877 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:29.583951  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.584679  1877 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:29.573180  5337 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.497s	user 0.003s	sys 0.026s
W20260709 12:23:29.585508  5337 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.536s	user 0.004s	sys 0.026s
I20260709 12:23:29.585911  5337 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:29.587208  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:29.587407  5337 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.595784  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:29.557102  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.377964  5538 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.509131  5476 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.517995  5520 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.600521  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.618345  1877 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:29.603487  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:29.604732  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.573210  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:29.622151  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.537834  1725 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:29.510264  5558 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:29.520037  5398 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:29.638230  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:29.613013  1802 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:29.640616  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:29.520524  5541 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:29.647533  1803 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:29.575774  5191 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:29.651638  5028 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.530s	user 0.009s	sys 0.001s
I20260709 12:23:29.652091  5558 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.509660  5312 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.495805  5491 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.678167  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:29.613880  5476 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:29.677269  1878 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:29.687116  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:29.687978  1802 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:29.504386  5561 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.143s
I20260709 12:23:29.691048  1872 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:29.694834  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:29.696287  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.660452  5488 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:29.704943  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.710682  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:29.716090  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 371136us
I20260709 12:23:29.610026  5509 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.720036  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:29.704110  5544 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.542s	user 0.011s	sys 0.000s
I20260709 12:23:29.720825  1872 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.653282  5542 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.698240  5249 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:29.700316  5348 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:29.741189  1872 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:29.742008  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.681231  5191 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.724205  5306 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:23:29.737730  5405 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.576s	user 0.011s	sys 0.000s
I20260709 12:23:29.739328  5564 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.733714  5475 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:29.769554  5306 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.741215  5540 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.777448  5405 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.791031  5564 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.804384  5499 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:29.789314  5567 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.781988  5552 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:29.779662  5570 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.816884  5570 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.804755  5491 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:29.804615  5565 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:23:29.760288  5541 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:29.809260  5510 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.223s	user 0.000s	sys 0.006s
I20260709 12:23:29.819062  5567 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:29.920675  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 462441us
I20260709 12:23:29.923736  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:29.939345  1725 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:29.780230  5269 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:29.992864  5604 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:29.994225  5604 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.995038  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:29.997421  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:29.815639  5355 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.690s	user 0.012s	sys 0.000s
I20260709 12:23:29.999785  5355 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.001704  5509 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.004379  5355 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.000818  5191 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.998733  5604 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.943869  5603 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:23:30.024386  5355 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.025207  5405 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.025763  5355 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:29.998592  1802 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
I20260709 12:23:29.800392  5306 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:29.839923  5541 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:30.050024  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.050153  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.051476  1872 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:30.053591  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.054533  5540 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.067193  5558 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:30.071604  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 463310us
I20260709 12:23:30.054764  1872 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:30.074872  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.076222  1872 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:30.077911  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:29.792225  4899 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:29.878759  5567 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.871564  5570 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:29.792698  5475 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:29.730883  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:30.127091  1872 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:30.090615  5306 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.146806  5191 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.147897  5558 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:30.147881  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 416882us
I20260709 12:23:30.151808  5289 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:30.161158  5312 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.169055  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1833) took 2673 ms. Trace:
I20260709 12:23:30.170142  5348 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:23:30.135854  4666 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.015s	sys 0.010s
I20260709 12:23:30.179455  1626 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 8, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 8 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:30.184382  5475 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:29.837914  5538 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.170674  5564 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:30.194190  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.199486  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:30.209468  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 262163us
I20260709 12:23:30.220405  5191 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.227530  5306 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.229956  5191 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.231184  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:30.231879  5577 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.245s	user 0.000s	sys 0.001s
I20260709 12:23:30.232148  1802 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:30.233598  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.235189  1881 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:30.238823  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.239845  1877 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:30.240031  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.243535  1875 raft_consensus.cc:2955] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:30.248013  5355 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.248909  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:30.202123  5289 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:30.254917  1800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.010s	sys 0.010s
W20260709 12:23:30.263343  1726 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.742s	user 0.020s	sys 0.009s
I20260709 12:23:30.264063  1726 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
W20260709 12:23:30.266955  1731 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.018s	sys 0.033s
I20260709 12:23:30.221271  5558 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:30.300056  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:30.300976  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52502.6us
I20260709 12:23:30.266633  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:30.309408  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49318.5us
I20260709 12:23:30.271180  5348 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:30.322676  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.330978  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 51533.1us
W20260709 12:23:30.333882  5535 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.201s	user 0.016s	sys 0.004s
I20260709 12:23:30.340164  5582 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:30.350121  5558 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.356299  5538 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:30.356741  5555 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.878s	user 0.012s	sys 0.008s
I20260709 12:23:30.357249  5538 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.358110  5556 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:30.358604  5556 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.359000  5556 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:30.371402  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:30.373106  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 106848us
I20260709 12:23:30.376240  5348 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.379354  5594 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:30.380004  5538 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:30.383023  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53208.1us
I20260709 12:23:30.384941  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:30.386145  5551 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.067s	user 0.004s	sys 0.014s
I20260709 12:23:30.386574  5365 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:30.357277  5555 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.204312  1804 rpcz_store.cc:276] 0709 12:23:27.495743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.496227 (+   484us) service_pool.cc:224] Handling call
0709 12:23:27.532931 (+ 36704us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.620705 (+ 87774us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:27.620754 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.666883 (+ 46129us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.175935 (+509052us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:28.175987 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.176091 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:29.702879 (+1526788us) spinlock_profiling.cc:243] Waited 54.6 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:30.168281 (+465402us) raft_consensus.cc:1685] finished
0709 12:23:30.168332 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.168776 (+   444us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87200,"raft.queue_time_us":87716,"raft.run_cpu_time_us":428,"raft.run_wall_time_us":25108,"spinlock_wait_cycles":125620992,"thread_start_us":432727,"threads_started":1,"wal-append.queue_time_us":781389}
I20260709 12:23:30.396270  1804 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 12:23:30.392602  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65955.4us
I20260709 12:23:30.388566  1799 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:30.413127  1877 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:30.389842  5365 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:30.421823  5365 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.423996  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1805) took 1060 ms. Trace:
I20260709 12:23:30.425518  5365 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:30.424544  1729 rpcz_store.cc:276] 0709 12:23:29.354381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.448659 (+ 94278us) service_pool.cc:224] Handling call
0709 12:23:29.449107 (+   448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.479583 (+ 30476us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:29.937632 (+458049us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.257729 (+320097us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:30.349429 (+ 91700us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.349627 (+   198us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:30.349671 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.363550 (+ 13879us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.411193 (+ 47643us) raft_consensus.cc:1685] finished
0709 12:23:30.411215 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.414760 (+  3545us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51959,"prepare.queue_time_us":32219,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":270105,"raft.run_cpu_time_us":470,"raft.run_wall_time_us":44261,"thread_start_us":687711,"threads_started":2,"wal-append.queue_time_us":413993}
I20260709 12:23:30.386160  5191 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260709 12:23:30.427479  5191 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:30.427937  5191 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:30.429217  5191 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.430158  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:30.380082  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68747.5us
I20260709 12:23:30.410992  1844 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:30.431125  5191 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:30.410903  5535 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:23:30.431674  1802 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:30.391613  5476 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.688s	user 0.011s	sys 0.008s
I20260709 12:23:30.435045  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2578) took 1121 ms. Trace:
I20260709 12:23:30.437193  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:30.455127  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68438.8us
I20260709 12:23:30.447861  1878 rpcz_store.cc:276] 0709 12:23:29.313709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.315156 (+  1447us) service_pool.cc:224] Handling call
0709 12:23:30.434546 (+1119390us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b5400026554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:30.434755 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":822789376}
I20260709 12:23:30.389436  5555 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.471922  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2580) took 1152 ms. Trace:
I20260709 12:23:30.475808  1877 rpcz_store.cc:276] 0709 12:23:29.314297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.753661 (+439364us) service_pool.cc:224] Handling call
0709 12:23:30.456499 (+702838us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:30.482307  5538 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:30.485236  5249 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.733s	user 0.011s	sys 0.005s
I20260709 12:23:30.491072  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:30.491909  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.492856  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:30.494325  5349 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.495846  5535 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:30.496474  5535 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:30.496940  5377 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:30.497645  5377 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:30.497578  5535 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.525146  1802 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:30.492900  1878 raft_consensus.cc:2376] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9. Re-sending same reply.
I20260709 12:23:30.386744  5551 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.481335  5594 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:30.499187  5499 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:30.490502  5335 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:30.506278  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:30.503103  4770 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:30.551008  5398 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.860s	user 0.017s	sys 0.001s
W20260709 12:23:30.551306  5520 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.892s	user 0.003s	sys 0.015s
I20260709 12:23:30.475502  5476 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.551903  5520 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.551855  5398 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:30.555177  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 125209us
I20260709 12:23:30.572593  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.561563  5476 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.563606  5551 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.563783  5594 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.564579  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1842) took 1059 ms. Trace:
W20260709 12:23:30.565321  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 128344us
I20260709 12:23:30.533592  5604 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:30.568085  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:30.569088  5555 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.570166  1880 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:30.570732  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:30.553886  5557 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:30.501458  5477 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.569140  5499 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:30.571125  5398 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.595517  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:30.577966  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 203206us
I20260709 12:23:30.607442  5480 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:30.608471  1732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2705) took 3141 ms (client timeout 1573 ms). Trace:
W20260709 12:23:30.608958  1732 rpcz_store.cc:269] 0709 12:23:27.466472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.474923 (+  8451us) service_pool.cc:224] Handling call
0709 12:23:30.607999 (+3133076us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54003cd2d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:30.608215 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3585909504}
I20260709 12:23:30.609880  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:30.611155  5555 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:30.612426  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2587) took 1294 ms (client timeout 1608 ms). Trace:
W20260709 12:23:30.612952  1880 rpcz_store.cc:269] 0709 12:23:29.317666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.452690 (+135024us) service_pool.cc:224] Handling call
0709 12:23:30.612090 (+1159400us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540021c5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:30.612344 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2566908288}
I20260709 12:23:30.613896  5555 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.614296  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.615036  2410 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:30.616925  4770 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.618736  5499 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.608206  5480 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.577741  1800 rpcz_store.cc:276] 0709 12:23:29.505236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.505432 (+   196us) service_pool.cc:224] Handling call
0709 12:23:29.515874 (+ 10442us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.598615 (+ 82741us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:29.598665 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:29.599346 (+   681us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.359093 (+759747us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:30.359228 (+   135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.359333 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.564208 (+204875us) raft_consensus.cc:1685] finished
0709 12:23:30.564232 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.564473 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26906,"raft.queue_time_us":171827,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":28510,"thread_start_us":583392,"threads_started":1,"wal-append.queue_time_us":785868}
W20260709 12:23:30.622452  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54571.2us
I20260709 12:23:30.615073  2414 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 5.
I20260709 12:23:30.605511  5365 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:30.620252  5477 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260709 12:23:30.623943  5538 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.641098  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.641996  5480 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.643554  5312 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:30.653261  5312 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.653785  5312 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:30.647578  5249 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:30.662694  1880 raft_consensus.cc:2376] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9. Re-sending same reply.
I20260709 12:23:30.664469  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:30.652391  5504 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.674264  5504 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.677217  5504 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.653985  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.684634  1800 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:30.687350  1807 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:30.687573  1806 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:23:30.648106  5538 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.693065  5538 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:30.694614  5520 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.704901  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1931) took 1365 ms. Trace:
I20260709 12:23:30.705454  1872 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:30.707155  5575 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.707908  5575 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.706599  1802 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:30.708604  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 475) took 1367 ms. Trace:
I20260709 12:23:30.717243  1872 rpcz_store.cc:276] 0709 12:23:29.341111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.327863 (+986752us) service_pool.cc:224] Handling call
0709 12:23:30.705170 (+377307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:30.706284 (+  1114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.708366 (+  2082us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b5400117950. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:30.708528 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":749362944}
I20260709 12:23:30.719131  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.719270  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:30.723647  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:30.725889  5563 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.982s	user 0.002s	sys 0.004s
I20260709 12:23:30.664321  5249 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:30.726289  1798 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:30.728420  5249 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.705420  1875 rpcz_store.cc:276] 0709 12:23:29.340084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.241304 (+901220us) service_pool.cc:224] Handling call
0709 12:23:30.241820 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:30.242646 (+   826us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:30.267041 (+ 24395us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.553607 (+286566us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.572979 (+ 19372us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.573165 (+   186us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:30.602544 (+ 29379us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.602737 (+   193us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:30.602772 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.602884 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.636651 (+ 33767us) spinlock_profiling.cc:243] Waited 1.16 ms on lock 0x7b1800398860. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1
0709 12:23:30.704600 (+ 67949us) raft_consensus.cc:1685] finished
0709 12:23:30.704625 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.704803 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10617,"prepare.queue_time_us":26974,"prepare.run_cpu_time_us":674,"prepare.run_wall_time_us":8351,"raft.queue_time_us":14386,"raft.run_cpu_time_us":451,"raft.run_wall_time_us":6324,"spinlock_wait_cycles":6880896,"thread_start_us":254437,"threads_started":1,"wal-append.queue_time_us":349920}
I20260709 12:23:30.732637  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.722365  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.740442  1804 raft_consensus.cc:2368] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9.
W20260709 12:23:30.723979  5614 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:30.755089  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.753494  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.757344  5523 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.757915  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.758245  5523 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.761247  5523 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.758980  1800 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:30.775416  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:30.741488  5575 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:30.775312  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2716) took 1193 ms. Trace:
I20260709 12:23:30.787262  1804 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:30.795428  1802 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:30.801900  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2574) took 1489 ms. Trace:
I20260709 12:23:30.803884  5537 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:30.804627  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2596) took 1483 ms. Trace:
W20260709 12:23:30.805325  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 167365us
I20260709 12:23:30.808276  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:30.809577  4770 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.812923  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.814949  1875 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:30.824011  5289 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.827344  5289 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:30.828712  5607 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.366s	user 0.001s	sys 0.000s
I20260709 12:23:30.830698  5524 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.770953  5557 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:30.771104  5538 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:30.845746  2411 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:30.855031  1798 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
W20260709 12:23:30.851902  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.843981  2417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 731) took 1725 ms (client timeout 1896 ms). Trace:
I20260709 12:23:30.805259  1881 rpcz_store.cc:276] 0709 12:23:29.320734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.119118 (+798384us) service_pool.cc:224] Handling call
0709 12:23:30.804302 (+685184us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b5400116f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:30.804527 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48594816}
I20260709 12:23:30.857954  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:30.859835  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.787875  1731 rpcz_store.cc:276] 0709 12:23:29.574877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.575009 (+   132us) service_pool.cc:224] Handling call
0709 12:23:29.612310 (+ 37301us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:30.269387 (+657077us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:30.269457 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:30.270292 (+   835us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.540554 (+270262us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:30.540610 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.573660 (+ 33050us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.763705 (+190045us) raft_consensus.cc:1685] finished
0709 12:23:30.763723 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.768460 (+  4737us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":159912,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":7431,"thread_start_us":923439,"threads_started":2,"wal-append.queue_time_us":372405}
I20260709 12:23:30.861081  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.802496  1876 rpcz_store.cc:276] 0709 12:23:29.312375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.313910 (+  1535us) service_pool.cc:224] Handling call
0709 12:23:29.314298 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.347514 (+ 33216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.801214 (+1453700us) spinlock_profiling.cc:243] Waited 63.3 ms on lock 0x7b5400233d20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abddba02 00007f16abdd7dd6 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:23:30.801660 (+   446us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145554688}
I20260709 12:23:30.859019  1874 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:30.844882  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 722) took 2881 ms. Trace:
I20260709 12:23:30.866297  2419 rpcz_store.cc:276] 0709 12:23:27.963641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.968092 (+  4451us) service_pool.cc:224] Handling call
0709 12:23:27.975938 (+  7846us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.176263 (+1200325us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.844286 (+1668023us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400840154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:30.844626 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2697385856}
W20260709 12:23:30.856099  2417 rpcz_store.cc:269] 0709 12:23:29.118767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.122140 (+  3373us) service_pool.cc:224] Handling call
0709 12:23:30.843358 (+1721218us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b5400488d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:30.843713 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1293350144}
I20260709 12:23:30.870889  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:30.878942  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 477) took 1631 ms (client timeout 2134 ms). Trace:
W20260709 12:23:30.879647  1798 rpcz_store.cc:269] 0709 12:23:29.224642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.224982 (+   340us) service_pool.cc:224] Handling call
0709 12:23:30.723353 (+1498371us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3eee 000000000036eb39 00007f16b0d271e0 00007f16b0df00c4 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:23:30.855624 (+132271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4090240}
I20260709 12:23:30.881614  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.845103  5398 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.851500  5191 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.757344  5556 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.851341  5575 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:30.895478  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.903616  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.905298  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.907562  1875 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:30.896827  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:30.909145  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 100970us
I20260709 12:23:30.910166  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.911113  1881 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:30.834692  5502 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:30.915165  1800 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:30.845268  5538 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.925437  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.930471  1798 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:30.931851  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.923113  1874 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:30.934684  1872 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:30.932675  1881 raft_consensus.cc:2376] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9. Re-sending same reply.
I20260709 12:23:30.936748  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:30.940182  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.941386  5487 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:30.942054  5487 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.942464  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.943280  1878 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:30.943722  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.944535  1876 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:30.945099  5619 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:30.945713  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 811) took 1622 ms. Trace:
I20260709 12:23:30.945849  5619 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.947160  5621 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.947841  5621 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.948418  5619 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:30.949105  5619 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:30.949630  5619 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:30.951856  5625 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:23:30.953321  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1927) took 1684 ms. Trace:
I20260709 12:23:30.954100  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:30.954869  1806 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:30.953728  1873 rpcz_store.cc:276] 0709 12:23:29.268711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.269171 (+   460us) service_pool.cc:224] Handling call
0709 12:23:29.298118 (+ 28947us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.641779 (+343661us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:30.255338 (+613559us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.531323 (+275985us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:30.565880 (+ 34557us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:30.566053 (+   173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.952908 (+386855us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b540021acd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:30.952956 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.953131 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":323,"prepare.queue_time_us":173761,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":6848,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":7538,"spinlock_wait_cycles":789605760,"thread_start_us":746433,"threads_started":2,"wal-append.queue_time_us":358975}
I20260709 12:23:30.946269  2415 rpcz_store.cc:276] 0709 12:23:29.323215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.323404 (+   189us) service_pool.cc:224] Handling call
0709 12:23:29.323818 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.578055 (+254237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.578083 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:29.578582 (+   499us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.149746 (+571164us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:30.149932 (+   186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.945021 (+795089us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b540048a3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:30.945069 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.945145 (+    76us) raft_consensus.cc:1685] finished
0709 12:23:30.945162 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.945476 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":584039680,"thread_start_us":490879,"threads_started":1,"wal-append.queue_time_us":762367}
I20260709 12:23:30.959440  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:30.960281  5312 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:30.960650  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.960851  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:30.961622  1800 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:30.946290  1876 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:30.961993  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.950062  5621 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:30.944537  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.804731  5537 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:30.925735  1880 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:30.968344  1626 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 1 to 5, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 5 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:30.319317   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:30.971866  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:23:30.973412  5597 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.722s	user 0.002s	sys 0.000s
I20260709 12:23:30.973610  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.976030  5556 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:30.961823  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:30.978575  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:30.978454  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:30.986742  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 477) took 1638 ms. Trace:
I20260709 12:23:30.987258  1876 rpcz_store.cc:276] 0709 12:23:29.341603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.945357 (+1603754us) service_pool.cc:224] Handling call
0709 12:23:30.945832 (+   475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:30.962634 (+ 16802us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.979970 (+ 17336us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:30.980739  1873 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:30.989445  2416 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:23:30.981385  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:30.990056  1878 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:23:30.985538  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:30.991732  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63560.8us
I20260709 12:23:30.991997  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:30.990480  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:30.944595  5487 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:30.978430  5603 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.920897  5398 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:31.000603  5603 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:31.001025  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.001976  2413 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:31.010171  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1936) took 1598 ms. Trace:
I20260709 12:23:31.010636  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:31.010913  5502 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.011462  5502 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.012094  5310 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:31.012648  5310 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.013000  5310 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.013954  5502 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:31.017771  5329 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:31.010665  1878 rpcz_store.cc:276] 0709 12:23:29.412027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.999484 (+1587457us) service_pool.cc:224] Handling call
0709 12:23:31.009538 (+ 10054us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.009757 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.009780 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:31.009854 (+    74us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:31.009887 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.009939 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.010075 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.036055  4666 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:31.035892  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.038259  1881 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.020308  1874 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:31.049116  1880 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:23:31.050223  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 483) took 1551 ms. Trace:
I20260709 12:23:31.050625  1880 rpcz_store.cc:276] 0709 12:23:29.498703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.007769 (+1509066us) service_pool.cc:224] Handling call
0709 12:23:31.040232 (+ 32463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.049853 (+  9621us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.050150 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.052042  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.062988  1807 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:31.064963  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 810) took 1743 ms. Trace:
I20260709 12:23:31.065335  2411 rpcz_store.cc:276] 0709 12:23:29.322078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.322251 (+   173us) service_pool.cc:224] Handling call
0709 12:23:30.845296 (+1523045us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:30.881027 (+ 35731us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.064737 (+183710us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400488d50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:31.064907 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3501568384}
I20260709 12:23:30.995877  5191 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.073531  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.074576  1807 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:31.072685  1873 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:31.070490  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59958.8us
I20260709 12:23:31.074885  1806 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:31.077044  1804 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:31.078869  1800 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:31.079250  1876 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:23:31.080230  1876 pending_rounds.cc:85] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 5
I20260709 12:23:31.036767  4666 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:31.081729  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.083068  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.081960  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 482) took 1732 ms. Trace:
I20260709 12:23:31.084810  1873 rpcz_store.cc:276] 0709 12:23:29.343712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.993892 (+1650180us) service_pool.cc:224] Handling call
0709 12:23:31.054521 (+ 60629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.075793 (+ 21272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.076061 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.089011  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.089718  1800 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:31.056681  5487 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:31.080539  1876 pending_rounds.cc:107] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:23:31.099290  1876 raft_consensus.cc:2889] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:31.075428  5575 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:31.105808  1806 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:31.106709  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 480) took 1764 ms. Trace:
I20260709 12:23:31.104362  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:31.107120  1876 rpcz_store.cc:276] 0709 12:23:29.342403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.998839 (+1656436us) service_pool.cc:224] Handling call
0709 12:23:31.070259 (+ 71420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.106455 (+ 36196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.106654 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.107812  1807 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.108850  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.109221  5581 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:31.109622  5581 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.109962  5581 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:31.110034  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.111222  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.112154  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:31.112854  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.114476  2414 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260709 12:23:31.110670  1800 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:31.105173  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:31.120434  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 142097us
I20260709 12:23:31.123692  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:31.127666  1874 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:31.129133  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.128314  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 484) took 1424 ms. Trace:
I20260709 12:23:31.130882  1873 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:23:31.107084  5575 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:31.130894  1874 rpcz_store.cc:276] 0709 12:23:29.704087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.052008 (+1347921us) service_pool.cc:224] Handling call
0709 12:23:31.097465 (+ 45457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.128106 (+ 30641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.128270 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:31.134363  2411 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:31.131325  1873 pending_rounds.cc:85] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 5
I20260709 12:23:31.136534  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.116462  5621 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:31.118954  5557 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:31.139108  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:31.120327  5312 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.143666  1873 pending_rounds.cc:107] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:23:31.145939  1873 raft_consensus.cc:2889] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:31.147044  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.114425  5524 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:23:31.151962  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:31.140730  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 475) took 1916 ms. Trace:
I20260709 12:23:31.153550  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1844) took 1637 ms. Trace:
I20260709 12:23:31.156713  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.136439  5575 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:31.160198  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:31.163460  1802 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:31.165220  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2579) took 1851 ms (client timeout 1779 ms). Trace:
W20260709 12:23:31.165630  1879 rpcz_store.cc:269] 0709 12:23:29.314013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.314979 (+   966us) service_pool.cc:224] Handling call
0709 12:23:31.164837 (+1849858us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:31.165016 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1525908864}
I20260709 12:23:31.168655  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.169819  1803 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
I20260709 12:23:31.171514  5538 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:31.175173  2410 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:31.175951  2416 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:31.143110  5557 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:31.155908  1799 rpcz_store.cc:276] 0709 12:23:29.516143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.516255 (+   112us) service_pool.cc:224] Handling call
0709 12:23:31.150643 (+1634388us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b540035f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:31.153483 (+  2840us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1860062592}
I20260709 12:23:31.180966  1803 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:31.181435  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:31.182515  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98009.7us
I20260709 12:23:31.185258  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:31.186674  5630 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:23:31.191110  5621 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.193703  5621 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.195050  5191 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:31.196568  5537 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:31.197896  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45632.3us
I20260709 12:23:31.201051  1802 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:31.201946  1804 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:31.179615  1806 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:31.203418  1800 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
W20260709 12:23:31.205938  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 306570us
W20260709 12:23:31.214850  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66654.4us
I20260709 12:23:31.155644  1801 rpcz_store.cc:276] 0709 12:23:29.224104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.224223 (+   119us) service_pool.cc:224] Handling call
0709 12:23:29.224575 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.612684 (+388109us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.612717 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:29.613297 (+   580us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.241879 (+628582us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:30.242005 (+   126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.242189 (+   184us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.972581 (+730392us) spinlock_profiling.cc:243] Waited 7.62 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:31.135324 (+162743us) raft_consensus.cc:1685] finished
0709 12:23:31.135351 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.140558 (+  5207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9056,"raft.queue_time_us":251812,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"spinlock_wait_cycles":17528064,"thread_start_us":872944,"threads_started":2,"wal-append.queue_time_us":588762}
I20260709 12:23:31.225411  1799 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:31.227391  1800 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:31.216838  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:31.201083  5557 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.214521  5249 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:23:31.237629  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 108994us
I20260709 12:23:31.241086  1877 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:31.213182  5487 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:31.221745  5191 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260709 12:23:31.245131  5504 raft_consensus.cc:2764] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:23:31.244500  5501 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260709 12:23:31.247123  5501 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:31.247565  5501 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:31.248680  5501 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.250165  5501 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.237643  5502 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 12:23:31.251885  5502 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:31.220300  5538 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:23:31.252975  5538 consensus_queue.cc:1243] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 10.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:31.253549  5335 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:31.252315  5502 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:31.253976  5335 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:31.255291  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101239us
I20260709 12:23:31.255331  5502 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.256858  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.257197  5502 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:31.262864  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.263525  1801 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: [6.6-8.7]
I20260709 12:23:31.264900  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:31.268414  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:31.268939  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.269865  5561 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:31.281292  1799 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:31.271128  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.276849  1874 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:31.282291  2416 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
W20260709 12:23:31.295832  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 197588us
I20260709 12:23:31.311791  5621 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:31.312638  5621 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.313158  5621 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:31.315201  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46886.7us
I20260709 12:23:31.316507  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:31.242092  5603 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:31.317319  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60526.2us
I20260709 12:23:31.323688  1799 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:23:31.325434  1800 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:23:31.329490  5249 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:31.330018  5249 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:31.330410  5249 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:31.215436  4666 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.297243  5619 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.340569  5581 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:31.330675  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50023.5us
I20260709 12:23:31.358002  1807 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:31.204938  5310 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.305019  5561 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:31.371131  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:31.371587  5561 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:31.372604  5561 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.373624  5499 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:31.373911  5561 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:31.364522  1807 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
I20260709 12:23:31.369501  5502 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.374372  5499 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:23:31.374989  1807 pending_rounds.cc:85] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 4
I20260709 12:23:31.390228  1807 pending_rounds.cc:107] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 9.5
W20260709 12:23:31.375418  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45247us
I20260709 12:23:31.390554  1807 raft_consensus.cc:2889] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:31.379757  1881 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:31.398877  1880 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [6.4-9.6]
I20260709 12:23:31.361129  5524 raft_consensus.cc:2764] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:23:31.410132  1845 leader_election.cc:400] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9.
I20260709 12:23:31.374998  5248 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:31.390122  5581 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:31.359754  5575 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.361316  5619 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
W20260709 12:23:31.423220  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54608.1us
I20260709 12:23:31.425520  1807 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:31.435307  5371 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.435653  5546 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 12:23:31.435851  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.436152  5371 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.436998  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:31.437963  5621 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.438622  5371 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:31.439139  5312 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:31.439328  5371 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:31.439805  5312 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:31.440137  5371 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.440987  5312 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.442617  5371 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.443245  5383 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:31.443441  5371 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:31.444209  5371 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.446038  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.411335  5504 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:31.397804  5502 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:31.454627  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.457624  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.460566  5492 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:31.464576  4899 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 12:23:31.466147  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.446452  5371 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:31.467703  5371 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:31.468072  5594 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:31.468676  5594 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:31.469726  5594 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.471596  2415 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
W20260709 12:23:31.472071  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 76413us
I20260709 12:23:31.465046  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:31.474625  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.474648  5504 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.468163  5371 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.472190  5522 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:31.457531  5621 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.497334  1800 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:31.483227  5487 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:31.499373  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.507531  1807 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:31.509567  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:31.510413  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.495443  5522 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.529702  5522 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:31.529520  5313 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:23:31.531454  5313 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 12:23:31.532567  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83964us
I20260709 12:23:31.532913  5310 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.529728  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.533715  5310 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:31.533980  1805 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:31.534250  5310 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:31.535079  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:31.535807  1805 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:31.535841  5557 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.537154  1805 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:23:31.537998  5557 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:31.538255  1805 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:31.538755  2412 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:31.539933  1800 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:31.544950  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2600) took 1169 ms. Trace:
I20260709 12:23:31.545392  1880 rpcz_store.cc:276] 0709 12:23:30.375629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.053314 (+677685us) service_pool.cc:224] Handling call
0709 12:23:31.396927 (+343613us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:31.402317 (+  5390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.544737 (+142420us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b5400052150. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:31.544886 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":789208448}
I20260709 12:23:31.547557  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 897) took 1042 ms. Trace:
I20260709 12:23:31.548061  2413 rpcz_store.cc:276] 0709 12:23:30.504873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.506533 (+  1660us) service_pool.cc:224] Handling call
0709 12:23:30.526339 (+ 19806us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.009142 (+482803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.547236 (+538094us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b5400487454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:31.547478 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1093040384}
I20260709 12:23:31.551146  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:31.507836  5594 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:31.552613  5310 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.554081  5487 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.554630  5487 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.556721  5487 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.563860  5393 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:31.565061  5393 consensus_queue.cc:1243] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 9.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:31.565654  5393 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:31.566011  5393 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:31.567323  5504 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:31.567205  5393 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.496776  5371 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.568879  5393 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:31.563745  5249 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.567750  5383 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:31.553056  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:31.533782  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.569947  5581 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.565238  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2614) took 1037 ms. Trace:
I20260709 12:23:31.588747  1872 rpcz_store.cc:276] 0709 12:23:30.527440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.718669 (+191229us) service_pool.cc:224] Handling call
0709 12:23:31.565044 (+846375us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b5400117e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:31.565171 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":480827776}
I20260709 12:23:31.563786  5522 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.596055  1725 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:23:31.597404  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:31.598031  2411 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:31.598088  5522 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.594256  5499 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:31.601531  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.605036  1732 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:31.606452  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.607618  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.608160  5249 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.608727  5581 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.611512  5581 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:31.616334  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.617044  1728 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
W20260709 12:23:31.623508  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89804.2us
I20260709 12:23:31.624944  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.565325  5502 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:31.626053  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:31.627213  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 76211.9us
I20260709 12:23:31.650838  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:31.651706  2415 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.629094  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.653400  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.654062  2412 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:31.564006  5535 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:31.657637  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.677809  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:31.680820  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.656250  5557 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:31.683296  5502 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:31.686478  5393 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:31.684060  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:31.688665  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.689420  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.691386  2412 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:23:31.656769  5504 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:31.700387  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:31.700760  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.700870  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.706110  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.707228  2418 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:31.715986  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:31.715210  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.696655  5557 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.659191  5535 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:31.697996  5502 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.728539  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.732035  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:31.740477  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.741080  2414 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:23:31.741271  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.702152  5393 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:31.708091  5289 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:31.708401  5504 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:31.746606  2411 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 6.
I20260709 12:23:31.727609  5557 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:31.745682  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:31.730144  5502 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:31.757325  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:31.757951  1728 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:23:31.743140  5393 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.744349  5501 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:31.763953  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.773445  1723 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
I20260709 12:23:31.774585  2418 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:31.772914  5501 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:31.785105  1873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.006s	sys 0.023s
I20260709 12:23:31.777086  5289 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:31.785626  1728 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:31.787176  1873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.006s	sys 0.023s
I20260709 12:23:31.784322  5501 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:31.782402  5365 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:31.800367  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.808404  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1860) took 1001 ms. Trace:
W20260709 12:23:31.839656  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 416671us
I20260709 12:23:31.850349  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.808900  5365 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.780865  5335 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:31.859704  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.860767  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:31.861644  1729 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:31.861510  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.796751  5371 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.863158  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.863354  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:31.863926  1725 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:31.864104  1727 raft_consensus.cc:2376] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12. Re-sending same reply.
I20260709 12:23:31.864738  5371 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.865171  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.850349  5535 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.865864  1727 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:31.865948  1724 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:31.866483  5535 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:31.868259  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:31.869109  5603 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:31.871601  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:31.862504  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.874367  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:31.874980  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 392651us
I20260709 12:23:31.864825  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.879127  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:31.870620  5603 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 12:23:31.886461  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:31.829268  5248 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:31.898280  5546 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:31.855979  5365 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:31.918771  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:31.901684  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1861) took 1067 ms. Trace:
I20260709 12:23:31.901927  5575 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:31.921945  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45388us
I20260709 12:23:31.919556  1880 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:31.920383  1804 rpcz_store.cc:276] 0709 12:23:30.834316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.202649 (+368333us) service_pool.cc:224] Handling call
0709 12:23:31.230143 (+ 27494us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.251153 (+ 21010us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:31.482760 (+231607us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.901175 (+418415us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.901362 (+   187us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:31.901406 (+    44us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:31.901456 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.901521 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.901627 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52566,"prepare.queue_time_us":178842,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":2793,"raft.queue_time_us":195,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":164,"thread_start_us":550088,"threads_started":2}
I20260709 12:23:31.907737  5371 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:32.080245  5371 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:32.080722  5371 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:31.914743  5594 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 12:23:31.809643  5504 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.896286  5556 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.881359  5581 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:31.927955  5249 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:31.965281  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:31.968708  5535 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:32.086952  5556 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.089501  5535 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:31.969405  1726 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:32.090585  5556 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:32.091310  5556 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:32.091702  5556 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:32.091890  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:31.972220  5501 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.934321  1807 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 12:23:31.938539  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52186.3us
I20260709 12:23:31.940075  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:31.941366  1800 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:31.960547  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.960741  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1862) took 1126 ms. Trace:
I20260709 12:23:31.928691  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:32.096520  1877 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:31.985620  2415 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:31.934334  5310 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:31.953133  5393 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:31.935940  5248 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.086315  5289 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:32.108700  5393 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:31.959211  5502 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.111550  5502 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:31.936462  5383 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.112484  5383 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.112921  5501 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:31.936911  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:31.987960  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:31.951026  1770 leader_election.cc:336] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.573s (SENT)
I20260709 12:23:31.988108  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.094810  1806 rpcz_store.cc:276] 0709 12:23:30.834646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.203143 (+368497us) service_pool.cc:224] Handling call
0709 12:23:31.218157 (+ 15014us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.275543 (+ 57386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.275569 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.276002 (+   433us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.573108 (+297106us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:31.573176 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.573293 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:31.960391 (+387098us) raft_consensus.cc:1685] finished
0709 12:23:31.960420 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.960649 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6372,"raft.queue_time_us":10224,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"thread_start_us":296617,"threads_started":1,"wal-append.queue_time_us":578836}
W20260709 12:23:32.114835  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 163784us
I20260709 12:23:31.975281  1727 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:31.996590  1723 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:23:31.931545  5644 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:23:31.948962  5561 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.575s	user 0.006s	sys 0.006s
I20260709 12:23:31.921262  5546 consensus_queue.cc:1243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 11.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:32.011142  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.013811  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1948) took 1066 ms. Trace:
W20260709 12:23:31.998848  5645 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:23:31.974419  5365 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.079597  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:32.128213  1880 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:31.942116  5557 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.104442  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1932) took 2756 ms. Trace:
I20260709 12:23:32.117990  1805 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:32.100208  5535 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:32.123703  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.121706  1873 rpcz_store.cc:276] 0709 12:23:30.946942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.203285 (+256343us) service_pool.cc:224] Handling call
0709 12:23:31.235826 (+ 32541us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:31.275429 (+ 39603us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.275458 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:31.276007 (+   549us) log.cc:844] Serialized 52 byte log entry
0709 12:23:31.817834 (+541827us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:31.817907 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.818015 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.013454 (+195439us) raft_consensus.cc:1685] finished
0709 12:23:32.013479 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.013716 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1356,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":79,"thread_start_us":517954,"threads_started":1,"wal-append.queue_time_us":724305}
W20260709 12:23:32.209215  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.121273  1724 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:32.129035  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.118619  1732 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:32.130918  1875 rpcz_store.cc:276] 0709 12:23:29.347960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.995582 (+1647622us) service_pool.cc:224] Handling call
0709 12:23:31.049151 (+ 53569us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.105752 (+ 56601us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.105785 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.108732 (+  2947us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.441931 (+333199us) spinlock_profiling.cc:243] Waited 7.1 ms on lock 0x7b180015fdc0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:23:31.843308 (+401377us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:31.843383 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.843524 (+   141us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.104096 (+260572us) raft_consensus.cc:1685] finished
0709 12:23:32.104123 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.104370 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14184,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":5453,"spinlock_wait_cycles":16335616,"thread_start_us":372404,"threads_started":1,"wal-append.queue_time_us":993066}
W20260709 12:23:32.180960  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 240980us
I20260709 12:23:32.182279  1876 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:32.182751  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:32.212410  1807 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:32.128468  5546 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.195794  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:32.207685  2413 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.008s	sys 0.016s
W20260709 12:23:32.208153  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.009s	sys 0.017s
W20260709 12:23:32.214001  2413 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.009s	sys 0.017s
W20260709 12:23:32.208591  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 462974us
W20260709 12:23:32.214069  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.009s	sys 0.017s
W20260709 12:23:32.113907  1803 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.008s	sys 0.020s
W20260709 12:23:32.216144  1803 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.008s	sys 0.020s
W20260709 12:23:32.194428  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.037s	sys 0.068s
W20260709 12:23:32.225610  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.037s	sys 0.068s
W20260709 12:23:32.225940  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 596852us
I20260709 12:23:32.225744  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:32.228837  1881 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:32.234735  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.244803  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1872) took 1406 ms. Trace:
I20260709 12:23:32.245390  1801 rpcz_store.cc:276] 0709 12:23:30.838287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.322035 (+483748us) service_pool.cc:224] Handling call
0709 12:23:31.322400 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.368417 (+ 46017us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:31.808111 (+439694us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.244406 (+436295us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:32.244476 (+    70us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:32.244535 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.244604 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.244736 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2559,"prepare.queue_time_us":119530,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"thread_start_us":802821,"threads_started":2}
W20260709 12:23:32.255273  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 503352us
I20260709 12:23:32.257516  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:32.259770  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.260188  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:32.262748  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.262826  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.264160  1724 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:32.264747  1807 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:32.265224  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.265889  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:32.266415  5651 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.271364  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:32.272493  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:32.276611  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:32.278358  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:32.279115  1805 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:32.279449  1801 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:32.117344  5556 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.294448  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:32.301038  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:32.305011  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51535.5us
I20260709 12:23:32.303156  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:32.287840  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:32.316562  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:32.320713  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.321022  1873 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:32.326612  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.291648  4770 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:23:32.328536  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:32.328747  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:32.282063  5383 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:32.330652  1875 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:32.242624  5546 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.338487  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:32.319830  5501 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:32.346123  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:32.331475  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:32.258370  5502 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:23:32.349385  1805 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:32.323244  5377 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:32.352396  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:32.286255  1877 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:32.354580  1805 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:32.361443  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44964.1us
I20260709 12:23:32.362295  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:32.266971  5651 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:32.350538  1627 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 255) took 22527 ms (client timeout 15000 ms). Trace:
W20260709 12:23:32.363467  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 398219us
I20260709 12:23:32.366238  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:32.373054  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.374228  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.376305  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:32.376744  1873 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:32.377698  1877 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:32.343134  4770 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:31.820688  1802 rpcz_store.cc:276] 0709 12:23:30.792311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.201894 (+409583us) service_pool.cc:224] Handling call
0709 12:23:31.202186 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.217283 (+ 15097us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:31.320635 (+103352us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.790703 (+470068us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:31.790771 (+    68us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:31.793176 (+  2405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.793263 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.793414 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":152533,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":1852,"raft.queue_time_us":94,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":12163,"thread_start_us":400581,"threads_started":1}
W20260709 12:23:32.391055  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.347246  5535 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:32.346503  5383 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:32.330440  5621 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:23:32.352012  5556 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:32.397321  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 77299.2us
I20260709 12:23:32.399766  1805 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:32.333639  5487 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:23:32.355610  5501 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.354116  5654 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:23:32.413353  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47208.4us
I20260709 12:23:32.429993  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:32.415396  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1942) took 1662 ms. Trace:
W20260709 12:23:32.419253  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 211798us
I20260709 12:23:32.419858  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.424265  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.378134  5398 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:32.406056  1626 catalog_manager.cc:5697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 4 to 6, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:32.435236  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:32.436695  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 160159us
I20260709 12:23:32.441231  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:32.383683  4770 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:32.384370  5594 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:32.442981  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.402084  5310 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:32.448869  1728 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:32.457641  1875 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:32.459565  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:32.425967  5499 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:32.464694  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.467618  1732 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.469120  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:32.468559  1626 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 4 to 5, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 5 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:32.464426  5561 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:32.471400  4770 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.471896  5653 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:32.466434  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:32.429605  5365 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:32.436002  5501 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.495493  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:32.501173  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:32.501700  1875 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:32.495396  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 207003us
I20260709 12:23:32.447643  5487 consensus_queue.cc:1243] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:32.515051  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:32.517622  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:32.423388  5669 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:32.517292  1724 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:32.494357  4770 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.530972  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1864) took 1695 ms. Trace:
I20260709 12:23:32.532533  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:32.524127  5662 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260709 12:23:32.450552  5654 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:32.465634  5383 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:32.538148  2414 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:32.527107  5669 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.517995  5310 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:32.541661  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 117827us
I20260709 12:23:32.524977  5487 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:32.536165  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:32.535374  5658 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20260709 12:23:32.502727  5594 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.535807  5377 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:32.546373  2414 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:32.533743  4666 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:32.559154  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.569378  1875 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:32.575212  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:32.576180  2416 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:32.552496  5557 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:32.560163  5377 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:32.546213  5383 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:32.581691  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1873) took 1743 ms. Trace:
I20260709 12:23:32.561587  2414 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:32.584378  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 222171us
I20260709 12:23:32.593632  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:32.584406  2415 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
W20260709 12:23:32.570295  5661 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.235s	user 0.001s	sys 0.001s
I20260709 12:23:32.552668  5487 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:32.554121  5669 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:32.365422  1627 rpcz_store.cc:269] 0709 12:23:09.823174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.823392 (+   218us) service_pool.cc:224] Handling call
0709 12:23:09.833767 (+ 10375us) spinlock_profiling.cc:243] Waited 1.12 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd42090 00007f16abde0f3a 00007f16b1b96e9c 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:23:32.350130 (+22516363us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:19.129677 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:19.131842 (+  2165us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:19.131897 (+    55us) write_op.cc:435] Acquired schema lock
0709 12:23:19.131931 (+    34us) tablet.cc:662] Decoding operations
0709 12:23:19.134369 (+  2438us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:19.134496 (+   127us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:19.134541 (+    45us) tablet.cc:685] Acquiring locks for 26 operations
0709 12:23:19.137446 (+  2905us) tablet.cc:701] Row locks acquired
0709 12:23:19.137475 (+    29us) write_op.cc:260] PREPARE: finished
0709 12:23:19.137701 (+   226us) write_op.cc:270] Start()
0709 12:23:19.137847 (+   146us) write_op.cc:276] Timestamp: P: 1783599799137625 usec, L: 0
0709 12:23:19.137874 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:23:19.224599 (+ 86725us) log.cc:844] Serialized 10436 byte log entry
0709 12:23:19.293007 (+ 68408us) op_driver.cc:464] REPLICATION: finished
0709 12:23:19.922892 (+629885us) write_op.cc:301] APPLY: starting
0709 12:23:19.923014 (+   122us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:19.923680 (+   666us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:19.924176 (+   496us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:20.024577 (+100401us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:20.025314 (+   737us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=26
0709 12:23:20.025390 (+    76us) write_op.cc:312] APPLY: finished
0709 12:23:20.026365 (+   975us) log.cc:844] Serialized 176 byte log entry
0709 12:23:20.026659 (+   294us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:20.049258 (+ 22599us) write_op.cc:454] Released schema lock
0709 12:23:20.049400 (+   142us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:21.556883 (+1507483us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b5400156400. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16af742161 00007f16af81733d 00007f16af8103c2 00007f16af80fb53 00007f16af813d61 00007f16af813d19 00007f16af813ca9 00007f16af813c71 00007f16af812e9d 0000000000388634 0000000000388479
0709 12:23:21.557275 (+   392us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:21.557315 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7610,"reported_tablets":50,"spinlock_wait_cycles":2589312,"child_traces":[["op",{"apply.queue_time_us":394787,"apply.run_cpu_time_us":18672,"apply.run_wall_time_us":1634346,"num_ops":26,"prepare.queue_time_us":177774,"prepare.run_cpu_time_us":17570,"prepare.run_wall_time_us":156311,"raft.queue_time_us":2855,"raft.run_cpu_time_us":31450,"raft.run_wall_time_us":385264,"replication_time_us":154945,"spinlock_wait_cycles":1375321728,"thread_start_us":492948,"threads_started":4,"wal-append.queue_time_us":53277,"wal-append.run_cpu_time_us":7934,"wal-append.run_wall_time_us":1742250}]]}
I20260709 12:23:32.555861  1626 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 6 to 3, term changed from 6 to 5, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66), 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from VOTER to NON_VOTER, now has a pending config: VOTER 075704438992440697732181e49ccd09 (127.0.140.65), VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66), VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67), VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 5 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:32.584242  1799 rpcz_store.cc:276] 0709 12:23:30.838623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.325081 (+486458us) service_pool.cc:224] Handling call
0709 12:23:31.366569 (+ 41488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.401195 (+ 34626us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.401228 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.430180 (+ 28952us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.871798 (+441618us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:31.871841 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.871911 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.581423 (+709512us) raft_consensus.cc:1685] finished
0709 12:23:32.581446 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.581627 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14514,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":7214,"thread_start_us":400494,"threads_started":1,"wal-append.queue_time_us":678730}
I20260709 12:23:32.603550  1732 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
W20260709 12:23:32.604513  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84703.2us
I20260709 12:23:32.583832  5377 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:32.591907  5664 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.262s	user 0.000s	sys 0.002s
I20260709 12:23:32.587453  5383 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:32.619035  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149974us
I20260709 12:23:32.620550  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.623940  1732 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:32.624980  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.588752  5561 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:32.608513  5487 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:32.645150  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.650998  5561 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:32.652714  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:32.641687  5668 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.240s	user 0.001s	sys 0.001s
W20260709 12:23:32.658254  5670 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:23:32.659152  1626 catalog_manager.cc:5697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 5 to 6, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:32.660823  1626 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:32.646936  5654 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:32.663776  5671 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:23:32.662267  1626 catalog_manager.cc:5697] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 4 to 6, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:32.669914  1873 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:32.627544  5651 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:32.673632  1626 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 2 to 3, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:32.623183  5557 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 12:23:32.692677  5557 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:32.693130  5557 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:32.699132  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.699842  1732 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:32.700776  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.701853  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.702445  1732 raft_consensus.cc:2368] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10.
W20260709 12:23:32.703552  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 113309us
I20260709 12:23:32.694068  5557 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:32.710582  2416 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:32.711512  1728 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:32.720317  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.719775  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1809) took 1152 ms. Trace:
I20260709 12:23:32.722055  1731 rpcz_store.cc:276] 0709 12:23:31.567476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.582108 (+ 14632us) service_pool.cc:224] Handling call
0709 12:23:31.603036 (+ 20928us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.603445 (+   409us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.603473 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.604111 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:23:32.118464 (+514353us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:32.118511 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.118841 (+   330us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.719494 (+600653us) raft_consensus.cc:1685] finished
0709 12:23:32.719518 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.719700 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":195,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":474028,"threads_started":1,"wal-append.queue_time_us":671647}
I20260709 12:23:32.726812  5561 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:32.729661  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:32.730690  1799 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 11 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:32.703392  5329 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:23:32.734978  5690 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:23:32.735553  5329 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:32.735992  5329 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:32.736150  5690 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:32.736610  5690 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:32.736894  5329 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:32.737475  5690 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:32.738382  5329 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:32.710513  5504 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.738929  5691 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
W20260709 12:23:32.718691  5672 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:23:32.744966  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.717051  5651 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:32.739688  5678 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
W20260709 12:23:32.750325  5685 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:23:32.759264  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:32.760113  5684 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.189s	user 0.005s	sys 0.000s
W20260709 12:23:32.761695  5639 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.982s	user 0.000s	sys 0.007s
W20260709 12:23:32.760138  5675 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:23:32.764604  5524 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:32.775450  5524 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:32.762533  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:32.776716  5682 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 12:23:32.777138  1729 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:32.777669  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:32.778292  5524 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:32.778223  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.779134  5310 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:32.779359  1729 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:32.780161  5594 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:32.780267  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.758236  5504 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:32.783179  5651 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:32.783603  5504 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:32.784279  5651 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.784569  5487 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.787086  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:32.787809  1804 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:32.793099  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.759740  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.795080  1729 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:32.800202  5499 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:32.800388  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.800863  5499 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.759254  5690 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:32.801905  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.757022  5557 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:32.804478  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 136511us
I20260709 12:23:32.786580  5487 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:32.732059  5653 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:32.806932  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:32.813419  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:32.814667  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:32.818010  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:32.818322  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.821725  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:32.811615  5371 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.730s	user 0.011s	sys 0.003s
I20260709 12:23:32.831424  5371 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.839236  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.865099  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:32.869150  5660 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.584s	user 0.002s	sys 0.001s
W20260709 12:23:32.879647  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.893993  5653 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.895756  1726 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:32.896078  5487 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:32.896739  2410 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:32.899435  1728 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:32.899650  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.908965  1732 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
W20260709 12:23:32.900761  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 93921.1us
I20260709 12:23:32.909354  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.899024  5689 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:32.919337  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.920780  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.922276  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.922667  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:32.967242  5676 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:32.968093  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45505.3us
I20260709 12:23:32.967247  1724 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:32.969038  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:32.969857  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1818) took 1294 ms. Trace:
I20260709 12:23:32.970455  1730 rpcz_store.cc:276] 0709 12:23:31.675806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.675959 (+   153us) service_pool.cc:224] Handling call
0709 12:23:32.969612 (+1293653us) spinlock_profiling.cc:243] Waited 685 ms on lock 0x7b540035d754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:32.969758 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1575229824}
I20260709 12:23:32.972096  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.972549  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.973879  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.974697  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.975049  5524 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:32.976145  5524 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:32.977401  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 383846us
I20260709 12:23:32.978220  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:32.979202  2419 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:32.979589  5669 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:32.981230  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.930119  5537 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:32.981005  5504 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:32.935451  5689 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:32.937743  5335 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:32.945272  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.986452  1723 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 12:23:32.955813  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:32.987332  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.958528  2410 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:32.960317  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:32.964583  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:32.989427  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:32.964756  5666 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.599s	user 0.002s	sys 0.000s
I20260709 12:23:32.989985  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:32.966250  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 536359us
W20260709 12:23:32.990240  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 378204us
W20260709 12:23:32.983506  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:32.991778  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:32.992219  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.992354  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:32.992959  1726 raft_consensus.cc:2376] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10. Re-sending same reply.
I20260709 12:23:32.993660  5371 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:32.993813  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.995204  5687 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:32.995694  5687 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:32.996178  5687 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:32.996824  5689 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:32.896704  5377 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.002640  1730 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:32.994469  1726 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:33.004010  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.965054  5398 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:32.995877  5538 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:33.004886  5398 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.005270  5538 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:33.005807  5538 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:33.006127  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 738) took 1454 ms. Trace:
I20260709 12:23:33.005363  5398 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:32.980969  2415 raft_consensus.cc:2955] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:32.911685  5501 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.006765  5538 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.006489  2413 rpcz_store.cc:276] 0709 12:23:31.551255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.590923 (+ 39668us) service_pool.cc:224] Handling call
0709 12:23:31.653976 (+ 63053us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.658538 (+  4562us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.658581 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.682677 (+ 24096us) log.cc:844] Serialized 26 byte log entry
0709 12:23:32.219780 (+537103us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:32.219846 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.219946 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.963884 (+743938us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:33.005777 (+ 41893us) raft_consensus.cc:1685] finished
0709 12:23:33.005803 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.006049 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":151,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":13890,"spinlock_wait_cycles":31571328,"thread_start_us":527900,"threads_started":1,"wal-append.queue_time_us":610799}
I20260709 12:23:33.012377  5538 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:33.012835  5501 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:33.007658  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:33.013871  1726 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:33.004233  5377 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:33.014822  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:33.019449  1728 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:33.019482  1726 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:32.984426  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 193557us
I20260709 12:23:33.027215  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.028234  1873 raft_consensus.cc:2410] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:23:33.029230  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:32.976451  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:33.030125  1875 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:23:33.036255  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:33.039388  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:32.985379  5537 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:33.039719  1723 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:23:33.040408  1880 raft_consensus.cc:2368] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
I20260709 12:23:33.040503  1724 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:33.048306  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.048872  1728 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:33.049469  1875 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:33.043493  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:33.051591  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.052625  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:33.053544  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.051594  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.985285  5504 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.056931  1872 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:33.020545  5688 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:33.059309  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:33.060367  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:33.061424  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:33.064354  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:32.982926  5487 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.070425  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:33.032742  5674 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:33.076828  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 166291us
I20260709 12:23:33.076831  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:33.088220  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.089350  1875 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:33.090962  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:33.092228  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:33.079690  1724 raft_consensus.cc:2376] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8. Re-sending same reply.
I20260709 12:23:33.095173  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:33.097047  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:33.042260  5594 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:33.080497  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:33.104446  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52980.3us
I20260709 12:23:33.104660  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:33.105024  1732 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:33.105422  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:33.107244  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48655.4us
I20260709 12:23:33.110368  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:33.072173  5289 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:33.116799  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.118105  1875 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:23:33.112401  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.071427  5676 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:33.062717  5688 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:33.076032  5501 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:33.127020  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:33.124491  1729 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:33.125144  1872 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:33.044502  5365 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.065s
I20260709 12:23:33.131549  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:33.132336  1800 raft_consensus.cc:2376] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9. Re-sending same reply.
I20260709 12:23:33.127717  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:33.105101  5524 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:33.139868  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:33.142861  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:33.143982  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:33.145875  1804 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:33.113416  5313 raft_consensus.cc:2764] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:23:33.111792  5651 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:33.071919  5667 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.061s
I20260709 12:23:33.096990  5687 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.111507  5594 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.109218  5674 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:33.152068  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 90687.6us
W20260709 12:23:33.209794  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50251.4us
W20260709 12:23:33.202601  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 97250.3us
I20260709 12:23:33.160576  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:33.200557  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:33.248843  1726 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:33.264554  1805 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:33.201285  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:33.265682  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.200260  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:33.186353  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:33.209594  1723 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 10 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:33.201782  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:33.117225  5535 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:33.261468  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:33.122543  5676 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:33.161643  1729 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:33.291046  5705 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:33.291694  5705 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:33.193720  5383 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.566s	user 0.007s	sys 0.003s
I20260709 12:23:33.292949  5383 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.294036  5705 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:33.295392  5700 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:23:33.295504  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:33.162199  5669 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:33.266953  1802 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:33.130219  5575 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:33.285475  1726 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
W20260709 12:23:33.300539  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114399us
I20260709 12:23:33.276201  1798 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:33.128194  5688 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:33.308476  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47853.2us
I20260709 12:23:33.270744  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:33.266323  1806 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:33.309546  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:33.208894  5501 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.307157  1873 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:33.302199  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:33.207852  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.160941  5689 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:33.156769  5524 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.157510  5371 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.277132  5651 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.305739  5705 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:33.317480  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:33.353629  1873 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:33.213868  5708 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.299947  5676 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:33.351337  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 80680.9us
I20260709 12:23:33.348462  5535 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:33.319418  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:33.159219  5289 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.186s
I20260709 12:23:33.299021  5654 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:33.315673  5383 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:33.316603  5312 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:33.345484  5524 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:33.326267  5697 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:33.369575  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:33.368924  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.355489  5365 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:33.350124  5689 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:33.377502  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.365155  5704 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:33.211798  5537 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:33.387813  5706 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:23:33.368992  5705 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:33.359019  5535 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.350478  5651 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:33.361949  5654 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:33.394663  1806 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:33.388204  5594 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:33.349656  5501 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.398957  5487 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:33.401189  1771 leader_election.cc:400] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
I20260709 12:23:33.401160  5312 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:33.399333  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 81961.4us
W20260709 12:23:33.402899  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 93442us
W20260709 12:23:33.404677  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 141984us
I20260709 12:23:33.380781  5365 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.349762  5398 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.392098  5697 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:32.433080  1879 rpcz_store.cc:276] 0709 12:23:30.755597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.166484 (+410887us) service_pool.cc:224] Handling call
0709 12:23:31.175911 (+  9427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.224398 (+ 48487us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:31.582318 (+357920us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.002093 (+419775us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.347810 (+345717us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.375847 (+ 28037us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.414912 (+ 39065us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.415038 (+   126us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:32.415073 (+    35us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:32.415119 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.415207 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.415326 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65281,"prepare.queue_time_us":74859,"prepare.run_cpu_time_us":171,"prepare.run_wall_time_us":6516,"raft.queue_time_us":262465,"raft.run_cpu_time_us":404,"raft.run_wall_time_us":25384,"spinlock_wait_cycles":24277632,"thread_start_us":1019136,"threads_started":3,"wal-append.queue_time_us":669484}
I20260709 12:23:33.394408  5575 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:33.421778  5667 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.422483  5289 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.458000  5717 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.362466  5694 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:23:33.418401  5535 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:33.352557  5371 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.361899  5621 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:33.412611  5709 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:33.441639  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.428483  5654 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:33.426723  5504 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:33.456548  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.457142  5667 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.455392  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.461925  1878 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:33.495186  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.463800  1875 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:33.458055  5289 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.447134  5365 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:33.362454  5708 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.478966  5724 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:33.484268  1874 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:23:33.487636  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46083.9us
W20260709 12:23:33.490427  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94596.2us
W20260709 12:23:33.496184  5557 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.682s	user 0.006s	sys 0.006s
I20260709 12:23:33.484018  5709 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:33.512981  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.494728  5313 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:33.517740  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:33.519801  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:33.522406  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 73837.6us
I20260709 12:23:33.523717  1878 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:33.525389  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.499730  5556 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:33.527616  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72315.7us
I20260709 12:23:33.528673  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.532305  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.533918  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:33.535746  5701 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.274s	user 0.002s	sys 0.000s
I20260709 12:23:33.513746  5504 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:33.459805  5689 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:33.522830  5707 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 12:23:33.539855  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.464274  5717 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.510488  5704 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:33.559850  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 115152us
I20260709 12:23:33.560868  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.531719  5575 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:33.527858  5312 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:33.566763  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.535802  5383 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:33.541592  5557 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:33.602886  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:33.605594  5704 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.507890  5705 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.599350  1878 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:33.603395  5709 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.613996  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:33.614740  5709 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:33.615119  1878 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:33.617157  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.538442  5556 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.569438  5575 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:33.628484  5721 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260709 12:23:33.629655  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 183653us
I20260709 12:23:33.514032  5697 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.635522  5312 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:33.636472  5501 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:33.636199  5312 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.637658  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 511) took 1528 ms. Trace:
I20260709 12:23:33.638091  1725 rpcz_store.cc:276] 0709 12:23:32.109403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.025671 (+916268us) service_pool.cc:224] Handling call
0709 12:23:33.046491 (+ 20820us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.046716 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.046753 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:33.057352 (+ 10599us) log.cc:844] Serialized 26 byte log entry
0709 12:23:33.329771 (+272419us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:33.329866 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.330038 (+   172us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.637393 (+307355us) raft_consensus.cc:1685] finished
0709 12:23:33.637420 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.637591 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":216276,"threads_started":1,"wal-append.queue_time_us":349763}
I20260709 12:23:33.640195  1725 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:33.644897  4666 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:33.645453  4666 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:33.645995  5538 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.633s	user 0.010s	sys 0.000s
I20260709 12:23:33.646265  1798 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:33.647534  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:33.647661  5717 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:33.649576  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.649705  1879 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:23:33.650158  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:33.653460  5716 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:23:33.656965  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 117194us
I20260709 12:23:33.652146  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:33.654606  5312 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:33.658174  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:33.618733  5557 raft_consensus.cc:2764] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
W20260709 12:23:33.659869  5715 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:23:33.669524  1806 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:33.633376  5575 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:33.658597  5499 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.848s	user 0.006s	sys 0.007s
I20260709 12:23:33.675944  2413 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:33.677088  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:33.677238  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74446us
I20260709 12:23:33.673758  5393 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:33.695782  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:33.704360  5722 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
I20260709 12:23:33.672921  5501 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.035s
I20260709 12:23:33.683928  5538 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:33.746215  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69182.5us
I20260709 12:23:33.752887  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.671236  5313 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
W20260709 12:23:33.753965  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87203.1us
I20260709 12:23:33.757650  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:33.758147  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.696933  5736 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.699088  5667 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:33.770015  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2725) took 1465 ms. Trace:
I20260709 12:23:33.770606  1732 rpcz_store.cc:276] 0709 12:23:32.304372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.114697 (+810325us) service_pool.cc:224] Handling call
0709 12:23:33.115073 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.156968 (+ 41895us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.157010 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:33.203499 (+ 46489us) log.cc:844] Serialized 26 byte log entry
0709 12:23:33.403280 (+199781us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:33.403349 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.403445 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.769660 (+366215us) raft_consensus.cc:1685] finished
0709 12:23:33.769703 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.769921 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":128,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":5252,"thread_start_us":148730,"threads_started":1,"wal-append.queue_time_us":298923}
I20260709 12:23:33.681450  5499 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.747367  5393 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:33.709172  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:33.754150  5371 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:33.821202  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 60212.4us
I20260709 12:23:33.762207  5557 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:33.752441  5504 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:33.839016  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:33.839859  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 83111.2us
I20260709 12:23:33.769133  5248 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:33.850320  1729 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:33.852157  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 87353.5us
I20260709 12:23:33.766847  5734 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:33.855376  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:33.856364  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:33.784205  5289 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:33.707336  5729 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.156s
W20260709 12:23:33.880705  5725 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.263s	user 0.001s	sys 0.000s
I20260709 12:23:33.882071  5708 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:33.887678  1881 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:23:33.888501  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1960) took 1635 ms. Trace:
I20260709 12:23:33.888928  1881 rpcz_store.cc:276] 0709 12:23:32.253411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.887060 (+1633649us) service_pool.cc:224] Handling call
0709 12:23:33.887444 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.888197 (+   753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.888419 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.889281  5504 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.890223  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.891439  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.872351  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:33.892035  1881 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:33.863864  5248 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 12:23:33.892879  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:33.854610  5371 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:23:33.878525  5676 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.517s	user 0.005s	sys 0.003s
I20260709 12:23:33.913466  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:33.914963  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:33.886600  4666 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.935660  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:33.890491  5546 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:33.943382  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:33.944360  1877 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:33.781865  5736 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:33.911413  5735 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.909870  5289 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.909860  5312 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:33.920537  5557 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:33.907243  5744 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:33.920516  5738 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.920568  5734 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:33.958673  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45325.5us
I20260709 12:23:33.934105  5708 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:34.009855  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 154562us
I20260709 12:23:34.052657  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:23:34.045536  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:34.045603  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.933032  5313 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260709 12:23:33.888398  5575 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:34.049332  5365 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.540s	user 0.010s	sys 0.002s
W20260709 12:23:34.073688  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.934710  5676 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.060374  1877 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:34.110612  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.047672  5743 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:34.075961  5729 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:34.129730  1877 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:34.045962  5289 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.076550  5557 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:33.956436  5393 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:33.978652  5735 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.136353  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2624) took 1104 ms. Trace:
W20260709 12:23:34.124720  5709 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.505s	user 0.007s	sys 0.006s
I20260709 12:23:34.109266  5365 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.063690  5708 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.148965  1881 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:34.149065  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 99720.8us
W20260709 12:23:34.144237  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 91652.7us
I20260709 12:23:34.141568  1877 rpcz_store.cc:276] 0709 12:23:33.031534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.123526 (+1091992us) service_pool.cc:224] Handling call
0709 12:23:34.129320 (+  5794us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.135993 (+  6673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.136263 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.147498  1878 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:23:34.152726  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:34.154028  1877 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:23:34.053097  5738 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.138840  5743 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:34.161839  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2626) took 1115 ms. Trace:
I20260709 12:23:34.162725  1877 rpcz_store.cc:276] 0709 12:23:33.040778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.153316 (+1112538us) service_pool.cc:224] Handling call
0709 12:23:34.153737 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.154562 (+   825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.155900 (+  1338us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.164952  1877 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:34.165652  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2627) took 1110 ms. Trace:
I20260709 12:23:34.165999  1877 rpcz_store.cc:276] 0709 12:23:33.055673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.164440 (+1108767us) service_pool.cc:224] Handling call
0709 12:23:34.164785 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.165390 (+   605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.165585 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.167884  1877 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:34.169879  1881 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:34.169814  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2634) took 1053 ms. Trace:
I20260709 12:23:34.145629  5499 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:34.172940  1877 rpcz_store.cc:276] 0709 12:23:33.116071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.167211 (+1051140us) service_pool.cc:224] Handling call
0709 12:23:34.167533 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.169344 (+  1811us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.169732 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:34.178653  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 116031us
I20260709 12:23:34.180840  1877 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:23:34.183564  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1953) took 2305 ms. Trace:
I20260709 12:23:34.140107  5557 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.184072  1876 rpcz_store.cc:276] 0709 12:23:31.876867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.897359 (+ 20492us) service_pool.cc:224] Handling call
0709 12:23:31.924214 (+ 26855us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.222368 (+298154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.181571 (+1959203us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b5400223b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:34.181775 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":589495552}
I20260709 12:23:34.184942  1807 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:34.183697  5743 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.186666  1876 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:34.187166  5383 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.586s	user 0.013s	sys 0.006s
I20260709 12:23:34.187665  5383 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:34.188505  1876 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:23:34.189688  5759 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:34.190312  5759 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:34.191411  5752 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:23:34.192435  1876 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:23:34.195936  1800 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:34.196489  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:34.182562  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2635) took 1021 ms. Trace:
W20260709 12:23:34.250111  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2748) took 1563 ms (client timeout 2079 ms). Trace:
I20260709 12:23:34.203995  5676 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:34.205360  5289 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:34.207214  5735 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:34.208544  5746 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.317s	user 0.002s	sys 0.000s
W20260709 12:23:34.209472  5741 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:23:34.192349  5734 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:34.212738  5621 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.728s	user 0.009s	sys 0.002s
I20260709 12:23:34.216120  5575 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:34.219027  4666 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:34.220290  5761 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:34.279747  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:34.191476  5738 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:34.223034  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 366681us
I20260709 12:23:34.179206  5756 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:34.229382  1876 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260709 12:23:34.229995  5524 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.837s	user 0.011s	sys 0.003s
I20260709 12:23:34.232514  5365 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:34.230526  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8bb813e4fab14a61b810a24b56899586 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:34.227984  5755 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:34.203840  5708 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:34.246762  5651 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.808s	user 0.011s	sys 0.002s
I20260709 12:23:34.291841  5651 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:34.294190  5651 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:34.245343  5747 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:34.191357  5759 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.294911  5747 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:34.295851  5747 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.248072  1872 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 12:23:34.296693  5674 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:34.297451  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 497) took 1024 ms. Trace:
I20260709 12:23:34.297796  1872 rpcz_store.cc:276] 0709 12:23:33.273496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.247228 (+973732us) service_pool.cc:224] Handling call
0709 12:23:34.247752 (+   524us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.297165 (+ 49413us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.297395 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.254936  1877 rpcz_store.cc:276] 0709 12:23:33.160995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.180144 (+1019149us) service_pool.cc:224] Handling call
0709 12:23:34.180487 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.182243 (+  1756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.182481 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:34.255296  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 104729us
I20260709 12:23:34.299196  1872 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: [10.8-12.9]
I20260709 12:23:34.299868  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 498) took 1026 ms. Trace:
I20260709 12:23:34.300184  1872 rpcz_store.cc:276] 0709 12:23:33.273796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.298521 (+1024725us) service_pool.cc:224] Handling call
0709 12:23:34.298917 (+   396us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:34.299604 (+   687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.299817 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.301342  1877 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:34.302712  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 508) took 2194 ms. Trace:
I20260709 12:23:34.303153  1728 rpcz_store.cc:276] 0709 12:23:32.108586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.036588 (+928002us) service_pool.cc:224] Handling call
0709 12:23:33.048393 (+ 11805us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.049471 (+  1078us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.302496 (+1253025us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b54002aa650. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:34.302648 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26323200}
I20260709 12:23:34.304276  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:34.255407  1730 rpcz_store.cc:269] 0709 12:23:32.686418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.971813 (+285395us) service_pool.cc:224] Handling call
0709 12:23:34.249804 (+1277991us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b54001bf3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:34.250027 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":67673472}
I20260709 12:23:34.305531  1877 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:23:34.306066  1877 pending_rounds.cc:85] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 4
W20260709 12:23:34.255038  5688 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.912s	user 0.011s	sys 0.000s
I20260709 12:23:34.220916  5557 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:34.284178  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:34.285775  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:34.286021  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 496) took 1012 ms. Trace:
I20260709 12:23:34.310518  1876 rpcz_store.cc:276] 0709 12:23:33.273186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.227608 (+954422us) service_pool.cc:224] Handling call
0709 12:23:34.229140 (+  1532us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.285731 (+ 56591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.285965 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:34.266286  5535 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.781s	user 0.011s	sys 0.004s
I20260709 12:23:34.288177  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:34.288441  1615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58114us
I20260709 12:23:34.279493  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:34.289390  5524 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.289603  5755 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:34.244374  5736 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:34.257989  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 105370us
I20260709 12:23:34.306388  1877 pending_rounds.cc:107] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:23:34.272754  5621 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:34.312104  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:34.318154  1726 raft_consensus.cc:606] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:23:34.318041  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:34.322885  1726 raft_consensus.cc:614] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:34.322338  1730 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:34.318475  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:34.319005  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:34.319067  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:34.320860  1877 raft_consensus.cc:2889] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:34.324558  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:34.328047  1730 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:34.328359  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:34.328576  1801 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:34.330960  1725 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:34.331863  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:34.333736  1802 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 12:23:34.334226  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:34.333418  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:34.334942  1876 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:34.335706  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:34.336272  1732 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:34.311641  5688 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.337602  5676 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:34.338717  1772 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:34.289477  5575 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.039s
I20260709 12:23:34.338999  1877 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:34.347254  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:34.359894  1801 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:34.359020  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:34.360622  1728 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:34.361099  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:34.362310  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43949.2us
I20260709 12:23:34.340950  1880 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:34.320637  5755 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:34.361989  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:34.327816  5365 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:34.363969  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:34.366513  1725 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:34.321565  5761 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:34.367949  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:34.369640  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.371953  1726 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:34.369917  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:34.370529  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2623) took 1339 ms. Trace:
I20260709 12:23:34.363528  4666 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 12:23:34.373173  1881 rpcz_store.cc:276] 0709 12:23:33.031297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.892617 (+861320us) service_pool.cc:224] Handling call
0709 12:23:34.370326 (+477709us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400026554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:34.370451 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":548839808}
I20260709 12:23:34.374224  5524 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:34.374675  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.374997  5524 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:34.370293  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:34.379784  1801 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:23:34.380453  1877 raft_consensus.cc:2410] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:23:34.380873  1732 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:23:34.330004  5708 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:34.338508  5674 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:34.319991  5535 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:34.388024  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:34.388798  1807 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:34.385428  5764 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.082s
I20260709 12:23:34.394151  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:34.366685  5735 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:34.401758  5708 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:34.330842  5504 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.048s
I20260709 12:23:34.324692  5756 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 12:23:34.366542  5289 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:34.369455  5557 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:34.369817  5393 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:34.399663  5654 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.901s	user 0.014s	sys 0.001s
I20260709 12:23:34.405059  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.376181  5524 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:23:34.410279  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70293.9us
I20260709 12:23:34.348385  5651 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.068s
I20260709 12:23:34.414882  1877 raft_consensus.cc:2368] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11.
I20260709 12:23:34.398427  5499 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.403205  5365 raft_consensus.cc:2764] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:23:34.404925  5735 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.450117  1881 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:34.363451  5676 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.405047  5765 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:34.454584  1770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 2.079s (SENT) [suppressed 44 similar messages]
I20260709 12:23:34.415685  5524 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:34.470253  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.450105  5556 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:34.457535  5735 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:34.444737  5393 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.446136  5654 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.492559  5524 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:34.500284  5772 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:23:34.501037  5708 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.465651  4666 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:23:34.509361  5676 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:34.492868  5761 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.512708  1731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:34.458570  5499 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.519604  5524 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:34.556320  5755 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.559068  5777 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:34.559543  1807 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:34.559732  5777 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.560384  5535 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:34.469424  5767 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.233s
I20260709 12:23:34.469023  5770 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.245s
I20260709 12:23:34.561945  5770 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:34.562261  5777 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:34.562351  5770 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:34.563331  5654 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:34.563544  5770 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.564982  5770 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:34.466012  5765 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.517139  5774 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.052s
I20260709 12:23:34.568156  5760 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.077s
W20260709 12:23:34.569137  5743 consensus_queue.cc:1175] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "075704438992440697732181e49ccd09" responder_term: 7 status { last_received { term: 6 index: 4 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:34.573709  5755 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:34.547398  1731 raft_consensus.cc:606] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 9 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:23:34.567272  5765 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:34.583716  1725 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:23:34.518246  5651 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:34.599390  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 144861us
W20260709 12:23:34.601756  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 187964us
I20260709 12:23:34.592975  1876 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:34.545635  5249 raft_consensus.cc:2955] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 9 LEADER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:34.604600  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:34.598465  1801 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:34.606276  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.607084  1876 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:34.615178  5765 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:34.604399  5755 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.647495  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.648412  1876 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:34.605121  5736 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.652451  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.653129  5756 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:34.653752  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.653815  5756 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.656972  5756 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:34.654649  1876 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:34.658954  5784 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260709 12:23:34.640848  5393 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:34.616479  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:34.605515  5735 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.602617  5651 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.675179  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:34.681155  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64776us
I20260709 12:23:34.721841  1876 raft_consensus.cc:2368] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10.
I20260709 12:23:34.686707  5329 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:34.668076  5755 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.670441  5736 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.727084  5760 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:34.728819  5775 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:34.748848  5750 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.764475  1872 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:34.751434  5676 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.733613  5653 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.746857  5755 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:34.761338  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2631) took 1691 ms (client timeout 2157 ms). Trace:
W20260709 12:23:34.769776  1880 rpcz_store.cc:269] 0709 12:23:33.070269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.111482 (+ 41213us) service_pool.cc:224] Handling call
0709 12:23:34.760926 (+1649444us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400026054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:34.761225 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2771904384}
I20260709 12:23:34.752077  5765 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.789800  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:34.790668  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:34.768165  5676 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.816135  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.765197  5760 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.821640  1880 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:34.768878  5750 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.800086  5761 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:34.771011  5653 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:34.773823  5765 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.767127  5775 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.822221  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 1972) took 1297 ms. Trace:
I20260709 12:23:34.883910  1878 rpcz_store.cc:276] 0709 12:23:33.525235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.616092 (+ 90857us) service_pool.cc:224] Handling call
0709 12:23:34.821973 (+1205881us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b540001ead4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:34.822151 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1212871552}
I20260709 12:23:34.828016  1800 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:34.829034  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.886838  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.825292  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:34.887971  1880 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
W20260709 12:23:34.898834  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 73611.5us
W20260709 12:23:34.899601  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.883567  5734 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.902244  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:34.900853  5790 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:23:34.906143  1880 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:34.827423  5651 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:34.916896  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:34.921958  1880 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:34.923766  1880 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:23:34.924397  1880 pending_rounds.cc:85] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 5
W20260709 12:23:34.904969  5788 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:23:34.894028  5735 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:34.924911  1880 pending_rounds.cc:107] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:23:34.931595  1880 raft_consensus.cc:2889] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:34.905356  5504 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:34.947592  1880 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:34.950223  5736 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:34.900928  5734 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:34.953358  5775 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:35.100811  5653 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:34.958504  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:34.956452  5750 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:34.948760  5504 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.102113  5504 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:35.003237  5782 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.262s	user 0.002s	sys 0.000s
I20260709 12:23:34.955188  5765 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:34.924314  5329 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:35.133823  5524 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.577s	user 0.008s	sys 0.003s
I20260709 12:23:35.136509  5760 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:35.137439  5760 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:35.125008  5708 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:35.139912  5708 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:35.021881  5792 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:23:35.138098  5760 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:35.174320  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 488) took 3804 ms. Trace:
I20260709 12:23:35.174883  1874 rpcz_store.cc:276] 0709 12:23:31.369501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.392249 (+ 22748us) service_pool.cc:224] Handling call
0709 12:23:31.411528 (+ 19279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.498065 (+2086537us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.173948 (+1675883us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b5400051754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:35.174237 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4765866752}
I20260709 12:23:35.176853  1874 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:35.178108  1874 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:23:35.161495  5708 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:35.170485  5676 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:35.190393  1874 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:35.186753  5329 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:35.200438  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.208348  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:35.210736  5803 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:35.211400  5687 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:35.188539  5653 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260709 12:23:35.209403  1880 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:35.215273  5731 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:35.215641  5653 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:35.215837  5731 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:35.216185  5653 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:35.216915  5731 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.217855  1806 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:35.223188  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:35.223779  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:35.226117  1880 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:35.230810  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:35.218761  5760 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:32.531721  1803 rpcz_store.cc:276] 0709 12:23:30.835253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.213767 (+378514us) service_pool.cc:224] Handling call
0709 12:23:31.236353 (+ 22586us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.278518 (+ 42165us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:32.223334 (+944816us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.530164 (+306830us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:32.530225 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:32.530279 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.530674 (+   395us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.530885 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88042,"prepare.queue_time_us":201157,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":2647,"thread_start_us":1058200,"threads_started":2}
I20260709 12:23:35.248361  5504 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:35.250165  1874 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:23:35.231775  1880 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:35.254362  1880 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:35.256467  1880 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:35.259037  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2746) took 2573 ms. Trace:
I20260709 12:23:35.259817  1729 rpcz_store.cc:276] 0709 12:23:32.685927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.292060 (+606133us) service_pool.cc:224] Handling call
0709 12:23:33.304640 (+ 12580us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.854663 (+550023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.252306 (+1397643us) spinlock_profiling.cc:243] Waited 545 ms on lock 0x7b5400414954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:35.258968 (+  6662us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1253700864}
I20260709 12:23:35.262929  1801 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:23:35.273989  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2637) took 2005 ms (client timeout 1923 ms). Trace:
I20260709 12:23:35.276206  1881 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:23:35.276620  1881 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 6
I20260709 12:23:35.276832  1881 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:23:35.277086  1881 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:35.217183  5653 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:35.280669  5653 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:35.280732  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1761) took 1623 ms. Trace:
I20260709 12:23:35.281358  1802 rpcz_store.cc:276] 0709 12:23:33.657838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.668720 (+ 10882us) service_pool.cc:224] Handling call
0709 12:23:33.682029 (+ 13309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.334137 (+652108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.280463 (+946326us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:35.280670 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1498175232}
W20260709 12:23:35.274526  1879 rpcz_store.cc:269] 0709 12:23:33.269028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.535033 (+266005us) service_pool.cc:224] Handling call
0709 12:23:35.273690 (+1738657us) spinlock_profiling.cc:243] Waited 58.6 ms on lock 0x7b5400222754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:35.273898 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":134851840}
I20260709 12:23:35.279569  5499 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:35.279667  1874 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:23:35.289517  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.285820  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.286669  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:35.267045  5731 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:35.291545  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84771.9us
I20260709 12:23:35.292075  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2633) took 2196 ms. Trace:
I20260709 12:23:35.297003  1875 rpcz_store.cc:276] 0709 12:23:33.092388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.126632 (+ 34244us) service_pool.cc:224] Handling call
0709 12:23:35.288537 (+2161905us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b5400116f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:35.288659 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":768946432}
I20260709 12:23:35.292857  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.290387  5669 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:23:35.293792  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:35.294009  5249 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.533s	user 0.010s	sys 0.003s
I20260709 12:23:35.295207  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.291770  5787 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.291488  1801 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:35.301594  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:35.302774  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.303655  1801 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:35.298372  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.305830  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:35.306576  1799 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:35.299556  1802 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:35.300441  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:35.298317  1803 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:35.307711  1806 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:35.289812  5504 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:35.292038  5734 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:35.310873  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.312508  5787 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.318288  5787 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:35.320362  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.321719  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.321779  5674 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.322348  1803 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:35.326678  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:35.331774  1802 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:35.331871  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.332765  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.333173  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.333833  1803 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:35.333920  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.327988  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:35.301170  5774 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:23:35.322383  5674 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:35.300067  5313 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:23:35.312494  5669 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:35.336930  5669 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:35.337836  5669 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.339161  5669 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:35.342571  5717 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:35.342924  5310 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:35.343423  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:35.343991  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:35.344218  5717 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:35.344873  5717 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.344774  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:35.345434  5717 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:35.345992  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:35.346987  5734 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:35.347182  5687 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:35.347000  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:35.344710  1804 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:35.347471  5734 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.351817  5734 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.354315  5705 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:35.354825  5705 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:35.355705  5705 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.356954  5705 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.361270  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:35.364732  1802 raft_consensus.cc:2410] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:23:35.374109  5738 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:35.374636  5738 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:35.375551  5738 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.376545  5738 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.368850  5499 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:35.378803  5499 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.379244  5499 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.380704  1798 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:35.381290  1798 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:35.382336  1798 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.385586  1798 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:35.388113  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:35.392930  1804 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:35.389729  5760 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:23:35.397512  1723 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:35.398679  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:35.407241  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:35.410038  5687 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:35.410660  5687 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:35.411616  5687 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.412806  2414 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:35.423617  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:35.431531  5813 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:35.432134  5813 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:35.432559  5813 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:35.434866  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.439652  5814 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:35.443184  5816 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:23:35.448742  5813 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:35.444017  5814 consensus_queue.cc:1243] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:35.451365  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2721) took 3286 ms. Trace:
I20260709 12:23:35.451573  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1974) took 1925 ms. Trace:
I20260709 12:23:35.451948  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:35.453035  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:35.459179  1878 rpcz_store.cc:276] 0709 12:23:33.526242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.918588 (+1392346us) service_pool.cc:224] Handling call
0709 12:23:34.945436 (+ 26848us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.945730 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:34.946102 (+   372us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.386120 (+440018us) spinlock_profiling.cc:243] Waited 1.7 ms on lock 0x7b180036f340. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:23:35.451195 (+ 65075us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:35.451255 (+    60us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:35.451317 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.451400 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.451517 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":203,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":41399,"raft.queue_time_us":4461,"spinlock_wait_cycles":3906304,"thread_start_us":64807,"threads_started":1}
I20260709 12:23:35.420156  5313 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:35.404618  5777 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.457772  1724 rpcz_store.cc:276] 0709 12:23:32.133647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.111801 (+978154us) service_pool.cc:224] Handling call
0709 12:23:33.112164 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.124063 (+ 11899us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:33.352711 (+228648us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.160109 (+1807398us) spinlock_profiling.cc:243] Waited 8.93 ms on lock 0x7b18003958c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:23:35.391063 (+230954us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.419985 (+ 28922us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:35.420051 (+    66us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:35.420110 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.420188 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.420354 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14421,"prepare.queue_time_us":251948,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"raft.queue_time_us":59928,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":4807,"spinlock_wait_cycles":20531072,"thread_start_us":380306,"threads_started":4,"wal-append.queue_time_us":1913471}
I20260709 12:23:35.461593  5313 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:35.465341  5313 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:35.466645  5813 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:35.469107  5499 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:35.469856  5745 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.350s	user 0.005s	sys 0.003s
I20260709 12:23:35.455541  5689 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:35.409101  5674 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.473104  5398 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:35.474336  5674 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:35.456431  5805 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:35.482283  5805 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:35.424352  5697 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:35.475427  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.484107  5717 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.475072  5777 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.477347  5688 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:35.404995  5768 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 12:23:35.479113  5760 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:35.480504  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.463110  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:35.485603  1798 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:23:35.489552  1798 pending_rounds.cc:85] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 4
I20260709 12:23:35.489910  1798 pending_rounds.cc:107] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:23:35.490226  1798 raft_consensus.cc:2889] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:35.495613  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:35.495810  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:35.496233  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:35.501652  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:35.503118  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1896) took 1176 ms. Trace:
I20260709 12:23:35.503753  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:35.503508  1798 rpcz_store.cc:276] 0709 12:23:34.327018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.327178 (+   160us) service_pool.cc:224] Handling call
0709 12:23:34.363346 (+ 36168us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.380575 (+1017229us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b5400403420. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:23:35.502823 (+122248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.503055 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":456311168}
I20260709 12:23:35.508309  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:35.508996  5781 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 12:23:35.498354  5760 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:35.509827  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:35.506112  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:35.507071  5705 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.501317  5704 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.501101  5813 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.531555  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:35.535323  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:35.551416  2414 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:23:35.555189  1726 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:35.483947  5697 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:35.555867  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:35.556069  5717 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:35.556448  5704 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.557153  5705 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.557667  5697 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:35.528851  1799 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:35.548867  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:35.549718  1802 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:35.553813  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 123720us
I20260709 12:23:35.550997  1725 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:35.549714  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:35.560161  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:35.560773  1726 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:35.561051  1697 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:35.555048  2414 pending_rounds.cc:85] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 7
I20260709 12:23:35.567448  2414 pending_rounds.cc:107] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:23:35.567888  2414 raft_consensus.cc:2889] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:35.569195  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 912) took 3051 ms. Trace:
I20260709 12:23:35.571449  5736 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.572029  5736 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:35.579138  4899 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260709 12:23:35.580150  1732 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:23:35.583138  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:35.583907  1798 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:35.556172  5813 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.595765  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:35.596853  5813 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:35.597335  5249 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:35.597882  5249 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:35.597683  5313 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:35.598961  5704 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:35.599360  1729 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:35.600055  5249 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:35.601349  5249 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.603595  1878 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:23:35.569625  2414 rpcz_store.cc:276] 0709 12:23:32.517475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.517708 (+   233us) service_pool.cc:224] Handling call
0709 12:23:32.532605 (+ 14897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.412534 (+2879929us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b58007805d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd340aa 00007f16abdb6447 00007f16abdcefe7 00007f16abdd07de 00007f16abddab49 00007f16abddb416 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4
0709 12:23:35.568913 (+156379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.569137 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5851961984}
I20260709 12:23:35.622150  1798 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:35.500720  5814 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.101s
I20260709 12:23:35.645401  1806 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:35.602034  5734 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:35.632578  5717 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:35.645010  5765 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.655450  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:36.287504  1802 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:36.287519  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:35.797326  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2625) took 2765 ms (client timeout 2079 ms). Trace:
I20260709 12:23:35.798988  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1898) took 1459 ms. Trace:
W20260709 12:23:36.178977  1771 thread.cc:641] server-negotiator [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.016s	sys 0.032s
W20260709 12:23:35.880199  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:36.289535  1873 rpcz_store.cc:269] 0709 12:23:33.031780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.054430 (+ 22650us) service_pool.cc:224] Handling call
0709 12:23:35.796981 (+2742551us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b540001efd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:35.797210 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489675648}
W20260709 12:23:36.279839  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.291523  1873 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:35.677549  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:36.292806  1874 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:36.286063  1875 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260709 12:23:36.294888  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.307154  1880 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260709 12:23:36.309057  1771 thread.cc:608] server-negotiator [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.019s	sys 0.033s
W20260709 12:23:36.067520  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.290997  1807 rpcz_store.cc:276] 0709 12:23:34.339345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.397292 (+ 57947us) service_pool.cc:224] Handling call
0709 12:23:34.397747 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.560044 (+162297us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.798700 (+1238656us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b540026f054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:35.798921 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":371625856}
I20260709 12:23:36.329012  1875 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:35.655099  5732 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.720s	user 0.000s	sys 0.027s
W20260709 12:23:35.793641  5749 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.645s	user 0.004s	sys 0.000s
I20260709 12:23:35.605592  5653 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:35.631839  5759 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:35.787813  5812 log.cc:927] Time spent T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:23:36.325722  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:36.341207  1807 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:36.341589  1879 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:36.341877  5669 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:36.344156  5734 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:36.343901  5669 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:35.798602  5313 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:35.737421  5789 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:36.345661  5789 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.346298  5669 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:36.348075  5789 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:35.878656  5777 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:36.349181  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:36.349859  1806 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:35.999599  5687 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.342446  1802 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:36.051545  5705 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:36.306919  5736 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:35.772324  5760 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:36.383997  5736 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.351452  5777 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.384555  5687 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:36.385221  5777 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.335038  5740 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:36.386986  5740 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:36.388134  5777 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:36.321635  5499 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:36.351277  5759 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.390528  5759 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:36.379914  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1891) took 2735 ms. Trace:
I20260709 12:23:36.391178  1805 rpcz_store.cc:276] 0709 12:23:33.644164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.651336 (+  7172us) service_pool.cc:224] Handling call
0709 12:23:33.673200 (+ 21864us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.678868 (+  5668us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.678909 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:33.687682 (+  8773us) log.cc:844] Serialized 26 byte log entry
0709 12:23:33.895314 (+207632us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:33.895413 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.895566 (+   153us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:35.792696 (+1897130us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:36.379548 (+586852us) raft_consensus.cc:1685] finished
0709 12:23:36.379573 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.379843 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4021,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":6948,"spinlock_wait_cycles":99907456,"thread_start_us":147156,"threads_started":1,"wal-append.queue_time_us":371353}
I20260709 12:23:36.379274  5603 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:36.384595  5736 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.394958  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:36.392365  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:36.396202  5687 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:36.326375  5765 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:36.379741  5738 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.996s	user 0.012s	sys 0.000s
I20260709 12:23:36.398149  1805 raft_consensus.cc:2410] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:23:36.396096  1802 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:36.400941  1874 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:36.401974  5737 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.402854  5719 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.403465  5719 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.404058  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2765) took 3622 ms. Trace:
I20260709 12:23:36.402648  5737 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:36.381595  5249 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.780s	user 0.011s	sys 0.000s
I20260709 12:23:36.411170  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:36.400211  5805 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:36.404553  1723 rpcz_store.cc:276] 0709 12:23:32.781673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.633982 (+852309us) service_pool.cc:224] Handling call
0709 12:23:33.669207 (+ 35225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.397935 (+1728728us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.403803 (+1005868us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b54003cd2d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:36.403990 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3974132480}
I20260709 12:23:36.410000  1807 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
W20260709 12:23:36.338583  5732 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.405s	user 0.008s	sys 0.069s
I20260709 12:23:36.331534  5816 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:36.387851  5740 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:36.412231  1802 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:36.412318  1806 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:36.404434  5653 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:36.404825  5765 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:36.396063  5736 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:36.340159  5717 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:36.425472  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.425778  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:36.426151  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.420914  5740 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:36.421595  5687 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:36.426489  2383 connection.cc:256] RPC call timeout handler was delayed by -1.5721701829997983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3264s for a response
I20260709 12:23:36.435873  1802 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:23:36.451730  5717 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.416697  5581 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 12:23:36.418380  5816 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:36.470561  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.470954  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.474576  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:36.474642  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:36.468762  5687 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:36.430162  5686 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:36.476033  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.476841  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.434679  5621 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.501711  5621 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.504052  5621 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:36.492816  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.494364  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:36.506969  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.431135  5691 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.499228  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:36.500905  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:36.509438  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:36.510978  5765 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:36.511723  5765 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:36.501204  5753 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 2.321s	user 0.003s	sys 0.039s
I20260709 12:23:36.515038  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:36.516566  1728 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:36.515866  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:36.478708  5667 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.519325  5667 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:36.521489  5667 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:36.523347  5687 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:36.526036  5766 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:36.527827  5816 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:36.528684  1724 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:36.529606  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.530817  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:36.531340  1724 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:36.532845  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1970) took 3010 ms. Trace:
I20260709 12:23:36.533377  1728 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:36.549985  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:36.603142  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:36.533452  1872 rpcz_store.cc:276] 0709 12:23:33.522238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.300950 (+778712us) service_pool.cc:224] Handling call
0709 12:23:34.762237 (+461287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.765227 (+  2990us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.532499 (+1767272us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b5400026050. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:36.532733 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1059888384}
I20260709 12:23:36.613742  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:36.615206  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49249.9us
I20260709 12:23:36.619743  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.630302  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.630944  1728 raft_consensus.cc:2376] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10. Re-sending same reply.
I20260709 12:23:36.631798  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.632758  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.635353  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:36.637156  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:36.644840  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:36.645157  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:36.646054  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.647668  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:36.655746  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1541985830003796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
W20260709 12:23:36.656132  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1415985830003592s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
I20260709 12:23:36.656773  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:36.657106  1726 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:36.659397  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:36.655519  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.526774  5766 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:36.665778  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:36.643352  5687 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:36.620186  5735 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:36.629707  5674 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:36.692260  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.643558  5667 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.570865  5691 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.643653  5546 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:36.661248  5819 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.655267  5492 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.662974  5765 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:36.711886  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46397.4us
I20260709 12:23:36.762144  2383 leader_election.cc:400] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11.
I20260709 12:23:36.687295  5724 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:36.762645  2383 leader_election.cc:403] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:36.714071  5377 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
W20260709 12:23:36.774442  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 502) took 2365 ms (client timeout 2149 ms). Trace:
W20260709 12:23:36.779827  1876 rpcz_store.cc:269] 0709 12:23:34.408614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.663632 (+255018us) service_pool.cc:224] Handling call
0709 12:23:36.774179 (+2110547us) spinlock_profiling.cc:243] Waited 38 ms on lock 0x7b54002df0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:36.774347 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87440128}
I20260709 12:23:36.717549  5537 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:36.759675  5718 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.693337  5487 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.074s
I20260709 12:23:36.783990  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:36.764647  5654 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.765954  5656 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.764906  5730 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:36.781574  5724 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:36.799847  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:36.757114  5546 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.801950  1730 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:36.765172  5492 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.817327  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:36.820185  1724 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:36.783365  5594 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:36.506032  5686 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.692412  5621 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:36.804044  1728 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 8. Ops: []
I20260709 12:23:36.796707  5654 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:36.806529  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 336540us
W20260709 12:23:36.806640  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.000s	sys 0.012s
W20260709 12:23:36.939724  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.001s	sys 0.012s
W20260709 12:23:36.939934  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.942113  1807 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:36.942026  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:36.943065  1806 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:36.943321  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 277596us
I20260709 12:23:36.949993  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:36.809237  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 521809us
I20260709 12:23:36.813158  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2799) took 1252 ms. Trace:
I20260709 12:23:36.803593  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:36.954420  1727 rpcz_store.cc:276] 0709 12:23:35.560612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.560735 (+   123us) service_pool.cc:224] Handling call
0709 12:23:36.326599 (+765864us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:36.326875 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:36.339886 (+ 13011us) log.cc:844] Serialized 14 byte log entry
0709 12:23:36.812630 (+472744us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:36.812711 (+    81us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:36.812804 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.812897 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:36.813077 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":121,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":20805,"thread_start_us":472232,"threads_started":1}
I20260709 12:23:36.794581  5377 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:23:36.799736  5487 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:36.689579  5669 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:36.807933  5499 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:36.772832  5766 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.957593  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:36.957749  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.958210  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:36.972148  5766 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:36.972877  1802 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:23:36.979143  5669 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:36.980515  5546 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:36.981385  5546 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:36.980937  5669 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:36.964023  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.001s	sys 0.013s
W20260709 12:23:36.982366  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.002s	sys 0.013s
W20260709 12:23:36.808596  1798 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.024s	user 0.005s	sys 0.007s
W20260709 12:23:36.992996  1798 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.208s	user 0.005s	sys 0.010s
W20260709 12:23:36.937515  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 98923.2us
W20260709 12:23:36.997498  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.960719  5487 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:36.985677  5829 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.793833  5718 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.816069  5656 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.937892  5724 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.840385  5594 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.834508  5694 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:37.002529  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:37.051182  1807 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:36.847703  5736 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:37.008534  2382 connection.cc:256] RPC call timeout handler was delayed by -2.2892995229999542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
W20260709 12:23:37.053846  2382 connection.cc:256] RPC call timeout handler was delayed by -2.2622995229999106s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
W20260709 12:23:37.054180  2382 connection.cc:256] RPC call timeout handler was delayed by -1.9778995229999055s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
W20260709 12:23:37.054486  2382 connection.cc:256] RPC call timeout handler was delayed by -1.4750552669997887s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
I20260709 12:23:37.007686  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:36.903503  5621 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:23:37.014672  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61541.6us
I20260709 12:23:36.715268  5667 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:36.835752  5702 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.176s
W20260709 12:23:37.047724  1770 leader_election.cc:336] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 2.079s (SENT)
I20260709 12:23:36.995447  5687 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.010380  5674 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:36.833956  5819 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.007905  5764 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:23:37.056854  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49273.4us
W20260709 12:23:37.121570  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73884.2us
I20260709 12:23:37.067987  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:36.960968  5756 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.088s
I20260709 12:23:37.071148  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:36.816390  5730 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.124115  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:37.128281  2411 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:37.130098  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.072995  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.169628  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.172885  2410 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:37.060109  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:37.130980  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.171342  2418 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:23:37.180989  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 121005us
I20260709 12:23:37.182991  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:37.174459  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.177146  2417 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:37.124562  2414 raft_consensus.cc:2410] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 13 because replica has last-logged OpId of term: 12 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:23:37.048772  5653 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.086s
I20260709 12:23:37.178642  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 750) took 4275 ms. Trace:
W20260709 12:23:37.118866  5717 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.027s	sys 0.058s
I20260709 12:23:37.177973  1731 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:37.176914  5691 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:37.193488  1618 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.813s	user 0.006s	sys 0.005s
I20260709 12:23:37.046491  5829 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.123469  5764 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:37.061483  5736 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:37.173640  1803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.118s	user 0.013s	sys 0.028s
I20260709 12:23:37.188443  2415 rpcz_store.cc:276] 0709 12:23:32.900326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.900784 (+   458us) service_pool.cc:224] Handling call
0709 12:23:32.901118 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:32.901934 (+   816us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:33.022449 (+120515us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.358006 (+335557us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.443069 (+ 85063us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.640988 (+197919us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b180039d480. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:36.402115 (+2761127us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b180039d480. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:23:36.402271 (+   156us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.402761 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:23:36.809542 (+406781us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:36.809605 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.809717 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:37.175244 (+365527us) raft_consensus.cc:1685] finished
0709 12:23:37.175263 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.175384 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2831261,"prepare.queue_time_us":300365,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":127,"raft.queue_time_us":3176595,"raft.run_cpu_time_us":608,"raft.run_wall_time_us":29428,"spinlock_wait_cycles":11396608,"thread_start_us":709087,"threads_started":4,"wal-append.queue_time_us":727316,"wal-append.run_cpu_time_us":4857,"wal-append.run_wall_time_us":1426409}
I20260709 12:23:37.209520  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:37.208884  2410 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:37.212152  1803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.157s	user 0.013s	sys 0.029s
I20260709 12:23:37.214731  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.052498  5724 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.221261  2415 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:37.215442  2416 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:37.220320  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.222719  2414 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:37.224231  2410 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:37.222841  2418 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 6.
I20260709 12:23:37.232740  2411 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:37.050616  5694 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:23:37.123098  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58395.4us
I20260709 12:23:37.179744  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.238528  2413 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:37.124540  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.188086  5313 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:37.179122  5836 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:37.239763  2412 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:37.241384  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:37.242394  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.242725  2411 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:23:37.251118  2415 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:23:37.192394  5717 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.028s	sys 0.069s
I20260709 12:23:37.239981  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.252635  2410 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:37.256883  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:37.196238  5654 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:37.254426  5696 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.330s	user 0.011s	sys 0.049s
I20260709 12:23:37.241353  2413 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:37.192260  5487 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.062562  5841 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:37.078403  1730 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:37.170682  5687 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.277100  5819 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:37.261852  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.227146  5686 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:37.250058  5842 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:37.290102  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 234744us
I20260709 12:23:37.264380  5581 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:37.225685  5621 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.279222  5830 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.296290  2416 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
W20260709 12:23:37.278033  5696 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.356s	user 0.012s	sys 0.049s
I20260709 12:23:37.272107  5492 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:37.314385  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 169028us
I20260709 12:23:37.282616  5841 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:37.260028  5836 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.369128  5830 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:37.336568  5718 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:37.371057  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:37.371589  5829 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:37.372119  1727 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 12 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:37.372189  5829 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:37.294773  5831 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.291904  5756 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:37.374504  5594 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:37.374635  5831 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.375094  5842 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:37.376497  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:37.377262  5831 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:37.275339  5764 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.347234  5724 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:37.378448  5764 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:37.378916  5756 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.381814  5756 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:37.383369  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:37.384001  5537 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.593s	user 0.005s	sys 0.005s
W20260709 12:23:37.348109  5844 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:23:37.348817  5852 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:37.352866  5832 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.138s	user 0.000s	sys 0.000s
I20260709 12:23:37.353235  5667 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:37.353768  5838 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:23:37.354676  5310 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:37.342028  5850 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.360972  5847 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:23:37.282940  5674 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.307217  5765 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:37.366544  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.373596  5841 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.390002  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:37.283438  5694 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.395364  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.398197  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.399240  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:37.406629  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.382546  5756 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
W20260709 12:23:37.402695  1771 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.923s (SENT)
I20260709 12:23:37.426020  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.408115  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1779) took 2107 ms. Trace:
I20260709 12:23:37.385114  5699 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.461843  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:37.462077  2412 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:37.462344  5699 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.464454  2410 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:23:37.389495  5537 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:37.395825  5718 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.465680  5718 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.407503  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:37.467352  1724 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:37.468561  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:37.469821  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:37.470450  1724 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:37.476985  5718 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:37.477597  1803 rpcz_store.cc:276] 0709 12:23:35.300496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.334445 (+ 33949us) service_pool.cc:224] Handling call
0709 12:23:35.334736 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.364296 (+ 29560us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:36.054807 (+690511us) log.cc:844] Serialized 14 byte log entry
0709 12:23:37.407677 (+1352870us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:37.407775 (+    98us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:37.407841 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.407927 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.408056 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1058948,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":2318,"raft.queue_time_us":134932,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":129,"thread_start_us":1659353,"threads_started":3,"wal-append.queue_time_us":1211847}
I20260709 12:23:37.479640  1728 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
W20260709 12:23:37.403630  5740 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.958s	user 0.013s	sys 0.000s
I20260709 12:23:37.459622  2414 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:37.482574  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:37.483273  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 243997us
I20260709 12:23:37.483610  2412 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:37.484999  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82347.4us
W20260709 12:23:37.485925  1771 leader_election.cc:336] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.079s (SENT)
I20260709 12:23:37.486459  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:37.471084  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 156894us
I20260709 12:23:37.491920  1732 raft_consensus.cc:2410] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:23:37.492430  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:37.421602  5383 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:37.407228  5310 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:37.496039  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.496661  2418 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:37.497502  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.500972  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1908) took 1931 ms. Trace:
I20260709 12:23:37.501438  1801 rpcz_store.cc:276] 0709 12:23:35.569872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.570056 (+   184us) service_pool.cc:224] Handling call
0709 12:23:35.570397 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:35.570756 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:35.570804 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.325827 (+755023us) log.cc:844] Serialized 26 byte log entry
0709 12:23:36.807201 (+481374us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:36.807267 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.807380 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:37.500709 (+693329us) raft_consensus.cc:1685] finished
0709 12:23:37.500731 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.500898 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":251,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":9716,"thread_start_us":480963,"threads_started":1,"wal-append.queue_time_us":808560}
I20260709 12:23:37.502770  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1873) took 1003 ms. Trace:
I20260709 12:23:37.503414  1726 rpcz_store.cc:276] 0709 12:23:36.499498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.658329 (+158831us) service_pool.cc:224] Handling call
0709 12:23:36.695512 (+ 37183us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:36.695831 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:36.998959 (+303128us) log.cc:844] Serialized 14 byte log entry
0709 12:23:37.484900 (+485941us) log.cc:844] Serialized 14 byte log entry
0709 12:23:37.502240 (+ 17340us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:37.502324 (+    84us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:37.502392 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.502497 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.502672 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53012,"prepare.queue_time_us":596814,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":4497,"raft.queue_time_us":41404,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":5603,"spinlock_wait_cycles":1893632,"thread_start_us":623446,"threads_started":3,"wal-append.queue_time_us":372755}
I20260709 12:23:37.378459  5831 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.106s
I20260709 12:23:37.455289  5546 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:37.534157  1801 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:40096
W20260709 12:23:37.508893  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 271168us
I20260709 12:23:37.536468  1801 raft_consensus.cc:606] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:23:37.511591  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:37.536808  1801 raft_consensus.cc:614] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:37.501184  5855 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:23:37.494088  5766 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.521s	user 0.012s	sys 0.001s
I20260709 12:23:37.514879  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.545341  2412 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
W20260709 12:23:37.551879  1798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.000s	sys 0.032s
I20260709 12:23:37.551879  1806 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
W20260709 12:23:37.552660  1798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.000s	sys 0.032s
I20260709 12:23:37.554968  2415 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
W20260709 12:23:37.555687  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.398330  5852 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.567912  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.572214  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:37.573190  2412 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:37.558075  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:37.483520  5765 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.543164  5694 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:37.508059  5537 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:37.537681  5855 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:37.538715  5859 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:37.456205  5850 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:37.625893  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 67985.9us
I20260709 12:23:37.507884  5383 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:37.618077  5377 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.661s	user 0.012s	sys 0.000s
W20260709 12:23:37.562602  5851 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:23:37.629384  1846 connection.cc:256] RPC call timeout handler was delayed by -2.4659075399999892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1622s for a response
W20260709 12:23:37.650992  1846 connection.cc:256] RPC call timeout handler was delayed by -2.4596343880002678s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15610000000000002s for a response
I20260709 12:23:37.562116  5858 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:37.652328  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.622476  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:37.633432  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 67540.6us
W20260709 12:23:37.633965  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 122429us
W20260709 12:23:37.638955  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.640069  5876 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:37.648092  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.560432  5718 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 12:23:37.613377  5737 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:37.650159  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.618618  5687 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:37.664677  5876 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:37.745905  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 339308us
W20260709 12:23:37.746022  1846 connection.cc:256] RPC call timeout handler was delayed by -1.9128382630001397s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
I20260709 12:23:37.723029  5765 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:37.560901  5731 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20260709 12:23:37.553860  5857 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
W20260709 12:23:37.634066  5839 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.293s	user 0.000s	sys 0.002s
I20260709 12:23:37.661427  5377 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11.
I20260709 12:23:37.687202  5855 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.763094  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1852) took 1307 ms. Trace:
I20260709 12:23:37.763550  1723 rpcz_store.cc:276] 0709 12:23:36.456109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.458878 (+  2769us) service_pool.cc:224] Handling call
0709 12:23:36.479132 (+ 20254us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.479476 (+   344us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:36.479504 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.525832 (+ 46328us) log.cc:844] Serialized 26 byte log entry
0709 12:23:36.816942 (+291110us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:37.191037 (+374095us) log.cc:844] Serialized 14 byte log entry
0709 12:23:37.552195 (+361158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.552301 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:37.762849 (+210548us) raft_consensus.cc:1685] finished
0709 12:23:37.762873 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.763019 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26566,"prepare.queue_time_us":475499,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":42,"raft.queue_time_us":70042,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":9262,"thread_start_us":870955,"threads_started":3,"wal-append.queue_time_us":753184}
I20260709 12:23:37.633845  5667 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.564074  5696 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.096s
I20260709 12:23:37.767372  5852 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:37.767217  5667 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.630928  5724 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:37.768929  5724 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.770113  5667 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:37.631532  4666 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:37.763568  5855 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:37.623679  5310 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.772161  5310 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:37.774048  5862 consensus_queue.cc:1175] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" responder_term: 11 status { last_received { term: 10 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:37.635110  5797 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:37.775007  5797 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:37.775465  5797 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:37.771068  4666 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.721390  2412 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
W20260709 12:23:37.725358  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70731.4us
I20260709 12:23:37.695457  5537 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.743386  5876 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:37.788143  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:37.748052  1846 connection.cc:256] RPC call timeout handler was delayed by -1.7922382629999447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
I20260709 12:23:37.660199  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:37.788652  2414 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:23:37.792484  1846 connection.cc:256] RPC call timeout handler was delayed by -1.6856075400000918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2489s for a response
I20260709 12:23:37.791491  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1911) took 2200 ms. Trace:
I20260709 12:23:37.793465  1798 rpcz_store.cc:276] 0709 12:23:35.591546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.625941 (+ 34395us) service_pool.cc:224] Handling call
0709 12:23:35.626415 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.626733 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:36.993544 (+1366811us) log.cc:844] Serialized 14 byte log entry
0709 12:23:37.790768 (+797224us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:37.791036 (+   268us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:37.791156 (+   120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.791249 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.791414 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44959,"prepare.queue_time_us":810173,"prepare.run_cpu_time_us":5471,"prepare.run_wall_time_us":16472,"raft.queue_time_us":178386,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":2300,"thread_start_us":1990806,"threads_started":3,"wal-append.queue_time_us":648977}
W20260709 12:23:37.793080  1846 connection.cc:256] RPC call timeout handler was delayed by -1.5157444249998662s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2002s for a response
W20260709 12:23:37.796368  1846 connection.cc:256] RPC call timeout handler was delayed by -1.5087382629999411s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22120000000000004s for a response
W20260709 12:23:37.797648  1846 connection.cc:256] RPC call timeout handler was delayed by -1.3468366659999447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19630000000000003s for a response
I20260709 12:23:37.799642  1697 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:37.798523  4666 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:37.770716  5667 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.782105  5718 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:23:37.809136  5764 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:37.809531  5731 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:37.744062  5865 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.058s
I20260709 12:23:37.744506  5756 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:23:37.823112  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:37.823359  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:37.824003  1875 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:37.824136  1881 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:37.839218  5819 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:37.841787  5862 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:37.820786  5724 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:37.843182  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:37.847460  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:37.848107  1880 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:37.848207  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:37.849855  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:37.851269  5877 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:37.851475  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:37.851833  5877 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:37.852357  5877 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:37.853765  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:37.854419  2412 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:37.854854  2412 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:37.856317  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:37.854457  1876 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
W20260709 12:23:37.808030  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 147904us
I20260709 12:23:37.796983  5687 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.856642  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:37.812194  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:37.863904  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:37.796775  5765 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.855736  2412 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.815079  5764 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:37.869771  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 57692.8us
I20260709 12:23:37.796974  5762 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:37.875027  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:37.875998  1881 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:37.877549  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:37.857712  1880 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:37.882496  1875 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:37.885594  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:37.885728  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:37.887076  1875 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:37.821568  5731 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.889161  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:37.824606  5696 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:23:37.897980  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:37.902246  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:37.909088  1731 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:37.824842  5847 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.862732  4666 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:37.907393  1876 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:37.897699  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:37.903481  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:37.916900  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:37.902549  1800 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:37.892129  2412 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:37.905452  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:37.906590  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:37.923265  1799 raft_consensus.cc:2376] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10. Re-sending same reply.
I20260709 12:23:37.901504  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:37.914368  1798 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:37.914603  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:37.925191  1807 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:37.925333  1801 raft_consensus.cc:2410] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:23:37.914915  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.927532  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:37.928244  1803 raft_consensus.cc:2376] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13. Re-sending same reply.
I20260709 12:23:37.929883  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:37.931043  2417 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:23:37.932324  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.978s	user 0.013s	sys 0.021s
W20260709 12:23:37.932720  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.979s	user 0.013s	sys 0.021s
I20260709 12:23:37.933086  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:37.933451  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 1.12988e+06us
I20260709 12:23:37.934177  5581 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:37.879083  5737 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.888192  5734 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.934334  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:37.917492  1802 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:37.889395  5764 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:37.820779  5667 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.954072  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:37.889953  5881 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:37.944288  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:37.903929  5724 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:37.961418  5724 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.962002  5724 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:37.867687  5687 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.938603  5850 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:37.907080  5731 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:37.902143  5883 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:37.890473  5762 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.889819  5765 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.022111  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1792) took 1627 ms. Trace:
I20260709 12:23:38.022934  1805 rpcz_store.cc:276] 0709 12:23:36.394494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.394663 (+   169us) service_pool.cc:224] Handling call
0709 12:23:38.017409 (+1622746us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b5400066154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:38.021991 (+  4582us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5091200}
I20260709 12:23:38.024832  1726 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:38.026072  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:38.015821  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:37.954928  5621 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:37.939486  5861 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.030525  1872 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:37.958566  5881 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.035315  1803 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:23:38.038769  1803 raft_consensus.cc:606] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:23:38.039445  1803 raft_consensus.cc:614] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:38.041318  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.915871  5852 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.099s
I20260709 12:23:38.043963  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:38.047430  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.014477  5883 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.048851  1802 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:37.942899  5847 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:37.954676  5734 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:37.953325  5737 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.018127  5842 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:38.055439  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:38.056740  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 171246us
I20260709 12:23:38.042975  5881 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:38.057226  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 113033us
W20260709 12:23:38.064729  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.060257  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:38.070528  1846 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:38.067912  1802 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 12:23:38.072453  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.073874  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:38.081704  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:38.086074  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:37.893201  1618 maintenance_manager.cc:643] P 4002daa3304c4fb4be053abd4780c195: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 9.716s	user 0.216s	sys 0.115s Metrics: {"bytes_written":30313,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":483067,"dirs.run_cpu_time_us":707,"dirs.run_wall_time_us":18986,"drs_written":1,"lbm_read_time_us":34486,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":2,"lbm_write_time_us":331653,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":25,"peak_mem_usage":0,"reinsert_count":0,"rows_written":63,"thread_start_us":783966,"threads_started":2,"update_count":11,"wal-append.queue_time_us":678768}
I20260709 12:23:38.090963  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:38.097517  1803 raft_consensus.cc:2376] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10. Re-sending same reply.
I20260709 12:23:38.053103  5883 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:38.063671  5785 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 3.304s	user 0.011s	sys 0.044s
I20260709 12:23:38.103078  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:38.066310  5842 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:38.084544  5717 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.087846  5313 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.141160  5901 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:23:38.106061  5785 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.365s	user 0.011s	sys 0.044s
W20260709 12:23:38.143482  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61874.6us
I20260709 12:23:38.144994  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:38.145215  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63079.9us
I20260709 12:23:38.146639  2383 leader_election.cc:400] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10.
I20260709 12:23:38.147047  2383 leader_election.cc:403] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:38.106595  5621 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.108528  5842 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:38.108263  5886 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:38.157840  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51751us
I20260709 12:23:38.161334  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:38.145313  5674 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:38.176913  1731 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 11 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:23:38.178220  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:38.175943  5903 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:38.179301  5903 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:23:38.195549  5887 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:23:38.142767  5313 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.205724  5667 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:38.203609  5884 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:23:38.204221  1619 maintenance_manager.cc:419] P 4002daa3304c4fb4be053abd4780c195: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 9043 bytes on disk
I20260709 12:23:38.206338  5621 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:38.214691  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.275048  1803 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:38.219504  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.275931  1805 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:38.154342  5718 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.060s
I20260709 12:23:38.218201  5765 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.229636  5687 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.181579  5797 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.294024  5830 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:23:38.293823  5876 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:38.214890  5869 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.235s	user 0.001s	sys 0.001s
I20260709 12:23:38.169363  5891 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260709 12:23:38.297677  5881 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.174518  5674 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:38.299484  5674 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:38.300153  5881 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.300874  5581 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:38.301286  5876 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.256744  5313 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.160542  5895 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.078s
I20260709 12:23:38.302860  5719 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.166965  5886 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.262579  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.303609  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.279327  5717 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.304625  1803 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:23:38.279229  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:38.281394  5888 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.169s	user 0.003s	sys 0.000s
I20260709 12:23:38.306399  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:38.307032  1726 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:38.304719  5717 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:38.280718  5898 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.107s	user 0.003s	sys 0.000s
I20260709 12:23:38.284243  5847 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.308976  5717 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:38.309424  5892 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.060s	user 0.000s	sys 0.002s
I20260709 12:23:38.309520  5836 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.298352  5765 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.284420  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2791) took 2779 ms. Trace:
I20260709 12:23:38.286957  1618 maintenance_manager.cc:643] P 4002daa3304c4fb4be053abd4780c195: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s	user 0.002s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":354,"lbm_reads_lt_1ms":4}
I20260709 12:23:38.262555  5900 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10.
I20260709 12:23:38.288552  5764 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.287961  5877 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:38.314742  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.332518  1731 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:23:38.321384  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.341948  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:38.328186  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 174626us
I20260709 12:23:38.350941  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:38.324973  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 178422us
I20260709 12:23:38.353487  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:38.352360  1799 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:38.353940  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:38.329110  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:38.356163  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.357059  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.357906  1807 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:38.357962  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.358968  1805 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:38.356267  1799 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:38.323019  1729 rpcz_store.cc:276] 0709 12:23:35.505229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.518584 (+ 13355us) service_pool.cc:224] Handling call
0709 12:23:35.531458 (+ 12874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:35.599774 (+ 68316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.284117 (+2684343us) spinlock_profiling.cc:243] Waited 67.5 ms on lock 0x7b54004362d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:38.284336 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":155218176}
I20260709 12:23:38.360773  1799 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:38.361115  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:38.361999  1875 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:38.362579  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.363574  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.342468  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.368479  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:38.369155  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.369803  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:38.317191  5694 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.693s	user 0.011s	sys 0.000s
I20260709 12:23:38.372822  1798 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
W20260709 12:23:38.319571  5855 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.002s
I20260709 12:23:38.374624  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.379396  2416 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:38.379652  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.327895  5767 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:38.384093  1801 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:38.366381  5881 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:38.387969  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:38.389135  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.391484  1876 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:38.383742  1800 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:38.318392  5765 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.319866  5842 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.394229  1880 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 7: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:38.394069  1801 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 7. (term mismatch)
I20260709 12:23:38.395035  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.382674  5313 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:38.387318  5908 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.426730  1881 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:38.429149  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.428396  1801 pending_rounds.cc:85] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 6
I20260709 12:23:38.430132  1801 pending_rounds.cc:107] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 12:23:38.430491  1801 raft_consensus.cc:2889] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:38.388181  5546 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:38.429916  1807 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:38.433214  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.437268  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.439760  1805 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:38.442282  1807 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:38.430274  1803 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:38.392599  5905 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:38.387082  5767 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.435532  5908 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:38.436385  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:38.425462  5826 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.347s	user 0.000s	sys 0.004s
I20260709 12:23:38.445009  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.452842  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 768) took 1046 ms. Trace:
I20260709 12:23:38.471696  2415 rpcz_store.cc:276] 0709 12:23:37.406176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.406330 (+   154us) service_pool.cc:224] Handling call
0709 12:23:38.452588 (+1046258us) spinlock_profiling.cc:243] Waited 93.7 ms on lock 0x7b54005230d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:38.452719 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":215418240}
W20260709 12:23:38.458019  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.448817  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.428220  5313 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:38.448207  1805 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:23:38.474336  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.465602  1801 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:23:38.477770  2413 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:38.479220  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:38.470914  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:38.470211  1807 raft_consensus.cc:2410] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:23:38.481462  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.482118  2415 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:38.482127  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.483078  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.474772  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.483840  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.484174  1805 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 7: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:38.485621  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:38.486743  1799 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:38.451193  5760 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:38.450721  5747 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:38.475365  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 308980us
I20260709 12:23:38.474670  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:38.492324  2417 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:38.490396  2413 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:38.488696  5718 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.455972  5765 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.455090  5824 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:38.494851  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.517045  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.511902  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 1992) took 2904 ms. Trace:
I20260709 12:23:38.490597  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.451427  5764 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.459239  5855 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:38.458490  5767 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.464934  5908 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.512499  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:38.519284  1880 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:38.525020  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.486824  5857 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:38.450321  5877 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.518752  1878 rpcz_store.cc:276] 0709 12:23:35.607441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.613334 (+  5893us) service_pool.cc:224] Handling call
0709 12:23:35.657588 (+ 44254us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:35.657946 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:35.657974 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:36.281097 (+623123us) log.cc:844] Serialized 26 byte log entry
0709 12:23:36.807664 (+526567us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:36.807723 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:36.807975 (+   252us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:38.424748 (+1616773us) spinlock_profiling.cc:243] Waited 5.29 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:38.511615 (+ 86867us) raft_consensus.cc:1685] finished
0709 12:23:38.511636 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.511842 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19084,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":67351,"spinlock_wait_cycles":12160768,"thread_start_us":464976,"threads_started":1,"wal-append.queue_time_us":624521}
I20260709 12:23:38.470846  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.541078  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:38.526165  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.519850  5371 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:38.531150  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.532027  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.532823  5651 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:38.568171  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.533185  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 848) took 1154 ms. Trace:
W20260709 12:23:38.534449  5731 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.516s	user 0.011s	sys 0.010s
I20260709 12:23:38.569012  2410 rpcz_store.cc:276] 0709 12:23:37.378465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.383054 (+  4589us) service_pool.cc:224] Handling call
0709 12:23:38.532923 (+1149869us) spinlock_profiling.cc:243] Waited 79.9 ms on lock 0x7b5400573ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:38.533095 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183853952}
I20260709 12:23:38.585999  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:38.488883  5781 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:38.461925  5674 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.601759  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.519992  5789 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:38.535214  5737 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.535749  5855 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.467886  5883 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.552299  2413 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:23:38.559547  5767 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.566877  5371 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:38.546736  1798 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:38.472234  1881 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:38.581822  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:38.583174  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.649241  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.584728  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.650852  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:38.541527  5724 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.579s	user 0.015s	sys 0.008s
I20260709 12:23:38.652299  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.659824  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.513862  5824 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:38.519265  5718 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.637001  5907 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.609171  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.614341  5735 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:23:38.609982  1805 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:38.610657  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.613840  1801 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:38.539053  5857 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.547099  5886 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.632345  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:38.553129  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:38.697265  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 144240us
I20260709 12:23:38.632375  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.593149  5861 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:38.586553  2410 raft_consensus.cc:2376] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10. Re-sending same reply.
I20260709 12:23:38.699776  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:38.700344  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.655938  1804 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:38.651628  1880 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:38.593984  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.660512  1803 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:38.661188  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.701547  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:38.702275  2412 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:38.696146  5857 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:38.663607  1807 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:38.705289  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.593321  5731 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:38.667214  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 85482.5us
I20260709 12:23:38.652247  5656 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:38.603745  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 79136.9us
I20260709 12:23:38.606019  5669 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:23:38.673571  1799 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:38.609788  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.719131  1724 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:38.719434  1879 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:38.720577  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.720824  5731 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.721819  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.662739  5724 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.722731  1876 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:38.724077  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.724844  1876 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:38.725260  5724 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.725932  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:38.727675  5913 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:23:38.637233  5781 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:38.729688  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:38.729353  5781 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.730474  1726 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:38.731146  5855 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.731316  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:38.731974  1726 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:38.614354  5867 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:23:38.733880  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.683362  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:38.703332  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.734815  5731 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:38.735280  1798 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:38.547487  5313 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.735327  5731 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:38.704680  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 110765us
I20260709 12:23:38.685830  5907 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.739176  1873 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:38.739396  5907 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.740329  5762 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.741671  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.742072  5674 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.742290  1873 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:23:38.743906  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1915) took 1794 ms (client timeout 2095 ms). Trace:
I20260709 12:23:38.744144  5718 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.687266  5310 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.745337  5310 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.707059  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.712343  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:38.747133  1725 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:38.747790  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2798) took 3187 ms. Trace:
I20260709 12:23:38.748476  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.704228  5861 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.749337  5861 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:38.749467  5793 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:38.750138  5793 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:38.750581  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.751062  5793 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.751562  2410 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:38.711423  5881 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:38.713721  5371 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.752321  5881 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.748136  1725 rpcz_store.cc:276] 0709 12:23:35.560235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:35.569960 (+  9725us) service_pool.cc:224] Handling call
0709 12:23:38.746609 (+3176649us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:23:38.746812 (+   203us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.747548 (+   736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.747734 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7719168}
I20260709 12:23:38.755652  1873 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:23:38.704576  1878 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:38.735893  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.592980  5908 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:23:38.756919  1876 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:38.751617  5730 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:38.744341  1806 rpcz_store.cc:269] 0709 12:23:36.949488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.949630 (+   142us) service_pool.cc:224] Handling call
0709 12:23:38.743752 (+1794122us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b54001eb9d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:38.743850 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1381674240}
I20260709 12:23:38.760833  1726 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:38.758553  1876 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 6. (term mismatch)
I20260709 12:23:38.757015  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:38.733681  5764 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:38.767778  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.769119  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:38.769515  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:38.711587  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.758499  1873 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:23:38.765998  5651 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:38.760654  5881 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:38.771524  5651 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.771974  5651 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:38.773357  1878 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:23:38.775164  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:38.776398  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:38.776914  2412 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:38.708660  5767 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:38.778024  5767 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.778517  5767 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:38.752590  5371 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:38.736933  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:38.761898  1876 pending_rounds.cc:85] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 5
I20260709 12:23:38.706284  5877 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:38.791213  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:38.747866  5310 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.722949  5824 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.764093  5734 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.764732  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:38.705226  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:38.781024  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:38.768841  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.782953  1876 pending_rounds.cc:107] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 8.6
I20260709 12:23:38.735741  5883 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.710732  5789 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:38.770753  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:38.783196  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46377.3us
W20260709 12:23:38.808535  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 103406us
I20260709 12:23:38.780362  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:38.807255  1876 raft_consensus.cc:2889] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:38.809356  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:38.814559  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45793.8us
I20260709 12:23:38.816514  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:38.783159  5829 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.824129  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:38.783776  5718 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:38.779110  5862 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:38.783978  5855 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.822562  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.814276  5674 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:38.816215  5310 raft_consensus.cc:2764] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:23:38.815690  5830 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:38.846124  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:38.847158  1725 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:38.821800  5248 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:38.850219  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:38.838975  5689 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:38.828522  5883 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:38.827152  5718 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.855517  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:38.829061  5862 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:38.860800  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.863570  5862 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:38.864173  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:38.865401  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:38.865983  5717 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:38.866652  5717 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:38.868281  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1879) took 1567 ms. Trace:
I20260709 12:23:38.868769  1730 rpcz_store.cc:276] 0709 12:23:37.301285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.301510 (+   225us) service_pool.cc:224] Handling call
0709 12:23:37.301839 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:37.336377 (+ 34538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.336424 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:37.336482 (+    58us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:37.336535 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.867982 (+1531447us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b54000d0cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:38.868038 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.868218 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78783872}
I20260709 12:23:38.867815  5717 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.854295  5674 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:38.875022  5908 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:38.875787  5908 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.876302  5908 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:38.857451  5817 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:38.878110  5817 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.881762  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:38.885586  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.858668  5883 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.859151  5737 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.853261  5689 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.857594  5718 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:38.906723  5764 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.910467  1876 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: [10.10-10.10]
W20260709 12:23:38.912930  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 103636us
I20260709 12:23:38.914276  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:38.916594  5764 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:38.917155  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:38.918185  5764 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:38.918660  5764 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.919553  5764 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:38.921276  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 485987us
I20260709 12:23:38.922739  5908 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:38.928853  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.932606  5767 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.834775  5904 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.053s
I20260709 12:23:38.860080  5248 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:38.895077  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:38.893380  2412 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:38.886137  5724 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.861864  5310 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.835414  5731 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.899241  5883 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:38.902248  5920 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.079s	user 0.001s	sys 0.001s
W20260709 12:23:38.943089  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48084.2us
I20260709 12:23:38.897744  2417 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:38.957386  2410 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:23:38.960412  5737 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:38.958216  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:38.966235  1877 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:38.967803  5817 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.968102  5908 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.969381  5248 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.975096  5669 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:23:38.975309  5881 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.978215  5881 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:38.952739  5907 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:38.955366  5656 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.028s
I20260709 12:23:38.947870  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:38.943957  5891 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.939522  5857 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:38.894013  5674 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.967926  5731 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:38.951349  1725 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:38.902120  5689 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:38.986660  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:38.980932  5883 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.009562  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2001) took 1192 ms. Trace:
I20260709 12:23:38.953776  5764 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.902464  5734 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:39.011209  5734 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.010007  1874 rpcz_store.cc:276] 0709 12:23:37.816978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.817125 (+   147us) service_pool.cc:224] Handling call
0709 12:23:37.817496 (+   371us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:37.817892 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.817923 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:37.818492 (+   569us) log.cc:844] Serialized 52 byte log entry
0709 12:23:38.746100 (+927608us) spinlock_profiling.cc:243] Waited 893 us on lock 0x7b18002c8c00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:23:38.900073 (+153973us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:38.900120 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.900215 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:39.009252 (+109037us) raft_consensus.cc:1685] finished
0709 12:23:39.009269 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.009499 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":829,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":14449,"spinlock_wait_cycles":2052992,"thread_start_us":153759,"threads_started":1,"wal-append.queue_time_us":1163793}
I20260709 12:23:39.011744  5734 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:39.012697  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.013682  1874 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:38.993425  5737 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:39.015319  5737 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.015872  5737 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:39.016747  5883 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:38.985819  5499 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.019747  5499 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.020751  5764 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.959008  5310 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.023151  5718 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.023699  5310 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:39.025736  5718 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.985402  5891 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:38.986718  5719 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:38.993245  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:39.014742  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.037967  1874 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:39.039050  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:39.041525  1874 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:39.045187  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58590us
I20260709 12:23:39.047295  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:39.030676  5728 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:23:39.008644  5767 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:38.987708  5857 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:39.070147  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 299428us
W20260709 12:23:39.071018  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 238673us
I20260709 12:23:39.058020  5546 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:39.044762  5747 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.058698  5719 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:39.084838  1874 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:39.045879  5881 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:39.042208  5764 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:39.113097  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:38.981775  5861 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.117427  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.120013  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.114804  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.114550  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.121901  1874 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:39.123179  5907 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:39.052529  5651 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.101094  5546 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.123624  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:39.126317  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:39.131309  1881 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:39.129424  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.133656  1772 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:39.135291  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.135664  1874 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:39.114542  5881 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.140810  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:39.140880  1881 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:39.142831  1881 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:39.068995  5836 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 12:23:39.115700  5817 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:39.147713  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.149163  1874 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:39.150835  1874 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:39.173483  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.152033  1801 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 12:23:39.177264  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.178476  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.126991  5728 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:39.110504  5734 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.173892  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.113849  5764 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.152813  1876 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:23:39.131820  5546 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:39.114939  5747 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.144738  5767 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:39.146363  5881 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:39.186966  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.180497  5933 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:39.201781  5734 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:39.204746  1874 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:39.176496  5730 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.212880  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.145970  5877 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:39.190258  5847 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:39.173097  5817 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.218214  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.246398  1881 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:39.248940  1874 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:39.196956  5834 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
W20260709 12:23:39.251685  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 115544us
I20260709 12:23:39.250870  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:39.219964  5781 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.258737  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.255424  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.221822  5847 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:39.216846  5933 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:39.266815  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:39.273061  5931 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:23:39.273736  5764 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:39.274025  5921 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
W20260709 12:23:39.274757  5924 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:23:39.297197  5747 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:39.297622  5861 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:39.298008  5925 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:23:39.298394  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 503) took 4883 ms (client timeout 1859 ms). Trace:
W20260709 12:23:39.262225  5927 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:23:39.284655  1801 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:39.289811  5817 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:39.300493  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.311424  1807 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.300092  5862 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.301996  5764 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:39.277091  1874 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:39.307890  1877 rpcz_store.cc:269] 0709 12:23:34.422505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.448327 (+ 25822us) service_pool.cc:224] Handling call
0709 12:23:39.298174 (+4849847us) spinlock_profiling.cc:243] Waited 4.49 s on lock 0x7b5400073d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:39.298284 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10332196224}
I20260709 12:23:39.309002  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.320900  1801 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:23:39.329232  1732 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1878) took 2384 ms (client timeout 2077 ms). Trace:
W20260709 12:23:39.329823  1732 rpcz_store.cc:269] 0709 12:23:36.944633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.944757 (+   124us) service_pool.cc:224] Handling call
0709 12:23:39.328911 (+2384154us) spinlock_profiling.cc:243] Waited 519 ms on lock 0x7b54002b75d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:39.329121 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1194165632}
I20260709 12:23:39.264014  5781 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.334661  5847 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.336076  5781 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.337317  5694 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:39.337743  5694 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:39.307379  5731 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:23:39.335501  5911 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.670s	user 0.001s	sys 0.001s
I20260709 12:23:39.342465  2411 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:39.345700  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:39.347680  1877 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.344391  1874 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:39.351204  2415 pending_rounds.cc:85] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 6
I20260709 12:23:39.351935  2415 pending_rounds.cc:107] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:23:39.352262  2415 raft_consensus.cc:2889] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:39.353876  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:39.351048  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.355733  1801 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:39.345276  5847 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:39.361316  5651 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.365823  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.338307  5881 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:23:39.384837  5927 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:23:39.411857  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.346664  5731 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:39.403851  5737 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.407869  1877 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:39.426206  1877 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:39.410054  5937 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:23:39.287927  5877 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.419003  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:39.419530  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:39.409097  1807 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:39.432869  2413 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:39.434854  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:39.438010  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:39.441018  2411 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:39.441174  1877 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:23:39.429939  1878 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:23:39.338754  5694 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.444342  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:39.445369  2413 raft_consensus.cc:2410] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 11 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 9.
I20260709 12:23:39.426317  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.449928  5694 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.439356  5817 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:39.453963  5817 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:39.454427  5817 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:39.458355  2382 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.948s (SENT) [suppressed 46 similar messages]
W20260709 12:23:39.459548  5915 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.758s	user 0.004s	sys 0.000s
I20260709 12:23:39.461575  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2818) took 1587 ms. Trace:
I20260709 12:23:39.462162  1731 rpcz_store.cc:276] 0709 12:23:37.874164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.350318 (+476154us) service_pool.cc:224] Handling call
0709 12:23:38.350640 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:38.351053 (+   413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:38.351078 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:38.351504 (+   426us) log.cc:844] Serialized 26 byte log entry
0709 12:23:38.537236 (+185732us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:23:38.537288 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.537376 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:39.319876 (+782500us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:39.461137 (+141261us) raft_consensus.cc:1685] finished
0709 12:23:39.461167 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.461483 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6574,"raft.queue_time_us":101,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":12874,"spinlock_wait_cycles":8369024,"thread_start_us":185245,"threads_started":1,"wal-append.queue_time_us":295890}
I20260709 12:23:39.428303  5651 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:23:39.470270  5881 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:39.339102  5933 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.475378  5731 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:39.481971  1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2800) took 2092 ms (client timeout 2477 ms). Trace:
W20260709 12:23:39.482527  1728 rpcz_store.cc:269] 0709 12:23:37.389582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.389702 (+   120us) service_pool.cc:224] Handling call
0709 12:23:39.481733 (+2092031us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b54002d19d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:39.481890 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30845312}
W20260709 12:23:39.493077  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.525852  2411 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:39.529525  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:39.516966  5731 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:39.536932  5689 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.512s	user 0.010s	sys 0.006s
I20260709 12:23:39.537452  5689 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.539250  5702 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.539896  5702 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.542034  5702 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.545532  5546 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.546908  5784 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:39.547555  5784 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:39.548022  5689 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.548503  5784 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.547151  2417 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:39.542510  5730 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:39.563776  5546 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.567728  5737 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:39.575098  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.575353  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2004) took 1730 ms (client timeout 2108 ms). Trace:
W20260709 12:23:39.575743  1872 rpcz_store.cc:269] 0709 12:23:37.844661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.849592 (+  4931us) service_pool.cc:224] Handling call
0709 12:23:39.575102 (+1725510us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:39.575279 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413794688}
I20260709 12:23:39.578227  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.579342  1875 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:39.585253  1807 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:39.586305  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.587787  1807 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:39.584321  5689 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.588996  5689 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.590531  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.590832  5862 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:39.591614  5862 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:39.594095  5900 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.594298  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:39.594826  5900 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:39.596035  5827 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 2.466s	user 0.005s	sys 0.000s
I20260709 12:23:39.596122  1728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:39.596975  1728 raft_consensus.cc:606] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.597435  5900 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.599255  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.599382  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.600225  1728 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:23:39.599417  5736 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.609946  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.612542  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.625267  1732 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
W20260709 12:23:39.617683  1771 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.072s (SENT)
I20260709 12:23:39.626995  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:39.628271  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.618590  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.630340  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.633046  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.636420  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.638705  1731 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: [8.8-9.9]
I20260709 12:23:39.619908  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.641677  1799 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:39.620788  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:39.643986  2411 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:39.651547  1697 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:39.616495  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:39.653503  1728 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:39.623729  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.664626  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.665565  1800 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:39.667598  1725 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:39.613986  5699 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.617291  5842 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.668985  1732 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 11 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:39.610321  5731 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260709 12:23:39.674837  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:39.675297  5731 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:39.675719  5731 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:39.669819  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:39.676770  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.677296  5731 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.678352  5789 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:39.677340  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.678561  5731 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:39.680408  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:39.682308  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:39.683182  5248 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:39.683641  5248 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:39.684062  5248 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.685539  5785 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:39.662380  5903 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:39.670265  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:39.622617  5766 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.693490  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:39.694360  1723 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:39.686192  5785 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.671557  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.696602  1725 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 4.
I20260709 12:23:39.619481  5689 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:39.672760  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.673002  5310 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:39.673630  5797 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:39.664330  5718 raft_consensus.cc:2764] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:23:39.673897  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.610471  5819 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:23:39.662336  5857 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:23:39.689353  1729 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:23:39.691290  5903 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.691643  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.692698  5926 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.652760  5557 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:39.612182  5825 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.060s
I20260709 12:23:39.693012  5891 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.618209  5817 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.695343  5736 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.718317  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.697220  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:39.676520  5383 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.698971  1730 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:39.701534  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:39.720333  1725 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:39.720414  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.699007  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.653695  5760 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 12:23:39.723388  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.723593  1801 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
W20260709 12:23:39.697626  5537 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.515s	user 0.009s	sys 0.020s
I20260709 12:23:39.687887  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.678942  5789 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:39.709018  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.713132  5928 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:39.716662  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:39.699432  5785 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:39.724045  1730 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:39.708967  5903 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:39.697279  5766 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.725271  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.735920  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.736709  1730 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:39.726863  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:39.729411  1731 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:39.701419  5310 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.732986  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.742308  1725 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:39.722714  1799 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:39.738014  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.726944  5557 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.743337  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:39.712656  5926 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.751463  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:39.734193  5537 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.561s	user 0.010s	sys 0.021s
I20260709 12:23:39.753878  5883 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:39.755218  5487 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:39.730993  5383 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 12:23:39.723238  5736 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.763949  5736 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:39.776296  5766 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.749955  5789 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:39.779224  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1803) took 1834 ms (client timeout 2226 ms). Trace:
I20260709 12:23:39.779387  5789 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:39.779757  1802 rpcz_store.cc:269] 0709 12:23:37.945009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.945458 (+   449us) service_pool.cc:224] Handling call
0709 12:23:39.778968 (+1833510us) spinlock_profiling.cc:243] Waited 94.1 ms on lock 0x7b54003b83d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:39.779144 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":216328960}
I20260709 12:23:39.781208  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:39.745810  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 57964.8us
I20260709 12:23:39.746886  5891 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:23:39.787343  5939 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.492s	user 0.000s	sys 0.004s
I20260709 12:23:39.748746  1727 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:39.749068  5310 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.744204  5781 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.749765  5760 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:39.746889  5861 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:39.792327  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.770337  5557 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:39.793373  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.790710  5891 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:39.794924  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.804970  1729 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:39.798902  5781 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.800266  5926 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:39.800491  5861 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:39.807390  1724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2810) took 1935 ms (client timeout 1656 ms). Trace:
I20260709 12:23:39.807547  5861 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:39.807971  1724 rpcz_store.cc:269] 0709 12:23:37.872164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.905187 (+ 33023us) service_pool.cc:224] Handling call
0709 12:23:39.807142 (+1901955us) spinlock_profiling.cc:243] Waited 796 ms on lock 0x7b54002a9c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:39.807304 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1831017728}
W20260709 12:23:39.801967  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.811008  5891 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.812435  5797 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.812711  5909 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:39.813494  5909 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:39.802294  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:39.814849  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.815213  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.816197  1730 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:39.817095  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.817647  1730 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:39.802726  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.819260  1799 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:39.819674  1728 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:39.815217  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.818624  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.753181  5933 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:39.824177  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:39.825440  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.825717  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:39.826054  1799 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
W20260709 12:23:39.824476  2414 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 854) took 2331 ms (client timeout 2002 ms). Trace:
I20260709 12:23:39.828363  2419 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:39.828542  5740 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:39.829336  5834 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:39.826632  2414 rpcz_store.cc:269] 0709 12:23:37.492016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.492162 (+   146us) service_pool.cc:224] Handling call
0709 12:23:39.823306 (+2331144us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54005b0254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:39.823480 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":621971200}
I20260709 12:23:39.829823  1729 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:39.830127  5819 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:39.831683  5834 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.832459  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:39.833065  5909 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.834050  5696 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:39.834234  5909 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:39.834491  5696 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:39.835059  5696 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:39.832147  5819 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:39.836218  5819 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:39.831192  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.841558  5310 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:39.843644  5310 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:39.844060  5310 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:39.854496  1799 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:39.858332  5819 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.854249  5933 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 12:23:39.859935  5933 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:39.860319  5933 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:39.860847  5819 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.867441  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.870246  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.853754  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:39.871114  1729 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:39.871292  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:39.851066  5736 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:23:39.872370  5876 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:39.872854  5876 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:39.873230  5876 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:39.877300  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:39.878860  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.879359  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:39.861198  5950 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.190s	user 0.003s	sys 0.000s
W20260709 12:23:39.881662  1770 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.656s (SENT)
I20260709 12:23:39.886572  1728 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:39.888756  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.889271  1729 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:39.890295  5760 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:39.893018  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2677) took 1471 ms. Trace:
I20260709 12:23:39.893512  1873 rpcz_store.cc:276] 0709 12:23:38.421676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.771293 (+349617us) service_pool.cc:224] Handling call
0709 12:23:38.771610 (+   317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.771833 (+   223us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:38.772161 (+   328us) log.cc:844] Serialized 14 byte log entry
0709 12:23:39.102092 (+329931us) log.cc:844] Serialized 14 byte log entry
0709 12:23:39.778034 (+675942us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b18001da940. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:39.892339 (+114305us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b18001da940. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:23:39.892547 (+   208us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:39.892612 (+    65us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:39.892676 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.892755 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.892943 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8986,"prepare.queue_time_us":18439,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":113,"raft.queue_time_us":9686,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":5536,"spinlock_wait_cycles":47098496,"thread_start_us":328898,"threads_started":1,"wal-append.queue_time_us":404809}
W20260709 12:23:39.896605  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 698646us
I20260709 12:23:39.878640  5487 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.882952  5819 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:39.899662  5734 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:39.900691  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:39.909858  1877 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:39.901791  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.910984  1873 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:23:39.903731  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:39.912580  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:39.912712  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:39.913517  1729 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:23:39.913435  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.896144  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.914049  1872 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:39.917485  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.918118  1878 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:23:39.916721  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.922058  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:39.922060  1873 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:39.924119  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.909657  5487 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.927774  5785 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.928264  5903 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:39.929649  1723 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:39.930316  5785 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:39.888664  5933 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:39.928694  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.954668  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:39.945479  2382 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:39.930503  5903 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:39.956988  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 864) took 2326 ms. Trace:
W20260709 12:23:39.958016  2419 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 757) took 2886 ms (client timeout 1954 ms). Trace:
W20260709 12:23:40.036267  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 114316us
I20260709 12:23:39.937849  1846 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:40.035653  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:39.943578   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:40.017529  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:40.041800  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.993257  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:40.053189  1799 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:39.968624  2411 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:40.051178  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:40.055800  1872 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:39.957002  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:40.056938  1878 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:39.917279  5819 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:40.018220  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72843.9us
I20260709 12:23:40.017490  1877 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:39.925756  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:39.917218  5717 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:40.058926  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.103262  5902 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:40.108121  5902 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.108659  5902 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:40.059377  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 121616us
I20260709 12:23:40.103478  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:39.919087  5674 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:40.111368  5674 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.111982  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:40.112705  1881 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.040921  5377 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:40.096858  5537 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:40.059754  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 66586.3us
W20260709 12:23:40.114401  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.104667  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:40.132517  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:40.134035  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:40.056128  5785 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.079s
I20260709 12:23:40.135535  5909 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:40.135833  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:40.138068  1872 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:40.138144  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:39.980326  5728 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:40.113950  5674 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:39.973520  5834 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:40.119445  5910 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:23:40.120707  1845 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.226s (SENT)
I20260709 12:23:40.120339  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:40.117548  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:40.201615  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:40.126056  2416 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 863) took 2502 ms (client timeout 2396 ms). Trace:
I20260709 12:23:40.176966  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:40.130800  5537 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.204907  1798 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:40.205267  5537 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:40.206369  1872 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
W20260709 12:23:40.116703  1844 connection.cc:256] RPC call timeout handler was delayed by -1.026048861000163s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260709 12:23:40.203912  1807 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.224874  1873 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:40.050104  2419 rpcz_store.cc:269] 0709 12:23:37.072076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.072220 (+   144us) service_pool.cc:224] Handling call
0709 12:23:39.957802 (+2885582us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b54005f05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:39.957937 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6106414336}
I20260709 12:23:40.057973  1729 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:40.181248  1771 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.266s (SENT)
I20260709 12:23:40.179113  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:40.227787  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:40.227950  2417 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.228091  1881 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:40.188421  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:40.248188  2412 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:40.208007  5867 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:40.142076  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2660) took 1778 ms. Trace:
I20260709 12:23:40.048632  2418 rpcz_store.cc:276] 0709 12:23:37.630185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.668984 (+ 38799us) service_pool.cc:224] Handling call
0709 12:23:39.823585 (+2154601us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:39.827127 (+  3542us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:39.827159 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:39.827214 (+    55us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:39.827249 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.827495 (+   246us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.956765 (+129270us) spinlock_profiling.cc:243] Waited 2.15 s on lock 0x7b54005b0250. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:39.956914 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3918,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"spinlock_wait_cycles":4954091392}
I20260709 12:23:40.187865  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:40.257095  2415 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:40.257350  2418 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:40.179486  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:40.258252  2414 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:40.174099  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:40.259068  1803 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:40.206980  5728 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:40.183111  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 514) took 1748 ms. Trace:
I20260709 12:23:39.970043  5891 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:39.921315  5736 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:40.183827  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 512) took 1616 ms (client timeout 1948 ms). Trace:
I20260709 12:23:40.268492  1876 rpcz_store.cc:276] 0709 12:23:38.434632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.911990 (+477358us) service_pool.cc:224] Handling call
0709 12:23:38.912498 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:39.195633 (+283135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.182757 (+987124us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b5400222254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:40.183037 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":551940352}
W20260709 12:23:40.269181  1804 rpcz_store.cc:269] 0709 12:23:38.545674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.545895 (+   221us) service_pool.cc:224] Handling call
0709 12:23:40.162083 (+1616188us) spinlock_profiling.cc:243] Waited 53.5 ms on lock 0x7b54003b7ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:40.162231 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":122964352}
I20260709 12:23:40.180220  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:40.136543  5842 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.074s
I20260709 12:23:40.160671  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:40.115083  5933 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:40.110615  5941 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.812s	user 0.000s	sys 0.002s
I20260709 12:23:40.136021  5909 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:40.175047  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:40.239802  5843 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:40.173200  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:40.208037  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 90573.1us
I20260709 12:23:40.239517  5760 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:40.204599  2416 rpcz_store.cc:269] 0709 12:23:37.623499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.647725 (+ 24226us) service_pool.cc:224] Handling call
0709 12:23:40.125812 (+2478087us) spinlock_profiling.cc:243] Waited 718 ms on lock 0x7b54004560d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:40.125973 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1651480320}
I20260709 12:23:40.159921  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:40.137403  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:40.195704  5817 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:40.198444  5499 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:39.933241  5696 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.144600  5313 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:40.167709  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:40.125126  5781 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:40.211465  1802 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:40.216792  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:39.938134  5907 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:40.335407  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:40.171852  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:40.336153  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 215454us
I20260709 12:23:40.180195  2410 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:23:40.255162  5834 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 12:23:40.236008  2411 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:23:39.944132  5825 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:40.340852  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:40.341867  1802 raft_consensus.cc:2410] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260709 12:23:40.236116  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.344412  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:40.224372  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:40.186302  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:40.346305  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 160048us
W20260709 12:23:40.237684  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 117459us
W20260709 12:23:40.237778  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 56600.5us
I20260709 12:23:39.936857  5824 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:40.040753  5928 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:39.917528  5881 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:39.976174  5581 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:23:40.251607  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.255554  1875 rpcz_store.cc:276] 0709 12:23:38.363435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.368641 (+  5206us) service_pool.cc:224] Handling call
0709 12:23:39.578990 (+1210349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:39.579818 (+   828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.141810 (+561992us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400223150. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:40.142010 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2782789504}
I20260709 12:23:40.259281  1878 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:40.259729  2417 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:39.928328  5487 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:40.263043  1807 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:39.984490  5876 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.280125  1725 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:40.152628  5954 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.484s	user 0.002s	sys 0.000s
I20260709 12:23:40.358884  5875 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:40.363488  5875 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.363965  5875 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:40.363924  5881 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.364938  5876 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:40.366711  5881 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:40.368433  5581 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:40.340202  5499 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:40.369787  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1828) took 1230 ms. Trace:
I20260709 12:23:40.280750  1798 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:39.932502  5248 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.372591  5248 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:40.238549  5674 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:40.284144  1727 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:23:40.373854  5957 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:23:40.151599  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:40.374027  5674 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.375994  1800 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:40.376271  5696 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:40.376704  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 515) took 1236 ms. Trace:
I20260709 12:23:40.239738  5878 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:40.377035  5764 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:40.377449  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:40.378243  5764 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:40.378688  5764 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:39.957841  5731 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.306s
I20260709 12:23:40.379630  5696 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:40.286484  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:40.287652  1876 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:23:40.380347  5696 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:40.380820  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94407.5us
I20260709 12:23:40.381397  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1820) took 1244 ms. Trace:
I20260709 12:23:40.381614  5731 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:40.381896  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:40.382217  5731 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:40.382342  5862 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:40.382969  5928 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:40.383045  5862 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:40.383574  5928 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:40.384111  5862 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.384574  5928 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.161580  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:40.383430  5731 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.387223  1879 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:40.387823  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 508) took 1981 ms (client timeout 2241 ms). Trace:
W20260709 12:23:40.388229  1879 rpcz_store.cc:269] 0709 12:23:38.406418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.721513 (+315095us) service_pool.cc:224] Handling call
0709 12:23:40.161441 (+1439928us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:23:40.387752 (+226311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2704640}
I20260709 12:23:40.317267  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:40.389549  1723 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:40.359760  5825 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:40.319638  5310 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.391721  5825 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.360385  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:40.393221  1802 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:40.148244  5819 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.357703  5736 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:40.395862  5736 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.396260  5736 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:40.320063  1801 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:40.233781  5717 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.377070  1800 rpcz_store.cc:276] 0709 12:23:39.140187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.375547 (+1235360us) service_pool.cc:224] Handling call
0709 12:23:40.375824 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.376468 (+   644us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.376653 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:40.318485  1877 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:40.409977  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 516) took 1269 ms. Trace:
I20260709 12:23:40.340020  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:40.278012  2414 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:40.415161  1732 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
W20260709 12:23:40.319702  5797 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.507s	user 0.010s	sys 0.000s
I20260709 12:23:40.354203  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:40.348168  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:40.356165  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:40.359102  5781 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:23:40.369488  5581 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.285949  2419 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 12:23:40.288794  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 151483us
I20260709 12:23:40.407344  5881 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.420158  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:40.421077  1879 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:40.420909  5881 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.407377  5867 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.423102  5581 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:40.423327  5867 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:40.417735  5789 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.638s	user 0.005s	sys 0.007s
I20260709 12:23:40.425545  5867 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:40.420202  5797 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:40.427487  1730 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:40.357616  5793 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:23:40.397819  1875 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:23:40.393523  1728 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:40.374698  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:40.431214  1731 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:40.432502  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:40.350998  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:40.410195  1872 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:40.410823  1878 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:23:40.381855  1806 rpcz_store.cc:276] 0709 12:23:39.136702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.136820 (+   118us) service_pool.cc:224] Handling call
0709 12:23:39.174723 (+ 37903us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:39.191390 (+ 16667us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:39.191428 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:39.197069 (+  5641us) log.cc:844] Serialized 52 byte log entry
0709 12:23:39.257672 (+ 60603us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:39.415470 (+157798us) log.cc:844] Serialized 14 byte log entry
0709 12:23:39.428583 (+ 13113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.428665 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:40.109916 (+681251us) spinlock_profiling.cc:243] Waited 8.51 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:40.381059 (+271143us) raft_consensus.cc:1685] finished
0709 12:23:40.381170 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.381337 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16123,"prepare.queue_time_us":174621,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":32945,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":13327,"spinlock_wait_cycles":19563264,"thread_start_us":183877,"threads_started":2,"wal-append.queue_time_us":55599}
I20260709 12:23:40.430671  1727 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:40.436920  1725 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:40.437237  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:40.437757  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:40.233341  1799 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
W20260709 12:23:40.353991  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 129726us
I20260709 12:23:40.370358  1805 rpcz_store.cc:276] 0709 12:23:39.139256 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.620484 (+481228us) service_pool.cc:224] Handling call
0709 12:23:39.620907 (+   423us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:39.621422 (+   515us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:39.621833 (+   411us) log.cc:844] Serialized 14 byte log entry
0709 12:23:39.656454 (+ 34621us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:39.657970 (+  1516us) log.cc:844] Serialized 52 byte log entry
0709 12:23:39.750484 (+ 92514us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b18002ff2c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16ab98ed70 00007f16ab982293 00007f16ab977cea 00007f16ab9780d1 00007f16abd93a0f 00007f16abd4f621 00007f16abdd892b
0709 12:23:39.750768 (+   284us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:39.751255 (+   487us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.751362 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:40.152003 (+400641us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:40.156616 (+  4613us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b18002ff2c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f
0709 12:23:40.157961 (+  1345us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b48001c0158. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:23:40.369536 (+211575us) raft_consensus.cc:1685] finished
0709 12:23:40.369564 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.369719 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4720,"prepare.queue_time_us":233,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":2614,"raft.run_cpu_time_us":4692,"raft.run_wall_time_us":512809,"spinlock_wait_cycles":131838208,"thread_start_us":33594,"threads_started":1,"wal-append.queue_time_us":33482}
I20260709 12:23:40.354640  5728 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.436076  1801 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:40.438016  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:40.438547  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 82496us
W20260709 12:23:40.411391  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.443647  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:40.334415  1729 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:40.444986  1879 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
W20260709 12:23:40.447830  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.445914  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 518) took 1304 ms. Trace:
I20260709 12:23:40.455006  1801 rpcz_store.cc:276] 0709 12:23:39.141412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.397550 (+1256138us) service_pool.cc:224] Handling call
0709 12:23:40.435848 (+ 38298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.445634 (+  9786us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.445855 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:40.449000  1729 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:40.360635  1902 diagnostics_log.cc:230] Unable to collect stacks to diagnostics log: Incomplete: not collecting stack trace since debugger or strace is attached
W20260709 12:23:40.457865  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.463514  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.458109  1729 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:23:40.467778  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:40.447073  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 99000.5us
I20260709 12:23:40.450132  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:40.470571  1725 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.469208  1729 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:40.472254  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:40.473275  1729 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:40.475262  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:40.475849  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:40.475899  5901 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:40.493703  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:40.479065  5728 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.480793  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:40.480227  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 129295us
I20260709 12:23:40.485275  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:40.477589  5761 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:40.498958  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:40.569262  2419 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:23:40.571792  1902 diagnostics_log.cc:230] Unable to collect stacks to diagnostics log: Incomplete: not collecting stack trace since debugger or strace is attached
I20260709 12:23:40.509608  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:40.544191  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.381428  5696 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:40.575327  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 65827.2us
I20260709 12:23:40.417322  5842 raft_consensus.cc:2764] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:23:40.436208  5907 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:40.469436  5876 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:40.459863  5857 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.620833  1806 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:40.507078  5908 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:40.482218  5797 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:40.499881  5310 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:40.628849  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:40.574816  5669 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.407424  5909 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:40.606675  5377 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:40.634513  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:40.679507  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:40.579010  5867 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260709 12:23:40.669659  5910 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:40.669809  5760 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:40.569093  5781 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 12:23:40.579324  5901 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:40.621001  5766 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.671509  5487 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:40.665514  5383 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.012s	sys 0.034s
I20260709 12:23:40.687527  5760 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:40.616063  5735 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:40.379664  5764 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.631492  5908 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.673802  5674 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:40.687505  5910 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:40.633164  5902 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.524s	user 0.014s	sys 0.000s
I20260709 12:23:40.630084  5890 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
W20260709 12:23:40.712937  5537 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.507s	user 0.010s	sys 0.000s
I20260709 12:23:40.715273  5901 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.673524  5669 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.628275  5857 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.706712  5766 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.720279  5735 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.751449  5764 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:40.727833  5908 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:40.730809  5902 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.776784  5910 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:40.777730  5861 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.039s	sys 0.021s
I20260709 12:23:40.813901  5978 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:40.822153  5978 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:40.842063  5978 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.236100   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:23:40.877187  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.878491  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.880828  1626 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 251) took 35530 ms (client timeout 15000 ms). Trace:
W20260709 12:23:40.883590  1771 leader_election.cc:336] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.821s (SENT)
I20260709 12:23:40.877741   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:23:40.885983  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 413586us
W20260709 12:23:40.951323  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.887979  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.889232  1798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.005s	sys 0.005s
W20260709 12:23:40.959473  1798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.006s	sys 0.006s
I20260709 12:23:40.891479  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1934) took 1276 ms. Trace:
I20260709 12:23:40.892704  1732 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:40.684405  1878 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:40.964413  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:40.962569  1800 rpcz_store.cc:276] 0709 12:23:39.615289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.409370 (+794081us) service_pool.cc:224] Handling call
0709 12:23:40.445529 (+ 36159us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.445777 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:40.479094 (+ 33317us) log.cc:844] Serialized 14 byte log entry
0709 12:23:40.890105 (+411011us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:40.890300 (+   195us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:40.890516 (+   216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.890729 (+   213us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.891136 (+   407us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4889,"prepare.queue_time_us":129268,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":69,"raft.queue_time_us":45696,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":369,"thread_start_us":409980,"threads_started":1,"wal-append.queue_time_us":347006}
I20260709 12:23:40.965325  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
W20260709 12:23:40.896143  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 310810us
I20260709 12:23:40.967350  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:40.666541  1872 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:40.898144  1876 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:40.899456  1807 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:40.971601  5674 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:40.899624  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:40.972743  5674 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:40.902715  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:40.973685  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1937) took 1354 ms. Trace:
W20260709 12:23:40.903048  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 480745us
I20260709 12:23:40.974228  1800 rpcz_store.cc:276] 0709 12:23:39.616177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.966023 (+1349846us) service_pool.cc:224] Handling call
0709 12:23:40.969474 (+  3451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.970070 (+   596us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.970110 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:40.970204 (+    94us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:40.970268 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.970364 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.970559 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:40.976991  1800 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:40.977828  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1938) took 1361 ms. Trace:
I20260709 12:23:40.904064  1879 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:40.904706  1881 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.007s	sys 0.007s
W20260709 12:23:40.905020  2413 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.014s	sys 0.025s
W20260709 12:23:40.911540  2410 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.001s	sys 0.014s
W20260709 12:23:40.980015  1881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.007s	sys 0.007s
W20260709 12:23:40.980473  2410 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.002s	sys 0.014s
W20260709 12:23:40.708688  5383 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.012s	sys 0.034s
I20260709 12:23:40.978456  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:40.983479  5377 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:40.983927  5767 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:40.984329  5377 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.985304  5767 consensus_queue.cc:1243] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 9.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:40.702732  5487 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:40.985309  5383 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:40.986990  5383 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:40.987527  5487 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:40.988454  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:40.918694  5669 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:40.988716  5383 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.718562  5717 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:40.990341  5377 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:40.991163  5377 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:40.991238  5383 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:40.992432  5377 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:40.978246  1800 rpcz_store.cc:276] 0709 12:23:39.616426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.975937 (+1359511us) service_pool.cc:224] Handling call
0709 12:23:40.976624 (+   687us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.977470 (+   846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.977719 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:40.995962  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.009s	sys 0.031s
W20260709 12:23:40.932034  5824 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.574s	user 0.011s	sys 0.000s
I20260709 12:23:40.933874  5825 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:40.998085  5825 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:40.998925  1725 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.527s	user 0.014s	sys 0.009s
I20260709 12:23:40.999665  1725 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:40.999719  5825 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.001271  5902 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.003243  1731 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:41.004319  5785 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.004909  5785 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:40.940712  5766 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:40.766373  5735 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:41.010136  1806 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:41.010836  5876 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.011273  5876 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:40.902554  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:40.901085  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 401981us
W20260709 12:23:41.019035  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.788s	user 0.010s	sys 0.032s
W20260709 12:23:40.980170  2413 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.014s	sys 0.025s
I20260709 12:23:40.912405  5537 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.024974  5876 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.027520  5537 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:41.032884  1803 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.773s	user 0.009s	sys 0.012s
W20260709 12:23:41.038585  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 807867us
I20260709 12:23:41.047533  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:41.041813  1725 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:23:41.039242  5933 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.754s	user 0.003s	sys 0.015s
I20260709 12:23:41.050621  1803 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:41.053218  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1834) took 1061 ms. Trace:
I20260709 12:23:41.058180  1803 rpcz_store.cc:276] 0709 12:23:39.991651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.173835 (+182184us) service_pool.cc:224] Handling call
0709 12:23:41.053058 (+879223us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:40.881739  1626 rpcz_store.cc:269] 0709 12:23:05.350136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.350371 (+   235us) service_pool.cc:224] Handling call
0709 12:23:40.880411 (+35530040us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:33.646899 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:33.647555 (+   656us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:33.647608 (+    53us) write_op.cc:435] Acquired schema lock
0709 12:23:33.647626 (+    18us) tablet.cc:662] Decoding operations
0709 12:23:33.654142 (+  6516us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:33.654238 (+    96us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:33.654266 (+    28us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:23:33.704198 (+ 49932us) tablet.cc:701] Row locks acquired
0709 12:23:33.704265 (+    67us) write_op.cc:260] PREPARE: finished
0709 12:23:33.704523 (+   258us) write_op.cc:270] Start()
0709 12:23:33.704703 (+   180us) write_op.cc:276] Timestamp: P: 1783599813704440 usec, L: 0
0709 12:23:33.704736 (+    33us) op_driver.cc:348] REPLICATION: starting
0709 12:23:33.705755 (+  1019us) log.cc:844] Serialized 4664 byte log entry
0709 12:23:34.377914 (+672159us) op_driver.cc:464] REPLICATION: finished
0709 12:23:34.484506 (+106592us) write_op.cc:301] APPLY: starting
0709 12:23:34.484667 (+   161us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:34.486095 (+  1428us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:34.486267 (+   172us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:34.487701 (+  1434us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:34.488220 (+   519us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=22
0709 12:23:34.488294 (+    74us) write_op.cc:312] APPLY: finished
0709 12:23:34.488890 (+   596us) log.cc:844] Serialized 83 byte log entry
0709 12:23:34.489203 (+   313us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:34.489428 (+   225us) write_op.cc:454] Released schema lock
0709 12:23:34.489524 (+    96us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:34.579579 (+ 90055us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:34.579620 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4890185,"reported_tablets":44,"child_traces":[["op",{"apply.queue_time_us":106211,"apply.run_cpu_time_us":5732,"apply.run_wall_time_us":44081,"num_ops":11,"prepare.queue_time_us":265620,"prepare.run_cpu_time_us":15022,"prepare.run_wall_time_us":223897,"raft.queue_time_us":112735,"raft.run_cpu_time_us":55484,"raft.run_wall_time_us":99852,"replication_time_us":672999,"thread_start_us":322123,"threads_started":4,"wal-append.queue_time_us":184891,"wal-append.run_cpu_time_us":30412,"wal-append.run_wall_time_us":1635079}]]}
I20260709 12:23:41.126063  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:41.126670  1804 raft_consensus.cc:2410] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:41.048741  5537 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.074s
I20260709 12:23:40.996199  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1949) took 1301 ms. Trace:
W20260709 12:23:41.176393  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 127532us
I20260709 12:23:41.181917  1800 rpcz_store.cc:276] 0709 12:23:39.694592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.994454 (+1299862us) service_pool.cc:224] Handling call
0709 12:23:40.995189 (+   735us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.995707 (+   518us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.995747 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:40.995834 (+    87us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:40.995898 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.995975 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.996139 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.174309  5933 raft_consensus.cc:2764] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:23:41.203270  5499 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:41.218851  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46529us
I20260709 12:23:41.222957  5862 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 12:23:41.223999  5696 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.225142  5696 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:41.226411  5313 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.855s	user 0.001s	sys 0.021s
W20260709 12:23:41.227176  1729 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.690s	user 0.004s	sys 0.007s
I20260709 12:23:41.227391  1726 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:41.228155  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.228381  5740 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:41.227737  1729 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.740s	user 0.004s	sys 0.007s
I20260709 12:23:41.223115  5903 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:23:41.228919  5740 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.215037  5377 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:23:41.235251  5377 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:41.235532  5979 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:41.235940  5377 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:41.236465  5979 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:41.239019  1770 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.000s	sys 0.011s
W20260709 12:23:41.240031  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 251867us
I20260709 12:23:41.242646  1800 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
W20260709 12:23:41.244097  1770 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.000s	sys 0.012s
W20260709 12:23:41.249435  1801 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.606s	user 0.001s	sys 0.008s
W20260709 12:23:41.250102  1801 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.002s	sys 0.009s
I20260709 12:23:41.250296  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2038) took 1303 ms. Trace:
W20260709 12:23:41.235265  5960 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.552s	user 0.003s	sys 0.004s
W20260709 12:23:41.257207  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 207822us
I20260709 12:23:41.260577  5740 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.264376  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 957) took 1077 ms. Trace:
W20260709 12:23:41.270025  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 648142us
W20260709 12:23:41.279414  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 232150us
I20260709 12:23:41.267481  2413 rpcz_store.cc:276] 0709 12:23:40.187225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.189104 (+  1879us) service_pool.cc:224] Handling call
0709 12:23:40.189390 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.189619 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:41.023354 (+833735us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.218041 (+194687us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.241966 (+ 23925us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:41.263869 (+ 21903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.263996 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.264217 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":331,"prepare.queue_time_us":618235,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":908,"thread_start_us":1022503,"threads_started":3}
I20260709 12:23:41.267382  5964 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.290827  1725 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:41.296662  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1903) took 1178 ms. Trace:
I20260709 12:23:41.297104  1725 rpcz_store.cc:276] 0709 12:23:40.115761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.182935 (+1067174us) service_pool.cc:224] Handling call
0709 12:23:41.253563 (+ 70628us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.291495 (+ 37932us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.293723 (+  2228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.250999  1880 rpcz_store.cc:276] 0709 12:23:39.915828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.273121 (+357293us) service_pool.cc:224] Handling call
0709 12:23:40.273544 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:40.273974 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.274002 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:40.274575 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 12:23:40.695671 (+421096us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:41.051499 (+355828us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.185221 (+133722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.192166 (+  6945us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:41.205704 (+ 13538us) raft_consensus.cc:1685] finished
0709 12:23:41.205728 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.219587 (+ 13859us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9831,"prepare.queue_time_us":355666,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":73,"raft.queue_time_us":8858,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":6601,"spinlock_wait_cycles":145792,"thread_start_us":661001,"threads_started":3,"wal-append.queue_time_us":507007}
I20260709 12:23:41.302434  2382 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:41.300333  1805 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.010s	sys 0.003s
W20260709 12:23:41.309358  1723 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.850s	user 0.012s	sys 0.008s
W20260709 12:23:41.309820  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.309947  1723 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:41.310699  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1904) took 1194 ms. Trace:
I20260709 12:23:41.310366  5735 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.311064  1723 rpcz_store.cc:276] 0709 12:23:40.116058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.286898 (+170840us) service_pool.cc:224] Handling call
0709 12:23:41.310602 (+1023704us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.312894  5735 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:41.321187  5728 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.715s	user 0.009s	sys 0.014s
I20260709 12:23:41.325285  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:41.325977  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:41.329402  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.330647  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:41.331332  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46455.8us
I20260709 12:23:41.331650  1873 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:41.331840  5891 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.051s	user 0.003s	sys 0.019s
I20260709 12:23:41.332459  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:41.332805  5891 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:41.333454  5891 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:41.334394  1724 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
W20260709 12:23:41.327718  5760 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.582s	user 0.013s	sys 0.005s
I20260709 12:23:41.334688  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:41.327415  5537 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.301787  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1951) took 1508 ms. Trace:
W20260709 12:23:41.340536  5581 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.917s	user 0.005s	sys 0.016s
W20260709 12:23:41.344918  1805 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.011s	sys 0.003s
I20260709 12:23:41.303190  5979 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:41.340757  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.277081  5908 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.347640  5908 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:41.337419  1724 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 6
W20260709 12:23:41.343451  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.305404  5964 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:41.353607  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48630.3us
I20260709 12:23:41.354034  5731 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:41.353507  5537 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.354568  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:41.355293  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:41.355746  1725 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 12:23:41.348664  5736 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.952s	user 0.012s	sys 0.008s
I20260709 12:23:41.342059  1803 rpcz_store.cc:276] 0709 12:23:39.789964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.059862 (+1269898us) service_pool.cc:224] Handling call
0709 12:23:41.060336 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.062075 (+  1739us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:41.189475 (+127400us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.295868 (+106393us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.295929 (+    61us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:41.297664 (+  1735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.297755 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.298587 (+   832us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62072,"prepare.queue_time_us":130550,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":4726,"raft.queue_time_us":2612,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":8546,"thread_start_us":75090,"threads_started":1}
I20260709 12:23:41.352680  1724 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:23:41.366323  1724 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:41.368335  5377 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.369175  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:41.370144  5875 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.928s	user 0.013s	sys 0.013s
I20260709 12:23:41.370653  5875 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:41.372429  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.373212  5875 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.374900  5581 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.375511  5581 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.376859  5537 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.379865  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1935) took 1764 ms. Trace:
I20260709 12:23:41.365089  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2851) took 1118 ms. Trace:
I20260709 12:23:41.381497  1725 rpcz_store.cc:276] 0709 12:23:40.240986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.351409 (+1110423us) service_pool.cc:224] Handling call
0709 12:23:41.355229 (+  3820us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.358727 (+  3498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.359146 (+   419us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.334720  5891 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.385281  1873 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
W20260709 12:23:41.390024  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.388717  5728 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:41.391716  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2849) took 1278 ms. Trace:
I20260709 12:23:41.380540  5696 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.402155  5875 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:41.405040  5933 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:23:41.405318  5970 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
W20260709 12:23:41.404980  5962 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.131s	user 0.001s	sys 0.001s
I20260709 12:23:41.407084  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1830) took 2267 ms. Trace:
I20260709 12:23:41.407559  1798 rpcz_store.cc:276] 0709 12:23:39.139687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.370811 (+1231124us) service_pool.cc:224] Handling call
0709 12:23:40.371178 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:40.371564 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:40.371895 (+   331us) log.cc:844] Serialized 14 byte log entry
0709 12:23:40.960434 (+588539us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:40.961641 (+  1207us) log.cc:844] Serialized 26 byte log entry
0709 12:23:40.961953 (+   312us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:40.961999 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.962155 (+   156us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:41.406808 (+444653us) raft_consensus.cc:1685] finished
0709 12:23:41.406829 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.407015 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":626,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":6198,"raft.run_cpu_time_us":397,"raft.run_wall_time_us":2416,"thread_start_us":587801,"threads_started":1,"wal-append.queue_time_us":449788}
I20260709 12:23:41.410259  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1902) took 1294 ms. Trace:
I20260709 12:23:41.398353  5728 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:41.410843  5999 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:41.411394  5999 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:23:41.405472  5986 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:23:41.412357  6002 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:41.411777  5728 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.410341  1798 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
W20260709 12:23:41.411816  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.411731  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 514) took 2271 ms. Trace:
I20260709 12:23:41.416944  5875 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:41.380568  1801 rpcz_store.cc:276] 0709 12:23:39.615634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.455741 (+840107us) service_pool.cc:224] Handling call
0709 12:23:40.456099 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.456323 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:41.250583 (+794260us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.379045 (+128462us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.379181 (+   136us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:41.379364 (+   183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.379460 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.379733 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23297,"prepare.queue_time_us":214136,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":24077,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":171,"thread_start_us":761550,"threads_started":2}
I20260709 12:23:41.398690  5581 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.413728  1732 rpcz_store.cc:276] 0709 12:23:40.115490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.418855 (+303365us) service_pool.cc:224] Handling call
0709 12:23:41.410026 (+991171us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54002d19d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:41.410200 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":912991616}
I20260709 12:23:41.418124  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1908) took 1300 ms. Trace:
I20260709 12:23:41.392256  1729 rpcz_store.cc:276] 0709 12:23:40.113582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.486389 (+372807us) service_pool.cc:224] Handling call
0709 12:23:40.486876 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.487163 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:41.239117 (+751954us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.307790 (+ 68673us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.350969 (+ 43179us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:41.384070 (+ 33101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.391382 (+  7312us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.391615 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":739552,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":101,"raft.queue_time_us":5791,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":18873,"thread_start_us":793752,"threads_started":2,"wal-append.queue_time_us":146108}
I20260709 12:23:41.420569  1724 rpcz_store.cc:276] 0709 12:23:40.117319 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.332867 (+1215548us) service_pool.cc:224] Handling call
0709 12:23:41.333520 (+   653us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.414695 (+ 81175us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.418061 (+  3366us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.366076  5760 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.414695  1807 rpcz_store.cc:276] 0709 12:23:39.139956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.373171 (+1233215us) service_pool.cc:224] Handling call
0709 12:23:40.373539 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.972256 (+598717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.411395 (+439139us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b54002c0954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:41.411651 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1207682304}
I20260709 12:23:41.411791  5999 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:41.423418  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1956) took 1588 ms. Trace:
I20260709 12:23:41.425122  1798 rpcz_store.cc:276] 0709 12:23:39.831962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.409494 (+1577532us) service_pool.cc:224] Handling call
0709 12:23:41.410016 (+   522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:41.418910 (+  8894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.419932 (+  1022us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.404160  5377 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.427042  5764 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:41.429806  1803 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:23:41.431363  5793 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.419322  5933 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:41.403981  5537 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:41.432353  5909 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:41.447642  2411 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:41.448494  1725 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:41.433035  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63981.2us
I20260709 12:23:41.411921  5997 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:41.441830  1807 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:23:41.441718  5789 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:41.451494  5793 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.442829  5875 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.454030  2418 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:23:41.449331  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.452013  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1836) took 1459 ms. Trace:
I20260709 12:23:41.450436  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1955) took 1631 ms. Trace:
I20260709 12:23:41.381896  5736 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.455914  1807 rpcz_store.cc:276] 0709 12:23:39.992200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.440781 (+1448581us) service_pool.cc:224] Handling call
0709 12:23:41.441419 (+   638us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:41.451606 (+ 10187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.451897 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.450436  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2852) took 1002 ms. Trace:
I20260709 12:23:41.456151  1803 rpcz_store.cc:276] 0709 12:23:39.803465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.382108 (+1578643us) service_pool.cc:224] Handling call
0709 12:23:41.429467 (+ 47359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:41.434186 (+  4719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.434469 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.465140  1725 rpcz_store.cc:276] 0709 12:23:40.448305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.385685 (+937380us) service_pool.cc:224] Handling call
0709 12:23:41.421423 (+ 35738us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.448998 (+ 27575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.450366 (+  1368us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.475814  5964 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.477658  1724 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:23:41.479137  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:41.482009  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 129941us
I20260709 12:23:41.495402  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2854) took 1015 ms. Trace:
I20260709 12:23:41.497231  1724 rpcz_store.cc:276] 0709 12:23:40.479616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.424964 (+945348us) service_pool.cc:224] Handling call
0709 12:23:41.477163 (+ 52199us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.495109 (+ 17946us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.495341 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:41.498390  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1936) took 1882 ms. Trace:
I20260709 12:23:41.499886  5736 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.500945  2417 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:41.501269  1799 raft_consensus.cc:2368] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9.
W20260709 12:23:41.506037  5383 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.001s
I20260709 12:23:41.498942  1805 rpcz_store.cc:276] 0709 12:23:39.615910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.618861 (+1002951us) service_pool.cc:224] Handling call
0709 12:23:40.635103 (+ 16242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.635458 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:41.345626 (+710168us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.421945 (+ 76319us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.437005 (+ 15060us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:41.485768 (+ 48763us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.498142 (+ 12374us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.498311 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":35227,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":13131,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":162,"thread_start_us":727248,"threads_started":2,"wal-append.queue_time_us":98041}
I20260709 12:23:41.506973  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.508065  5760 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.509080  2416 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:41.509522  2416 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:23:41.509760  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.512674  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:41.519335  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.524091  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.525786  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.527384  1875 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:41.527895  5964 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:41.528389  5964 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:41.529433  5964 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.530895  5696 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.531852  5696 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.534153  5696 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.523898  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:41.526726  1872 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:41.549070  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:41.559008  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:41.546665  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:41.547453  5377 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.564915  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 960) took 1215 ms. Trace:
I20260709 12:23:41.568684  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:41.569588  1872 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:41.573454  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.574602  1875 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:41.576092  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.577014  1873 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:41.582208  2414 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:41.509107  5933 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.567516  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:41.589023  5760 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:41.589762  5760 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.590175  5760 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:41.591526  1873 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:41.588095  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:41.594837  5903 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.600251  6008 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.599980  5903 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.600973  6008 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.603003  5383 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:40.449275  1802 rpcz_store.cc:276] 0709 12:23:39.140422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.394425 (+1254003us) service_pool.cc:224] Handling call
0709 12:23:40.406997 (+ 12572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.409454 (+  2457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.409500 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:40.409563 (+    63us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:40.409616 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.409701 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.409896 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20801,"raft.queue_time_us":7587,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":98}
I20260709 12:23:41.606132  6008 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.595041  5377 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.611089  5881 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.619100  6010 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:41.619788  6010 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.620217  6010 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:41.620893  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:41.622383  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 125720us
I20260709 12:23:41.616051  6002 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:41.623509  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:41.625308  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:41.586540  5696 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:41.631523  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.633858  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.566844  2410 rpcz_store.cc:276] 0709 12:23:40.349305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.349532 (+   227us) service_pool.cc:224] Handling call
0709 12:23:40.349890 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.350139 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:40.981466 (+631327us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.349480 (+368014us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.498025 (+148545us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.516243 (+ 18218us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:41.543512 (+ 27269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.547294 (+  3782us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.564788 (+ 17494us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":528524,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":100,"raft.queue_time_us":101931,"raft.run_cpu_time_us":556,"raft.run_wall_time_us":29098,"thread_start_us":1025401,"threads_started":4,"wal-append.queue_time_us":299764}
I20260709 12:23:41.602444  5557 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:41.639556  1875 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:41.632463  5731 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.640211  1872 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:41.610606  5377 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.650826  1872 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:41.645540  5557 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:41.655646  5557 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:41.656823  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1891) took 1844 ms (client timeout 1933 ms). Trace:
W20260709 12:23:41.657419  1730 rpcz_store.cc:269] 0709 12:23:39.811973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.818308 (+  6335us) service_pool.cc:224] Handling call
0709 12:23:41.656371 (+1838063us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b54001bf3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:41.656585 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1393638144}
I20260709 12:23:41.658419  5557 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:41.657871  1872 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.634080  5696 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.640805  5767 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:41.650022  5731 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:41.663808  1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2845) took 1777 ms (client timeout 1912 ms). Trace:
I20260709 12:23:41.660418  5383 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:23:41.675419  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50266.2us
W20260709 12:23:41.673590  1728 rpcz_store.cc:269] 0709 12:23:39.886003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.888469 (+  2466us) service_pool.cc:224] Handling call
0709 12:23:41.663428 (+1774959us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b5400327dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:41.663629 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1158948096}
I20260709 12:23:41.676765  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:41.677099  5767 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:41.678082  5736 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:41.679792  1872 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:41.678727  5767 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.682098  1727 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:41.686993  5377 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.687677  5377 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.694772  1801 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:41.647734  5891 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:23:41.673273  5696 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:41.706072  5819 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.706930  5819 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:41.650707  5999 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.715929  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2846) took 1829 ms. Trace:
I20260709 12:23:41.716501  1731 rpcz_store.cc:276] 0709 12:23:39.886238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.432042 (+545804us) service_pool.cc:224] Handling call
0709 12:23:41.715643 (+1283601us) spinlock_profiling.cc:243] Waited 561 ms on lock 0x7b540030a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:41.715841 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1290649984}
I20260709 12:23:41.718233  5909 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.719545  5909 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.723383  5909 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.725560  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2693) took 1152 ms. Trace:
I20260709 12:23:41.727865  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2032) took 1814 ms. Trace:
I20260709 12:23:41.728377  1881 rpcz_store.cc:276] 0709 12:23:39.913781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.250014 (+336233us) service_pool.cc:224] Handling call
0709 12:23:40.250314 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.370685 (+120371us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:40.371389 (+   704us) log.cc:844] Serialized 14 byte log entry
0709 12:23:40.981121 (+609732us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:40.981335 (+   214us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:40.981473 (+   138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.727474 (+746001us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b540021b1d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:41.727525 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.727777 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":183,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":8949,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":179,"spinlock_wait_cycles":276109184,"thread_start_us":609156,"threads_started":1,"wal-append.queue_time_us":440723}
I20260709 12:23:41.733413  5785 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:41.736377  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1961) took 1323 ms (client timeout 1669 ms). Trace:
W20260709 12:23:41.736894  1806 rpcz_store.cc:269] 0709 12:23:40.412909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.435055 (+ 22146us) service_pool.cc:224] Handling call
0709 12:23:41.736087 (+1301032us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b540013a954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:41.736275 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319566592}
I20260709 12:23:41.739740  1798 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 11. Ops: []
I20260709 12:23:41.741478  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1831) took 1754 ms. Trace:
I20260709 12:23:41.741886  1798 rpcz_store.cc:276] 0709 12:23:39.987442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.435632 (+1448190us) service_pool.cc:224] Handling call
0709 12:23:41.739460 (+303828us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.740250 (+   790us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.741236 (+   986us) spinlock_profiling.cc:243] Waited 226 ms on lock 0x7b540013a950. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:41.741406 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":519789824}
I20260709 12:23:41.744716  5793 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.748270  5907 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:41.749019  5907 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.762868  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1954) took 1964 ms. Trace:
I20260709 12:23:41.763396  1800 rpcz_store.cc:276] 0709 12:23:39.798124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.183835 (+1385711us) service_pool.cc:224] Handling call
0709 12:23:41.211049 (+ 27214us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:41.244414 (+ 33365us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.762510 (+518096us) spinlock_profiling.cc:243] Waited 5.33 ms on lock 0x7b54001d4554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:41.762773 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12269312}
I20260709 12:23:41.766106  5581 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.767153  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:41.768348  5499 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:41.768967  5499 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:41.773325  5731 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.758097  2416 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.775965  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:41.776965  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:41.777313  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 1.43139e+06us
I20260709 12:23:41.777896  5834 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:41.778486  5834 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.778894  5834 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:41.779069  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.766901  5581 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.784881  5581 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.785849  5857 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.790319  5903 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.793008  5557 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:41.793805  5557 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.796371  5557 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.726073  1879 rpcz_store.cc:276] 0709 12:23:40.573059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.573202 (+   143us) service_pool.cc:224] Handling call
0709 12:23:40.573502 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.979226 (+405724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.725164 (+745938us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:41.725445 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":758418176}
I20260709 12:23:41.774801  2416 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:41.808472  1799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1962) took 1395 ms (client timeout 1744 ms). Trace:
I20260709 12:23:41.802132  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1946) took 2135 ms. Trace:
I20260709 12:23:41.805604  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:41.798049  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:23:41.817444  1799 rpcz_store.cc:269] 0709 12:23:40.413220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.442507 (+ 29287us) service_pool.cc:224] Handling call
0709 12:23:41.808213 (+1365706us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001eb9d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:41.808407 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2410259712}
W20260709 12:23:41.807152  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2831) took 3052 ms (client timeout 1718 ms). Trace:
I20260709 12:23:41.818867  1804 rpcz_store.cc:276] 0709 12:23:39.666449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.291240 (+624791us) service_pool.cc:224] Handling call
0709 12:23:41.801846 (+1510606us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b54001d4f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:41.802055 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1641728256}
W20260709 12:23:41.824743  1726 rpcz_store.cc:269] 0709 12:23:38.754635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.780053 (+ 25418us) service_pool.cc:224] Handling call
0709 12:23:41.806800 (+3026747us) spinlock_profiling.cc:243] Waited 2.42 s on lock 0x7b540044aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:41.807004 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5558817664}
I20260709 12:23:41.770095  5499 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:41.813585  6004 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.305s	user 0.003s	sys 0.000s
I20260709 12:23:41.805279  5487 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:41.834620  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 499975us
I20260709 12:23:41.805083  5891 consensus_queue.cc:1243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 11.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:41.798866  5933 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.820101  5557 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:41.845963  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.839357  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:41.807929  5907 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:41.850039  1872 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
I20260709 12:23:41.850472  1872 pending_rounds.cc:85] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 7
I20260709 12:23:41.850732  1872 pending_rounds.cc:107] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20260709 12:23:41.851058  1872 raft_consensus.cc:2889] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:41.856061  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58147.9us
I20260709 12:23:41.857868  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:41.858843  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2029) took 1924 ms. Trace:
I20260709 12:23:41.859086  5933 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.859463  1874 rpcz_store.cc:276] 0709 12:23:39.912869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.920867 (+  7998us) service_pool.cc:224] Handling call
0709 12:23:39.943869 (+ 23002us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.428206 (+484337us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:40.428554 (+   348us) log.cc:844] Serialized 14 byte log entry
0709 12:23:40.897821 (+469267us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:40.898006 (+   185us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:40.898237 (+   231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.814018 (+915781us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b54000f0854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:41.814067 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.836934 (+ 22867us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":588,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":16462,"raft.queue_time_us":22440,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":379,"spinlock_wait_cycles":1113311488,"thread_start_us":468287,"threads_started":1,"wal-append.queue_time_us":391601,"wal-append.run_cpu_time_us":1709,"wal-append.run_wall_time_us":1024588}
I20260709 12:23:41.861667  5933 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:41.858887  1879 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:41.848917  5377 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:41.881403  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 64695.9us
I20260709 12:23:41.881692  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:41.886499  5982 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.625s	user 0.000s	sys 0.003s
W20260709 12:23:41.897320  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 107088us
W20260709 12:23:41.898041  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.853997  5557 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.884680  5581 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:41.895040  6002 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:41.910854  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66250.3us
I20260709 12:23:41.875717  5767 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:41.915060  5907 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:41.917048  5907 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:41.914759  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:41.930776  2416 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:41.910660  6002 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:41.932716  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:41.933467  6014 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
W20260709 12:23:41.941143  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.918550  5907 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260709 12:23:41.938723  6002 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:41.951428  5819 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:41.938506  5875 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260709 12:23:41.935057  5581 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:41.955830  5581 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:41.957165  5875 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:41.964524  1877 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:23:41.966946  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52329.1us
I20260709 12:23:41.954558  1880 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
W20260709 12:23:41.966082  5990 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.703s	user 0.001s	sys 0.002s
I20260709 12:23:41.968899  5875 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.938364  5557 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.922744  6010 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.978637  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:41.972132  5785 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:41.987149  5816 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.972051  5999 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:41.976801  5674 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:41.976153  5902 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:41.989554  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:41.990154  5891 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:42.000617  5891 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.001034  5876 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:42.002434  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71252.6us
I20260709 12:23:42.001782  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.020769  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1894) took 2134 ms. Trace:
I20260709 12:23:42.067236  1727 rpcz_store.cc:276] 0709 12:23:39.886007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.436873 (+550866us) service_pool.cc:224] Handling call
0709 12:23:42.000053 (+1563180us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54002b66d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:42.020664 (+ 20611us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2859537152}
W20260709 12:23:42.028007  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.018015  5793 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.002736  5785 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.987905  5816 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:41.995982  5674 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:41.992619  5902 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:42.001132  5989 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.821s	user 0.004s	sys 0.000s
W20260709 12:23:42.073750  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 96244.7us
I20260709 12:23:42.076411  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.003638  5557 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.082394  5902 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:42.087638  5978 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:42.088810  1731 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:42.072606  5907 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:23:42.097381  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 118607us
I20260709 12:23:42.079051  5793 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.099810  5785 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:42.101557  5793 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:42.102486  5816 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:42.110234  5875 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:42.077694  5760 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:42.112198  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:42.113256  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:42.114305  5760 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:42.115150  1729 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
W20260709 12:23:42.117956  6012 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 12:23:42.123771  5891 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:42.123850  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.124895  1729 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:42.128208  5988 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 12:23:42.129501  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:42.112214  6011 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.209s	user 0.002s	sys 0.000s
I20260709 12:23:42.131397  5674 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:42.134239  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.137166  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:42.138343  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.141166  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.139009  1872 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:42.102679  5557 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.152931  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.154268  5557 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:42.133621  5902 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:42.155128  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.130136  1628 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 5 to 9, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 9 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:42.156904  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.157419  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1957) took 2292 ms. Trace:
I20260709 12:23:42.157915  1801 rpcz_store.cc:276] 0709 12:23:39.865503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.423152 (+1557649us) service_pool.cc:224] Handling call
0709 12:23:41.444481 (+ 21329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:41.698404 (+253923us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.157174 (+458770us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b5400066154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:42.157361 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":574943488}
I20260709 12:23:42.159211  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.102257  6002 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.139269  5767 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:42.162254  1731 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:42.164767  1726 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:42.165527  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.167542  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.146597  5875 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:42.156293  5988 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:42.176605  6017 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:23:42.177237  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:42.177376  2417 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:42.172525  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.184554  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:42.185335  2419 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:42.162212  5499 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:23:42.160584  5834 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.191160  2416 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:42.195453  2419 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
W20260709 12:23:42.202551  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69112.4us
I20260709 12:23:42.203073  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.203518  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:42.203969  1806 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:23:42.196871  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:42.197634  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.198143  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.205017  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.198679  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.205835  1806 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:23:42.206168  1807 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:42.206748  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.207260  5999 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:42.207660  5999 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.199093  1877 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:42.200855  5696 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:42.201491  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.202271  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.200958  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.212049  1798 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:42.214951  5999 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:42.211094  1800 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:42.223519  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.225215  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:42.168500  5767 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.219877  1724 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:42.162177  5964 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:23:42.236078  1880 pending_rounds.cc:85] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 7
I20260709 12:23:42.236544  1880 pending_rounds.cc:107] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:23:42.236827  1880 raft_consensus.cc:2889] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.237187  1880 pending_rounds.cc:107] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260709 12:23:42.237452  1880 raft_consensus.cc:2889] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.175264  5875 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.162912  5876 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 12:23:42.247284  1728 raft_consensus.cc:2368] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
W20260709 12:23:42.252179  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48773.5us
I20260709 12:23:42.260795  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.263640  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.238696  5767 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:42.264531  1800 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:42.267249  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.258006  1801 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:42.241624  5557 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:42.243192  6010 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:42.255842  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.265967  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.248353  5875 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:42.270289  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:42.289476  2419 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:42.272672  6002 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:42.291467  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.272912  1879 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:42.274718  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:42.249547  5979 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:42.277144  5487 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.292454  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.292821  1881 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:42.294473  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69408.2us
I20260709 12:23:42.291324  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.297183  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.304188  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.303299  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:42.310102  1877 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
W20260709 12:23:42.310547  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 162623us
I20260709 12:23:42.312623  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:42.320725  1881 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:42.321350  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:42.322016  6019 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:23:42.293282  5557 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.323184  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.309696  1876 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:42.326304  5499 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.335220  1878 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:42.339772  1725 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:23:42.327543  1628 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:42.341552  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.342936  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.341650  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.345290  1879 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:42.345417  1731 raft_consensus.cc:2410] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:23:42.344709  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.347436  5499 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.348922  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.346714  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.346441  1801 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:42.352304  1877 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:23:42.354372  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 166814us
W20260709 12:23:42.355952  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.303478  5487 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.362521  1878 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:23:42.363050  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:42.360329  5891 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 12:23:42.365001  5696 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:42.365074  5760 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:42.366204  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:42.386546  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.363808  1877 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:42.387360  2416 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:42.356618  5876 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:42.388622  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.390216  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.387326  1877 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:42.361445  1628 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 3 to 6, term changed from 5 to 6, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to <none>, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 075704438992440697732181e49ccd09 (127.0.140.65), VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66), VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67), VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 6 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:42.397154  1731 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:42.392766  1877 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:42.402472  1877 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:42.406147  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.398890  2419 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:42.407665  2416 raft_consensus.cc:2410] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260709 12:23:42.391644  5876 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.414045  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.415402  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.421257  1731 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:42.421721  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.422299  1727 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:42.421288  5819 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.002s
I20260709 12:23:42.423280  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.424368  2419 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 11 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:42.422533  2410 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:42.427650  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.429469  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.430303  1724 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:42.431761  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.433144  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.436091  5487 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:42.436928  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.437796  1725 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:42.443241  1872 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:23:42.413489  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.445508  1724 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:42.447211  1724 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:42.447504  1730 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:42.450572  1727 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:42.452076  5907 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:23:42.399358  5785 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:42.418771  1628 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 5 to 8, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 8 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:42.362215  5557 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.453328  1731 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:42.467872  1725 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:42.461180  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.463532  1726 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:42.411118  6010 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.475500  2413 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:42.495813  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:42.496543  1878 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:42.497694  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.502364  6020 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:23:42.454033  5876 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:42.502131  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.503345  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.489226  1628 catalog_manager.cc:5697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:42.459645  5785 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.526563  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:42.519186  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.523648  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.529850  1805 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:42.527156  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.525271  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:42.530411  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.528184  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.531252  2419 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 5: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:42.526072  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:42.509644  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.471997  5487 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:42.537611  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 110602us
I20260709 12:23:42.530683  2413 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:42.535938  1807 raft_consensus.cc:2368] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9.
I20260709 12:23:42.536166  2416 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 7.
I20260709 12:23:42.542363  1628 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 5 to 6, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 6 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:42.547395  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.553300  1729 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:42.555361  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.555567  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:42.556099  1801 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:42.577315  1804 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:42.578682  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:42.579789  1800 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:42.510555  6002 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.588246  1628 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 3 to 4, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:42.548852  5785 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:42.495832  6026 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.120s
I20260709 12:23:42.611984  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:42.612318  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.496758  6010 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.615936  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:42.619711  1872 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:42.619946  1628 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 4 to 6, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:42.623996  1628 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:42.555275  5487 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:42.653805  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.665756  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 167120us
I20260709 12:23:42.606829  6002 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.643391  5903 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:42.692003  1628 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 3 to 6, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 6 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:42.701238  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:42.701907  1874 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:42.669796  5964 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:42.712821  1872 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:42.714462  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:42.716794  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.692265  6032 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:42.726052  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 181480us
I20260709 12:23:42.727428  2410 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:42.727723  1628 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:42.752112  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:42.699550  5834 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:42.752913  2419 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:42.762631  1877 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:23:42.745604  6024 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 12:23:42.769517  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.775058  1872 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:42.769946  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:42.708014  5377 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:42.712946  5988 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:42.726768  6032 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:42.773268  5999 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.530s	user 0.012s	sys 0.000s
I20260709 12:23:42.790649  5377 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.792166  2410 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:42.805269  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.795990  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:42.788882  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 251803us
W20260709 12:23:42.770571  6015 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:42.795670  5816 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:42.816475  1877 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:42.825245  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.830493  2418 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260709 12:23:42.830650  1730 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:42.821027  1874 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:42.825953  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:42.839699  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 43780us
I20260709 12:23:42.776638  5891 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:42.727157  5964 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.861361  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:42.843595  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 129224us
I20260709 12:23:42.861536  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:42.860684  1628 catalog_manager.cc:5697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 6, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:23:42.871110  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:42.872210  5988 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:42.872726  5988 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.873303  5988 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:42.872536  1628 catalog_manager.cc:5697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 2 to 6, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 6 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:42.831071  2418 pending_rounds.cc:85] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 7
I20260709 12:23:42.875157  2418 pending_rounds.cc:107] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 12:23:42.875516  2418 raft_consensus.cc:2889] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.797930  5834 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:42.786772  5557 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:42.866837  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:42.796694  6050 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:23:42.887441  2419 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
W20260709 12:23:42.889195  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 123607us
I20260709 12:23:42.896077  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.917250  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.919260  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:42.805312  5999 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.885689  5834 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:23:42.929966  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 88875.6us
W20260709 12:23:42.934334  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.937055  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:42.907598  5557 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:42.898138  6044 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:42.877668  5891 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:42.943395  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.945675  2418 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:42.946943  2419 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260709 12:23:42.941752  5557 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:42.950594  5557 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.952078  5557 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:42.956897  6059 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.957535  6059 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.958623  6068 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.959157  6056 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.959257  6068 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:42.959753  6056 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:42.948092  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 43788.2us
I20260709 12:23:42.969251  1874 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:23:42.970029  2414 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:42.939360  5903 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:42.940840  6010 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:42.968916  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:42.971707  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 110283us
I20260709 12:23:42.920979  5696 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 12:23:42.976068  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:42.980327  2414 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
W20260709 12:23:42.989815  6041 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20260709 12:23:42.990041  6031 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.121s
I20260709 12:23:42.962491  6057 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:42.991215  6069 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:23:42.991557  6033 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.298s	user 0.000s	sys 0.001s
I20260709 12:23:42.991707  6057 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:42.990554  5876 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:42.970644  5487 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:42.972047  5903 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.023423  5903 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.024055  5999 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.025117  6027 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:23:42.967698  5793 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 12:23:43.021039  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.022140  6045 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:43.030578  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.022467  5377 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.022868  6031 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:43.041134  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.041553  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:43.041971  1874 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
W20260709 12:23:43.040855  6039 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.292s	user 0.002s	sys 0.000s
I20260709 12:23:43.044129  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.047791  2411 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:23:43.048161  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.050453  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:43.050923  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:43.051515  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 166378us
I20260709 12:23:43.026590  5834 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.053319  1805 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:43.061709  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:43.067381  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:43.068449  2414 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:42.972249  6044 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.073141  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:43.088145  1878 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.554s	user 0.012s	sys 0.000s
I20260709 12:23:43.088742  1878 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:43.094070  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:43.095003  2383 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.744s (SENT)
I20260709 12:23:43.095544  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:43.094455  1800 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:43.094419  1880 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:43.103765  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.104816  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.105403  1805 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:43.109416  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.110217  1806 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:43.105391  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.116641  6049 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.117110  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.054147  6056 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:43.123561  2414 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:23:43.125463  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56810.1us
I20260709 12:23:43.118574  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:43.128898  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.094020  6053 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:43.075366  6029 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:42.975176  6032 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.073829  6057 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:43.131884  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.136968  1805 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:43.101760  6068 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:43.136431  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71004.5us
I20260709 12:23:43.166496  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:43.170548  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53487.2us
I20260709 12:23:43.111164  5377 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:43.174232  2414 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:43.059219  6031 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.182534  1806 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:43.183599  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.134972  6029 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.132678  6053 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:43.190521  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:43.041409  6045 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.201339  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.201762  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.200227  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:43.173117  6049 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:43.180821  6059 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:43.196756  6029 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:43.236490  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:43.203372  5767 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.904s	user 0.006s	sys 0.007s
I20260709 12:23:43.239144  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:43.239307  5767 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.241690  5767 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.242455  1802 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:43.246167  1880 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:43.237862  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.204046  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:43.201175  6055 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.176s
W20260709 12:23:43.205082  6067 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:23:43.205574  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.135712  6056 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:23:43.208163  6070 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:43.208613  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:43.209173  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.005s	sys 0.026s
W20260709 12:23:43.263064  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.006s	sys 0.027s
I20260709 12:23:43.209703  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:43.264096  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54485.6us
I20260709 12:23:43.211361  6031 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.211956  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.206207  6073 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 12:23:43.213967  6082 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.214670  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:43.188257  6037 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.310s	user 0.002s	sys 0.000s
I20260709 12:23:43.219446  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.280254  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.222378  1805 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:43.280380  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.226209  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.227828  6045 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.201521  5964 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.196498  6065 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.196452  5816 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:43.231161  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.234308  6044 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.259865  1723 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
W20260709 12:23:43.275350  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.216740  5793 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:43.229305  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.231215  5876 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:43.191221  6068 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
W20260709 12:23:43.294555  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 99975.4us
W20260709 12:23:43.266775  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 335958us
I20260709 12:23:43.287325  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.294608  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.267747  6070 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.300315  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:43.322597  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 351754us
I20260709 12:23:43.282830  6073 consensus_queue.cc:1243] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 12.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260709 12:23:43.336328  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 119288us
I20260709 12:23:43.278339  5819 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:43.339780  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:43.290985  5793 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:43.281958  6025 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.017s	sys 0.007s
I20260709 12:23:43.344216  1806 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.343775  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:43.379738  6067 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:23:43.288220  5816 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:43.294958  5487 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:43.379387  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.381728  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:43.323364  6070 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.414041  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:43.384011  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 184275us
I20260709 12:23:43.385283  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.387931  1880 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:43.384268  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.295253  1801 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:43.397871  5979 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 12:23:43.416599  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.416960  6040 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:43.417407  1801 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:43.393779  5793 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:43.395364  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.300658  6065 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.421705  6065 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:43.422748  6044 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:43.291236  6082 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.423441  6044 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.297209  5903 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:43.386879  5875 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.039s	user 0.005s	sys 0.009s
I20260709 12:23:43.397019  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:43.393921  1799 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:43.418491  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.425638  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.384104  5819 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:43.393241  6025 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.018s	sys 0.007s
I20260709 12:23:43.428131  1697 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:43.395442  5785 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.776s	user 0.012s	sys 0.000s
W20260709 12:23:43.429934  6078 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:23:43.430241  6076 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:43.434542  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.384678  6043 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.472867  6043 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.475819  6043 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.456341  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.428400  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:43.503705  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47470.8us
I20260709 12:23:43.505592  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.397621  5816 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:43.439412  1803 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:43.459082  1696 leader_election.cc:400] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9.
I20260709 12:23:43.462106  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.511955  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.512085  1696 leader_election.cc:403] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:23:43.430112  6048 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.461313  5487 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:23:43.516848  6068 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.467995  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:43.520450  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52564.8us
I20260709 12:23:43.468770  6085 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:43.523054  6085 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.523162  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.523550  6085 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:43.525008  1799 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:43.526296  1799 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:43.468772  5903 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:43.477939  6071 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.270s	user 0.000s	sys 0.002s
I20260709 12:23:43.478722  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:43.532383  1873 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:43.395812  6030 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 12:23:43.535827  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1936) took 1016 ms. Trace:
I20260709 12:23:43.536231  1727 rpcz_store.cc:276] 0709 12:23:42.519727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.537911 (+ 18184us) service_pool.cc:224] Handling call
0709 12:23:42.538324 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:42.762576 (+224252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:42.762644 (+    68us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:42.784038 (+ 21394us) log.cc:844] Serialized 26 byte log entry
0709 12:23:42.928410 (+144372us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:42.928447 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.928533 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:43.476993 (+548460us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:43.535548 (+ 58555us) raft_consensus.cc:1685] finished
0709 12:23:43.535569 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.535765 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7446,"raft.queue_time_us":66807,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":3799,"spinlock_wait_cycles":87766784,"thread_start_us":232000,"threads_started":2,"wal-append.queue_time_us":349987}
I20260709 12:23:43.479831  6082 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.538702  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:43.539258  6030 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:43.539356  5999 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:43.539716  6030 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:43.541258  5999 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:43.541047  6030 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.541858  5999 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:43.542433  6030 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:43.393185  6073 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 12:23:43.511214  5964 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.543931  6073 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:23:43.513898  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 85652.2us
I20260709 12:23:43.544528  6073 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:43.514465  6059 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:43.515202  1875 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:43.544878  6073 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:43.545375  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:43.515928  6079 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.306s	user 0.002s	sys 0.000s
I20260709 12:23:43.546867  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.547510  1807 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:43.464262  5785 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.548408  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.549177  1807 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:43.550074  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.550280  5785 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.544934  1770 leader_election.cc:400] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
I20260709 12:23:43.551162  1770 leader_election.cc:403] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:43.551417  5499 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:43.552271  6065 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9.
I20260709 12:23:43.553141  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:43.460803  5819 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:43.466398  5875 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.385063  6052 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.471429  6074 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:43.516012  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:43.567831  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51883.8us
I20260709 12:23:43.537541  6048 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:43.539677  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.483086  1878 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:43.495796  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.523631  6029 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.555478  1806 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:43.516902  5487 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:43.554263  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:43.570598  1873 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:43.576341  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:43.584257  2411 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:23:43.585211  1798 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:43.593737  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.561975  6068 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:43.596264  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.546175  6073 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:43.603437  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 190853us
I20260709 12:23:43.604516  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:43.604933  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1971) took 1407 ms. Trace:
W20260709 12:23:43.599601  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 202649us
I20260709 12:23:43.599838  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.605652  1802 rpcz_store.cc:276] 0709 12:23:42.197027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.197192 (+   165us) service_pool.cc:224] Handling call
0709 12:23:43.604642 (+1407450us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540026f054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:43.604848 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2385080576}
I20260709 12:23:43.594558  1807 raft_consensus.cc:2376] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12. Re-sending same reply.
I20260709 12:23:43.602483  1805 raft_consensus.cc:2368] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13.
I20260709 12:23:43.605959  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.608623  1731 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 12:23:43.611372  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:43.612411  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:43.612540  1845 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.742s (SENT)
I20260709 12:23:43.613745  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.616415  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.611204  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:43.621970  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.622828  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.575687  5903 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.575076  5876 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.613744  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.648128  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.606788  1801 raft_consensus.cc:2376] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12. Re-sending same reply.
I20260709 12:23:43.617393  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.576817  6059 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:43.620311  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:43.636404  1845 leader_election.cc:336] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.816s (SENT)
I20260709 12:23:43.656546  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.661302  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.599946  6065 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:43.599231  5377 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.680548  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:43.665589  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.681649  1726 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:43.664354  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.679034  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.686821  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:43.679574  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 84446.4us
W20260709 12:23:43.688140  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.673570  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:43.663981  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:43.689821  2416 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.690384  1723 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:43.692094  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:43.599805  6049 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9.
I20260709 12:23:43.700644  1801 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:23:43.649266  6048 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:43.648488  5487 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:23:43.649953  5933 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:43.699118  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:43.699191  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.702937  1802 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260709 12:23:43.705219  1807 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:43.625703  6068 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:43.656762  6059 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:43.689118  1844 connection.cc:256] RPC call timeout handler was delayed by -1.1029977679995682s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1698s for a response
I20260709 12:23:43.708330  6075 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:43.625854  6040 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:23:43.711314  6082 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.666193  5876 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.712498  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.712754  6084 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:43.663775  6083 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:43.664608  5903 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.694435  5377 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.728358  5933 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:43.663923  6045 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.694964  5760 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:43.714351  2416 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:43.705536  6027 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.696194  6073 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:43.703841  5988 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.830s	user 0.009s	sys 0.007s
I20260709 12:23:43.582146  6052 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:43.731786  1844 connection.cc:256] RPC call timeout handler was delayed by -1.0633977679999589s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17420000000000002s for a response
W20260709 12:23:43.838299  1844 connection.cc:256] RPC call timeout handler was delayed by -1.0314735079996353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4502s for a response
W20260709 12:23:43.839180  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.787456  1801 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.705972  6048 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:43.617333  5999 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.793155  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:43.794148  6077 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:43.727090  5487 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:43.760651  5834 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.012s	sys 0.047s
I20260709 12:23:43.667991  5979 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:23:43.732801  5581 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.761020  6059 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:43.818853  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:43.787019  5964 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:43.826380  1845 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.896s (SENT)
I20260709 12:23:43.867977  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:43.828316  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 252102us
I20260709 12:23:43.791553  6084 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.729611  1802 pending_rounds.cc:85] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 7
I20260709 12:23:43.872920  1802 pending_rounds.cc:107] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 12:23:43.794581  5875 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:43.843438  1727 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:23:43.851699  1798 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:23:43.859014  5834 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.013s	sys 0.047s
I20260709 12:23:43.821290  6083 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:43.863286  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 176579us
W20260709 12:23:43.870178  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 164939us
I20260709 12:23:43.870689  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.873346  1802 raft_consensus.cc:2889] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:43.847326  5760 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:43.858695  1732 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:43.858670  5487 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:43.886433  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.846148  6048 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:43.851688  6065 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:43.862020  6077 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:43.892763  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66439.2us
I20260709 12:23:43.845098  5988 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.880924  1804 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:23:43.846796  6082 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 12:23:43.849386  6045 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.877494  6084 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:43.903964  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51423us
I20260709 12:23:43.911792  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:43.913473  1799 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:23:43.907194  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.892855  5979 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:23:43.914476  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:43.887954  6060 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:23:43.905952  5760 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.911423  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:43.918589  2418 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:23:43.859575  6068 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.921774  1804 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:43.914669  6105 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:43.873266  6085 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:43.907873  5499 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.927017  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:43.930374  1802 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:43.932111  6083 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.934018  5875 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:43.934599  5875 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.936801  5875 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:43.961886  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 50189.5us
I20260709 12:23:43.968061  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:43.929394  5834 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:23:43.971652  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 241526us
I20260709 12:23:43.975608  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:43.991466  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:43.924870  5979 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:43.992056  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:43.919415  6029 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:43.928165  6105 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:44.002545  1873 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:23:43.969920  6101 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:44.048177  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:44.049721  1881 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:23:44.022650  1801 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
W20260709 12:23:44.003646  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 132975us
I20260709 12:23:44.005960  6052 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:44.007982  6002 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:44.012049  5876 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:44.013367  6095 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:23:44.014427  6032 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:23:44.069311  6059 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:44.072624  6030 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.526s	user 0.009s	sys 0.000s
I20260709 12:23:44.075639  5988 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:44.077286  5793 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.659s	user 0.008s	sys 0.003s
I20260709 12:23:44.077742  5793 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:44.081467  6090 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.262s	user 0.002s	sys 0.000s
W20260709 12:23:44.083977  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 155671us
I20260709 12:23:44.084213  5903 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:44.085122  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:44.086967  5979 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:23:44.088464  5979 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:44.088573  6075 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:23:44.058512  5377 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:44.089645  6070 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.631s	user 0.012s	sys 0.000s
I20260709 12:23:44.090088  6070 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.092933  6070 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:44.014931  6045 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:44.015496  6077 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.024065  6082 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:43.916517  5933 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:43.993404  5499 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:44.116544  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.005s	sys 0.017s
I20260709 12:23:44.064442  6085 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:44.096175  1873 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:44.054768  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:44.041085  6105 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:44.058712  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:44.119392  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.005s	sys 0.017s
W20260709 12:23:44.110505  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:44.154095  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:44.154510  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.103065  5978 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:44.156637  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:44.141971  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:44.171335  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 116665us
I20260709 12:23:44.143020  1880 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:44.144922  6042 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:23:44.145922  5816 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.590s	user 0.007s	sys 0.001s
I20260709 12:23:44.148051  5891 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:44.149071  1878 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:23:44.141415  5793 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:44.178610  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:44.187114  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45226.3us
I20260709 12:23:44.199566  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:44.200452  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:44.202162  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:44.195151  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:44.237089  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:44.237361  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:44.238440  2417 raft_consensus.cc:2376] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9. Re-sending same reply.
I20260709 12:23:44.048539  6065 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.247582  1875 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:44.188508  5999 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:44.249261  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54178.1us
I20260709 12:23:44.248144  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:44.185853  5377 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:44.255267  1873 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:44.259307  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 891) took 1065 ms. Trace:
I20260709 12:23:44.155560  5487 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.324999  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:44.315563  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 60712.4us
I20260709 12:23:44.202463  6082 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:23:44.317175  6111 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
I20260709 12:23:44.218097  6032 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:23:44.321003  6106 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20260709 12:23:44.249123  5557 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:44.341154  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 373224us
I20260709 12:23:44.251626  5793 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:44.321949  6108 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20260709 12:23:44.320505  2414 rpcz_store.cc:276] 0709 12:23:43.193633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.193807 (+   174us) service_pool.cc:224] Handling call
0709 12:23:43.244423 (+ 50616us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.244647 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.244672 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.245142 (+   470us) log.cc:844] Serialized 26 byte log entry
0709 12:23:43.426683 (+181541us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:43.565171 (+138488us) log.cc:844] Serialized 14 byte log entry
0709 12:23:43.565417 (+   246us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.565505 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.080646 (+515141us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:44.248945 (+168299us) raft_consensus.cc:1685] finished
0709 12:23:44.259023 (+ 10078us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.259226 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":134355,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":120,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":11118,"spinlock_wait_cycles":6715648,"thread_start_us":281537,"threads_started":2,"wal-append.queue_time_us":436757}
I20260709 12:23:44.260419  5999 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:44.320639  6091 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.526s	user 0.003s	sys 0.000s
I20260709 12:23:44.197976  5816 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.237035  6085 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.099s
I20260709 12:23:44.399358  2416 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 7: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:44.403591  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:44.484902  1802 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:44.401209  1873 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:44.486421  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:44.486604  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89729.9us
I20260709 12:23:44.245041  5891 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.139s
I20260709 12:23:44.329191  6047 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:44.244028  6070 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.098s
I20260709 12:23:44.237089  5785 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.118s
W20260709 12:23:44.484133  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 225502us
W20260709 12:23:44.491446  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 310427us
I20260709 12:23:44.388888  6030 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.499394  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:44.512830  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:44.502488  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:44.505522  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.076s (SENT) [suppressed 37 similar messages]
I20260709 12:23:44.496693  6082 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:44.342703  6069 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.162s
W20260709 12:23:44.505527  1845 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.697s (SENT)
I20260709 12:23:44.493677  5999 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:44.548952  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.333122  5377 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.492431  5793 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:44.547992  2414 raft_consensus.cc:2410] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:23:44.491806  5696 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:44.547961  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:44.405711  5557 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.495661  6032 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.065s
I20260709 12:23:44.552738  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:44.594661  2411 raft_consensus.cc:2410] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:23:44.594993  1873 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:44.602478  2415 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:44.604779  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:44.545894  6109 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:44.574893  6048 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.667s	user 0.009s	sys 0.002s
I20260709 12:23:44.545743  6047 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:44.585978  6076 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:44.623101  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2081) took 1351 ms. Trace:
W20260709 12:23:44.603489  5819 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.037s	user 0.008s	sys 0.008s
I20260709 12:23:44.628446  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:44.629155  1873 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:44.626858  1876 rpcz_store.cc:276] 0709 12:23:43.271344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.272998 (+  1654us) service_pool.cc:224] Handling call
0709 12:23:43.289477 (+ 16479us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.293909 (+  4432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.293942 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.304858 (+ 10916us) log.cc:844] Serialized 26 byte log entry
0709 12:23:43.626675 (+321817us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:43.626742 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.626867 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.319493 (+692626us) spinlock_profiling.cc:243] Waited 871 us on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:44.602881 (+283388us) raft_consensus.cc:1685] finished
0709 12:23:44.602916 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.623001 (+ 20085us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41374,"raft.queue_time_us":4670,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":16489,"spinlock_wait_cycles":2002688,"thread_start_us":230744,"threads_started":1,"wal-append.queue_time_us":412673}
I20260709 12:23:44.634490  6057 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:23:44.635075  6057 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:44.635133  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:44.636240  1876 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 12 because replica has last-logged OpId of term: 11 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 10.
I20260709 12:23:44.636087  6057 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.637315  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:44.638453  6114 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.297s	user 0.000s	sys 0.001s
W20260709 12:23:44.639812  6119 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260709 12:23:44.639909  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 817) took 1110 ms. Trace:
I20260709 12:23:44.640487  2418 rpcz_store.cc:276] 0709 12:23:43.529966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.530186 (+   220us) service_pool.cc:224] Handling call
0709 12:23:43.530538 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.919283 (+388745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.639557 (+720274us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b540048a3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:44.639835 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":891512576}
I20260709 12:23:44.641024  5499 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:44.642261  5487 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:44.625914  5696 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.645761  5696 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:44.653420  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:44.653752  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 819) took 1074 ms. Trace:
I20260709 12:23:44.654114  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1846) took 1410 ms. Trace:
I20260709 12:23:44.551716  6030 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.660485  1805 rpcz_store.cc:276] 0709 12:23:43.244043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.294308 (+ 50265us) service_pool.cc:224] Handling call
0709 12:23:44.653883 (+1359575us) spinlock_profiling.cc:243] Waited 303 ms on lock 0x7b54001d4f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:44.654028 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":695922944}
I20260709 12:23:44.608661  6059 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:44.624732  5581 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:44.664418  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:44.646759  6115 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.399s	user 0.000s	sys 0.001s
I20260709 12:23:44.670563  1805 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:44.671777  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:44.672041  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:44.672465  1805 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:44.685204  6043 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:44.685853  6043 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:44.654074  6121 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:23:44.691133  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2874) took 1063 ms. Trace:
I20260709 12:23:44.660202  2412 rpcz_store.cc:276] 0709 12:23:43.573936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.678894 (+104958us) service_pool.cc:224] Handling call
0709 12:23:43.773352 (+ 94458us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.791845 (+ 18493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.791876 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.819650 (+ 27774us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.223685 (+404035us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:44.223745 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.223843 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.647984 (+424141us) raft_consensus.cc:1685] finished
0709 12:23:44.648005 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.648158 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":110585,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":2131,"thread_start_us":343275,"threads_started":1,"wal-append.queue_time_us":700478}
I20260709 12:23:44.694962  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:44.696674  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:44.699054  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:44.663110  6032 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.702845  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:44.705624  2410 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 881) took 1827 ms (client timeout 1697 ms). Trace:
W20260709 12:23:44.706086  2410 rpcz_store.cc:269] 0709 12:23:42.869344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.869516 (+   172us) service_pool.cc:224] Handling call
0709 12:23:44.696781 (+1827265us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b540046a354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:44.696943 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2291801856}
I20260709 12:23:44.709498  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:44.627293  6048 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.674029  5581 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:44.707715  6032 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.711750  5581 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.712954  5581 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:44.713195  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1851) took 1467 ms. Trace:
I20260709 12:23:44.713449  6032 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:44.713766  1798 rpcz_store.cc:276] 0709 12:23:43.245602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.664153 (+418551us) service_pool.cc:224] Handling call
0709 12:23:43.700893 (+ 36740us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.878477 (+177584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.712905 (+834428us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:44.713098 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":345597312}
I20260709 12:23:44.705729  6085 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:44.715353  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:44.715910  6085 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.717746  6040 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:44.718220  6040 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:44.718276  6085 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:44.694145  1728 rpcz_store.cc:276] 0709 12:23:43.611144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.611381 (+   237us) service_pool.cc:224] Handling call
0709 12:23:43.664736 (+ 53355us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:43.686653 (+ 21917us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.686704 (+    51us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:43.692819 (+  6115us) log.cc:844] Serialized 78 byte log entry
0709 12:23:43.886626 (+193807us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:43.996130 (+109504us) log.cc:844] Serialized 14 byte log entry
0709 12:23:43.997020 (+   890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.997400 (+   380us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.017154 (+ 19754us) spinlock_profiling.cc:243] Waited 4.4 ms on lock 0x7b1800410c20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:44.666226 (+649072us) raft_consensus.cc:1685] finished
0709 12:23:44.673725 (+  7499us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.673907 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32121,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":508,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":15044,"spinlock_wait_cycles":10127104,"thread_start_us":183602,"threads_started":2,"wal-append.queue_time_us":214112}
W20260709 12:23:44.723278  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.648865  5819 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:44.724992  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:44.725520  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:44.726438  1873 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:44.727487  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2089) took 1308 ms. Trace:
I20260709 12:23:44.730516  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:44.730861  6083 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:44.732792  1874 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:44.693233  6043 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:44.732533  1880 rpcz_store.cc:276] 0709 12:23:43.419176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.540908 (+121732us) service_pool.cc:224] Handling call
0709 12:23:43.541295 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.175052 (+633757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.727221 (+552169us) spinlock_profiling.cc:243] Waited 601 ms on lock 0x7b54001f8454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:44.727413 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1383280000}
I20260709 12:23:44.707485  6045 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:44.741451  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 239019us
I20260709 12:23:44.735968  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:44.746902  5785 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.747648  5785 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.750422  5785 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:44.751008  6069 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.753042  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:44.652952  6109 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.779445  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:44.756918  1724 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:44.762207  1873 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:44.763119  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:44.766395  5816 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:44.655788  6053 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.727494  6055 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:44.632225  6082 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.767954  1875 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:23:44.771502  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46575.8us
I20260709 12:23:44.773854  1876 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:44.737303  6032 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:44.704665  5978 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.051s
I20260709 12:23:44.791316  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:44.736616  6083 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:44.806043  1798 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:44.803233  1805 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:44.809949  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:44.810294  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:44.812278  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:44.784400  6106 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:23:44.819046  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:44.743232  6133 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:44.784664  6031 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:44.793772  6073 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.956s	user 0.010s	sys 0.003s
I20260709 12:23:44.751652  6045 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.785049  6026 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:44.794049  5979 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:44.813269  6075 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:44.770114  6040 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.834069  1874 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:44.833763  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:44.885545  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94353.4us
I20260709 12:23:44.816654  6083 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:44.797798  6032 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:44.889643  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:44.818621  6084 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.900s	user 0.011s	sys 0.001s
W20260709 12:23:44.827638  6120 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.324s	user 0.001s	sys 0.000s
I20260709 12:23:44.808465  5377 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:44.819721  5816 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:23:44.899861  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:44.925345  2410 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:44.924625  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:44.831070  6055 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:44.921927  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:44.898214  1730 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:44.785852  6069 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.745041  5793 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.933593  1802 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:44.944099  1728 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:23:44.881261  6141 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:44.945319  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:44.901899  6032 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:44.756234  6047 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.899379  5557 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:44.931461  6055 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:44.931701  6044 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:44.962671  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:44.942022  6030 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:44.933898  6075 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.956563  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:44.887053  6026 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:44.886632  6031 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:44.944159  6122 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.978699  1875 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
W20260709 12:23:44.995024  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.007252  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:44.957741  5816 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:45.010538  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 54135us
I20260709 12:23:45.011713  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:44.952031  6065 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.020954  6065 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.022514  6134 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.023674  6044 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:45.032456  1805 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:44.999912  6141 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:44.963446  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 237977us
I20260709 12:23:45.014453  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.041712  1875 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:44.983305  6055 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:45.023370  1802 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:44.998948  1798 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:45.051302  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:44.924552  6084 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.007370  6049 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:45.023424  6065 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:45.064173  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 118989us
W20260709 12:23:45.072741  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 175889us
I20260709 12:23:45.073627  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:45.014269  6135 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:45.071303  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 527) took 1999 ms (client timeout 2139 ms). Trace:
W20260709 12:23:45.080420  1803 rpcz_store.cc:269] 0709 12:23:43.072287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.072525 (+   238us) service_pool.cc:224] Handling call
0709 12:23:45.071056 (+1998531us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b540009fbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:45.071230 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":767129344}
I20260709 12:23:45.076659  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:45.023344  6134 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.044447  6140 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.003933  6075 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.120807  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:45.129550  2411 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:45.124743  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:45.070710  6048 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.127856  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:45.134879  1873 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:45.044452  6138 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:45.040740  6050 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:23:45.070501  5819 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:45.081216  6044 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.136680  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:45.099627  6049 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:45.133122  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:45.138444  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64920.9us
I20260709 12:23:45.111886  5978 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:45.162181  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:45.096274  5557 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:45.166541  1803 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:45.169637  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:45.130577  1805 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
W20260709 12:23:45.172505  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 95943.1us
I20260709 12:23:45.014403  6122 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.140190  6109 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.139468  6073 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:45.179630  1873 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:45.189646  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.190451  1723 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:45.186328  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:45.185150  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:45.063760  6141 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.191408  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:45.192512  1803 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:45.192962  2411 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:45.195546  1723 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:45.191038  6077 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:45.163276  6044 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:45.211061  6073 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.184696  5819 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:45.238448  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:45.095885  6135 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:45.243570  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 192342us
I20260709 12:23:45.249943  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:45.251479  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:45.244436  1803 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:45.215339  6069 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.171870  5978 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.194298  6045 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.268541  6133 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:45.174839  5816 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.141173  6140 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.279340  6132 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.280984  6146 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:23:45.281519  6160 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:23:45.282404  6164 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:45.283150  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:45.263579  6137 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.297s	user 0.002s	sys 0.000s
I20260709 12:23:45.299404  2410 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:45.239084  6077 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:45.243686  5819 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:45.263342  5767 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:45.212641  6148 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.062s
I20260709 12:23:45.338986  2414 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:45.293403  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:45.270751  6109 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:45.343072  1803 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:45.348834  5581 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.635s	user 0.013s	sys 0.004s
I20260709 12:23:45.349076  6141 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.349853  6141 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:45.351157  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:45.351415  6134 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.351899  2410 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:45.353874  1729 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
W20260709 12:23:45.347793  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 211219us
W20260709 12:23:45.355544  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.283602  6069 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:45.297453  6152 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:23:45.361227  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 365919us
I20260709 12:23:45.275753  6133 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.342165  6077 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:45.385848  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:45.388698  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 173244us
I20260709 12:23:45.385586  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:45.309402  6030 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:45.396395  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:45.398538  1803 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:45.375284  6069 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.386967  6133 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.403038  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:45.406502  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:45.409139  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 225112us
I20260709 12:23:45.427186  5696 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:45.409060  6069 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:45.437021  2411 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:45.429191  5696 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:45.360011  6164 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:23:45.446130  5978 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:45.345244  6132 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.464213  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:45.465621  2414 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:45.447157  6160 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.448006  6158 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
W20260709 12:23:45.449538  5999 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.865s	user 0.008s	sys 0.004s
I20260709 12:23:45.446465  6159 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:45.468864  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:45.297928  6049 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.431486  6075 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.498557  6168 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.499698  5999 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.544035  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.412647  5819 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.551898  1803 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:45.464128  6073 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.467262  6084 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.420696  6031 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.502985  6149 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:45.554116  2411 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:23:45.573414  6082 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.573941  6082 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.575961  6082 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.468290  5767 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:45.442934  6030 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.082s
I20260709 12:23:45.577550  5767 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:45.577968  5767 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:45.551024  6043 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.784s	user 0.006s	sys 0.007s
I20260709 12:23:45.578624  6043 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.580962  6043 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:45.557210  6153 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.313s	user 0.002s	sys 0.000s
W20260709 12:23:45.562111  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 176599us
I20260709 12:23:45.546988  6044 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:23:45.582231  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:45.582978  1731 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:45.584861  2418 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:45.585517  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.568248  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:45.593364  1803 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:45.594887  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:45.597524  1803 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:45.599455  2411 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
W20260709 12:23:45.601172  6151 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
W20260709 12:23:45.602278  6147 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:23:45.602969  6171 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.604755  5999 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.604868  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2005) took 1365 ms. Trace:
I20260709 12:23:45.602586  6026 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:45.607513  6155 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
W20260709 12:23:45.608466  6167 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
W20260709 12:23:45.609365  6107 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 1.414s	user 0.005s	sys 0.000s
I20260709 12:23:45.571657  5819 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:45.610008  6140 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.613482  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:45.635236  6043 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.615895  6132 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:45.558900  6159 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.563818  6168 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.534971  6160 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.572463  6073 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.637393  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:45.637154  6171 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.571061  6045 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:45.643697  1805 rpcz_store.cc:276] 0709 12:23:44.239673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.818664 (+578991us) service_pool.cc:224] Handling call
0709 12:23:45.604667 (+786003us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b540026f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:45.604795 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":482163584}
I20260709 12:23:45.643755  2411 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:23:45.583853  6044 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:45.644169  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:45.687356  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:45.646482  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:45.635321  6032 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.652s	user 0.011s	sys 0.002s
W20260709 12:23:45.649461  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 83273.4us
I20260709 12:23:45.622597  6040 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.650970  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.651798  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.614079  6164 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:23:45.634991  6030 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:45.669203  6150 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:23:45.596884  6149 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.676993  6159 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.681403  1729 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:45.683936  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.720243  1770 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:45.660001  6047 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:45.693436  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:45.684077  6135 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.681701  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:45.699527  1732 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:23:45.724592  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:45.726570  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44967.4us
I20260709 12:23:45.695937  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:45.706938  6183 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:45.678428  6140 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:23:45.714919  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 68527.7us
I20260709 12:23:45.728780  1846 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:45.701391  6083 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.768s	user 0.011s	sys 0.002s
I20260709 12:23:45.694070  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:45.733388  1805 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:45.737663  6043 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:45.728523  6177 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:45.711089  6132 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:45.732054  6068 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:45.739946  6068 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.740351  6068 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.738662  6140 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.744500  1803 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:45.690814  6073 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.749003  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:45.751581  6160 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.753343  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.753818  6133 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.755990  6132 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:45.756916   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:45.724519  6047 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.762959  1805 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:45.775900  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:45.776542  1805 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:45.777359  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:45.746932  6026 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.699441  6032 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.693624  6044 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.767763  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.779973  1729 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:45.695873  6045 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.721890  6030 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:45.773272  6105 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.585s	user 0.008s	sys 0.009s
I20260709 12:23:45.787696  1803 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:45.789053  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.790696  1729 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:45.780144  6047 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:45.744189  6083 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.794808  1881 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:45.794845  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.798322  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.799094  1728 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:45.799983  1805 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:45.785933  6132 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:45.804550  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:45.807065  1846 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.883s (SENT)
I20260709 12:23:45.807586  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:45.807958  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.809039  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.809223  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.809904  1726 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
W20260709 12:23:45.810920  6175 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:23:45.794436  6105 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.810088  1728 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:45.812530  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.813390  6105 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:45.813956  6161 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.400s	user 0.002s	sys 0.000s
I20260709 12:23:45.813795  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.814821  1728 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:45.815568  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.817795  1728 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:45.818977  1728 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:45.820039  6140 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.820139  1728 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:23:45.744660  6183 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.822096  1801 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:45.824355  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.825275  1878 raft_consensus.cc:2368] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11.
I20260709 12:23:45.826362  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2105) took 1103 ms. Trace:
I20260709 12:23:45.826174  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.811122  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:45.827405  1878 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:45.827978  6149 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:45.828194  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.828590  1726 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:45.829985  1726 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:45.829715  6077 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.831858  6133 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:45.835397  6132 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.826924  1875 rpcz_store.cc:276] 0709 12:23:44.723070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.062164 (+339094us) service_pool.cc:224] Handling call
0709 12:23:45.062584 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.072476 (+  9892us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.072523 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:45.099669 (+ 27146us) log.cc:844] Serialized 26 byte log entry
0709 12:23:45.211098 (+111429us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:45.211155 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.211260 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.281955 (+ 70695us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b48000ce7d8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:23:45.813313 (+531358us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:45.826030 (+ 12717us) raft_consensus.cc:1685] finished
0709 12:23:45.826052 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.826184 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4555,"raft.queue_time_us":2629,"raft.run_cpu_time_us":2289,"raft.run_wall_time_us":202629,"spinlock_wait_cycles":37540096,"thread_start_us":52774,"threads_started":1,"wal-append.queue_time_us":217777}
I20260709 12:23:45.838215  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.788651  6026 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.839928  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.840174  6030 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.840636  1875 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:45.840790  6032 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:45.799502  6055 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.706s	user 0.013s	sys 0.000s
I20260709 12:23:45.841732  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.841542  6055 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.849326  6055 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.852438  1771 leader_election.cc:400] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11.
I20260709 12:23:45.852945  1771 leader_election.cc:403] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:45.854938  6098 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:45.855571  6098 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:45.856264  6130 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.248s	user 0.003s	sys 0.000s
I20260709 12:23:45.857995  6098 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.818540  6069 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:45.863072  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1429988690001665s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1879s for a response
I20260709 12:23:45.826679  6068 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.871372  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 822) took 1171 ms. Trace:
I20260709 12:23:45.871781  2414 rpcz_store.cc:276] 0709 12:23:44.699684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.552982 (+853298us) service_pool.cc:224] Handling call
0709 12:23:45.564692 (+ 11710us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.564857 (+   165us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:45.679558 (+114701us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.870979 (+191421us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:45.871036 (+    57us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:45.871093 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.871173 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.871312 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":132679,"prepare.queue_time_us":190134,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":151860,"threads_started":2,"wal-append.queue_time_us":94107}
I20260709 12:23:45.794062  5767 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.866288  1805 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:45.869933  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.877552  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:45.885535  2414 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:45.867473  1726 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:45.886351  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:45.831864  6077 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:45.849309  6044 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.861565  5834 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.870858  6059 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:45.888108  1875 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:45.882875  1729 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:45.861404  6002 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:45.878945  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.906807  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:45.907327  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:45.921903  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1993) took 2508 ms. Trace:
I20260709 12:23:45.922351  1804 rpcz_store.cc:276] 0709 12:23:43.413104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.909153 (+496049us) service_pool.cc:224] Handling call
0709 12:23:43.909536 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.925868 (+ 16332us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.921617 (+1995749us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b540035fa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:45.921829 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27322752}
I20260709 12:23:45.885794  6072 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.926295  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1943) took 1036 ms. Trace:
I20260709 12:23:45.926994  1725 rpcz_store.cc:276] 0709 12:23:44.889483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.889636 (+   153us) service_pool.cc:224] Handling call
0709 12:23:44.890026 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.924290 (+ 34264us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:44.924330 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.934230 (+  9900us) log.cc:844] Serialized 26 byte log entry
0709 12:23:45.874359 (+940129us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b1800326140. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:23:45.901850 (+ 27491us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:45.901937 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.902417 (+   480us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.925862 (+ 23445us) raft_consensus.cc:1685] finished
0709 12:23:45.925893 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.926201 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14037,"raft.queue_time_us":60964,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":8391,"spinlock_wait_cycles":25716480,"thread_start_us":27154,"threads_started":1,"wal-append.queue_time_us":946640}
I20260709 12:23:45.915788  6059 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.929262  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:45.929937  1875 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:45.930753  6059 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:45.936663  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 83871.2us
W20260709 12:23:45.937927  1846 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 2.077s (SENT)
I20260709 12:23:45.939435  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:45.941358  6040 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.938280  5487 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:45.942214  5487 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:45.941936  6040 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.942633  5487 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:45.942889  6069 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.943768  6110 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.944234  6040 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:45.944392  6110 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.947002  2412 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:45.947726  6100 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.919744  6002 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.950490  6002 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.923611  5988 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.951269  5988 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.952646  6047 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:45.953537  5988 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.954895  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:45.955227  1878 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:45.956550  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:45.956928  5377 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:45.957486  5377 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:45.939007  6072 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.958076  5377 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.963826  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 554) took 2336 ms. Trace:
I20260709 12:23:45.964382  1727 rpcz_store.cc:276] 0709 12:23:43.627542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.694269 (+ 66727us) service_pool.cc:224] Handling call
0709 12:23:43.733457 (+ 39188us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.878161 (+144704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.963442 (+2085281us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b540035d754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:45.963696 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":235139968}
I20260709 12:23:45.919721  5999 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.968640  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:45.969448  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 899) took 1729 ms. Trace:
I20260709 12:23:45.974004  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:45.978117  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:45.972314  2417 rpcz_store.cc:276] 0709 12:23:44.240310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.335172 (+ 94862us) service_pool.cc:224] Handling call
0709 12:23:44.398214 (+ 63042us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.852023 (+1453809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.852055 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:45.862082 (+ 10027us) log.cc:844] Serialized 26 byte log entry
0709 12:23:45.940115 (+ 78033us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:45.940161 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.940231 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.969189 (+ 28958us) raft_consensus.cc:1685] finished
0709 12:23:45.969212 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.969381 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1455171,"raft.queue_time_us":1461149,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":10311,"thread_start_us":36690,"threads_started":1,"wal-append.queue_time_us":80440}
I20260709 12:23:45.980340  5988 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:45.985109  5988 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:45.985670  5988 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.988534  6068 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:45.975651  5999 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.999414  6117 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:45.999882  6072 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:45.999979  6117 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:46.000906  6117 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:45.915374  5834 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:45.967036  6110 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:46.002720  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 64837.2us
W20260709 12:23:46.006752  1846 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 2.208s (SENT)
I20260709 12:23:46.007440  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:46.012233  5834 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.014008  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 555) took 2386 ms. Trace:
I20260709 12:23:46.014480  1724 rpcz_store.cc:276] 0709 12:23:43.627859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.711204 (+ 83345us) service_pool.cc:224] Handling call
0709 12:23:43.711732 (+   528us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.781913 (+1070181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.013706 (+1231793us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54002d14d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:46.013930 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2403081984}
I20260709 12:23:46.015080  1872 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:45.995018  5487 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.017603  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 972) took 1522 ms. Trace:
I20260709 12:23:46.018271  2418 rpcz_store.cc:276] 0709 12:23:44.494830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.641563 (+146733us) service_pool.cc:224] Handling call
0709 12:23:44.641956 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.598372 (+956416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.017272 (+418900us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b5400591ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:46.017514 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2168029568}
W20260709 12:23:46.023415  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2002) took 2344 ms (client timeout 1757 ms). Trace:
W20260709 12:23:46.023941  1806 rpcz_store.cc:269] 0709 12:23:43.679442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.767646 (+ 88204us) service_pool.cc:224] Handling call
0709 12:23:46.023125 (+2255479us) spinlock_profiling.cc:243] Waited 741 ms on lock 0x7b540026f054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:46.023285 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1703679872}
I20260709 12:23:46.025910  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1998) took 2366 ms. Trace:
I20260709 12:23:46.027128  6074 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:46.026317  1806 rpcz_store.cc:276] 0709 12:23:43.659734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.024907 (+2365173us) service_pool.cc:224] Handling call
0709 12:23:46.025308 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.025526 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.025563 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.025609 (+    46us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:46.025654 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.025710 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.025819 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.028782  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1863) took 2491 ms. Trace:
I20260709 12:23:46.002843  5999 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.004638  6047 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:46.029295  1804 rpcz_store.cc:276] 0709 12:23:43.537510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.974409 (+2436899us) service_pool.cc:224] Handling call
0709 12:23:45.974902 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.975161 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:45.984057 (+  8896us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.028080 (+ 44023us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.028409 (+   329us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.028469 (+    60us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:46.028532 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.028599 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.028724 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":29893,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":61,"raft.queue_time_us":1665,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":106,"spinlock_wait_cycles":12808064,"thread_start_us":35611,"threads_started":1}
W20260709 12:23:46.035825  6124 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.509s	user 0.005s	sys 0.002s
I20260709 12:23:46.036760  6168 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.037207  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1995) took 2623 ms. Trace:
I20260709 12:23:46.038224  6171 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.039383  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2000) took 2379 ms. Trace:
W20260709 12:23:46.039573  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.039762  1806 rpcz_store.cc:276] 0709 12:23:43.660284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.034935 (+2374651us) service_pool.cc:224] Handling call
0709 12:23:46.038911 (+  3976us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.039088 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.039111 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.039151 (+    40us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:46.039190 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.039239 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.039334 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.040324  5487 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.041904  5933 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.043965  6073 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.045625  6083 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.047415  6045 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:46.048403  6117 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.049073  6183 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.050771  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2001) took 2376 ms. Trace:
I20260709 12:23:46.050951  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2014) took 1254 ms. Trace:
I20260709 12:23:46.051270  1802 rpcz_store.cc:276] 0709 12:23:44.796682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.955537 (+158855us) service_pool.cc:224] Handling call
0709 12:23:46.050774 (+1095237us) spinlock_profiling.cc:243] Waited 37.5 ms on lock 0x7b540035f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:46.050898 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86330624}
I20260709 12:23:46.018630  6122 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:46.053098  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2003) took 1814 ms. Trace:
I20260709 12:23:46.053455  1802 rpcz_store.cc:276] 0709 12:23:44.238953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.052262 (+1813309us) service_pool.cc:224] Handling call
0709 12:23:46.052582 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.052761 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.052824 (+    63us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.052864 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:46.052901 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.052951 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.053050 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.054358  1875 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:46.055500  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.056136  1875 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:46.056993  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 1999) took 2397 ms. Trace:
I20260709 12:23:46.057878  1804 rpcz_store.cc:276] 0709 12:23:43.660033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.034681 (+2374648us) service_pool.cc:224] Handling call
0709 12:23:46.056262 (+ 21581us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.056545 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.056579 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.056633 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:46.056692 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.056772 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.056912 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.058764  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2008) took 1569 ms. Trace:
I20260709 12:23:46.051183  1806 rpcz_store.cc:276] 0709 12:23:43.679030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.040721 (+2361691us) service_pool.cc:224] Handling call
0709 12:23:46.041058 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.041250 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.041275 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.041315 (+    40us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:46.041350 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.041403 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.050713 (+  9310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.059360  1802 rpcz_store.cc:276] 0709 12:23:44.489231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.054295 (+1565064us) service_pool.cc:224] Handling call
0709 12:23:46.058430 (+  4135us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.058559 (+   129us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.058573 (+    14us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.058600 (+    27us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:46.058625 (+    25us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.058657 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.058724 (+    67us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.037588  1799 rpcz_store.cc:276] 0709 12:23:43.413505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.922201 (+508696us) service_pool.cc:224] Handling call
0709 12:23:43.938444 (+ 16243us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.938741 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.938770 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.994308 (+ 55538us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.241528 (+247220us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:44.241577 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.241650 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.010032 (+1768382us) spinlock_profiling.cc:243] Waited 4.32 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:46.036911 (+ 26879us) raft_consensus.cc:1685] finished
0709 12:23:46.036933 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.037129 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9943424,"thread_start_us":184855,"threads_started":1,"wal-append.queue_time_us":478636}
I20260709 12:23:46.062316  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.074608  6100 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11.
I20260709 12:23:46.079247  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2011) took 1374 ms. Trace:
I20260709 12:23:46.079756  1798 rpcz_store.cc:276] 0709 12:23:44.705301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.715018 (+  9717us) service_pool.cc:224] Handling call
0709 12:23:46.079060 (+1364042us) spinlock_profiling.cc:243] Waited 89.9 ms on lock 0x7b540013a6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:46.079179 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":206860672}
I20260709 12:23:46.083225  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.086951  1875 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:46.082262  6117 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:46.092944  6002 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:46.093169  5988 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.096819  5988 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.101811  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 892) took 1924 ms. Trace:
I20260709 12:23:46.102331  2419 rpcz_store.cc:276] 0709 12:23:44.177351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.189487 (+ 12136us) service_pool.cc:224] Handling call
0709 12:23:44.190048 (+   561us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.190750 (+   702us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:44.248185 (+ 57435us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.846402 (+1598217us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b18003fa1e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16afcac58f 00007f16a8ceb571 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd
0709 12:23:46.002072 (+155670us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b18003fa1e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:23:46.091687 (+ 89615us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.091916 (+   229us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.092438 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.092595 (+   157us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:46.092632 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.092718 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.101544 (+  8826us) raft_consensus.cc:1685] finished
0709 12:23:46.101567 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.101740 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16177,"prepare.queue_time_us":60183,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":7580,"raft.queue_time_us":420,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":23670,"spinlock_wait_cycles":53470208,"thread_start_us":84559,"threads_started":1,"wal-append.queue_time_us":1761105}
I20260709 12:23:46.106902  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:46.110349  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.116626  2419 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:46.119243  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.120299  2410 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:46.123970  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:46.124889  2414 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:46.115506  6138 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:46.136919  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72663.6us
I20260709 12:23:46.137131  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.137388  6138 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.137915  2410 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:46.137931  6138 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:46.147218  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.148036  2410 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:46.150318  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.159147  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.163456  6163 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:46.168668  5377 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.170284  6026 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.182087  6134 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:46.184021  1807 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 13 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:23:46.190032  6053 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:46.186357  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:46.194365  6002 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:46.238651  2412 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.195312  5876 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:46.196530  6044 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:46.197753  2414 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:46.244205  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.245015  2414 raft_consensus.cc:2368] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10.
I20260709 12:23:46.198341  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.245182  2417 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:46.245347  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.200816  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1861) took 2663 ms. Trace:
I20260709 12:23:46.246296  1801 rpcz_store.cc:276] 0709 12:23:43.537002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.051451 (+514449us) service_pool.cc:224] Handling call
0709 12:23:44.054014 (+  2563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.822572 (+1768558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.200515 (+377943us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54002f2454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:46.200743 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4065651072}
W20260709 12:23:46.204780  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:46.247748  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.207299  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.208729  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:46.212044  1799 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:23:46.212101  6138 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:46.215240  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:46.235718  6163 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:46.210526  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.220256  1805 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:46.254073  1806 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:46.254124  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 541) took 1305 ms. Trace:
I20260709 12:23:46.254642  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:46.240617  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:46.215708  1802 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:46.215219  6172 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.239684  6109 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.238458  6134 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:46.263612  5377 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:46.263557  6134 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.264911  6134 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.267022  5988 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:46.267767  5988 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:46.268802  5988 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.254577  1805 rpcz_store.cc:276] 0709 12:23:44.948465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.219541 (+1271076us) service_pool.cc:224] Handling call
0709 12:23:46.219894 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.253858 (+ 33964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.254065 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.252408  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2012) took 1529 ms. Trace:
I20260709 12:23:46.270871  6132 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:46.271466  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.271842  6074 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:46.272127  2418 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:46.272495  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2009) took 1643 ms. Trace:
I20260709 12:23:46.271392  1799 rpcz_store.cc:276] 0709 12:23:44.722635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.211403 (+1488768us) service_pool.cc:224] Handling call
0709 12:23:46.211784 (+   381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.252134 (+ 40350us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.252344 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.272464  6074 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:46.273527  2418 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:46.273514  6172 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.273926  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2010) took 1643 ms. Trace:
I20260709 12:23:46.256729  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 537) took 1309 ms. Trace:
I20260709 12:23:46.241747  6002 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:46.256090  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.261423  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:46.253034  6144 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:46.259841  5487 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:23:46.288571  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:46.276547  1802 rpcz_store.cc:276] 0709 12:23:44.947312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.213144 (+1265832us) service_pool.cc:224] Handling call
0709 12:23:46.214655 (+  1511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.256376 (+ 41721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.256656 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.290277  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.273037  1804 rpcz_store.cc:276] 0709 12:23:44.629244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.073139 (+1443895us) service_pool.cc:224] Handling call
0709 12:23:46.083793 (+ 10654us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.084170 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.084197 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.086844 (+  2647us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.155593 (+ 68749us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:46.155858 (+   265us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.156401 (+   543us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.156495 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.272230 (+115735us) raft_consensus.cc:1685] finished
0709 12:23:46.272266 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.272431 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":131,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":217,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":218,"thread_start_us":44483,"threads_started":1,"wal-append.queue_time_us":173079}
I20260709 12:23:46.275831  1798 rpcz_store.cc:276] 0709 12:23:44.630774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.080543 (+1449769us) service_pool.cc:224] Handling call
0709 12:23:46.080920 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.081266 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.081287 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.083603 (+  2316us) log.cc:844] Serialized 26 byte log entry
0709 12:23:46.154859 (+ 71256us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:46.154900 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.155441 (+   541us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:46.273693 (+118252us) raft_consensus.cc:1685] finished
0709 12:23:46.273717 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.273864 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":124,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":1549,"thread_start_us":32882,"threads_started":1,"wal-append.queue_time_us":139940}
I20260709 12:23:46.279874  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.291787  2419 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:46.253185  6126 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:46.274461  6074 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.297111  2413 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:46.284173  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:46.297900  1799 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:46.284804  5999 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:46.285463  6138 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.286314  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.300662  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.294467  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:46.295229  2416 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 897) took 2096 ms (client timeout 2208 ms). Trace:
I20260709 12:23:46.294111  6126 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:46.301569  2416 rpcz_store.cc:269] 0709 12:23:44.199129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.199285 (+   156us) service_pool.cc:224] Handling call
0709 12:23:46.295028 (+2095743us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54003fc854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:46.295173 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":456928512}
I20260709 12:23:46.283535  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.301463  1804 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:46.303262  1877 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:46.260361  6117 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:46.293460  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.305300  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:46.307585  6117 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:46.287735  6144 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:46.271924  6109 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:46.316429  1801 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:46.300426  5999 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:46.304678  6059 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 12:23:46.319198  1877 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:23:46.319445  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:46.324573  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:46.325932  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:46.326831  1806 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:46.317793  6144 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:46.304536  5377 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:23:46.328974  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:46.322266  5999 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:46.331845  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2113) took 1257 ms. Trace:
I20260709 12:23:46.336784  1881 rpcz_store.cc:276] 0709 12:23:45.074307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.389856 (+315549us) service_pool.cc:224] Handling call
0709 12:23:45.407954 (+ 18098us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.796888 (+388934us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.331569 (+534681us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b5400052154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:46.331777 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":889179008}
I20260709 12:23:46.332849  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1940) took 1445 ms. Trace:
I20260709 12:23:46.337903  1731 rpcz_store.cc:276] 0709 12:23:44.887647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.894657 (+  7010us) service_pool.cc:224] Handling call
0709 12:23:44.896044 (+  1387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.583410 (+687366us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.332558 (+749148us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b540044af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:46.332759 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1579285760}
I20260709 12:23:46.333817  6110 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:46.340291  6110 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:46.340796  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 971) took 1857 ms. Trace:
I20260709 12:23:46.319126  6074 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.341318  2413 rpcz_store.cc:276] 0709 12:23:44.483038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.652326 (+169288us) service_pool.cc:224] Handling call
0709 12:23:46.295165 (+1642839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.297658 (+  2493us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.340559 (+ 42901us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54003fc850. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:46.340723 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3725341952}
I20260709 12:23:46.296183  6163 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:46.345273  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.338760  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.335386  1801 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.346314  1878 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:23:46.349011  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.308784  6117 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.354703  1799 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:46.356164  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.365512  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:46.356185  6118 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.709s	user 0.006s	sys 0.000s
I20260709 12:23:46.362370  1801 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:46.367218  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:46.370322  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.367422  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.372073  1799 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:46.372225  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.373112  1801 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:46.378767  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.374435  6072 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:46.391070  6072 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.392481  1806 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:46.393350  6072 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.383141  6163 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:46.396049  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 702664us
I20260709 12:23:46.396544  6159 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:46.410507  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.397532  2417 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:46.398567  1802 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260709 12:23:46.399312  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.341485  6110 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.399655  6042 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.401295  6059 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:46.401757  1798 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:46.403000  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2018) took 1008 ms. Trace:
I20260709 12:23:46.403077  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:46.403901  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.427086  6065 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:46.428174  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.428364  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.403676  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.429863  1799 raft_consensus.cc:2376] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12. Re-sending same reply.
I20260709 12:23:46.404592  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.395946  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1947) took 1200 ms. Trace:
I20260709 12:23:46.408308  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:46.415743  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2019) took 1015 ms. Trace:
I20260709 12:23:46.432998  1802 rpcz_store.cc:276] 0709 12:23:45.400078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.397976 (+997898us) service_pool.cc:224] Handling call
0709 12:23:46.398248 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.411659 (+ 13411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.415694 (+  4035us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.432516  1723 rpcz_store.cc:276] 0709 12:23:45.195667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.221607 (+ 25940us) service_pool.cc:224] Handling call
0709 12:23:45.239330 (+ 17723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.239596 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:45.377130 (+137534us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.713553 (+336423us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.850758 (+137205us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b18002b4fe0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:46.180867 (+330109us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b18002b4fe0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:46.395439 (+214572us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b18002b4fe0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:23:46.395589 (+   150us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.395637 (+    48us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:46.395690 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.395756 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.395880 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":551215,"prepare.queue_time_us":174494,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":109,"raft.queue_time_us":682175,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":8429,"spinlock_wait_cycles":50195456,"thread_start_us":437036,"threads_started":2,"wal-append.queue_time_us":373366}
I20260709 12:23:46.436452  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.438344  2413 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:46.440470  1798 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:46.431867  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:46.424826  1806 rpcz_store.cc:276] 0709 12:23:45.394536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.387441 (+992905us) service_pool.cc:224] Handling call
0709 12:23:46.392261 (+  4820us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.402550 (+ 10289us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.402807 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.444557  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.447321  1802 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:46.448026  1731 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:46.449153  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.450417  1802 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:23:46.450292  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.451429  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.452342  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.454931  1726 raft_consensus.cc:2368] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10.
I20260709 12:23:46.453650  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.455579  1729 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:23:46.447386  6159 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:46.456923  6027 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:46.428319  5767 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:46.457440  6159 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:46.458134  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.445183  5785 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:46.459337  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.456828  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.460129  1875 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:46.460462  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.461329  1732 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:23:46.461176  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.462447  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.463569  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.463752  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.457504  6105 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:46.464679  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.464802  6057 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:23:46.457180  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.465837  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.466347  1875 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:46.466991  6105 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:46.468710  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.470320  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.456821  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.472067  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.472237  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.473269  1731 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:46.473228  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.475541  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.476538  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.422544  6172 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:46.479432  1802 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:23:46.484903  1875 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:46.488613  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.489835  1875 raft_consensus.cc:2410] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 12 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 10.
W20260709 12:23:46.481666  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78050.7us
I20260709 12:23:46.478531  5499 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:23:46.502583  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.470271  5903 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.486891  6105 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:46.503756  5903 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.504509  6105 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.504990  6105 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.506321  5903 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:46.507221  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.507419  1726 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:46.508631  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.509387  1875 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:23:46.510331  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.512322  1802 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:46.513152  1875 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:46.513864  1802 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:23:46.514083  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:46.514986  6157 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.515187  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.515838  1875 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:46.515920  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.516816  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.413928  5988 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:23:46.507126  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.519291  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.521904  2412 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:46.515587  6157 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:46.520939  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 68425us
I20260709 12:23:46.526293  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2015) took 1451 ms. Trace:
W20260709 12:23:46.545094  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 132371us
I20260709 12:23:46.545940  1803 rpcz_store.cc:276] 0709 12:23:45.075398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.695609 (+620211us) service_pool.cc:224] Handling call
0709 12:23:46.526046 (+830437us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b5400167bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:46.526228 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37547264}
I20260709 12:23:46.527304  1799 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:46.496649  6059 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:23:46.547314  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74159.4us
I20260709 12:23:46.548600  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:46.529678  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 97881.6us
I20260709 12:23:46.534992  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.534507  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.551339  1724 raft_consensus.cc:2410] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 12 because replica has last-logged OpId of term: 10 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 10.
I20260709 12:23:46.538864  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.530767  1802 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:46.485160  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.552196  1731 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:23:46.423694  6042 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.496315  6180 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:46.555655  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.558252  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.560567  1731 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:46.562166  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.470433  5999 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.580715  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:46.529119  5785 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:46.575765  1800 raft_consensus.cc:2410] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260709 12:23:46.531945  6069 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:46.585518  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.585502  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.548413  6048 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:46.586050  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.544276  5834 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:46.586608  1727 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:46.554236  6065 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:46.591604  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.591670  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.580488  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.605566  2415 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:46.598094  1731 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:46.606703  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:46.594551  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.604508  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.598105  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.529011  6172 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:23:46.554584  6059 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:46.627657  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.628430  1727 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:46.590735  6179 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:46.615000  2412 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 5.
W20260709 12:23:46.614619  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98762.3us
I20260709 12:23:46.615501  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.616631  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.599186  6029 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:46.576201  6044 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.038s
I20260709 12:23:46.630998  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.592321  6048 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.635535  1730 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:46.647152  6048 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:46.647856  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.629935  1770 leader_election.cc:400] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10.
I20260709 12:23:46.648630  1770 leader_election.cc:403] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:46.649304  6027 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10.
I20260709 12:23:46.650104  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.635842  1731 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:46.651036  6027 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:46.651415  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.583249  6174 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:23:46.600885  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:46.652436  6027 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:46.653101  6027 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.581662  5767 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:23:46.653596  6027 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.653753  5875 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:46.655551  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.658972  5767 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:46.660009  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.661005  5767 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:46.616472  6142 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:46.613660  6084 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:46.637463  6065 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:46.590829  6026 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:23:46.585074  6138 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:23:46.663111  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:46.636425  6209 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.269s	user 0.002s	sys 0.000s
I20260709 12:23:46.664959  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.665220  2412 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:46.616806  6180 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:46.599601  5557 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:23:46.637333  6179 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:46.675674  6210 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.228s	user 0.001s	sys 0.000s
I20260709 12:23:46.643790  6172 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:46.606329  6049 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.058s
I20260709 12:23:46.645767  6043 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:23:46.632541  6053 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:46.614348  6069 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:46.663851  5903 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.049s
I20260709 12:23:46.690580  1881 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 5.
I20260709 12:23:46.686213  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.645516  6135 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20260709 12:23:46.701198  1726 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 4. Current term is 5. Ops: []
I20260709 12:23:46.689394  2411 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:23:46.695755  6142 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:46.705621  1875 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:23:46.714073  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.730626  1803 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:46.732941  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.780900  2417 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:46.734056  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:46.736109  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:46.719355  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:46.782886  1802 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:23:46.633885  6059 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:46.755059  2415 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260709 12:23:46.762202  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.756958  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:46.704119  6042 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:46.674739  6213 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.637027  6077 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.108s
I20260709 12:23:46.678311  5377 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:46.721856  6065 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.704282  5876 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:46.680949  6074 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:46.676717  5999 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.690366  6057 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.818485  6057 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.785600  5767 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:46.821115  6057 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:46.821782  6057 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:46.822193  6057 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:46.818518  5999 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:46.806198  5377 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.823863  6221 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.824482  5377 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:46.786589  6029 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:23:46.773203  6048 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:46.693687  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.682292  6109 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.701393  1877 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:46.704286  6163 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:46.781937  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 95805.1us
W20260709 12:23:46.782729  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63532.8us
W20260709 12:23:46.744248  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 79370.6us
I20260709 12:23:46.784883  1772 leader_election.cc:400] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10.
I20260709 12:23:46.890883  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:23:46.891906  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 107119us
I20260709 12:23:46.748018  6084 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:46.751314  6044 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:46.756652  6174 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.757357  6179 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:46.794358  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 60389.8us
I20260709 12:23:46.907086  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:46.800374  6148 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:46.713630  6027 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:46.765980  5988 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:46.763031  6172 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:46.771641  6180 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:46.775278  6141 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.812741  5876 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:46.812922  6065 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:46.926712  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.817363  6222 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.817173  6074 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.775527  5785 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:46.830749  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 1997) took 3172 ms (client timeout 2226 ms). Trace:
W20260709 12:23:46.932045  1807 rpcz_store.cc:269] 0709 12:23:43.658670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.791361 (+132691us) service_pool.cc:224] Handling call
0709 12:23:46.830499 (+3039138us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b54001eaad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:46.830666 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5339337472}
I20260709 12:23:46.837749  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2737) took 1192 ms. Trace:
I20260709 12:23:46.931416  6141 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.933447  1875 rpcz_store.cc:276] 0709 12:23:45.645270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.528963 (+883693us) service_pool.cc:224] Handling call
0709 12:23:46.837513 (+308550us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b5400222754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:46.837680 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":358120192}
I20260709 12:23:46.918131  6116 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:46.935103  6116 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.935344  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.935870  6116 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:46.936879  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.831440  1801 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 6. (term mismatch)
I20260709 12:23:46.937682  1801 pending_rounds.cc:85] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 5
I20260709 12:23:46.937963  1801 pending_rounds.cc:107] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:23:46.938196  1801 raft_consensus.cc:2889] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:46.831351  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2725) took 2904 ms (client timeout 2694 ms). Trace:
I20260709 12:23:46.938995  6218 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:46.939353  1872 rpcz_store.cc:269] 0709 12:23:43.926557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.926730 (+   173us) service_pool.cc:224] Handling call
0709 12:23:46.831061 (+2904331us) spinlock_profiling.cc:243] Waited 2.08 s on lock 0x7b54002058d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:46.831235 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4794986240}
I20260709 12:23:46.940292  6074 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:46.939845  6218 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.940546  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:46.839936  6142 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:46.942214  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2114) took 1867 ms. Trace:
W20260709 12:23:46.773182  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 217627us
I20260709 12:23:46.943257  6180 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:46.866041  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 231095us
I20260709 12:23:46.912467  6174 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.945430  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:46.946457  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.948091  2411 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:46.808531  6213 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.949172  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:46.795351  5903 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260709 12:23:46.931424  6144 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.590s	user 0.008s	sys 0.004s
I20260709 12:23:46.951175  6144 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.951890  6048 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.918285  6049 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 12:23:46.953473  6144 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:46.954807  6172 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:46.956542  6172 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:46.911435  6148 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.959748  6148 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:46.960930  6220 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:46.961872  6220 consensus_queue.cc:1243] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:23:46.824723  6221 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.890585  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:46.964226  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73802us
I20260709 12:23:46.822614  6057 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:46.965512  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:46.918457  6043 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:23:46.967427  5876 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.968510  5876 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:46.968554  6027 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:46.834563  6002 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.538s	user 0.008s	sys 0.000s
I20260709 12:23:46.802181  6219 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:46.917495  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:46.828467  6069 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.836563  6157 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.970574  6222 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.931285  6105 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.970557  6002 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.973348  6059 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.974406  2413 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:46.904381  5487 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:46.975057  6002 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:46.975272  5487 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:46.975888  2410 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:46.976722  5487 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.978286  6144 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:46.942546  1872 rpcz_store.cc:276] 0709 12:23:45.074606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.941472 (+1866866us) service_pool.cc:224] Handling call
0709 12:23:46.941734 (+   262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.941919 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.941944 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.941984 (+    40us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:46.942025 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.942074 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.942167 (+    93us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.979511  6126 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:46.980777  1872 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:46.892860  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:46.959143  1878 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:46.921025  5988 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:46.902998  6163 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:46.981446  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2732) took 1344 ms. Trace:
I20260709 12:23:47.037946  1729 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:47.038368  1872 rpcz_store.cc:276] 0709 12:23:45.637408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.980213 (+1342805us) service_pool.cc:224] Handling call
0709 12:23:46.980588 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.981194 (+   606us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.981388 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:47.015370  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70002.1us
I20260709 12:23:46.982039  1875 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
W20260709 12:23:47.040623  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 75200.3us
W20260709 12:23:47.041808  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 149071us
I20260709 12:23:46.988308  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2736) took 1349 ms. Trace:
I20260709 12:23:47.052414  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:47.042965  1877 rpcz_store.cc:276] 0709 12:23:45.638476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.987077 (+1348601us) service_pool.cc:224] Handling call
0709 12:23:46.987628 (+   551us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.987884 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.987922 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:46.987981 (+    59us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:46.988031 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.988101 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:46.988231 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:47.057248  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:47.089519  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45849us
I20260709 12:23:46.976133  6059 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:47.093119  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.021s	sys 0.037s
I20260709 12:23:47.093536  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:47.110417  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.021s	sys 0.037s
I20260709 12:23:47.101413  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2099) took 2752 ms. Trace:
I20260709 12:23:47.102150  1872 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:23:47.112572  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2740) took 1351 ms. Trace:
I20260709 12:23:47.113209  1872 rpcz_store.cc:276] 0709 12:23:45.761602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.057151 (+1295549us) service_pool.cc:224] Handling call
0709 12:23:47.057512 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.112217 (+ 54705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.112495 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:46.999481  6042 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:46.981689  6144 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:47.103520  1770 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:47.057660  5819 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:46.981328  6126 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:47.116073  1876 rpcz_store.cc:276] 0709 12:23:44.349179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.815089 (+465910us) service_pool.cc:224] Handling call
0709 12:23:44.834566 (+ 19477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.834920 (+   354us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:45.005513 (+170593us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.757264 (+1751751us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.100974 (+343710us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:47.101032 (+    58us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:47.101085 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.101220 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.101350 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3244396,"prepare.queue_time_us":1726968,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":94,"raft.queue_time_us":128228,"raft.run_cpu_time_us":396,"raft.run_wall_time_us":44274,"thread_start_us":598976,"threads_started":4,"wal-append.queue_time_us":387440,"wal-append.run_cpu_time_us":2392,"wal-append.run_wall_time_us":1251529}
I20260709 12:23:47.122506  1877 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:47.058153  6109 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:47.058349  6132 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:47.142638  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.144484  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:47.145123  1726 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:47.149564  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:47.150271  1723 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:23:47.156014  1726 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:23:47.157032  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:47.158213  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:47.159730  6117 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s	user 0.020s	sys 0.015s
I20260709 12:23:47.140025  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2746) took 1374 ms. Trace:
I20260709 12:23:47.162688  1877 rpcz_store.cc:276] 0709 12:23:45.765574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.098444 (+1332870us) service_pool.cc:224] Handling call
0709 12:23:47.122038 (+ 23594us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.139631 (+ 17593us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.139937 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:47.163790  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2101) took 2814 ms. Trace:
I20260709 12:23:47.164458  6043 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:47.163856  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 111605us
I20260709 12:23:47.164348  1874 rpcz_store.cc:276] 0709 12:23:44.349626 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.884019 (+534393us) service_pool.cc:224] Handling call
0709 12:23:44.884593 (+   574us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.894341 (+  9748us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:44.950465 (+ 56124us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.424529 (+474064us) spinlock_profiling.cc:243] Waited 877 us on lock 0x7b18003264a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:46.719054 (+1294525us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.163404 (+444350us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:47.163465 (+    61us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:47.163523 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.163607 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.163731 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3264265,"prepare.queue_time_us":1732557,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":102,"raft.queue_time_us":44559,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":21743,"spinlock_wait_cycles":2016640,"thread_start_us":643866,"threads_started":3,"wal-append.queue_time_us":258146,"wal-append.run_cpu_time_us":2919,"wal-append.run_wall_time_us":1367589}
I20260709 12:23:47.167476  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:47.162849  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.016s	sys 0.008s
I20260709 12:23:47.168660  1807 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:47.169190  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:47.161962  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:47.168730  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.016s	sys 0.008s
W20260709 12:23:47.171789  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 520412us
I20260709 12:23:47.172000  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:47.177510  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 232604us
W20260709 12:23:47.179615  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.114827  6049 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:46.986702  6065 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:47.180522  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:47.179641  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:47.182924  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:47.093374  6057 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:47.093936  6157 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:23:47.121393  6105 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:47.119663  6042 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:47.133630  6044 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:47.193754  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:47.196005  1802 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:47.197909  1876 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:47.205878  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 558) took 1191 ms. Trace:
I20260709 12:23:47.206372  1876 rpcz_store.cc:276] 0709 12:23:46.014591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.175103 (+1160512us) service_pool.cc:224] Handling call
0709 12:23:47.189403 (+ 14300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.205586 (+ 16183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.205815 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:47.207746  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:47.199048  1803 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:47.129431  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:47.213975  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:47.175987  5819 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:47.175544  6040 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:47.218808  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2091) took 2873 ms. Trace:
I20260709 12:23:47.219373  1877 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:47.216718  1807 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:47.221628  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:47.185740  6059 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:47.229672  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:47.231405  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:47.186918  6110 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:47.216998  1874 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:47.141445  6027 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.078s
I20260709 12:23:47.261013  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:47.262138  2411 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:47.219306  1880 rpcz_store.cc:276] 0709 12:23:44.345153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.757316 (+412163us) service_pool.cc:224] Handling call
0709 12:23:44.757667 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.757891 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:44.925543 (+167652us) log.cc:844] Serialized 14 byte log entry
0709 12:23:46.711593 (+1786050us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b1800326440. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf 00007f16abdf2038
0709 12:23:46.711837 (+   244us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.218308 (+506471us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:47.218374 (+    66us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:47.218452 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.218606 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.218732 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38945,"prepare.queue_time_us":1848922,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":1259,"raft.queue_time_us":12437,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":12045,"spinlock_wait_cycles":39982336,"thread_start_us":784863,"threads_started":3,"wal-append.queue_time_us":229239,"wal-append.run_cpu_time_us":3234,"wal-append.run_wall_time_us":1182337}
I20260709 12:23:47.224216  5979 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:47.249666  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:47.277343  1874 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:47.165544  6117 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.021s	sys 0.016s
I20260709 12:23:47.226897  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:47.188190  6049 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:47.255532  6144 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:47.282637  2415 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:47.288065  6218 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:47.289499  5819 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.227244  5834 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:47.254156  1723 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:47.251268  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:47.202466  6044 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.271867  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:47.223577  6109 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:47.248174  6219 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:47.300166  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:47.247898  6002 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:47.328959  6002 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.329427  6002 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:47.282469  6049 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:47.247025  6223 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.119762  6084 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.387185  6163 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:47.393321  6223 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:47.317198  6044 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:47.306202  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:47.316057  6217 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:23:47.267740  6174 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:47.316546  5760 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.259805  6220 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:23:47.317248  6218 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:23:47.328310  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:47.332962  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61228.9us
I20260709 12:23:47.416365  1880 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:47.294457  6159 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:23:47.336850  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1847) took 4092 ms (client timeout 2999 ms). Trace:
W20260709 12:23:47.213001  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 83717us
I20260709 12:23:47.236038  6040 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.429724  6065 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:47.235371  5377 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.431995  5377 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:47.375929  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.338924  6226 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:47.304677  6042 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.406279  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:47.415128  6220 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.436604  6220 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:47.436512  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:47.423645  1800 rpcz_store.cc:269] 0709 12:23:43.244343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.434248 (+189905us) service_pool.cc:224] Handling call
0709 12:23:47.331497 (+3897249us) spinlock_profiling.cc:243] Waited 3.11 s on lock 0x7b54003dbdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:47.336752 (+  5255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7150095616}
I20260709 12:23:47.336583  6109 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:47.437887  1802 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:23:47.438316  1802 pending_rounds.cc:85] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 7
I20260709 12:23:47.438525  1802 pending_rounds.cc:107] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:23:47.423328  6159 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:47.438742  1802 raft_consensus.cc:2889] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:47.439157  6213 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:47.439551  6159 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:47.441010  1803 pending_rounds.cc:85] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 7
I20260709 12:23:47.441131  6159 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:47.414512  5760 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:47.429505  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:47.435091  1876 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:23:47.187656  5988 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.441468  1803 pending_rounds.cc:107] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 12:23:47.452090  2415 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:47.453528  1803 raft_consensus.cc:2889] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:47.402372  6049 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.452314  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:47.460845  2411 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
W20260709 12:23:47.454659  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.455426  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:47.424048  6227 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.106s	user 0.001s	sys 0.001s
I20260709 12:23:47.539099  1807 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:23:47.527701  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.539817  1807 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:47.439100  6223 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:47.504371  5785 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.012s	sys 0.047s
W20260709 12:23:47.507709  6217 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
W20260709 12:23:47.566431  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:47.513291  6160 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.002s	sys 0.018s
I20260709 12:23:47.501797  6222 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:47.559101  1807 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:47.629326  2384 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.014s	sys 0.050s
W20260709 12:23:47.629771  2384 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.014s	sys 0.050s
W20260709 12:23:47.631765  6232 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
W20260709 12:23:47.561317  6089 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.015s	sys 0.006s
I20260709 12:23:47.549898  5988 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:47.633172  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:47.638563  2411 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:47.551523  6065 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:47.502313  6218 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.166s
I20260709 12:23:47.636569  1807 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:47.563680  5785 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.013s	sys 0.048s
I20260709 12:23:47.508761  6049 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.632339  6117 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.559278  6132 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:47.566622  5581 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:47.559010  6221 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:47.650012  2412 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:47.636893  6089 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.015s	sys 0.007s
I20260709 12:23:47.633183  6057 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:47.438555  6109 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:47.628216  6160 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.002s	sys 0.018s
I20260709 12:23:47.669063  1723 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:47.672362  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 553) took 1450 ms. Trace:
I20260709 12:23:47.672976  1799 rpcz_store.cc:276] 0709 12:23:46.222284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.444508 (+222224us) service_pool.cc:224] Handling call
0709 12:23:46.527127 (+ 82619us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.547647 (+ 20520us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.672125 (+1124478us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b5400167bd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:47.672295 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72964480}
I20260709 12:23:47.683234  2411 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:47.656213  6132 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:47.696179  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 484933us
W20260709 12:23:47.700909  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:47.703068  1873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.006s	sys 0.017s
W20260709 12:23:47.705586  1873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.006s	sys 0.017s
W20260709 12:23:47.703727  1729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.004s	sys 0.019s
W20260709 12:23:47.707933  1729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.004s	sys 0.019s
I20260709 12:23:47.663040  5581 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:47.715677  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2111) took 2642 ms. Trace:
I20260709 12:23:47.648478  6065 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.716301  1873 rpcz_store.cc:276] 0709 12:23:45.073622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.182366 (+108744us) service_pool.cc:224] Handling call
0709 12:23:45.182625 (+   259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.182871 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:47.129799 (+1946928us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.711666 (+581867us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:47.715222 (+  3556us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:47.715295 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.715394 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:47.715580 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1623963,"prepare.queue_time_us":1676907,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":72,"raft.queue_time_us":8937,"thread_start_us":903559,"threads_started":2}
W20260709 12:23:47.724858  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 321571us
I20260709 12:23:47.725390  2411 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:47.725981  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:47.661551  6117 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.722366  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:47.730924  2412 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
W20260709 12:23:47.719550  6234 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.377s	user 0.001s	sys 0.000s
W20260709 12:23:47.731962  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.013s	sys 0.017s
I20260709 12:23:47.745728  2417 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:47.754753  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:47.760049  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.015s	sys 0.023s
I20260709 12:23:47.753810  1772 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:47.720795  6089 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:47.717641  6212 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:47.733712  6148 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.773s	user 0.013s	sys 0.004s
W20260709 12:23:47.734293  6179 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.809s	user 0.011s	sys 0.002s
I20260709 12:23:47.778574  2412 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:47.772632  6142 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:47.775267  6148 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.785241  6148 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:47.785737  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:47.787549  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:47.788949  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:47.789209  1873 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:47.790179  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:47.792146  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:47.791172  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:47.794389  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:47.794448  6142 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260709 12:23:47.765251  6089 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:47.760753  5785 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 12:23:47.772617  5760 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:47.811666  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2750) took 1759 ms (client timeout 1992 ms). Trace:
I20260709 12:23:47.893028  1725 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
W20260709 12:23:47.888170  1878 rpcz_store.cc:269] 0709 12:23:46.052048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.399875 (+347827us) service_pool.cc:224] Handling call
0709 12:23:46.713832 (+313957us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:23:47.811394 (+1097562us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:47.811555 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":423795968}
I20260709 12:23:47.800158  1873 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:47.895951  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:47.896948  1880 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:47.898239  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:47.896881  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:47.897150  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:47.813337  6172 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:47.771878  6212 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:47.913058  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.903689  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:47.901577  1880 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:47.934345  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:47.902576  6172 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:47.901705  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:47.809646  6241 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 12:23:47.776073  6179 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:47.859906  6074 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:47.828244  6245 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:23:47.812321  6163 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:47.938138  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:47.940666  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:47.910043  6240 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
I20260709 12:23:48.007529  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:47.809054  6233 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260709 12:23:47.899305  6221 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:47.815318  6229 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
W20260709 12:23:47.935298  6116 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.999s	user 0.015s	sys 0.000s
I20260709 12:23:47.899551  6218 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:47.831374  6249 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.831089  6050 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:47.891245  6057 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.904160  5499 raft_consensus.cc:2764] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:23:48.008093  6049 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:47.823673  6224 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:47.903748  6065 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:47.912299  6117 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:47.996362  1880 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:48.043502  6223 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:47.911882  6248 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:48.003029  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:48.070204  1725 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:47.903945  5581 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:48.010265  1727 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:48.005959  6163 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:48.048861  6040 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:48.014587  6251 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:23:48.061317  6224 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:48.080271  6224 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.081723  6224 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:47.941636  5891 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.076s
I20260709 12:23:48.034394  6221 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:48.084280  6220 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.647s	user 0.010s	sys 0.004s
I20260709 12:23:48.084784  6220 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.085131  6221 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:48.024600  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86565.3us
I20260709 12:23:48.052606  6050 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:48.086910  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:48.087733  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.087531  6050 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.018919  6259 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:48.088385  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:48.020742  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2919) took 1094 ms. Trace:
W20260709 12:23:48.038606  6247 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.291s	user 0.000s	sys 0.003s
I20260709 12:23:48.089095  1724 rpcz_store.cc:276] 0709 12:23:46.929433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.929615 (+   182us) service_pool.cc:224] Handling call
0709 12:23:46.930131 (+   516us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:47.013823 (+ 83692us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:47.013882 (+    59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:47.063594 (+ 49712us) log.cc:844] Serialized 52 byte log entry
0709 12:23:47.404439 (+340845us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:47.404506 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.404593 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:48.020429 (+615836us) raft_consensus.cc:1685] finished
0709 12:23:48.020455 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.020669 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22794,"raft.queue_time_us":84135,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":39188,"thread_start_us":286294,"threads_started":1,"wal-append.queue_time_us":570306}
I20260709 12:23:47.903035  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:48.091403  1878 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:48.092139  1724 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:48.088853  6259 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:48.093853  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.093444  6259 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.094386  6040 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:48.046770  6002 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.717s	user 0.008s	sys 0.004s
W20260709 12:23:47.938313  6044 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.523s	user 0.012s	sys 0.001s
I20260709 12:23:48.008275  6089 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.102918  6089 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:48.048346  6116 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.054016  6249 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.112828  6116 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:48.113420  6249 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:48.115226  1876 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:48.095367  6213 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:48.095887  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:48.095654  6044 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.117982  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:48.095053  6040 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:48.120054  6040 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.095191  6002 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.123574  6044 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:48.095950  6259 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:48.125327  6002 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:48.094734  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.066591  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:48.005200  6074 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.012607  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:48.158845  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 64204.8us
I20260709 12:23:48.159508  1873 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
W20260709 12:23:48.055380  6126 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.915s	user 0.015s	sys 0.000s
I20260709 12:23:48.075809  6248 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.053114  6218 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:47.941301  6256 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:48.088766  6050 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:48.044631  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 158155us
I20260709 12:23:48.089061  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 556) took 2206 ms. Trace:
I20260709 12:23:48.306756  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:48.157413  1724 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:23:48.059347  6047 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:48.161042  1873 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:48.059429  6057 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:48.279819  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.297312  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 569) took 1788 ms. Trace:
I20260709 12:23:48.298622  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 906) took 1903 ms. Trace:
W20260709 12:23:48.302284  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 141167us
I20260709 12:23:48.330108  2410 rpcz_store.cc:276] 0709 12:23:46.394618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.394765 (+   147us) service_pool.cc:224] Handling call
0709 12:23:46.395148 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.976396 (+581248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.298140 (+1321744us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b5400561b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:48.298455 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1335065856}
I20260709 12:23:48.129478  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:48.308542  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2762) took 1901 ms. Trace:
I20260709 12:23:48.332119  1873 rpcz_store.cc:276] 0709 12:23:46.407501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.160356 (+1752855us) service_pool.cc:224] Handling call
0709 12:23:48.160802 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.308129 (+147327us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.308449 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:48.333886  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1869) took 3382 ms. Trace:
W20260709 12:23:48.333978  2414 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 904) took 2125 ms (client timeout 1977 ms). Trace:
W20260709 12:23:48.334753  6215 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.516s	user 0.012s	sys 0.008s
W20260709 12:23:48.334637  2414 rpcz_store.cc:269] 0709 12:23:46.208259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.257743 (+ 49484us) service_pool.cc:224] Handling call
0709 12:23:48.333749 (+2076006us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540048a3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:48.333886 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2334436736}
W20260709 12:23:48.335467  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 206100us
I20260709 12:23:48.127334  6132 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.066843  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:48.165433  5499 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:48.311545  6049 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:48.312337  6047 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.316975  6218 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:48.317461  6239 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:48.066066  6172 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.322932  6002 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.325758  6074 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:48.339275  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.334326  1801 rpcz_store.cc:276] 0709 12:23:44.951347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.378921 (+1427574us) service_pool.cc:224] Handling call
0709 12:23:46.830986 (+452065us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.938660 (+107674us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.333683 (+1395023us) spinlock_profiling.cc:243] Waited 452 ms on lock 0x7b54001eaad0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:48.333829 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1038795136}
I20260709 12:23:48.164074  6213 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:23:48.345702  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 278939us
I20260709 12:23:48.301522  5834 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:23:48.291847  6117 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:48.331391  1731 rpcz_store.cc:276] 0709 12:23:46.508610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.651834 (+143224us) service_pool.cc:224] Handling call
0709 12:23:46.655277 (+  3443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.673046 (+ 17769us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.673092 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:47.766016 (+1092924us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b2c0005a338. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:23:47.773697 (+  7681us) log.cc:844] Serialized 26 byte log entry
0709 12:23:48.051655 (+277958us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:48.068239 (+ 16584us) log.cc:844] Serialized 14 byte log entry
0709 12:23:48.069375 (+  1136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.069458 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:48.157958 (+ 88500us) raft_consensus.cc:1685] finished
0709 12:23:48.297042 (+139084us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.297233 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20747,"prepare.queue_time_us":19783,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":65,"raft.queue_time_us":18392,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"spinlock_wait_cycles":113551616,"thread_start_us":268001,"threads_started":1,"wal-append.queue_time_us":317057}
I20260709 12:23:48.353533  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 560) took 2312 ms. Trace:
I20260709 12:23:48.305704  1872 rpcz_store.cc:276] 0709 12:23:45.882914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.115431 (+1232517us) service_pool.cc:224] Handling call
0709 12:23:47.115833 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.141871 (+ 26038us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:47.175932 (+ 34061us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.500664 (+324732us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.558230 (+ 57566us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:47.558301 (+    71us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:47.558362 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.037649 (+479287us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:48.088811 (+ 51162us) spinlock_profiling.cc:243] Waited 25.7 ms on lock 0x7b5400223654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:48.088860 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.089002 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19403,"prepare.queue_time_us":16364,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":81522,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":94465,"spinlock_wait_cycles":129682304,"thread_start_us":279797,"threads_started":1,"wal-append.queue_time_us":526319}
I20260709 12:23:48.348438  6077 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:48.371901  1875 rpcz_store.cc:276] 0709 12:23:46.041488 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.936601 (+895113us) service_pool.cc:224] Handling call
0709 12:23:48.353202 (+1416601us) spinlock_profiling.cc:243] Waited 44.4 ms on lock 0x7b5400073354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:48.353426 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102009728}
I20260709 12:23:48.375586  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2125) took 1170 ms. Trace:
I20260709 12:23:48.351068  5760 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:48.378934  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.359666  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:48.376245  1872 rpcz_store.cc:276] 0709 12:23:47.205734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.374354 (+1168620us) service_pool.cc:224] Handling call
0709 12:23:48.374803 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.375085 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:48.375126 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:48.375190 (+    64us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:48.375251 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.375326 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.375507 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:48.387892  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:48.354183  6220 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:48.382567  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50972.7us
W20260709 12:23:48.393016  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.389243  1872 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:48.394487  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:48.405567  1875 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:23:48.406320  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2124) took 1208 ms. Trace:
I20260709 12:23:48.406724  1875 rpcz_store.cc:276] 0709 12:23:47.198038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.373562 (+1175524us) service_pool.cc:224] Handling call
0709 12:23:48.393900 (+ 20338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.406055 (+ 12155us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.406262 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:48.407600  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1878) took 2182 ms. Trace:
I20260709 12:23:48.407699  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:48.407908  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:48.408041  1806 rpcz_store.cc:276] 0709 12:23:46.225328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.443778 (+218450us) service_pool.cc:224] Handling call
0709 12:23:46.480109 (+ 36331us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:46.480685 (+   576us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.480715 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:46.511249 (+ 30534us) log.cc:844] Serialized 52 byte log entry
0709 12:23:46.667869 (+156620us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:46.667924 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.668404 (+   480us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:48.334057 (+1665653us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:48.390871 (+ 56814us) raft_consensus.cc:1685] finished
0709 12:23:48.390915 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.407535 (+ 16620us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9860,"raft.queue_time_us":6271,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":139,"spinlock_wait_cycles":9467136,"thread_start_us":155133,"threads_started":1,"wal-append.queue_time_us":307378}
I20260709 12:23:48.356603  6249 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:23:48.408859  1875 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:48.414780  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:48.374934  5785 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:48.376731  6077 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.361990  6221 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:23:48.392102  6074 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:48.362169  6256 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 12:23:48.369221  6223 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:23:48.396482  5760 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:48.438210  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:48.368194  6002 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.439200  1875 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:48.397396  6179 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:48.411303  6057 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:48.432206  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:48.444793  1872 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:48.420935  6077 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:48.409847  6266 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260709 12:23:48.453814  1875 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:48.454759  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2127) took 1235 ms. Trace:
I20260709 12:23:48.455245  1875 rpcz_store.cc:276] 0709 12:23:47.219155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.441338 (+1222183us) service_pool.cc:224] Handling call
0709 12:23:48.453530 (+ 12192us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.454407 (+   877us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.454688 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:48.457161  1875 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:48.457914  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2129) took 1222 ms. Trace:
I20260709 12:23:48.410622  6070 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:23:48.468600  2411 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:48.458325  1875 rpcz_store.cc:276] 0709 12:23:47.237151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.456278 (+1219127us) service_pool.cc:224] Handling call
0709 12:23:48.456688 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.457601 (+   913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.457833 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:48.475606  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:48.476851  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.482911  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:48.483263  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:48.483876  2410 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 6: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:48.484371  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:48.485984  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:48.486513  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.423375  5785 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.487350  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:48.485119  2415 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:48.488343  2412 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:48.487828  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.471645  6239 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.471410  6220 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:48.488220  5988 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.842s	user 0.014s	sys 0.000s
W20260709 12:23:48.506611  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 562) took 2453 ms (client timeout 1950 ms). Trace:
W20260709 12:23:48.510109  1881 rpcz_store.cc:269] 0709 12:23:46.053058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.365223 (+312165us) service_pool.cc:224] Handling call
0709 12:23:48.506327 (+2141104us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b5400025b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:48.506503 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":745948416}
I20260709 12:23:48.511775  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:48.512509  1881 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:23:48.509411  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:48.487600  5834 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:48.491262  6221 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.518254  1872 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:23:48.508057  6085 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:48.530972  6248 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:48.520546  5834 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:48.531553  6047 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:48.531710  5834 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:48.520524  6117 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.535419  6117 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.525836  6221 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.547978  6221 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:48.549095  1875 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
W20260709 12:23:48.549938  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70248.9us
I20260709 12:23:48.550976  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.552580  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.553058  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.553920  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.561451  1881 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6.
I20260709 12:23:48.518980  6220 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:48.559919  6159 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.108s	user 0.010s	sys 0.006s
I20260709 12:23:48.566326  5731 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:48.569795  5785 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:48.556249  6085 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:48.554770  5988 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.575119  5988 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.577085  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 549) took 4403 ms. Trace:
I20260709 12:23:48.519239  6239 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.566131  6142 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:48.592543  6224 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.510s	user 0.012s	sys 0.002s
I20260709 12:23:48.592741  6280 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:23:48.570420  6002 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:48.581161  6218 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:48.577617  1879 rpcz_store.cc:276] 0709 12:23:44.174117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.174941 (+   824us) service_pool.cc:224] Handling call
0709 12:23:46.831413 (+2656472us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:46.831809 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.831841 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:46.832500 (+   659us) log.cc:844] Serialized 52 byte log entry
0709 12:23:47.216001 (+383501us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:47.307220 (+ 91219us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.338314 (+ 31094us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.338400 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:47.442982 (+104582us) raft_consensus.cc:1685] finished
0709 12:23:47.443001 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.576845 (+1133844us) spinlock_profiling.cc:243] Waited 2.63 s on lock 0x7b54002058d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:48.577003 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30778,"prepare.queue_time_us":100397,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":167699,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":262,"spinlock_wait_cycles":6057810816,"thread_start_us":382906,"threads_started":1,"wal-append.queue_time_us":580621,"wal-append.run_cpu_time_us":2698,"wal-append.run_wall_time_us":1028715}
I20260709 12:23:48.591089  6085 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:48.578209  5988 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:48.595983  6280 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:48.629351  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:48.631443  1879 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:48.570294  5834 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.635499  5834 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:48.606761  6218 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:48.645079  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:48.645659  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:48.646126  6218 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.646967  2419 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:48.647488  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:48.647811  6132 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:48.648313  2412 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:23:48.604033  6257 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.603s	user 0.005s	sys 0.005s
I20260709 12:23:48.608137  6281 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:48.608438  6077 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.609983  6220 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:48.573341  5731 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:48.649324  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:48.649699  6117 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:48.611369  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1966) took 1332 ms. Trace:
I20260709 12:23:48.616276  6239 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:48.651082  2410 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:23:48.650955  1723 rpcz_store.cc:276] 0709 12:23:47.279482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.436185 (+156703us) service_pool.cc:224] Handling call
0709 12:23:48.611175 (+1174990us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54003cc3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:48.611301 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":532827520}
I20260709 12:23:48.650204  5731 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.652658  6085 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.654711  6077 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:48.655704  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2759) took 2556 ms. Trace:
I20260709 12:23:48.656224  1877 rpcz_store.cc:276] 0709 12:23:46.100127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.163844 (+1063717us) service_pool.cc:224] Handling call
0709 12:23:47.164160 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.219842 (+ 55682us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.655449 (+1435607us) spinlock_profiling.cc:243] Waited 55 ms on lock 0x7b54001f8954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:48.655632 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":126540160}
I20260709 12:23:48.657358  6212 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:48.657527  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:48.658187  1877 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:23:48.659610  6141 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:48.664640  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:48.666625  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:48.667171  1874 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:48.667644  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.668558  1874 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:48.672991  6077 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:48.673415  6077 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.674835  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 911) took 1422 ms. Trace:
I20260709 12:23:48.675246  2417 rpcz_store.cc:276] 0709 12:23:47.252134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.260037 (+  7903us) service_pool.cc:224] Handling call
0709 12:23:47.260465 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.763532 (+503067us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.674582 (+911050us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54000e6d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:48.674748 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1115224960}
I20260709 12:23:48.672603  1874 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:48.681404  6077 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:48.673856  6259 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:23:48.686077  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:48.691821  6272 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.195s	user 0.003s	sys 0.000s
I20260709 12:23:48.693426  2417 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:48.707396  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 153533us
I20260709 12:23:48.711002  1874 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:23:48.712141  2412 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:48.718719  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:48.721873  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.723393  1874 raft_consensus.cc:2376] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 6. Re-sending same reply.
I20260709 12:23:48.724531  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:48.725272  1874 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:48.726905  1877 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:48.727927  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:48.711257  6218 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:48.728291  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:48.683557  6163 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.736249  6163 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:48.732473  6259 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:23:48.766163  6224 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:48.766625  6224 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:48.767555  6224 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.768874  6224 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:48.773921  6077 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.788342  2417 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:48.790413  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:48.791631  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:48.792374  1877 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:48.797591  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2917) took 1882 ms. Trace:
I20260709 12:23:48.798162  1727 rpcz_store.cc:276] 0709 12:23:46.915175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.915476 (+   301us) service_pool.cc:224] Handling call
0709 12:23:46.931252 (+ 15776us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:48.071209 (+1139957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.797210 (+726001us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400212ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:48.797511 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2475514240}
I20260709 12:23:48.815886  1877 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:48.829777  6148 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:23:48.865839  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.867995  6042 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:48.872159  6269 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.419s	user 0.004s	sys 0.000s
I20260709 12:23:48.855562  6148 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:48.874220  6148 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.876560  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:48.878880  2416 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:48.887755  6285 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:23:48.854147  6077 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:48.896210  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 565) took 2388 ms. Trace:
I20260709 12:23:48.896811  1730 rpcz_store.cc:276] 0709 12:23:46.507556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.647548 (+139992us) service_pool.cc:224] Handling call
0709 12:23:48.895874 (+2248326us) spinlock_profiling.cc:243] Waited 2.03 s on lock 0x7b54002aa154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:48.896119 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4664259456}
I20260709 12:23:48.886489  2416 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:48.899204  1728 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:48.900540  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:48.900797  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 1960) took 2181 ms. Trace:
I20260709 12:23:48.901496  6042 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:48.901281  1728 rpcz_store.cc:276] 0709 12:23:46.719819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.720023 (+   204us) service_pool.cc:224] Handling call
0709 12:23:48.898984 (+2178961us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.899730 (+   746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.900565 (+   835us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b54002aa150. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:48.900728 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4918244224}
I20260709 12:23:48.903007  6042 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:48.903471  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.904261  6042 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:48.911978  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.913386  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2119) took 2656 ms. Trace:
I20260709 12:23:48.915977  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2924) took 1030 ms. Trace:
I20260709 12:23:48.916509  1724 rpcz_store.cc:276] 0709 12:23:47.885876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.093167 (+207291us) service_pool.cc:224] Handling call
0709 12:23:48.093549 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:48.307610 (+214061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.898851 (+591241us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b5400327dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:48.915887 (+ 17036us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145996672}
I20260709 12:23:48.919857  6172 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:48.923314  6212 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:48.926216  1846 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.977s (SENT)
I20260709 12:23:48.926724  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:48.929370  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:48.929716  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:48.929900  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.930437  2416 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:23:48.948252  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.924031  6212 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:48.961824  1729 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:48.964351  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:48.966485  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:48.968886  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:48.969820  1723 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:48.972693  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:48.975836  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:48.977797  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:48.977845  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:48.980579  1730 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:48.925128  6089 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:48.981930  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:48.958999  6286 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:23:48.988430  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:48.988699  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76817.6us
I20260709 12:23:48.989507  1728 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:48.948110  6219 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:48.990897  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:48.991652  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:48.988490  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 62322us
I20260709 12:23:48.950433  6141 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.984153  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:48.987083  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 61805.5us
I20260709 12:23:48.993836  1725 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:23:48.993721  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:48.994401  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:48.983597  6148 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:48.980376  6283 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.001425  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:48.987054  6259 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.014256  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:49.023146  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:49.058264  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 64645.8us
I20260709 12:23:49.061551  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:48.994390  6085 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:48.996727  6144 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:49.064936  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50686.6us
W20260709 12:23:49.065527  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 58006.1us
I20260709 12:23:49.066454  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:49.070858  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2120) took 1899 ms. Trace:
I20260709 12:23:49.007057  6159 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:49.071291  1873 rpcz_store.cc:276] 0709 12:23:47.171856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.333065 (+1161209us) service_pool.cc:224] Handling call
0709 12:23:48.333362 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.392488 (+ 59126us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:48.392532 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:48.392595 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:48.392651 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.070553 (+677902us) spinlock_profiling.cc:243] Waited 58.9 ms on lock 0x7b540021b1d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:49.070602 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:49.070766 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135366528}
I20260709 12:23:49.081395  6132 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:23:49.072646  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71289.8us
I20260709 12:23:49.077126  1846 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:49.088451  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:49.065116  6221 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:49.006880  6084 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.073s
W20260709 12:23:49.135556  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45544.3us
I20260709 12:23:49.002880  6141 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.068044  6238 raft_consensus.cc:2764] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:23:49.083356  6159 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:49.142542  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58313.8us
I20260709 12:23:49.078583  6249 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:49.078084  6047 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:49.079001  6276 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:49.148185  1725 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:49.063301  6259 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.010250  6142 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.078s
I20260709 12:23:49.072971  6213 raft_consensus.cc:2764] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:23:49.197013  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:48.913937  1880 rpcz_store.cc:276] 0709 12:23:46.257288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.130851 (+1873563us) service_pool.cc:224] Handling call
0709 12:23:48.131290 (+   439us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:48.160350 (+ 29060us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:23:48.380224 (+219874us) log.cc:844] Serialized 14 byte log entry
0709 12:23:48.475250 (+ 95026us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:48.489145 (+ 13895us) log.cc:844] Serialized 52 byte log entry
0709 12:23:48.489380 (+   235us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:48.489636 (+   256us) log.cc:844] Serialized 14 byte log entry
0709 12:23:48.638275 (+148639us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x7b1800326620. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:48.876367 (+238092us) spinlock_profiling.cc:243] Waited 8.14 ms on lock 0x7b1800326620. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:48.902429 (+ 26062us) spinlock_profiling.cc:243] Waited 3.98 ms on lock 0x7b1800326620. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:23:48.902678 (+   249us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.912998 (+ 10320us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:48.913072 (+    74us) raft_consensus.cc:1685] finished
0709 12:23:48.913095 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.913301 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35755,"prepare.queue_time_us":292956,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":3601,"raft.queue_time_us":449655,"raft.run_cpu_time_us":481,"raft.run_wall_time_us":142032,"spinlock_wait_cycles":47822464,"thread_start_us":106953,"threads_started":2,"wal-append.queue_time_us":85493}
I20260709 12:23:49.009896  6148 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.062124  6283 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.139516  6132 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:49.159808  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.203317  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:49.082294  6163 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.070s
I20260709 12:23:49.205670  1880 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:49.158211  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:49.211920  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53787.1us
I20260709 12:23:49.162639  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:49.163090  6290 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
W20260709 12:23:49.165294  6271 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.660s	user 0.004s	sys 0.000s
I20260709 12:23:49.167215  6249 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.169955  6212 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:49.170967  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1876) took 2946 ms. Trace:
I20260709 12:23:49.171658  6266 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:49.175498  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:49.176159  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 1.4224e+06us
I20260709 12:23:49.199273  6225 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:49.179831  6105 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20260709 12:23:49.233255  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 57876.4us
I20260709 12:23:49.151834  6070 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:23:49.218576  1798 rpcz_store.cc:276] 0709 12:23:46.224826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.441441 (+216615us) service_pool.cc:224] Handling call
0709 12:23:46.484135 (+ 42694us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:46.501944 (+ 17809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.501990 (+    46us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:46.502899 (+   909us) log.cc:844] Serialized 78 byte log entry
0709 12:23:46.889015 (+386116us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:47.560847 (+671832us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.621418 (+ 60571us) log.cc:844] Serialized 14 byte log entry
0709 12:23:47.822433 (+201015us) spinlock_profiling.cc:243] Waited 2.88 ms on lock 0x7b180003c460. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:49.170353 (+1347920us) spinlock_profiling.cc:243] Waited 8.53 ms on lock 0x7b180003c460. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:23:49.170573 (+   220us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.170654 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:49.170708 (+    54us) raft_consensus.cc:1685] finished
0709 12:23:49.170722 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:49.170904 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1579685,"prepare.queue_time_us":535909,"prepare.run_cpu_time_us":140,"prepare.run_wall_time_us":3249,"raft.queue_time_us":1570263,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":58701,"spinlock_wait_cycles":26250368,"thread_start_us":857296,"threads_started":2,"wal-append.queue_time_us":644003,"wal-append.run_cpu_time_us":4546,"wal-append.run_wall_time_us":1614149}
I20260709 12:23:49.204545  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:49.208371  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.244052  1801 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:49.244769  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.245194  1801 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:49.246011  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.245858  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:49.217056  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:49.160382  6159 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:49.257632  2410 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:23:49.221616  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:49.195480  6065 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:49.177719  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 561) took 1740 ms. Trace:
I20260709 12:23:49.259179  6065 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.259228  1807 rpcz_store.cc:276] 0709 12:23:47.436864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.442818 (+  5954us) service_pool.cc:224] Handling call
0709 12:23:47.502523 (+ 59705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:49.177182 (+1674659us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:49.177233 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:49.177293 (+    60us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:49.177355 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.177471 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:49.177643 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1343631,"raft.queue_time_us":1353746,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":3406}
I20260709 12:23:49.260847  6065 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:49.261116  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.261422  6065 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.261845  6065 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.263029  6109 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:49.263640  6109 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.265192  6065 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.265980  6065 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.266233  6109 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.269855  1801 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:49.207818  6219 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:49.273952  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.231930  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:49.234262  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:49.262617  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:49.239380  1799 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:23:49.240470  6276 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:49.247009  1728 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:49.251048  1724 pending_rounds.cc:85] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 9
W20260709 12:23:49.254240  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49795.8us
I20260709 12:23:49.254817  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:49.220523  6069 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:49.281119  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46985.2us
I20260709 12:23:49.122169  6248 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:49.283419  1724 pending_rounds.cc:107] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:23:49.289532  1724 raft_consensus.cc:2889] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:49.275416  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.282747  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:49.236567  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:49.291761  1727 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 6: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:49.276301  2411 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:49.289987  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.218077  6157 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.279608  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:49.286108  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.315150  1798 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:49.277339  1801 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:49.317407  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.233462  6180 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.215840  6048 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.072s
I20260709 12:23:49.208074  6132 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.327925  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:49.279189  6072 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:23:49.279376  6179 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:49.336553  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.266556  6065 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.286643  5988 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:49.304957  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:49.331617  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:49.292338  6069 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.281683  5557 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:49.280596  6225 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:49.342043  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.344336  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.383639  1798 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:49.333808  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.303162  6138 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:49.320662  6218 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:23:49.330020  6285 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:23:49.274767  6163 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:23:49.325824  6048 consensus_queue.cc:1243] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 9.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:23:49.380335  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48800.5us
W20260709 12:23:49.388070  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.50602e+06us
I20260709 12:23:49.286226  6238 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.068s
W20260709 12:23:49.354233  6293 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:23:49.358718  6221 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:49.392750  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 87887.3us
I20260709 12:23:49.394078  1801 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:49.295121  6276 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.443014  6276 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:49.446844  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 558) took 2719 ms. Trace:
I20260709 12:23:49.447397  1805 rpcz_store.cc:276] 0709 12:23:46.727398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.727555 (+   157us) service_pool.cc:224] Handling call
0709 12:23:46.728019 (+   464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.745150 (+ 17131us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:47.116762 (+371612us) log.cc:844] Serialized 14 byte log entry
0709 12:23:49.171726 (+2054964us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:49.172582 (+   856us) log.cc:844] Serialized 26 byte log entry
0709 12:23:49.325191 (+152609us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:49.325241 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.325327 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:49.446529 (+121202us) raft_consensus.cc:1685] finished
0709 12:23:49.446554 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:49.446740 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1530438,"prepare.queue_time_us":510977,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":20407,"raft.queue_time_us":1532395,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":5039,"thread_start_us":988294,"threads_started":3,"wal-append.queue_time_us":1000676,"wal-append.run_cpu_time_us":2564,"wal-append.run_wall_time_us":1194925}
I20260709 12:23:49.449749  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:49.403990  1799 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:49.407287  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:49.408231  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:49.410156  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:49.411057  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:49.453359  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.454102  1801 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:49.455062  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.455734  1798 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:23:49.453359  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:49.456681  1723 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:49.326403  6044 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.078s
W20260709 12:23:49.418404  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.424710  6224 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.655s	user 0.018s	sys 0.005s
W20260709 12:23:49.425630  6292 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.172s	user 0.000s	sys 0.001s
I20260709 12:23:49.405000  6299 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:49.430766  6283 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:49.378252  6069 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:49.431625  6259 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.359486  5988 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:49.358662  6179 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.434115  6148 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.326980  6157 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.422149  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:49.455075  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.518098  1801 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:49.390918  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:49.458632  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.519632  1805 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:49.520949  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.459388  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:49.522434  1723 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:49.451982  1727 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:49.332324  6109 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 12:23:49.391646  6144 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:23:49.525796  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.524151  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:49.404311  6220 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.097s
I20260709 12:23:49.527073  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.481243  2412 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:49.483062  5581 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:49.389293  6072 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:49.487035  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.330669  6180 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:49.488101  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 209473us
I20260709 12:23:49.489395  6219 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:49.491923  6224 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:49.495070  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 199493us
I20260709 12:23:49.421406  6138 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:49.484985  5557 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:49.430856  6172 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.510s	user 0.011s	sys 0.000s
I20260709 12:23:49.480773  6141 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:49.528525  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.553186  1807 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:49.528949  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.554669  1801 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 10 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:49.450474  6225 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.541208  1800 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:49.515117  5988 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:49.517086  6259 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.544216  1728 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:49.548352  1726 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:23:49.571805  1732 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:49.467207  6221 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.574055  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:49.549548  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.575959  1807 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:23:49.533241  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.577847  1805 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:23:49.538401  6047 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.566946  6306 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:49.532846  5581 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:49.538842  6213 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:49.591284  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.592317  6180 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.592546  1806 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:49.593797  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.585724  1874 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:49.596589  1806 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:23:49.550102  6299 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:49.601730  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.597362  6072 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.608410  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:49.610312  5979 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.610926  5979 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.611963  6172 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:49.610532  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:49.602747  1806 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:49.600735  6065 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.621677  6157 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:49.624157  6132 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.625238  6163 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:49.625790  6163 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:49.626845  6172 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:49.627359  6172 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:49.626799  6163 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.627754  6172 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:49.628405  6221 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.634356  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:49.634979  1806 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:23:49.637171  1806 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:49.582417  6213 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.640586  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:49.644547  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:49.558039  6117 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.059s
I20260709 12:23:49.650812  6213 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:49.558226  6224 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.585079  6302 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.079s
I20260709 12:23:49.573702  5557 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.651322  6132 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:49.652330  6141 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:49.598920  6029 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:49.693346  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 271333us
I20260709 12:23:49.719120  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:23:49.697897  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53455.5us
I20260709 12:23:49.722056  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:49.579175  6144 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 12:23:49.584723  6259 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.640553  5979 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.595355  6047 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.574656  6138 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.643229  6159 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.713758  6218 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:49.713464  6221 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.751138  5499 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:49.716588  6132 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:49.750672  6029 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.714865  6315 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:49.808898  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 573) took 2557 ms. Trace:
I20260709 12:23:49.837383  1729 rpcz_store.cc:276] 0709 12:23:47.251113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.027495 (+776382us) service_pool.cc:224] Handling call
0709 12:23:48.050996 (+ 23501us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.962392 (+911396us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.808616 (+846224us) spinlock_profiling.cc:243] Waited 910 ms on lock 0x7b54002d14d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:49.808826 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2094129408}
W20260709 12:23:49.829685  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.840674  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:49.837451  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:49.808208  6307 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:23:49.769003  6180 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:49.635877  6299 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.806046  6144 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:49.851116  1725 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:49.790195  6157 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:49.804713  6065 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.854857  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.855810  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:49.790550  6109 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:23:49.650498  6219 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.850167  6132 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:49.833444  5499 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:49.870792  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.874694  1799 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:23:49.830876  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 262700us
I20260709 12:23:49.849352  6029 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:49.872241  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 150305us
I20260709 12:23:49.847388  6072 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:49.877065  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:49.854180  6306 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:49.857326  6050 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.853564  6312 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:49.883453  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:49.887832  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:49.888597  1880 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:49.890929  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:49.891192  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:49.891964  1881 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:23:49.891983  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:49.847774  6221 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.871039  1801 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:40096
W20260709 12:23:49.896977  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.920012  1801 raft_consensus.cc:606] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:23:49.920444  1801 raft_consensus.cc:614] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:49.903286  1800 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:49.904307  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:49.908296  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:49.922922  1880 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:49.925453  1872 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:49.910990  6218 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:49.932420  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:49.932655  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:49.911854  6259 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:49.912420  6047 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.894224  6050 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.885612  6072 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.015672  6072 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:49.909279  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:50.016820  1876 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:49.919495  6302 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.870085  6065 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:49.939896  6221 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.940472  6163 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:49.885797  6306 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:49.944090  5557 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:49.953311  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.040113  2412 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
W20260709 12:23:49.956723  6310 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:23:49.956892  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:50.052448  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 95647us
I20260709 12:23:50.053746  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:49.957849  6298 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:23:49.962111  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:49.966142  6317 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.879132  6179 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:49.879426  6314 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.968482  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.071920  2410 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:50.072351  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:50.080072  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:50.001668  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:49.936434  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:49.889482  6144 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:49.938637  5499 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.059762  6220 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:49.869287  6157 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:49.967242  6138 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:49.921175  6315 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:50.058561  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:50.009447  6172 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:50.107275  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48871.6us
I20260709 12:23:50.068867  6224 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:50.035921  6163 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:50.095474  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 159164us
I20260709 12:23:50.156208  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:49.965734  6311 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:50.096237  1803 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:23:50.088796  6221 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:50.097229  6144 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:50.102645  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 140620us
W20260709 12:23:50.108425  6069 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.586s	user 0.005s	sys 0.008s
I20260709 12:23:50.170011  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.171583  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.171983  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.172840  1881 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:50.171655  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.067150  6072 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:50.120137  6138 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:50.119346  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2135) took 2286 ms. Trace:
I20260709 12:23:50.176985  1875 rpcz_store.cc:276] 0709 12:23:47.832862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.479671 (+646809us) service_pool.cc:224] Handling call
0709 12:23:48.494340 (+ 14669us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.549564 (+ 55224us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:50.119021 (+1569457us) spinlock_profiling.cc:243] Waited 54.4 ms on lock 0x7b5400222754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:50.119269 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125052672}
W20260709 12:23:50.123458  6322 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
I20260709 12:23:50.124544  1805 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:23:50.124828  5731 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:50.165705  6144 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.186303  6144 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.165962  6040 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:50.162182  6311 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:50.126807  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 584) took 1406 ms. Trace:
I20260709 12:23:50.162321  6163 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:50.135380  1802 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:50.136217  6089 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:50.198109  6089 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:50.094765  6314 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.137992  6065 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:50.138541  6321 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:50.105620  6315 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.034408  6302 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.119314  6157 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:50.113648  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:50.204725  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 91163.2us
I20260709 12:23:50.140370  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 559) took 3411 ms. Trace:
I20260709 12:23:49.930174  6225 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.140s
I20260709 12:23:49.918838  6312 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.141969  6159 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:50.119589  6325 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
I20260709 12:23:50.068945  6320 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:50.158771  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:50.165441  6221 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.165282  6050 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:50.190037  1874 rpcz_store.cc:276] 0709 12:23:48.719997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.726970 (+  6973us) service_pool.cc:224] Handling call
0709 12:23:50.126494 (+1399524us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b540021bbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:50.126683 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1955147008}
W20260709 12:23:50.133056  6304 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.745s	user 0.004s	sys 0.000s
I20260709 12:23:50.179270  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:50.209043  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:50.070679  6317 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.214445  6072 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:50.230470  6072 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.230959  6072 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:50.232002  6317 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:50.233006  6317 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:50.233484  6317 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:50.233978  6317 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:50.234393  6302 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:50.235446  6159 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:50.203843  6321 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.236377  6159 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.237242  6321 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:50.237461  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58269us
I20260709 12:23:50.207255  1804 rpcz_store.cc:276] 0709 12:23:46.728900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.729059 (+   159us) service_pool.cc:224] Handling call
0709 12:23:46.753981 (+ 24922us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.779916 (+ 25935us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:46.829841 (+ 49925us) spinlock_profiling.cc:243] Waited 7.77 ms on lock 0x7b180036dc00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf 00007f16abdf2038
0709 12:23:47.245188 (+415347us) log.cc:844] Serialized 14 byte log entry
0709 12:23:49.173229 (+1928041us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:49.173795 (+   566us) log.cc:844] Serialized 26 byte log entry
0709 12:23:49.545817 (+372022us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:49.546758 (+   941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.546890 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:50.139941 (+593051us) raft_consensus.cc:1685] finished
0709 12:23:50.140087 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:50.140285 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1544192,"prepare.queue_time_us":544057,"prepare.run_cpu_time_us":164,"prepare.run_wall_time_us":916,"raft.queue_time_us":1535678,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":8977,"spinlock_wait_cycles":17879936,"thread_start_us":1146802,"threads_started":3,"wal-append.queue_time_us":1164436,"wal-append.run_cpu_time_us":1608,"wal-append.run_wall_time_us":1072931}
W20260709 12:23:50.220678  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62012.4us
I20260709 12:23:50.236925  6157 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:50.241324  6157 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.241873  6157 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:50.242947  6047 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:50.206499  6315 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:50.241353  1802 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "908089442ca44109875257bfbc9d06c2"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40096
W20260709 12:23:50.225745  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.248494  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:50.251183  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:50.252274  1724 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:50.199120  6089 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:50.214680  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.259712  1876 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:50.265110  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:50.267614  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:50.177466  6069 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.279146  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:50.279927  1724 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:50.214221  6040 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.281980  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:50.288184  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:50.289304  1726 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:23:50.228339  6221 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:50.287185  6089 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.263633  6042 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:50.328871  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.272657  6320 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:50.266930  6328 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:50.282553  6180 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:50.275416  6326 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:50.300834  6319 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:50.324625  6065 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:50.326442  6069 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:50.332031  6050 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:50.218392  6311 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.276093  6302 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
W20260709 12:23:50.347455  5988 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.763s	user 0.015s	sys 0.000s
I20260709 12:23:50.364837  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.377312  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.332682  6218 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
W20260709 12:23:50.406404  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 200271us
I20260709 12:23:50.327647  6040 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.349642  6326 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.413164  1730 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:50.344796  6042 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.413582  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:50.455536  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.374818  6159 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:50.462339  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49734.2us
I20260709 12:23:50.470301  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:50.374387  6065 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:50.404238  6239 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:50.455134  6302 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.455483  6326 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:50.377529  6050 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.383164  5988 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:50.529736  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.003s	sys 0.021s
W20260709 12:23:50.530309  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.003s	sys 0.021s
I20260709 12:23:50.494220  6327 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.502019  6239 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:50.530660  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 676177us
I20260709 12:23:50.504619  6172 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:50.570502  1880 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260709 12:23:50.568819  1770 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:50.574561  1878 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:50.574935  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.585040  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:50.528280  6219 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:50.586086  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:50.590001  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.599117  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.504998  6220 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.067s
I20260709 12:23:50.606835  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.607620  1879 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:50.607856  1804 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:50.609306  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.591459  1880 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:23:50.613349  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.614171  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.614446  1804 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 12 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:23:50.595359  6109 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:50.595120  6340 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:50.606611  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:50.490337  6320 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.560510  6319 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:23:50.623617  6331 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.212s	user 0.000s	sys 0.000s
I20260709 12:23:50.577873  6218 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.572503  6239 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:50.624922  6324 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:50.627985  6042 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:50.631829  6040 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:50.571398  6327 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.635434  5581 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.639647  6283 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:50.640610  6213 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.925s	user 0.011s	sys 0.001s
I20260709 12:23:50.704100  6141 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:50.641916  5499 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:50.642899  6065 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.566375  6314 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:50.644678  6249 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:50.646186  6027 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:23:50.649840  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2455973340001947s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16970000000000002s for a response
I20260709 12:23:50.655411  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:50.739341  5581 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:50.755173  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.655373  1879 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:50.756803  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:50.757157  6317 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.523s	user 0.011s	sys 0.000s
I20260709 12:23:50.757422  1879 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:50.757723  6317 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.758373  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:50.759500  1879 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.524617  6302 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:50.763473  6338 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:23:50.763055  6319 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:50.764441  6315 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.517s	user 0.009s	sys 0.003s
I20260709 12:23:50.765017  6302 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:50.765054  6315 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:50.766291  6306 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.767669  6306 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:50.769030  6315 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:50.735088  6213 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.770205  6220 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:50.770941  6220 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.742506  6141 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:50.773484  6029 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.884s	user 0.013s	sys 0.002s
I20260709 12:23:50.735263  6283 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.748725  6027 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:50.681936  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:50.688431  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 81898.9us
I20260709 12:23:50.579821  6345 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:50.701365  1806 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:50.597259  6219 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:50.797623  6219 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:50.777767  6126 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:50.799341  6126 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.777688  6352 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.775503  5988 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:50.802156  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:50.802457  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:50.774755  6332 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:50.804273  6332 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:50.805091  6283 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:50.803143  1732 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:50.774113  6029 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.586174  6313 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.115s
W20260709 12:23:50.715619  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.814700  1844 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 2.085s (SENT) [suppressed 10 similar messages]
I20260709 12:23:50.527918  6180 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:50.707154  6341 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.066s	user 0.000s	sys 0.002s
I20260709 12:23:50.686319  6346 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:23:50.641433  6336 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20260709 12:23:50.714277  6050 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:50.716725  6324 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:50.740835  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 268492us
I20260709 12:23:50.728833  6047 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:50.656687  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.743206  6342 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:50.720813  6218 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.786590  1879 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:50.789724  1728 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:23:50.852032  1728 pending_rounds.cc:85] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 7
I20260709 12:23:50.852300  1728 pending_rounds.cc:107] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:23:50.789975  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:50.852559  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.778448  1881 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: [11.8-13.9]
W20260709 12:23:50.775921  6144 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.589s	user 0.009s	sys 0.003s
I20260709 12:23:50.795554  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:50.852901  1727 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:50.862496  1729 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:23:50.796774  6350 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:50.796909  6327 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:50.599357  6312 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:50.689181  6132 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.803s	user 0.003s	sys 0.012s
I20260709 12:23:50.807837  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:50.805894  6317 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:50.815804  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:50.819025  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.840023  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:50.843120  1799 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:23:50.904663  2418 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:50.850725  6324 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:50.853715  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.852599  1728 raft_consensus.cc:2889] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:50.856282  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:50.893251  6163 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.698s	user 0.013s	sys 0.000s
I20260709 12:23:50.936329  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.937204  2411 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:23:50.860488  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.807979  6345 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:50.867092  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.777909  6027 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:50.850276  6321 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.612s	user 0.011s	sys 0.003s
I20260709 12:23:50.842089  6220 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:50.893935  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.895956  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.898262  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.899266  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.962603  2419 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:50.866326  6144 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.900743  1723 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:50.845809  6213 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:50.906028  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 90303.7us
I20260709 12:23:50.966689  2415 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:50.967777  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:50.801648  6352 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.904143  2412 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:50.970659  2412 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:50.971067  6352 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:50.851500  6342 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:50.972929  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2038) took 1764 ms (client timeout 2010 ms). Trace:
I20260709 12:23:50.973174  1801 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [10.5-12.6]
W20260709 12:23:50.973459  1800 rpcz_store.cc:269] 0709 12:23:49.208583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.217928 (+  9345us) service_pool.cc:224] Handling call
0709 12:23:50.972693 (+1754765us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:50.972830 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":589484288}
I20260709 12:23:50.936167  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.862208  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.840023  6346 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.975462  2416 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:50.976651  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.825934  6315 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:23:50.953473  6072 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.722s	user 0.015s	sys 0.000s
I20260709 12:23:50.955013  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2772) took 1062 ms. Trace:
I20260709 12:23:50.960115  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.981446  1873 rpcz_store.cc:276] 0709 12:23:49.892711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.892877 (+   166us) service_pool.cc:224] Handling call
0709 12:23:49.893205 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:50.094205 (+201000us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:50.094248 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:50.094854 (+   606us) log.cc:844] Serialized 26 byte log entry
0709 12:23:50.211090 (+116236us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:50.211141 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:50.796322 (+585181us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b540021acd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:50.796396 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:50.933447 (+137051us) raft_consensus.cc:1685] finished
0709 12:23:50.933468 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:50.954914 (+ 21446us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4646,"raft.queue_time_us":10084,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":2147,"spinlock_wait_cycles":450329344,"thread_start_us":113068,"threads_started":1,"wal-append.queue_time_us":234315}
I20260709 12:23:50.986733  2412 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:50.978983  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.864496  6348 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:50.983963  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.862880  6047 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:50.992499  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.992568  2415 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 10. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
W20260709 12:23:50.957379  6350 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:51.003057  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:50.996325  1881 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:50.992841  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:50.963194  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:51.004093  1800 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:23:51.004863  2419 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:51.000234  1873 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260709 12:23:51.005519  1873 pending_rounds.cc:85] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 7
I20260709 12:23:51.005558  1876 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:51.006111  1873 pending_rounds.cc:107] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 12:23:51.006848  6332 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:51.006626  1873 raft_consensus.cc:2889] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:51.008898  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 194227us
I20260709 12:23:51.009362  1874 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
W20260709 12:23:51.015657  1803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.014s	sys 0.006s
I20260709 12:23:50.963943  6126 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:51.017625  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:50.933007  6132 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:51.051539  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 232741us
W20260709 12:23:51.052418  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1285973340004603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1827s for a response
I20260709 12:23:50.980291  6342 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:50.959080  6163 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:51.057257  2419 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: []
I20260709 12:23:51.058718  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:51.060885  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:51.053285  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:51.007551  6359 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:51.021159  6332 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:23:51.108726  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.068054  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:51.073769  1880 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:51.003726  6315 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:51.112625  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:51.117000  1800 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:50.975546  6321 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.114679  6049 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:51.114681  6050 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.004s
I20260709 12:23:51.110091  2415 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:23:51.113659  1873 pending_rounds.cc:85] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 6
I20260709 12:23:51.062670  6126 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:51.065426  5903 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.079820  6135 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:51.001483  6072 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.121603  1873 pending_rounds.cc:107] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 12:23:51.138191  1873 raft_consensus.cc:2889] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:51.124541  2419 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:23:51.118064  2414 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:51.010821  6348 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.074831  6359 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:51.117630  6157 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.875s	user 0.013s	sys 0.010s
W20260709 12:23:51.179140  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62914.1us
I20260709 12:23:51.181208  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:51.178272  1800 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
W20260709 12:23:51.195521  6357 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:23:51.195431  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:51.196136  6315 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:51.197232  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:51.197975  6049 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:51.200258  6049 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:51.200666  6049 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:51.125252  6135 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:51.119839  6321 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:51.126024  6299 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.212076  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:51.229061  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47965.9us
I20260709 12:23:51.229790  2413 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:23:51.230039  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:51.230754  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 180738us
I20260709 12:23:51.167320  6213 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.165259  6359 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:51.123723  6126 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:51.124863  5903 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:51.244748  6239 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.003s
I20260709 12:23:51.202374  6117 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.245008  6355 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:51.252424  6174 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:51.264827  5581 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:51.248545  6042 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.292385  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:51.293579  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2054) took 1936 ms. Trace:
I20260709 12:23:51.316882  1799 rpcz_store.cc:276] 0709 12:23:49.356944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.900895 (+543951us) service_pool.cc:224] Handling call
0709 12:23:50.008389 (+107494us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:50.904923 (+896534us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.293248 (+388325us) spinlock_profiling.cc:243] Waited 834 ms on lock 0x7b54003b79d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:51.293508 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1918709504}
I20260709 12:23:51.213302  6135 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:51.211486  6314 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.078s
I20260709 12:23:51.202929  6157 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:51.290160  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 161439us
I20260709 12:23:51.333326  2415 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:51.330849  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:51.201711  6049 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:51.385296  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:51.340068  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1912) took 1748 ms. Trace:
I20260709 12:23:51.393918  1803 rpcz_store.cc:276] 0709 12:23:49.591963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.164773 (+572810us) service_pool.cc:224] Handling call
0709 12:23:50.169559 (+  4786us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:50.403231 (+233672us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:50.403281 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:50.406173 (+  2892us) log.cc:844] Serialized 26 byte log entry
0709 12:23:51.016262 (+610089us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:51.072647 (+ 56385us) log.cc:844] Serialized 14 byte log entry
0709 12:23:51.275057 (+202410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.275152 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:51.339813 (+ 64661us) raft_consensus.cc:1685] finished
0709 12:23:51.339840 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:51.339995 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":50206,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":201731,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":11190,"thread_start_us":935879,"threads_started":3,"wal-append.queue_time_us":744434}
I20260709 12:23:51.288535  6239 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:51.310890  6221 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.969s	user 0.014s	sys 0.000s
I20260709 12:23:51.266270  6213 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.321398  6141 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:51.409322  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78484us
I20260709 12:23:51.304402  6117 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.387495  6050 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.392992  5581 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.386703  6345 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:51.431233  2411 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:51.382588  6029 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.346647  6144 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.399834  6218 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.421502  6105 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:23:51.487452  6366 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
W20260709 12:23:51.462077  6365 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.157s	user 0.001s	sys 0.000s
I20260709 12:23:51.373850  5785 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
W20260709 12:23:51.470152  5499 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.730s	user 0.009s	sys 0.001s
I20260709 12:23:51.471803  5903 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.423866  6132 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:51.476079  6072 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:51.315743  6355 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:51.478440  6219 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.680s	user 0.011s	sys 0.000s
I20260709 12:23:51.480049  6239 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:51.481714  6117 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:51.458039  6050 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:51.482698  5581 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.338835  6042 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:51.616564  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:51.609615  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 1977) took 2586 ms (client timeout 2085 ms). Trace:
I20260709 12:23:51.628064  1802 raft_consensus.cc:606] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
W20260709 12:23:51.617520  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 349031us
I20260709 12:23:51.666147  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:51.669392  5499 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.694522  5499 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.663340  5581 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.686429  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1904) took 2287 ms. Trace:
I20260709 12:23:51.651830  1872 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:51.690755  6050 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:51.705601  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2776) took 1535 ms. Trace:
I20260709 12:23:51.698390  5499 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.708362  5499 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.637806  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:51.672317  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:51.711651  1873 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:51.713727  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:51.721535  6157 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:51.722627  6141 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:51.723343  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2158) took 1133 ms. Trace:
I20260709 12:23:51.485399  6213 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.521751  6219 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.723877  6141 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:51.725013  6213 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:51.725937  6213 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:51.727593  2417 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:51.727845  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:51.729280  1876 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:51.634706  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2161) took 1023 ms. Trace:
I20260709 12:23:51.421816  6049 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:51.638211  1725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 2926) took 2231 ms (client timeout 1685 ms). Trace:
I20260709 12:23:51.723917  1878 rpcz_store.cc:276] 0709 12:23:50.589531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.589706 (+   175us) service_pool.cc:224] Handling call
0709 12:23:50.590051 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:50.980786 (+390735us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:50.980825 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:50.980872 (+    47us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:50.980921 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.722682 (+741761us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b540021b1d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:51.722759 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:51.723069 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":898017152}
W20260709 12:23:51.730444  1725 rpcz_store.cc:269] 0709 12:23:49.406476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.407900 (+  1424us) service_pool.cc:224] Handling call
0709 12:23:51.637682 (+2229782us) spinlock_profiling.cc:243] Waited 398 ms on lock 0x7b540030b454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:51.638005 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":916292352}
I20260709 12:23:51.732162  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:51.494601  6089 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.162s	user 0.005s	sys 0.009s
I20260709 12:23:51.734161  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:51.687790  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2780) took 1448 ms. Trace:
I20260709 12:23:51.420804  6047 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.735280  1878 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:51.734990  1874 rpcz_store.cc:276] 0709 12:23:50.239523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.239707 (+   184us) service_pool.cc:224] Handling call
0709 12:23:50.240149 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.010577 (+770428us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.687188 (+676611us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b54001f7a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:51.687553 (+   365us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1768433536}
I20260709 12:23:51.736526  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:51.737264  1878 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:51.737545  6089 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:51.688301  6132 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.421577  6221 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.656297  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2065) took 1002 ms. Trace:
I20260709 12:23:51.691697  6126 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:51.692385  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:51.656500  1730 rpcz_store.cc:269] 0709 12:23:49.022390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.022711 (+   321us) service_pool.cc:224] Handling call
0709 12:23:51.608459 (+2585748us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:51.609039 (+   580us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3110756096}
I20260709 12:23:51.662506  6042 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.693008  6109 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:51.668605  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.692731  6135 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:51.656953  1802 raft_consensus.cc:614] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:51.669056  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:51.700704  1801 rpcz_store.cc:276] 0709 12:23:49.399187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.055552 (+656365us) service_pool.cc:224] Handling call
0709 12:23:50.972889 (+917337us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:50.979439 (+  6550us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.685771 (+706332us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b54002f3350. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:51.686155 (+   384us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2108128640}
W20260709 12:23:51.675534  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.706202  1872 rpcz_store.cc:276] 0709 12:23:50.169892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.171237 (+  1345us) service_pool.cc:224] Handling call
0709 12:23:51.705260 (+1534023us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b5400222254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:51.705467 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3401279744}
I20260709 12:23:51.673509  5903 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:51.709978  5499 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.721662  6105 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:51.714217  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:51.750557  1880 rpcz_store.cc:276] 0709 12:23:50.612824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.613943 (+  1119us) service_pool.cc:224] Handling call
0709 12:23:50.614410 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.110598 (+496188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.633981 (+523383us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b540021b6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:51.634433 (+   452us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055705856}
I20260709 12:23:51.729470  6219 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:51.642176  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.566169  6312 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:51.807171  6312 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:51.809154  6312 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.812798  6135 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:51.814056  2415 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:51.815534  6141 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:51.819742  5499 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:51.821877  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:51.823527  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.824988  1875 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:51.826280  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.801002  6132 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:51.565721  6324 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.604s	user 0.009s	sys 0.000s
W20260709 12:23:51.560602  6326 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.025s	user 0.010s	sys 0.001s
W20260709 12:23:51.553653  6337 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.819s	user 0.001s	sys 0.001s
I20260709 12:23:51.568574  6332 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.792098  6105 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:51.844724  6332 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:51.561470  6348 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.846635  6105 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:51.547017  6362 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.255s	user 0.001s	sys 0.000s
I20260709 12:23:51.417851  6361 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:23:51.550031  6367 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
W20260709 12:23:51.551054  6376 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:23:51.497560  6380 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:51.557183  6346 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:51.536427  6356 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.308s	user 0.001s	sys 0.001s
I20260709 12:23:51.919217  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:51.920660  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:51.921787  1726 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:23:51.922690  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40096 (request call id 3) took 1681 ms. Trace:
I20260709 12:23:51.923228  1802 rpcz_store.cc:276] 0709 12:23:50.240825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.241180 (+   355us) service_pool.cc:224] Handling call
0709 12:23:51.922340 (+1681160us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001e3a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd2853 00007f16b0da060a 00007f16aa3267d4 00007f16aa326762 00007f16aa326691 00007f16aa32660c 00007f16aa3258c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:51.922507 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3173425408}
I20260709 12:23:51.924077  1729 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
W20260709 12:23:51.923961  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8bb813e4fab14a61b810a24b56899586 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica a20e99e125b84513a44e88860667ed61 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 9 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:51.670919  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:51.737267  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:51.729600  6213 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:51.767781  1806 rpcz_store.cc:276] 0709 12:23:50.654197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.688065 (+ 33868us) service_pool.cc:224] Handling call
0709 12:23:50.688541 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:50.803282 (+114741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.655577 (+852295us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b54003b7ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:51.656058 (+   481us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27670784}
W20260709 12:23:51.850646  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.830953  6311 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:51.835582  6324 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.556960  6355 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:51.833786  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:51.829746  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2169) took 1179 ms. Trace:
I20260709 12:23:51.827226  6148 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:51.834409  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:51.835536  1873 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:51.833319  1880 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:23:51.861866  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87111.1us
I20260709 12:23:51.826596  6221 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.830572  6132 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:51.830940  6144 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.863242  5903 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:51.834877  6109 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:51.914433  6027 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:51.865556  6042 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:51.886205  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:51.898927  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:51.899362  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.836285  6326 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.873845  5499 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.572484  6342 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.913213  6312 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:51.913877  6299 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:51.914502  6163 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:51.934432  6359 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.662s	user 0.012s	sys 0.004s
I20260709 12:23:51.926687  1727 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:23:51.929391  2413 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:51.899479  6174 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.910528  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:51.919505  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 999) took 1027 ms. Trace:
I20260709 12:23:51.949092  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2784) took 1127 ms. Trace:
I20260709 12:23:51.953852  1728 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:51.943071  1875 rpcz_store.cc:276] 0709 12:23:50.650835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.956914 (+306079us) service_pool.cc:224] Handling call
0709 12:23:51.829341 (+872427us) spinlock_profiling.cc:243] Waited 836 ms on lock 0x7b54001f7f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:51.829633 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1922721536}
I20260709 12:23:51.934917  6105 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:51.892023  1723 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
W20260709 12:23:51.938897  6159 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.014s	sys 0.011s
W20260709 12:23:51.985941  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.987046  1880 rpcz_store.cc:276] 0709 12:23:50.817114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.831774 (+1014660us) service_pool.cc:224] Handling call
0709 12:23:51.832470 (+   696us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.943559 (+111089us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.944658 (+  1099us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:51.943737  6148 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:51.985344  2411 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:23:51.986934  2417 rpcz_store.cc:276] 0709 12:23:50.893456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.893610 (+   154us) service_pool.cc:224] Handling call
0709 12:23:51.919201 (+1025591us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54004993d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:51.919436 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1760378624}
I20260709 12:23:51.992511  2419 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:51.993619  1730 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
W20260709 12:23:51.994861  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.008s	sys 0.017s
I20260709 12:23:51.994531  6174 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:51.994992  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:51.995437  1798 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:51.996284  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:52.000152  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 584) took 1042 ms. Trace:
I20260709 12:23:52.001307  1877 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:51.969660  6050 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:52.006520  1724 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.019s	sys 0.015s
I20260709 12:23:52.013262  1874 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:52.010084  1875 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:52.008602  6042 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.015200  6050 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:52.018445  1724 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.020s	sys 0.015s
I20260709 12:23:52.009589  2414 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:52.008876  6148 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:52.050908  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2167) took 1366 ms. Trace:
I20260709 12:23:52.051512  1881 rpcz_store.cc:276] 0709 12:23:50.642849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.004965 (+362116us) service_pool.cc:224] Handling call
0709 12:23:51.005311 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.005578 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:51.031800 (+ 26222us) log.cc:844] Serialized 14 byte log entry
0709 12:23:51.695810 (+664010us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b180002dfa0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:23:52.002726 (+306916us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.002852 (+   126us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:52.002999 (+   147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.003110 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.003325 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26192,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":14034,"spinlock_wait_cycles":23421824,"thread_start_us":251592,"threads_started":1}
I20260709 12:23:51.952476  6144 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.052781  6342 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:52.046468  6361 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:52.045533  6042 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:52.060786  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.019140  6219 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:52.039826  6132 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:52.003765  6359 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.054255  6221 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.011020  1803 rpcz_store.cc:276] 0709 12:23:50.957786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.397399 (+439613us) service_pool.cc:224] Handling call
0709 12:23:51.400572 (+  3173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.401176 (+   604us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:51.419125 (+ 17949us) log.cc:844] Serialized 14 byte log entry
0709 12:23:51.999086 (+579961us) log.cc:844] Serialized 14 byte log entry
0709 12:23:51.999401 (+   315us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:51.999470 (+    69us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:51.999585 (+   115us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.999720 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:51.999945 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":36619,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":79,"raft.queue_time_us":50400,"raft.run_cpu_time_us":426,"raft.run_wall_time_us":40940,"thread_start_us":450885,"threads_started":1,"wal-append.queue_time_us":582625}
I20260709 12:23:52.018672  5499 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.077914  2415 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
W20260709 12:23:52.079250  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70024.7us
I20260709 12:23:52.004818  5903 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.078476  1731 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:23:52.098425  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 43970.9us
W20260709 12:23:52.104822  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 719552us
I20260709 12:23:52.107038  6391 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:23:52.108182  6050 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:52.110361  6359 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.111390  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 579) took 1218 ms. Trace:
I20260709 12:23:52.111186  6221 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.111836  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:52.113009  1879 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:52.111760  1804 rpcz_store.cc:276] 0709 12:23:50.892923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.260566 (+367643us) service_pool.cc:224] Handling call
0709 12:23:51.260860 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.261070 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:51.404017 (+142947us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.079022 (+675005us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.093864 (+ 14842us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:52.110848 (+ 16984us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.110992 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.111178 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72715,"prepare.queue_time_us":24457,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":5286,"raft.queue_time_us":83833,"thread_start_us":681354,"threads_started":2,"wal-append.queue_time_us":692490}
I20260709 12:23:52.114106  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:52.122534  6361 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.107450  6324 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:52.114774  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2782) took 1701 ms (client timeout 1520 ms). Trace:
W20260709 12:23:52.124631  1877 rpcz_store.cc:269] 0709 12:23:50.413077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.413269 (+   192us) service_pool.cc:224] Handling call
0709 12:23:52.114496 (+1701227us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b54002df5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:52.114699 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3523749248}
I20260709 12:23:52.116735  1875 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:23:52.084543  6355 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.107419  1800 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:52.123260  1879 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:52.119131  6043 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:23:52.141021  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.012097  6299 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.162904  1875 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:52.165084  1875 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:52.149299  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 446787us
W20260709 12:23:52.174014  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 503729us
I20260709 12:23:52.230706  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:52.244966  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.243851  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.204841  1873 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:52.251459  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.213331  1805 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:23:52.211205  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 577) took 1346 ms. Trace:
I20260709 12:23:52.211480  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.243350  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:52.211153  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43700.6us
I20260709 12:23:52.182402  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 2936) took 1388 ms. Trace:
I20260709 12:23:52.266528  1800 rpcz_store.cc:276] 0709 12:23:50.865007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.196959 (+331952us) service_pool.cc:224] Handling call
0709 12:23:52.210893 (+1013934us) spinlock_profiling.cc:243] Waited 839 ms on lock 0x7b54001e4454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:52.211103 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1930223360}
W20260709 12:23:52.278249  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.198189  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:52.230965  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 583) took 1301 ms (client timeout 1501 ms). Trace:
I20260709 12:23:52.273144  2414 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
W20260709 12:23:52.280089  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 102885us
I20260709 12:23:52.284435  1879 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
W20260709 12:23:52.281517  1876 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.551s	user 0.016s	sys 0.000s
I20260709 12:23:52.285218  1876 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:52.239243  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:52.237839  6306 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.470s	user 0.011s	sys 0.010s
W20260709 12:23:52.284353  1798 rpcz_store.cc:269] 0709 12:23:50.899623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.960856 (+ 61233us) service_pool.cc:224] Handling call
0709 12:23:52.176745 (+1215889us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b5400133654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:52.200544 (+ 23799us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2225552640}
I20260709 12:23:52.291895  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:52.292608  1730 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:52.300590  1875 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:52.294234  1726 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:23:52.302594  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.230432  5499 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:52.247473  6326 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:52.291508  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.238286  6380 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:52.181985  6213 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:52.305742  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.320211  1727 raft_consensus.cc:2410] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:23:52.212631  5903 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:52.230576  6378 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:23:52.321889  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 145073us
I20260709 12:23:52.279793  6355 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.324991  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.313978  6027 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:52.334223  6324 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:52.340730  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:52.346741  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 107981us
I20260709 12:23:52.309978  1627 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 4, term changed from 6 to 9, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to <none>, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 075704438992440697732181e49ccd09 (127.0.140.65), VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66), VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67), VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 9 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:52.351002  1726 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:52.351183  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.351819  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.281736  1724 rpcz_store.cc:276] 0709 12:23:50.788938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.790586 (+  1648us) service_pool.cc:224] Handling call
0709 12:23:50.791083 (+   497us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:50.791329 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:50.956400 (+165071us) log.cc:844] Serialized 14 byte log entry
0709 12:23:51.359844 (+403444us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b1800456160. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:23:51.568401 (+208557us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b1800456160. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:51.734603 (+166202us) spinlock_profiling.cc:243] Waited 8.02 ms on lock 0x7b4800041c58. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:23:52.073873 (+339270us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.085095 (+ 11222us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:52.094402 (+  9307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.160266 (+ 65864us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.177751 (+ 17485us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75988,"prepare.queue_time_us":436462,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":113318,"raft.run_cpu_time_us":4512,"raft.run_wall_time_us":369092,"spinlock_wait_cycles":87255680,"thread_start_us":1081677,"threads_started":3,"wal-append.queue_time_us":417793}
I20260709 12:23:52.356472  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:52.324574  6109 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.361078  6321 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.360899  6378 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.361541  6213 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.361938  6174 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:52.362259  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.362766  6380 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.356599  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:52.364507  6044 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:23:52.365279  6027 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.382623  6027 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.383239  6027 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:52.383594  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:52.383870  6027 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.385398  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:52.386564  6348 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:52.386868  6109 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.387274  1873 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:23:52.388545  1879 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:52.396319  6138 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.396867  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:52.397156  6138 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.397832  2415 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:23:52.398890  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:52.399056  6174 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:52.400115  6174 consensus_queue.cc:1243] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:23:52.400947  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 575) took 1541 ms (client timeout 1904 ms). Trace:
I20260709 12:23:52.401748  6042 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:23:52.366683  6049 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.636s	user 0.007s	sys 0.011s
I20260709 12:23:52.399422  6144 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:52.401017  6174 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:52.413868  6174 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.416049  6380 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.415122  1627 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 7, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 7 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:52.417411  6395 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:23:52.414510  6174 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:52.419482  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 580) took 1520 ms. Trace:
I20260709 12:23:52.402335  6135 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 12:23:52.403592  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.420908  1723 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:52.419963  1799 rpcz_store.cc:276] 0709 12:23:50.898737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.356815 (+458078us) service_pool.cc:224] Handling call
0709 12:23:51.357263 (+   448us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:51.365462 (+  8199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:51.365493 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:51.366129 (+   636us) log.cc:844] Serialized 52 byte log entry
0709 12:23:51.907461 (+541332us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:51.907541 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.907714 (+   173us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:52.419040 (+511326us) raft_consensus.cc:1685] finished
0709 12:23:52.419065 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.419260 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8761,"raft.queue_time_us":8680,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":20364,"thread_start_us":483584,"threads_started":1,"wal-append.queue_time_us":912086}
I20260709 12:23:52.392976  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.418503  1627 catalog_manager.cc:5697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 6 to 10, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 10 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:52.377439  6324 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.424723  1627 catalog_manager.cc:5697] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 5 to 10, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 10 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:52.360963  6306 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:52.401630  1805 rpcz_store.cc:269] 0709 12:23:50.859434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.860108 (+   674us) service_pool.cc:224] Handling call
0709 12:23:52.400696 (+1540588us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54001e3f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:52.400868 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2933065472}
I20260709 12:23:52.407763  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:52.387560  6109 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:52.427843  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59351.3us
I20260709 12:23:52.427278  1627 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 8. New cstate: current_term: 8 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:52.429208  6306 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:52.429597  6109 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.429713  6306 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:52.430197  6306 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.431071  6144 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:52.431485  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:52.431803  6144 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.432945  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:52.434203  6144 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:52.414213  1732 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:52.435216  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:52.432199  2415 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:52.372907  6299 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.406253  6321 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.436131  2411 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:52.437445  2410 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 12 index: 12. (index mismatch)
I20260709 12:23:52.438845  6321 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.438953  6378 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.439206  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:52.439483  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 929) took 1692 ms. Trace:
I20260709 12:23:52.439985  2413 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:52.439888  2410 rpcz_store.cc:276] 0709 12:23:50.746775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.747026 (+   251us) service_pool.cc:224] Handling call
0709 12:23:50.747467 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.437833 (+1690366us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.439108 (+  1275us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54005609d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:52.439388 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3884918656}
I20260709 12:23:52.400952  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:52.441046  6213 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:52.441730  6313 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:52.442178  6313 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:52.442559  6313 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.442749  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:52.443819  6049 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.444458  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:52.445508  6043 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:52.446008  6043 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.446406  6043 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:52.447781  1873 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:52.448709  6391 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:52.449698  6391 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:52.450045  6391 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:52.447801  6393 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:52.451290  6393 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.451896  6393 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.452893  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.453763  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.454411  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.456156  6109 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:52.455999  6391 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.204968  6148 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:52.430477  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 106053us
I20260709 12:23:52.426443  6324 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.428643  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:52.402194  6355 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:52.430531  1627 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 6 to 7, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 7 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:52.465463  1730 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:23:52.435614  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:52.467250  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:52.387533  6348 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:52.402467  6042 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:52.420663  6135 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:52.485730  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:52.484803  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 56256.7us
I20260709 12:23:52.463660  6109 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:23:52.488783  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53405us
I20260709 12:23:52.490054  2410 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:52.492720  1730 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:52.403926  5903 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260709 12:23:52.476279  6378 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.480451  6027 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.513115  1878 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:52.517920  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:52.518247  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:52.489266  6135 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:52.493743  6213 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:52.533106  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.527019  2417 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:52.533118  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:52.541795  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:52.543808  6299 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:52.544420  6299 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:52.544876  6299 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:52.544610  6042 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.546233  6042 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:52.546959  2415 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:52.547297  2410 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:52.547945  6378 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:52.551997  6148 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:52.552896  6050 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:52.553838  6355 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:52.554327  6355 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.554687  6355 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.558465  6318 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:52.558830  6138 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:52.540218  6391 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:52.544019  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:52.565853  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:52.566599  6109 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:52.567974  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:52.568470  6109 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:52.527021  6324 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:52.568993  6321 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:52.569814  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:52.569459  6324 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.567519  6326 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:52.571280  6326 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.573225  6138 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:52.573066  6324 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.574455  6326 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:52.568990  6109 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:52.511757  6348 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 12:23:52.571255  6321 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:52.576144  6221 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:52.575331  1723 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:23:52.578802  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:52.550496  6378 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
W20260709 12:23:52.532648  6332 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.683s	user 0.018s	sys 0.000s
I20260709 12:23:52.577232  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:52.577337  1726 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:52.574860  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:52.609336  1805 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:52.546988  2411 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:52.590930  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.603137  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.612571  2410 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:23:52.612241  6135 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.614171  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.593005  6321 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.616014  6321 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:52.617434  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:52.573966  6138 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.619051  6221 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:52.619591  6221 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.619925  6221 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:52.620121  6138 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.621119  6326 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:52.621582  6326 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.621994  6326 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:52.618522  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:52.603871  6311 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.643s	user 0.008s	sys 0.005s
I20260709 12:23:52.577572  6213 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:52.636137  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70386.2us
I20260709 12:23:52.637923  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:52.641322  1804 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:52.601137  6027 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:52.631103  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.639078  2414 raft_consensus.cc:2368] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12.
W20260709 12:23:52.641988  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49985.6us
I20260709 12:23:52.635233  5499 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:52.645531  5499 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.645969  5499 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:52.614562  6135 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:52.649014  5903 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:52.650499  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.650887  6380 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.651372  1731 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:52.651522  6380 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.654385  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.655514  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.656507  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.657027  6174 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.657217  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:52.657857  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.658526  1807 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:52.660545  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:52.660873  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.658542  1798 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:52.660447  6361 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:52.642804  1800 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:52.665315  1723 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:23:52.671514  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 291957us
I20260709 12:23:52.677451  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.672576  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:52.678500  1724 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 6.
W20260709 12:23:52.687392  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49196us
I20260709 12:23:52.696740  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.695617  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:52.696512  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.710662  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.711213  1806 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:52.738222  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.714248  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:52.740444  1798 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:52.742182  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76118.4us
I20260709 12:23:52.735674  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.750366  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:52.690939  6312 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.706s	user 0.010s	sys 0.003s
W20260709 12:23:52.753893  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.755683  1732 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:23:52.627028  6332 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.633430  6306 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.765162  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:52.766676  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:52.770541  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:52.770687  6359 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:52.770838  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.770541  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.772128  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:52.789326  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.774091  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.774932  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.759660  1627 catalog_manager.cc:5697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 7, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 7 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:52.758384  1801 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:52.665532  6148 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.122s
I20260709 12:23:52.794055  1727 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:52.796747  6359 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:52.797691  1731 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:52.798271  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:52.798830  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.798125  6359 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:52.799645  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.797832  1802 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:23:52.800899  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:52.801846  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:52.793164  1803 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:52.809628  1730 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:23:52.813428  1875 raft_consensus.cc:2410] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:23:52.703640  6219 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260709 12:23:52.810263  1730 pending_rounds.cc:85] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 5
I20260709 12:23:52.735970  6163 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.092s
I20260709 12:23:52.820433  6306 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:52.625159  6299 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:52.830595  1805 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:52.892565  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:53.012454  1803 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
W20260709 12:23:52.882710  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 279018us
I20260709 12:23:53.013485  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:53.009799  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.820508  1879 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:52.821061  1725 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:52.929620  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:53.004779  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 344354us
W20260709 12:23:52.854560  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.018304  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.000329  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:52.894356  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 248543us
I20260709 12:23:52.904767  1724 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:23:52.814473  6348 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:53.028823  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.820652  1878 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:53.008529  1730 pending_rounds.cc:107] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:23:53.032876  1730 raft_consensus.cc:2889] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:52.846521  1806 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:52.821312  6402 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:23:53.033332  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:52.869477  6174 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.007229  6332 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.048708  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.049830  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:53.054585  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:53.060426  1875 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:52.846141  1801 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae23688e76624153b5a7089586669bb0"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:23:53.061921  1801 raft_consensus.cc:606] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:23:53.062408  1801 raft_consensus.cc:614] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:53.061950  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:53.070001  1728 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:53.065237  1802 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:53.068529  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.074455  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 593) took 1412 ms. Trace:
I20260709 12:23:53.074935  1881 rpcz_store.cc:276] 0709 12:23:51.661893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.053511 (+391618us) service_pool.cc:224] Handling call
0709 12:23:52.053900 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.061933 (+  8033us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:52.346290 (+284357us) log.cc:844] Serialized 14 byte log entry
0709 12:23:53.074084 (+727794us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:53.074148 (+    64us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:53.074196 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.074264 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.074384 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20482,"prepare.queue_time_us":172897,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"thread_start_us":878852,"threads_started":3}
I20260709 12:23:53.079124  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:52.847011  6318 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.203s
I20260709 12:23:53.084880  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:52.847601  6378 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.203s
I20260709 12:23:53.087066  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:52.783699  6393 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.087953  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.093582  1728 raft_consensus.cc:2410] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:23:52.820478  6312 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.246s
I20260709 12:23:53.094830  1803 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:53.098676  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.099715  1730 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:23:53.067381  6306 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:52.911820  6213 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.108420  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.119110  1875 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:53.110553  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.115263  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2790) took 1209 ms. Trace:
I20260709 12:23:53.121028  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.123292  1723 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 12: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:53.124769  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:53.050163  6348 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.127907  1873 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 8: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:53.131603  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 483411us
I20260709 12:23:53.145370  1730 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:53.120076  1876 rpcz_store.cc:276] 0709 12:23:51.905615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.287312 (+381697us) service_pool.cc:224] Handling call
0709 12:23:52.318595 (+ 31283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.358715 (+ 40120us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:52.470492 (+111777us) log.cc:844] Serialized 14 byte log entry
0709 12:23:53.114730 (+644238us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:53.114832 (+   102us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:53.114897 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.114999 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.115188 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":106600,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"thread_start_us":616937,"threads_started":2}
I20260709 12:23:53.107424  6332 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:53.153681  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.149430  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:53.159382  1728 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:53.124934  6306 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:53.168180  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:53.177174  1807 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.518s	user 0.008s	sys 0.004s
I20260709 12:23:53.178107  1807 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:23:53.183916  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:53.184651  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:53.188490  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.226269  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:53.228291  1807 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:53.207773  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:53.229601  1803 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
W20260709 12:23:53.184756  6407 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.543s	user 0.005s	sys 0.000s
I20260709 12:23:53.215025  6332 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:53.231017  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 46517.2us
I20260709 12:23:53.231878  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:53.231950  1772 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:53.233187  6332 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.233573  1727 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:53.234015  6332 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:53.184839  6394 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:53.224599  1724 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:23:53.234869  6393 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.225428  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:53.238864  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.213757  1798 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
W20260709 12:23:53.221012  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 95214.1us
I20260709 12:23:53.186743  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:53.242177  1805 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:53.224836  1801 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:23:53.184077  6313 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.741s	user 0.013s	sys 0.000s
W20260709 12:23:53.227622  5499 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.581s	user 0.008s	sys 0.004s
I20260709 12:23:53.250270  6306 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.249995  5499 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:53.252416  6135 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.606s	user 0.008s	sys 0.004s
I20260709 12:23:53.253160  5499 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.247622  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:53.254127  1802 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:53.254359  6299 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:53.254149  6400 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.331s	user 0.001s	sys 0.001s
I20260709 12:23:53.256361  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 589) took 1596 ms. Trace:
I20260709 12:23:53.256727  1872 rpcz_store.cc:276] 0709 12:23:51.659908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.811588 (+151680us) service_pool.cc:224] Handling call
0709 12:23:51.811994 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:51.812507 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:51.812546 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:51.813318 (+   772us) log.cc:844] Serialized 26 byte log entry
0709 12:23:51.961342 (+148024us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:51.961432 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.961549 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.256146 (+1294597us) raft_consensus.cc:1685] finished
0709 12:23:53.256167 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.256300 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":147340,"threads_started":1,"wal-append.queue_time_us":959313}
W20260709 12:23:53.258193  6027 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.586s	user 0.011s	sys 0.000s
I20260709 12:23:53.259181  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:53.263125  1875 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:53.177089  6163 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.264321  6163 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:53.235571  6394 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.268922  6394 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.269526  6311 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:53.270344  6049 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.826s	user 0.011s	sys 0.003s
W20260709 12:23:53.270807  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84196.1us
I20260709 12:23:53.272132  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.272292  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:53.273703  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.273788  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:53.275030  6174 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:53.180665  6420 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.102s
W20260709 12:23:53.192152  6412 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:23:53.281052  1732 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 12:23:53.193022  6405 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:23:53.281787  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:53.286679  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.290510  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.295980  1725 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:53.297175  1802 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
W20260709 12:23:53.301280  1804 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.659s	user 0.010s	sys 0.003s
I20260709 12:23:53.301793  1804 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:23:53.303297  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:53.298884  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.304103  1727 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:53.307106  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:53.307360  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:53.213265  6348 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.314040  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.316377  1725 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:53.317269  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.266687  6163 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:53.317866  1728 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:53.320933  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.258677  6027 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:53.322742  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:53.289580  6049 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:53.275970  6174 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:23:53.314625  1627 catalog_manager.cc:5697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 6 to 9, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 9 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:53.304701  6135 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:23:53.320202  6355 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.765s	user 0.012s	sys 0.001s
I20260709 12:23:53.337121  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.341064  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.334256  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:53.339217  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 591) took 1678 ms. Trace:
I20260709 12:23:53.339475  1873 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:23:53.285005  6148 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 12:23:53.351368  1875 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:53.277362  6313 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.360795  2414 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:53.329633  6324 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:23:53.355616  6043 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.909s	user 0.015s	sys 0.000s
I20260709 12:23:53.339278  6391 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:53.343345  1874 rpcz_store.cc:276] 0709 12:23:51.660936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.037231 (+376295us) service_pool.cc:224] Handling call
0709 12:23:52.074475 (+ 37244us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:52.206395 (+131920us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.206433 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:52.405122 (+198689us) spinlock_profiling.cc:243] Waited 42.3 ms on lock 0x7b2c00009c38. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:23:52.421396 (+ 16274us) log.cc:844] Serialized 26 byte log entry
0709 12:23:52.666436 (+245040us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:52.666479 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.997680 (+331201us) spinlock_profiling.cc:243] Waited 96.7 ms on lock 0x7b5400073854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:52.997772 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.338919 (+341147us) raft_consensus.cc:1685] finished
0709 12:23:53.338951 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.339127 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9180,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":266,"spinlock_wait_cycles":319542272,"thread_start_us":244603,"threads_started":1,"wal-append.queue_time_us":458050}
I20260709 12:23:53.370221  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.353221  6355 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:53.377847  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.378096  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.376855  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:53.328857  6421 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
W20260709 12:23:53.381186  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.381642  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.391163  1807 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:53.391324  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:53.366042  6324 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:53.365095  6135 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:53.401684  2415 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:53.381713  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:53.403472  1725 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:53.369129  6361 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:53.424503  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:53.392418  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 357325us
I20260709 12:23:53.376398  6391 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:53.411376  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 88788.2us
I20260709 12:23:53.427354  6324 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.428830  6324 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:53.431137  6391 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:53.432862  6423 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:53.420317  1806 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:23:53.368609  6043 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.435478  6135 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:53.437702  1729 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:53.438657  6225 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.439118  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:53.440132  2415 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:53.439484  6225 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:53.433215  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:53.441429  6423 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:53.453611  6391 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:53.468748  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 88417.9us
I20260709 12:23:53.474239  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 590) took 1813 ms. Trace:
I20260709 12:23:53.412169  6219 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:53.493736  6430 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:53.494366  6430 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.494748  6430 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:53.495882  6321 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.879s	user 0.013s	sys 0.002s
I20260709 12:23:53.496296  6225 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:53.496441  6321 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.498471  6427 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:53.499533  6427 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.500948  1627 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 10. New cstate: current_term: 10 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:23:53.505808  6413 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:23:53.509404  1731 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:53.511503  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1010) took 1040 ms. Trace:
I20260709 12:23:53.482492  1880 rpcz_store.cc:276] 0709 12:23:51.660459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.018929 (+358470us) service_pool.cc:224] Handling call
0709 12:23:52.067607 (+ 48678us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:52.104720 (+ 37113us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.104774 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:52.105707 (+   933us) log.cc:844] Serialized 52 byte log entry
0709 12:23:52.498350 (+392643us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:52.498427 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.498554 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.471558 (+973004us) raft_consensus.cc:1685] finished
0709 12:23:53.471597 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.474103 (+  2506us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10506,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":177,"thread_start_us":391862,"threads_started":1,"wal-append.queue_time_us":939803}
W20260709 12:23:53.530894  1877 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.008s	sys 0.020s
W20260709 12:23:53.545410  1877 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.010s	sys 0.022s
I20260709 12:23:53.525838  1875 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:23:53.545470  1723 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: [11.7-13.8]
W20260709 12:23:53.547637  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 315704us
I20260709 12:23:53.512015  2413 rpcz_store.cc:276] 0709 12:23:52.482105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.488235 (+  6130us) service_pool.cc:224] Handling call
0709 12:23:52.532676 (+ 44441us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:52.554816 (+ 22140us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:23:52.554852 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:52.555668 (+   816us) log.cc:844] Serialized 52 byte log entry
0709 12:23:53.048035 (+492367us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:23:53.048142 (+   107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.048351 (+   209us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.508179 (+459828us) raft_consensus.cc:1685] finished
0709 12:23:53.508202 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.511434 (+  3232us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":102,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":147,"thread_start_us":489270,"threads_started":1,"wal-append.queue_time_us":769144}
I20260709 12:23:53.498618  6321 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:53.510536  6163 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:23:53.551383  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:53.552353  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 278627us
I20260709 12:23:53.562885  2415 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:53.551942  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:53.564116  2417 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:23:53.514495  6042 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.968s	user 0.010s	sys 0.006s
I20260709 12:23:53.507972  6219 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:53.598335  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.507865  6429 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:53.554039  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:53.584002  6043 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.586047  6417 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:23:53.587389  6424 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:23:53.571782  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:53.558133  6326 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.936s	user 0.008s	sys 0.008s
I20260709 12:23:53.595441  1726 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
W20260709 12:23:53.597118  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.613901  1726 pending_rounds.cc:85] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 11
W20260709 12:23:53.551942  6221 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.932s	user 0.010s	sys 0.005s
I20260709 12:23:53.614176  1726 pending_rounds.cc:107] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 12:23:53.616178  1726 raft_consensus.cc:2889] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:53.618038  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 64087.3us
W20260709 12:23:53.620558  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 423752us
I20260709 12:23:53.621227  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:53.612720  6417 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:53.625387  6417 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:53.625874  6219 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.627451  6219 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:53.629258  6417 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:53.631143  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.630934  6429 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.633706  6429 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.642156  1880 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:23:53.642750  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 209735us
I20260709 12:23:53.610760  1627 catalog_manager.cc:5697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 9 to 11, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 11 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:53.643745  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:53.623518  2415 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:53.624337  6324 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:53.650208  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.652016  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:53.652963  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:53.660669  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 219665us
I20260709 12:23:53.661968  1727 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:53.662206  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.663138  1880 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:53.578209  6427 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:53.667014  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.668267  2413 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:53.613008  6163 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:53.622176  6380 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:53.676855  1627 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 10, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 10 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:53.676210  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:53.657131  6324 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:53.663288  6225 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.682449  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:53.684353  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 139724us
I20260709 12:23:53.663200  6321 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:53.686426  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72777.8us
I20260709 12:23:53.663681  6429 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:53.690682  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:53.694078  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.676231  6306 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:53.676141  6433 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.682031  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.644646  6221 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.620890  6326 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.695959  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.696987  1874 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 16 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:23:53.697722  6225 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.689070  6324 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:53.691372  6321 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:53.707568  6418 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.340s	user 0.002s	sys 0.000s
I20260709 12:23:53.709643  6355 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:53.721119  6429 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.693172  6359 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:53.698459  6138 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:23:53.729831  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59088.2us
I20260709 12:23:53.725663  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 594) took 2057 ms. Trace:
I20260709 12:23:53.731369  6428 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:53.762284  6321 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:53.770639  6359 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:53.731057  1627 catalog_manager.cc:5697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 6 to 8, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 8 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:53.732649  6332 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:53.733397  6391 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:53.734906  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1920) took 1160 ms. Trace:
I20260709 12:23:53.738912  5499 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:53.740617  6213 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 12:23:53.741578  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.708577  6437 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:53.742512  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47580.3us
I20260709 12:23:53.698330  6163 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:53.794610  6435 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 12:23:53.794824  6225 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.797386  6380 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:23:53.684604  6109 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.092s	user 0.014s	sys 0.000s
I20260709 12:23:53.799196  6109 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:53.800100  6221 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:53.750754  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55277.7us
I20260709 12:23:53.750864  1731 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:23:53.780406  6436 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:23:53.683359  6313 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:53.759827  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 141018us
I20260709 12:23:53.810316  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:53.761233  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:53.779275  6359 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:53.811767  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 50697.5us
I20260709 12:23:53.812338  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:53.770552  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:23:53.782616  1799 rpcz_store.cc:276] 0709 12:23:52.574271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.576406 (+  2135us) service_pool.cc:224] Handling call
0709 12:23:52.578237 (+  1831us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:52.608974 (+ 30737us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.609022 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:52.638011 (+ 28989us) log.cc:844] Serialized 26 byte log entry
0709 12:23:53.038624 (+400613us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:23:53.038687 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.038797 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.734662 (+695865us) raft_consensus.cc:1685] finished
0709 12:23:53.734687 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.734846 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14627,"raft.queue_time_us":21173,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"thread_start_us":279003,"threads_started":1,"wal-append.queue_time_us":675198}
W20260709 12:23:53.793712  1628 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 271) took 41424 ms (client timeout 15000 ms). Trace:
I20260709 12:23:53.786791  1627 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 5 to 10. New cstate: current_term: 10 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:53.771415  6361 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:53.757623  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 83472us
I20260709 12:23:53.803987  1800 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:53.768155  6306 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:53.771330  1877 rpcz_store.cc:276] 0709 12:23:51.668749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.125726 (+456977us) service_pool.cc:224] Handling call
0709 12:23:52.126253 (+   527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.126744 (+   491us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:52.506398 (+379654us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.928428 (+422030us) log.cc:844] Serialized 14 byte log entry
0709 12:23:53.218282 (+289854us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b180041a880. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:53.693488 (+475206us) log.cc:844] Serialized 14 byte log entry
0709 12:23:53.693749 (+   261us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:53.693802 (+    53us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:53.693859 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.693930 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.725571 (+ 31641us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11464,"prepare.queue_time_us":114830,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":78,"raft.queue_time_us":485891,"raft.run_cpu_time_us":460,"raft.run_wall_time_us":27681,"spinlock_wait_cycles":6788352,"thread_start_us":1345881,"threads_started":4,"wal-append.queue_time_us":576637}
I20260709 12:23:53.815217  1726 raft_consensus.cc:606] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 11 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:23:53.810602  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:53.825847  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:53.819109  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:53.822923  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:53.835394  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.841523  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1935) took 1186 ms. Trace:
I20260709 12:23:53.758399  6433 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:53.847746  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.871851  6324 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.842133  1800 rpcz_store.cc:276] 0709 12:23:52.662423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.738990 (+ 76567us) service_pool.cc:224] Handling call
0709 12:23:53.841207 (+1102217us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54001671d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:53.841434 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2334871296}
I20260709 12:23:53.824653  6043 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:53.875566  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.846045  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:53.878118  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59688.4us
W20260709 12:23:53.879990  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60955.1us
W20260709 12:23:53.880278  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 70080.1us
W20260709 12:23:53.884596  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82262.7us
I20260709 12:23:53.837998  6225 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.062s
I20260709 12:23:53.893266  1627 catalog_manager.cc:5697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 9, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 9 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:53.895438  1627 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 10, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 10 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:53.897186  1627 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 2 to 4, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 4 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:53.900619  6332 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:23:53.911574  6332 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:53.911998  6332 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:23:53.911844  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 123057us
I20260709 12:23:53.913461  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:53.824918  6391 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.083s
I20260709 12:23:53.916347  6391 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:53.916785  6391 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:53.920236  5499 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:53.876250  6428 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:53.921579  6428 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:53.922153  6428 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:53.840153  6313 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:53.927326  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:53.909006  6109 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:53.843514  6361 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:53.928359  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:53.929337  1875 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:53.929469  6361 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:53.930968  6438 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 12:23:53.931528  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.899089  6043 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:53.933545  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:53.934229  1878 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:53.933678  6043 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.927376  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:53.938062  1772 leader_election.cc:400] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12.
I20260709 12:23:53.938459  1772 leader_election.cc:403] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:23:53.940100  1874 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:53.866621  6348 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:53.875990  6417 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:53.945237  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 55076us
W20260709 12:23:53.945676  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 99699.2us
I20260709 12:23:53.898749  1627 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 12. New cstate: current_term: 12 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:53.949190  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.886176  6380 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:53.948426  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:53.899547  6306 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:53.954622  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:53.979034  1723 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:23:53.981640  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:53.982594  1727 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:53.958408  1877 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:53.957264  6138 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:53.968950  1878 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:53.990464  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:53.971020  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:53.982344  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:53.993170  2415 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:53.907449  6433 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:53.957037  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:53.981388  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:53.947245  2383 connection.cc:256] RPC call timeout handler was delayed by -1.3157228609998128s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1904s for a response
I20260709 12:23:53.999651  1803 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:23:54.002241  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:54.017732  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.036455  1726 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:23:54.037919  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:54.036960  1726 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:54.041077  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:54.041467  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 64182us
I20260709 12:23:54.042001  2417 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:53.909454  6434 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.083s
I20260709 12:23:53.951527  6109 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12.
I20260709 12:23:54.038722  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:54.044407  2411 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:54.042273  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.044428  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.048441  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:54.049947  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:54.038277  6437 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.051765  6391 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.052613  6437 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.053457  6391 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:54.055641  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:54.056103  6324 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:54.059130  6042 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:23:54.060726  6430 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.565s	user 0.005s	sys 0.005s
I20260709 12:23:54.061299  6430 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.063446  6174 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:54.064111  6174 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:54.065722  6174 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:54.074295  1874 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:54.033922  6313 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:54.071429  2413 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 12:23:54.067767  6429 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:54.075057  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:54.082864  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:54.075977  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 187579us
W20260709 12:23:54.044785  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 74030us
I20260709 12:23:54.036090  6027 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:54.042205  6445 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:23:53.951288  6440 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.097s
I20260709 12:23:54.073434  1726 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.093389  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:54.072202  6427 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:54.096519  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.075692  6417 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:54.077793  6433 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:54.111709  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:54.112275  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72044.8us
I20260709 12:23:54.103727  2412 raft_consensus.cc:2368] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12.
W20260709 12:23:54.104238  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 62042.5us
I20260709 12:23:54.086364  6434 consensus_queue.cc:1243] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:23:54.098409  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:54.099238  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:54.090469  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:54.117080  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.107223  1627 catalog_manager.cc:5697] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 3 to 8, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 8 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:54.119823  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:54.120285  6451 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:54.120927  1726 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:54.112943  6429 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:23:54.143792  6429 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:54.144200  6429 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:54.145154  6429 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.122632  2416 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:54.124277  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:54.133384  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:54.126580  6174 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:54.107396  6163 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:54.127712  6430 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:54.090466  6326 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:54.127815  6437 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:54.108395  6138 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:54.129117  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:54.133875  2411 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:54.137672  6225 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:23:54.138387  6219 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.510s	user 0.010s	sys 0.002s
I20260709 12:23:54.138720  6213 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:53.815346  1628 rpcz_store.cc:269] 0709 12:23:12.319096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.322376 (+  3280us) service_pool.cc:224] Handling call
0709 12:23:53.720835 (+41398459us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:43.429574 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:43.430377 (+   803us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:43.430432 (+    55us) write_op.cc:435] Acquired schema lock
0709 12:23:43.430458 (+    26us) tablet.cc:662] Decoding operations
0709 12:23:43.431881 (+  1423us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:43.431988 (+   107us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:43.432024 (+    36us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:23:43.433329 (+  1305us) tablet.cc:701] Row locks acquired
0709 12:23:43.433376 (+    47us) write_op.cc:260] PREPARE: finished
0709 12:23:43.435137 (+  1761us) write_op.cc:270] Start()
0709 12:23:43.435302 (+   165us) write_op.cc:276] Timestamp: P: 1783599823435057 usec, L: 0
0709 12:23:43.435355 (+    53us) op_driver.cc:348] REPLICATION: starting
0709 12:23:43.682187 (+246832us) log.cc:844] Serialized 5210 byte log entry
0709 12:23:45.607232 (+1925045us) spinlock_profiling.cc:243] Waited 4.8 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:45.974589 (+367357us) op_driver.cc:464] REPLICATION: finished
0709 12:23:46.066371 (+ 91782us) write_op.cc:301] APPLY: starting
0709 12:23:46.066524 (+   153us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:46.144838 (+ 78314us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:46.144901 (+    63us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:46.284576 (+139675us) spinlock_profiling.cc:243] Waited 9.48 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd532a 00007f16abd5af14 00007f16abd5aec3 00007f16abd5ae38 00007f16abd5adf3 00007f16abd5a0dc 00007f16abd83e16 00007f16abd6bcbb 00007f16abd5639d 00007f16abd59d75
0709 12:23:46.466345 (+181769us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:46.466714 (+   369us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=26,mrs_lookups=0
0709 12:23:46.466758 (+    44us) write_op.cc:312] APPLY: finished
0709 12:23:46.502432 (+ 35674us) log.cc:844] Serialized 121 byte log entry
0709 12:23:46.502624 (+   192us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:46.502810 (+   186us) write_op.cc:454] Released schema lock
0709 12:23:46.502886 (+    76us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:46.503391 (+   505us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:46.503420 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1777379,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":91327,"apply.run_cpu_time_us":28935,"apply.run_wall_time_us":437024,"cfile_cache_hit":50,"cfile_cache_hit_bytes":52300,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6219,"cfile_init":2,"lbm_read_time_us":10603,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":11,"num_ops":13,"prepare.queue_time_us":222702,"prepare.run_cpu_time_us":24012,"prepare.run_wall_time_us":479577,"raft.queue_time_us":149725,"raft.run_cpu_time_us":19576,"raft.run_wall_time_us":380282,"replication_time_us":2472181,"spinlock_wait_cycles":32851584,"thread_start_us":377200,"threads_started":4,"wal-append.queue_time_us":301479,"wal-append.run_cpu_time_us":23105,"wal-append.run_wall_time_us":3472776}]]}
I20260709 12:23:54.151806  2415 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:54.146602  6429 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:54.154609  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:54.155475  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.156035  1726 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:23:54.159359  6451 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:23:54.159811  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62136.6us
I20260709 12:23:54.160391  6221 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:54.159915  6109 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:54.093075  6348 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:54.163067  1627 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 5 to 7, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 7 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:54.184789  6434 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:54.161827  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:54.207270  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45531us
I20260709 12:23:54.158227  6432 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:54.119853  6027 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:54.202222  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:54.186097  6213 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.209203  1723 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:54.198976  6219 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20260709 12:23:54.189745  6326 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:54.221076  1627 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 9, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 9 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:54.193281  6221 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:54.232431  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.236246  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:54.202167  6348 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:54.205180  6109 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:54.205864  6451 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:54.240451  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:54.215703  6135 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.214609  6213 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:54.224776  6326 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.215372  6027 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:23:54.252684  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 98113.8us
I20260709 12:23:54.265146  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:54.269412  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:54.270308  1731 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:23:54.260785  1627 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:54.275519  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:54.277241  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:54.277473  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:54.249500  6348 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:54.281561  6423 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:54.282227  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:54.289304  1732 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:54.284734  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:54.287616  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 197238us
I20260709 12:23:54.287001  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:54.282675  6313 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:54.294049  1627 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 5 to 7, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 7 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:54.295529  1627 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 10, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 10 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:54.297873  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:54.315907  1730 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:54.293083  6313 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.267298  6326 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:54.301425  6453 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:23:54.301900  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87399.3us
I20260709 12:23:54.267032  6027 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:54.293659  6423 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:54.302634  6454 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:23:54.267097  6417 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:54.303603  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:54.303794  6457 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:23:54.267716  6135 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.347491  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.333956  6423 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:54.350173  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.351547  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.305142  1627 catalog_manager.cc:5697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 10. New cstate: current_term: 10 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.352906  1627 catalog_manager.cc:5697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 7, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 7 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.310700  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.355295  6138 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:54.355505  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44880.2us
I20260709 12:23:54.356264  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:54.357308  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:54.242419  6432 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:23:54.363548  6109 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:54.367060  6442 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.536s	user 0.003s	sys 0.003s
W20260709 12:23:54.370620  6455 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
I20260709 12:23:54.371902  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2025) took 1055 ms. Trace:
W20260709 12:23:54.373140  6462 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:23:54.372622  1724 rpcz_store.cc:276] 0709 12:23:53.316379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:53.328329 (+ 11950us) service_pool.cc:224] Handling call
0709 12:23:54.371490 (+1043161us) spinlock_profiling.cc:243] Waited 775 ms on lock 0x7b54002b7ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:54.371780 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1783358336}
I20260709 12:23:54.285365  6463 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:54.345353  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.346133  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:54.389307  1798 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:23:54.346796  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:54.298049  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:54.381680  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:54.347353  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.390702  1802 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:23:54.389978  1801 raft_consensus.cc:2376] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14. Re-sending same reply.
I20260709 12:23:54.321197  6313 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:54.398767  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65202.6us
W20260709 12:23:54.404130  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.403939  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.394047  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 890) took 1009 ms. Trace:
I20260709 12:23:54.406188  1807 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:54.268054  6043 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.111s
I20260709 12:23:54.349568  6135 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:54.411434  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.387086  5499 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.406594  2414 rpcz_store.cc:276] 0709 12:23:53.384744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:53.403925 (+ 19181us) service_pool.cc:224] Handling call
0709 12:23:53.446480 (+ 42555us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:53.446911 (+   431us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:53.446939 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:53.447522 (+   583us) log.cc:844] Serialized 26 byte log entry
0709 12:23:53.652536 (+205014us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:53.652590 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.652691 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:54.219518 (+566827us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:54.378347 (+158829us) raft_consensus.cc:1685] finished
0709 12:23:54.378380 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:54.393981 (+ 15601us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":116823,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":23659,"spinlock_wait_cycles":27656064,"thread_start_us":204548,"threads_started":1,"wal-append.queue_time_us":328603}
I20260709 12:23:54.413681  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:54.410063  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:54.423476  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:54.385314  6306 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.413475  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:54.441799  1805 raft_consensus.cc:2376] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14. Re-sending same reply.
I20260709 12:23:54.437227  1807 raft_consensus.cc:2376] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10. Re-sending same reply.
I20260709 12:23:54.414587  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:54.439525  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 306193us
I20260709 12:23:54.388540  6463 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:54.443603  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:54.394752  6138 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:54.449869  1800 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:54.450924  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:54.410151  6427 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:23:54.451758  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.397127  6432 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:54.460834  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 173978us
I20260709 12:23:54.442896  6042 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.489190  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.492157  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:54.477174  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 179168us
I20260709 12:23:54.398759  6332 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.494277  1804 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:23:54.494518  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 80963.4us
I20260709 12:23:54.454710  6138 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.497675  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.455600  6440 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:54.498072  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:54.504077  1807 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:54.496877  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 264504us
I20260709 12:23:54.505334  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.505374  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:54.506722  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:54.507419  1800 raft_consensus.cc:2376] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8. Re-sending same reply.
I20260709 12:23:54.461386  6427 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:54.497924  6471 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:54.481109  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:54.509383  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:54.506215  1806 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:54.460532  6464 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:54.514379  6464 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.514884  6464 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:54.515105  6461 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.141s	user 0.002s	sys 0.000s
I20260709 12:23:54.446691  5499 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.449986  6463 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:54.518283  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:54.403906  6221 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.519881  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:54.513442  1805 raft_consensus.cc:2376] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15. Re-sending same reply.
I20260709 12:23:54.525988  1807 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:54.504407  6138 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:54.512420  6432 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:54.503723  6440 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:54.452363  6306 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.517288  6427 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:23:54.539718  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 206052us
I20260709 12:23:54.541023  1798 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260709 12:23:54.541868  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 85656.5us
I20260709 12:23:54.516808  6471 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:54.499747  6042 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.552229  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:54.558315  6458 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260709 12:23:54.559791  6469 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.533314  6477 raft_consensus.cc:2764] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election decision vote started in defunct term 11: won
I20260709 12:23:54.571871  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:54.542944  6432 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:54.545230  6440 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:54.571521  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.569772  6472 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:54.582772  1729 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: [11.7-13.8]
I20260709 12:23:54.583101  6472 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:54.583469  6472 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:54.583817  6472 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:54.584288  6324 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:54.584707  6324 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:54.585577  6324 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.588191  6042 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:54.560392  6469 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.591174  6219 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:54.591935  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 591) took 1408 ms. Trace:
I20260709 12:23:54.609664  1803 rpcz_store.cc:276] 0709 12:23:53.183799 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:53.243201 (+ 59402us) service_pool.cc:224] Handling call
0709 12:23:54.591728 (+1348527us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b5400167bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:54.591877 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1251917696}
I20260709 12:23:54.578948  6432 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:54.597357  1723 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:23:54.552080  6471 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:54.605511  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 87290.7us
I20260709 12:23:54.606256  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:54.617002  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:54.632146  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.608942  6324 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:54.635746  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.635977  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:54.639854  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:54.617005  6332 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:54.649005  5499 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:54.649308  2414 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:23:54.657253  6474 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:23:54.644586  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:54.662384  6174 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.507s	user 0.007s	sys 0.002s
I20260709 12:23:54.663180  6483 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:54.667135  6321 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.892s	user 0.008s	sys 0.007s
I20260709 12:23:54.672921  6306 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:54.673974  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45820.8us
I20260709 12:23:54.674352  6451 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:54.618255  5903 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
W20260709 12:23:54.675613  6476 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
W20260709 12:23:54.681722  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75539.7us
W20260709 12:23:54.683262  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 79002.2us
I20260709 12:23:54.639935  6430 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:54.713220  6464 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.714658  1801 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:54.715896  6434 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:54.703927  6332 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 12:23:54.718078  6430 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:54.719619  6430 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.721493  6464 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:54.723416  6472 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.727077  6440 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:54.684787  6428 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.752s	user 0.008s	sys 0.002s
I20260709 12:23:54.734665  6433 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:54.737668  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.738122  6332 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:54.687136  5499 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:54.705982  6451 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:54.774171  6451 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:54.774052  6027 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.776496  6027 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:54.777355  6469 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:54.745707  1731 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:23:54.755395  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:54.761307  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:54.761894  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.782630  1725 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:23:54.765252  1723 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:54.783327  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.741904  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:54.761941  6359 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.900s	user 0.010s	sys 0.004s
I20260709 12:23:54.769954  5499 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:54.771136  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:54.771178  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:54.772181  6481 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
W20260709 12:23:54.772416  6429 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.588s	user 0.010s	sys 0.003s
I20260709 12:23:54.692803  6321 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.772573  6468 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.079s
I20260709 12:23:54.763603  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.768085  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:54.787688  6420 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:54.760149  6433 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:54.794061  6027 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:54.788048  6463 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.638912  6423 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.791473  6427 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.781692  1873 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
W20260709 12:23:54.793073  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51248.2us
I20260709 12:23:54.825623  1804 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: [11.8-13.9]
I20260709 12:23:54.795928  1772 leader_election.cc:400] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12.
I20260709 12:23:54.796505  6464 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:23:54.798518  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 227604us
I20260709 12:23:54.762163  6428 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.820639  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:54.821707  2419 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:54.823408  1876 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
W20260709 12:23:54.841986  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 344840us
W20260709 12:23:54.843559  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 107719us
I20260709 12:23:54.831250  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:54.780205  6348 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.843135  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:54.856254  1879 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:54.857751  2413 raft_consensus.cc:2410] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:54.857906  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:54.855921  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:54.800693  6359 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.865164  1804 raft_consensus.cc:2376] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 13. Re-sending same reply.
I20260709 12:23:54.863000  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:54.866060  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:54.866554  1731 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:54.864372  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 223828us
I20260709 12:23:54.878548  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:54.885396  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:54.885722  1802 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:54.889024  1730 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:54.814597  6471 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.855677  6135 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.906328  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:54.907204  1881 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:23:54.821692  6313 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.859649  6429 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:54.911525  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:54.924676  1729 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:54.915169  6485 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:54.906697  6219 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.009s
I20260709 12:23:54.915910  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:54.927318  1799 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:54.928977  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:54.873613  6488 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:54.922892  6487 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:54.931074  6489 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.950634  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:54.933183  6440 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:54.934857  6428 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:54.935472  6472 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:54.963601  1872 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:54.935746  6348 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:54.937176  6313 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:54.937685  6429 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:54.957355  1731 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:54.940726  6427 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:54.942349  6423 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:54.872972  6332 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:54.872969  6138 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.945679  6475 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:54.959466  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.010519  1803 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:54.931574  6321 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:54.950129  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:54.974769  1874 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:54.939910  6432 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:54.977555  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 140370us
W20260709 12:23:54.945238  6213 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.655s	user 0.006s	sys 0.005s
W20260709 12:23:54.945238  6458 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:23:54.930801  6326 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.592s	user 0.012s	sys 0.001s
W20260709 12:23:55.034610  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149307us
W20260709 12:23:54.997983  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.942574  6471 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:54.961488  6174 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:54.947088  6359 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:54.943478  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.020289  1798 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: [13.7-14.8]
I20260709 12:23:54.980129  6487 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:54.943336  6135 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.075630  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.076591  1805 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:23:55.024856  6430 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 12:23:55.061431  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.100818  1800 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:55.047428  1806 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:23:55.105533  1806 raft_consensus.cc:606] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:23:55.008345  6483 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:55.065317  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:55.066552  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 188583us
I20260709 12:23:55.017031  6488 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.926149  6433 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.015628  6492 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.138981  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:55.046545  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:55.045935  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:55.079847  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.387102  1804 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:23:55.080113  6487 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:54.983611  6489 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.088821  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:55.091244  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:55.389339  1879 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:55.092247  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:55.093160  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:55.093983  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:55.097164  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.016541  6428 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:55.106166  1806 raft_consensus.cc:614] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:55.017128  6472 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:55.102259  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:55.071230  6391 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.017s	user 0.015s	sys 0.003s
I20260709 12:23:55.060810  6174 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:55.125743  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.126837  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:55.446475  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.219398  6463 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:55.174256  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.450150  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.262300  6451 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.076952  6326 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.268262  6478 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.213294  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:55.181113  1725 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:55.268246  6432 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:55.231125  6027 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:55.201704  6466 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:55.123548  6109 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
I20260709 12:23:55.467062  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.268373  6467 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.469688  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.118510  6359 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:55.476681  6359 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.477134  6466 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.479061  6515 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:55.423257  6472 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:55.480059  6472 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:55.480994  1805 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:55.320729  1875 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:55.179737  6488 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.042745  6213 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.437737  6428 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:55.189240  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 90085.7us
W20260709 12:23:55.269300  6503 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 12:23:55.229943  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 478542us
I20260709 12:23:55.484849  1873 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:55.485335  6427 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:55.486117  6427 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:55.487264  6427 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.488468  6321 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.489068  6321 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.400926  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.491392  6430 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:55.189045  6492 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.436254  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:55.496290  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:55.496894  1881 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:55.095316  6135 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:55.447543  1727 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 13
W20260709 12:23:55.242336  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 177118us
I20260709 12:23:55.400205  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.448940  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:55.499317  1730 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:55.499572  1879 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:55.500833  1879 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:55.183034  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.502080  6499 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:55.502537  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.503742  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.498601  6221 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:55.504552  1728 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:23:55.504707  6499 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:55.505251  6499 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:55.505347  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:55.451574  1874 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:23:55.507022  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:55.507187   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:55.452575  1806 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:55.474283  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.509069  6499 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.510396  6499 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:55.508824  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:55.510944  6138 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:55.510620  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.461735  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:55.514353  1878 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:55.462183  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:55.463408  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.450953  1804 raft_consensus.cc:2368] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10.
I20260709 12:23:55.232652  1872 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:23:55.470482  1726 raft_consensus.cc:2376] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14. Re-sending same reply.
W20260709 12:23:55.472642  6501 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:23:55.474238  6326 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:55.474504  6484 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.475171  6467 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.475924  6451 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:55.467062  6478 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.538450  1878 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:55.479705  6515 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.492416  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.401580  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.493017  6359 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:55.501435  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.540475  1732 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:55.540618  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:55.512995  1806 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 8. Current term is 9. Ops: [6.7-8.8]
I20260709 12:23:55.541683  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:55.460364  1731 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:23:55.465958  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 420088us
I20260709 12:23:55.529335  1879 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 12:23:55.441645  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:55.531111  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.435711  6429 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:55.536417  1801 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:23:55.543159  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:55.545615  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:55.545847  1877 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:55.547293  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.548030  1725 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:55.552031  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.552196  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:55.553043  1872 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:55.552035  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:55.554108  1726 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:55.554646  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:55.557004  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 115917us
I20260709 12:23:55.559667  1727 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:55.562448  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:55.565083  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:55.566977  1801 raft_consensus.cc:2376] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13. Re-sending same reply.
I20260709 12:23:55.567253  1723 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:55.570021  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:55.570961  1801 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:55.613778  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:55.614005  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:55.614449  1801 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:55.615403  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.616248  1727 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:55.617578  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59392.6us
I20260709 12:23:55.619745  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:55.617894  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:55.624185  1731 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:55.625623  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:55.627136  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86589.7us
I20260709 12:23:55.635051  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:55.551214  6027 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:23:55.652264  1729 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.726s	user 0.008s	sys 0.003s
I20260709 12:23:55.653952  1729 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:55.657402  1725 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:23:55.642886  1806 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:55.655573  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.664178   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:55.664505  6467 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:55.665445  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:55.556188  6213 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:55.563088  6432 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:55.670933  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:55.680018  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 117362us
I20260709 12:23:55.680766  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:55.614573  6135 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:55.667861  1872 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:23:55.635939  1731 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:55.684100  1727 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:23:55.618081  6440 raft_consensus.cc:2764] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:23:55.688298  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:55.685885  1804 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:23:55.688925  1725 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:55.637894  6451 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:55.730820  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50115.3us
I20260709 12:23:55.736681  1806 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:23:55.740063  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:55.740768  1872 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:55.737468  1725 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.6-12.6]
I20260709 12:23:55.736693  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:55.753968  1731 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:55.757380  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2173) took 1002 ms. Trace:
I20260709 12:23:55.758041  1880 rpcz_store.cc:276] 0709 12:23:54.754530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.755188 (+   658us) service_pool.cc:224] Handling call
0709 12:23:54.755556 (+   368us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:54.764640 (+  9084us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:54.764678 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:54.770126 (+  5448us) log.cc:844] Serialized 52 byte log entry
0709 12:23:55.008809 (+238683us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:55.008865 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.008956 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:55.757055 (+748099us) raft_consensus.cc:1685] finished
0709 12:23:55.757087 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:55.757293 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13603,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"thread_start_us":203533,"threads_started":1,"wal-append.queue_time_us":371030}
I20260709 12:23:55.764106  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:55.765024  1880 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:55.766134  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:55.769073  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149343us
I20260709 12:23:55.632974  6393 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.106s
I20260709 12:23:55.667909  6520 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:55.687099  6027 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:55.778812  1725 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 12:23:55.690866  6135 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.774804  1728 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 6. Current term is 7. Ops: []
I20260709 12:23:55.561684  6174 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.789448   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:55.746742  6429 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:55.745920  6483 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:55.790380  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:55.792129  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:55.795660  6488 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:55.797111  6466 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:55.840363  2419 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1024) took 1718 ms (client timeout 1912 ms). Trace:
I20260709 12:23:55.545718  6484 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:55.840953  2419 rpcz_store.cc:269] 0709 12:23:54.113152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.128747 (+ 15595us) service_pool.cc:224] Handling call
0709 12:23:55.831327 (+1702580us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b5400525154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:55.831528 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1501926784}
I20260709 12:23:55.792201  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.849018  1802 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:55.850449  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:55.851358  1875 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:55.803647  6485 raft_consensus.cc:2955] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:23:55.808945  6477 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:55.818817  6135 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:55.811470  1806 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:55.811968  6492 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:55.820362  1872 raft_consensus.cc:2410] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:23:55.825592  6324 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.750280  6451 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:23:55.831809  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 92951.7us
I20260709 12:23:55.880251  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:55.831791  6483 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:55.859377  1799 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1951) took 1513 ms (client timeout 1769 ms). Trace:
W20260709 12:23:55.864987  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.818641  6348 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.737708  6219 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:55.825902  6505 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:55.862066  1772 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.912s (SENT) [suppressed 33 similar messages]
I20260709 12:23:55.848649  6488 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:55.844105  6496 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:55.889292  6478 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:55.888828  6027 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.892199  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.929639  1806 raft_consensus.cc:2410] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:23:55.845468  6521 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:55.836813  6391 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:55.837461  6489 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:55.898947  1799 rpcz_store.cc:269] 0709 12:23:54.345516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.347220 (+  1704us) service_pool.cc:224] Handling call
0709 12:23:55.859133 (+1511913us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b54001d4a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:55.859299 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1291506944}
I20260709 12:23:55.896029  6432 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.906699  6483 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:55.924108  1880 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:55.866716  6393 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:23:55.882438  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:55.903231  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.897729  6451 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:55.843223  1800 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
W20260709 12:23:55.949283  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 157330us
I20260709 12:23:55.910117  6429 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:55.917167  6321 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:55.962015  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:55.937441  6477 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.982725  6477 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.936334  6391 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:55.898358  6324 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.939975  6495 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:55.975162  1799 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:55.921358  6219 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:55.979413  6502 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:23:55.980163  1878 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:55.938218  6496 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:55.988942  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:23:55.992143  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 130112us
W20260709 12:23:55.995694  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 113343us
I20260709 12:23:55.998301  1880 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:23:55.917469  6348 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:55.985809  6477 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.004608  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:56.006259  1799 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:55.937107  6488 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.987174  6467 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:55.991312  6492 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.089507  6219 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.089996  6492 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.092314  6492 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:56.094403  6527 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
W20260709 12:23:56.095429  6511 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.334s	user 0.001s	sys 0.001s
W20260709 12:23:56.027421  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.096405  6496 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:55.956688  6521 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.100510  6521 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:55.950160  6432 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.103087  6432 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:55.936645  6505 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:56.049880  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:56.104285  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.055143  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:56.073254  1806 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:56.105782  6505 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.107051  1802 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:56.112239  1805 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:23:56.112644  1803 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:56.033268  6478 raft_consensus.cc:2764] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election decision vote started in defunct term 13: won
I20260709 12:23:56.119415  6469 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:56.120049  6469 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:56.121224  6429 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:23:56.114591  1803 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:23:56.125126  1803 pending_rounds.cc:85] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 5
I20260709 12:23:56.125664  1803 pending_rounds.cc:107] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 12.6
I20260709 12:23:56.126309  1803 raft_consensus.cc:2889] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:56.074873  1881 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.577s	user 0.009s	sys 0.007s
I20260709 12:23:56.016992  6306 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.089115  6488 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.127137  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:56.139643  6478 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:56.121286  6469 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.141552  6469 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:56.141731  2411 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:56.147571  1799 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:23:56.137486  2414 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
W20260709 12:23:56.149151  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 173857us
I20260709 12:23:56.153421  1880 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:56.074100  1801 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:56.140340  6478 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.140638  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:56.129071  1881 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:56.151033  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:56.160429  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:56.160698  1875 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:56.139869  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:56.162317  1801 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:23:56.163869  2416 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:23:56.085556  6515 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.163432  6225 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.044s
W20260709 12:23:56.205749  6518 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 12:23:56.208841  6478 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:56.215075  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57436us
I20260709 12:23:56.164786  6488 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.223989  1799 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:56.076112  6391 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.153018  6306 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.196430  6492 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:56.243824  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:56.192587  6324 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.284526  1801 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
W20260709 12:23:56.267692  6472 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.787s	user 0.013s	sys 0.000s
I20260709 12:23:56.288726  2413 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:56.288276  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44893.4us
I20260709 12:23:56.230813  6484 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.229527  6488 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.293673  1696 leader_election.cc:400] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10.
W20260709 12:23:56.277511  6534 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:23:56.295115  1696 leader_election.cc:403] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:56.261344  6492 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:56.296698  6532 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:23:56.303720  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2076) took 1329 ms. Trace:
I20260709 12:23:56.304179  1798 rpcz_store.cc:276] 0709 12:23:54.974325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.974490 (+   165us) service_pool.cc:224] Handling call
0709 12:23:54.979517 (+  5027us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:55.063819 (+ 84302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.303364 (+1239545us) spinlock_profiling.cc:243] Waited 40.4 ms on lock 0x7b540013a6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:56.303651 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92983168}
I20260709 12:23:56.292065  6536 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:23:56.313730  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 749787us
I20260709 12:23:56.314726  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:56.251343  6521 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:23:56.320000  6523 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:23:56.308967  6492 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:56.322813  1799 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 7. Current term is 8. Ops: []
I20260709 12:23:56.316557  6391 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:56.325908  6536 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.004s
I20260709 12:23:56.298930  6472 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:56.334748  6520 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.543s	user 0.010s	sys 0.004s
I20260709 12:23:56.335754  6495 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:56.353597  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:56.354823  1874 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:56.355937  6536 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:56.356498  6536 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.358306  6536 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.348276  6520 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:56.339000  6451 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.367755  1805 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:56.373070  2413 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:56.352190  6348 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.363603  6536 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:56.381575  6499 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.871s	user 0.005s	sys 0.008s
I20260709 12:23:56.390205  6483 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.393260  1878 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:23:56.389369  6306 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:56.398923  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 343849us
I20260709 12:23:56.399770  6451 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.400028  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:56.401295  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:56.402174  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.402962  1880 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
W20260709 12:23:56.404201  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89585us
I20260709 12:23:56.403209  6483 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.407799  6469 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:56.411271  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.406247  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:23:56.419682  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 66693.2us
I20260709 12:23:56.421085  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:56.425006  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:56.425565  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:56.405687  6483 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:23:56.423033  6485 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.533s	user 0.008s	sys 0.005s
I20260709 12:23:56.427446  6466 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:23:56.430114  6484 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:56.431180  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.431727  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:56.431957  1877 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:56.432338  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:56.432650  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.433350  1723 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:23:56.433495  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.434229  1726 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:56.434468  6484 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:56.435256  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.435798  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.436141  1728 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:23:56.436691  6428 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.437383  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.438285  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.438711  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:56.441224  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:56.427351  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.444545  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.411374  6495 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:56.447446  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:56.448377  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:56.406260  6523 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10.
I20260709 12:23:56.449877  5903 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:56.469143  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43642.8us
W20260709 12:23:56.449992  6221 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.943s	user 0.014s	sys 0.003s
I20260709 12:23:56.491971  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.439626  6488 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:56.485288  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.485496  6516 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:56.490932  6321 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:56.491283  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.471082  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:56.491299  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:56.451035  6484 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:56.498185  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:56.430123  6174 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:56.453812  6213 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:56.460016  6466 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:23:56.477357  6467 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:56.477200  6432 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:56.525635  1723 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:23:56.497319  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:56.470021  6483 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:56.515193  6429 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:23:56.531095  6393 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:56.531922  6393 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:56.530870  6483 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.536342  6504 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:23:56.538316  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2836) took 1033 ms. Trace:
I20260709 12:23:56.564541  6219 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:23:56.565059  6219 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:56.564839  1874 rpcz_store.cc:276] 0709 12:23:55.504920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.506743 (+  1823us) service_pool.cc:224] Handling call
0709 12:23:56.538025 (+1031282us) spinlock_profiling.cc:243] Waited 847 ms on lock 0x7b54002058d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:56.538220 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1947870592}
I20260709 12:23:56.565980  6469 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:56.540197  6332 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:56.549721  6434 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:56.550042  6492 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.552508  1724 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
W20260709 12:23:56.553709  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82438.2us
I20260709 12:23:56.581732  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:56.582718  1730 raft_consensus.cc:2376] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13. Re-sending same reply.
I20260709 12:23:56.554651  1876 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
W20260709 12:23:56.555267  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57275.2us
I20260709 12:23:56.524385  6432 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:56.551052  6135 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.000s
I20260709 12:23:56.559791  6429 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:56.560304  6221 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:56.534524  6472 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.553000  6393 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.539687  6464 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.499910  6321 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.609798  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:56.566371  6219 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.611121  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:56.567575  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70360.5us
I20260709 12:23:56.577011  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:56.584493  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:56.515758  6516 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:56.558585  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.587188  6213 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:23:56.588976  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 93914.7us
I20260709 12:23:56.590317  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:23:56.568014  6545 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:23:56.596362  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:56.567129  6332 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.630501  6043 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:56.596839  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:56.562616  6468 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:56.637590  6429 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.638049  6468 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:23:56.638173  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:56.638676  6468 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:56.639348  6348 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:56.639940  6348 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:56.639757  6468 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.641122  6468 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:23:56.640973  6348 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.607939  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:56.642591  1881 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:56.643121  1799 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:56.644198  6516 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.597944  6042 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 12:23:56.647776  6042 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:56.581897  6467 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:56.637080  6043 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.644722  2417 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:23:56.631223  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.613704  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:56.631304  1730 raft_consensus.cc:2376] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11. Re-sending same reply.
I20260709 12:23:56.635471  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:56.651017  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:56.632511  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:56.652519  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:56.661093  1807 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:56.658015  2410 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:56.658150  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:56.662133  1875 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 13 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
W20260709 12:23:56.662845  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.664199  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:56.664197  1879 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:23:56.666900  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.599078  6432 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:56.609484  5903 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:56.668071  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.669080  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:56.667590  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:56.680364  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:56.670316  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 57181.8us
I20260709 12:23:56.682096  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:56.684022  2411 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.12-12.12]
I20260709 12:23:56.584127  6434 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.684453  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:56.690376  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.703554  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:56.705962  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.711474  1878 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:23:56.597481  6523 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260709 12:23:56.711330  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:56.655843  6042 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:56.714025  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:56.714495  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.715287  1880 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:56.607591  6477 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 12:23:56.722671  6332 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.635095  6464 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.666332  6043 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:56.723047  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:56.723922  6042 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.726861  6433 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:56.673152  6225 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:56.759326  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.727393  6213 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:56.730824  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2180) took 1642 ms (client timeout 1862 ms). Trace:
W20260709 12:23:56.762039  1876 rpcz_store.cc:269] 0709 12:23:55.088090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.092886 (+  4796us) service_pool.cc:224] Handling call
0709 12:23:56.730565 (+1637679us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54001f7554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:56.730720 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2529981184}
I20260709 12:23:56.730544  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:56.732043  6554 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:23:56.664491  6221 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:56.735592  6417 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:56.735646  1875 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:56.668799  6324 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:56.608817  6135 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.609493  6549 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.106s
I20260709 12:23:56.678964  6469 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:56.748080  2417 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:56.683768  6428 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:56.717566  6326 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:56.745318  6027 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:56.773443  1805 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:56.685487  6536 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260709 12:23:56.704571  5903 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.816012  5903 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.817616  6523 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:56.818439  6523 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:56.819064  6464 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.819761  6523 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:56.808399  6548 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 12:23:56.747742  6467 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:23:56.812678  6326 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:56.743294  6496 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:23:56.757328  6332 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
W20260709 12:23:56.829139  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 171070us
I20260709 12:23:56.780546  6225 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:56.808203  6135 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.784346  6221 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.801501  6027 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.869452  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:56.838196  6042 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:56.842875  6391 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.004s
W20260709 12:23:56.872068  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 204641us
W20260709 12:23:56.872128  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 152325us
W20260709 12:23:56.875983  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 213968us
I20260709 12:23:56.861254  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:56.891436  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.892570  1880 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:23:56.801182  6428 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:56.883867  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:56.919903  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:56.928665  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2823) took 2024 ms (client timeout 1820 ms). Trace:
I20260709 12:23:56.923168  1724 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:56.903542  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:23:56.929198  1873 rpcz_store.cc:269] 0709 12:23:54.904138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.906011 (+  1873us) service_pool.cc:224] Handling call
0709 12:23:56.928399 (+2022388us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b54001f8954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:56.928568 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1314747264}
I20260709 12:23:56.943286  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:56.947887  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:56.949702  6326 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:56.883226  6027 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:56.958619  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:56.961308  1880 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:56.897271  6492 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:56.935917  6523 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:56.968726  1879 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:56.918280  6464 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:56.949126  6534 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:23:56.926252  6429 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:23:56.969944  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66496us
I20260709 12:23:56.915139  6496 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:23:56.980659  1728 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:23:56.981070  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:23:56.981812  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75939.9us
I20260709 12:23:56.963574  6569 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.963990  6516 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.944245  6391 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:23:56.966082  6321 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.998594  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:56.977027  6464 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.974563  6135 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:56.981247  6492 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:23:57.015403  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.995692  6434 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:56.983544  6225 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.033232  6576 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:57.034265  6576 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:57.035631  6576 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.015599  6135 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.038969  6135 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:57.041793  6477 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:57.042352  6477 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:23:57.043937  6565 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:23:57.044961  6551 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.262s	user 0.001s	sys 0.001s
I20260709 12:23:57.058503  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:57.060031  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:57.060997  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:57.063148  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:57.063822  2414 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:57.015808  6572 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.067700  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:57.020954  6485 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:57.016247  6464 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:56.987134  6429 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.071033  2412 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:57.075824  1728 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:23:57.049939  6391 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:57.017596  6432 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.004312  6569 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.059744  6492 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:57.019268  6564 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
W20260709 12:23:57.099565  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 101064us
I20260709 12:23:57.006301  6516 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:57.102479  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 159296us
I20260709 12:23:57.081225  6391 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:57.103538  6488 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.089648  6485 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:57.107486  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.108093  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.109375  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.110934  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.113821  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.124063  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:57.114142  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.120137  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:57.135526  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 187186us
I20260709 12:23:57.132082  1801 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
W20260709 12:23:57.145334  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 140465us
I20260709 12:23:57.146688  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:57.152238  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.132303  2418 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:57.153894  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.154863  1801 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:57.155490  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.155784  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.156554  1801 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:57.156749  1798 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:23:57.160938  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:57.163977  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.164680  1799 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:57.164227  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:57.168412  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.168661  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.106067  6505 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:23:57.111610  6496 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:57.111135  6579 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:57.113703  6428 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.106040  6324 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:57.113787  6391 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:57.173447  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 612) took 1385 ms. Trace:
I20260709 12:23:57.115156  6516 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.143479  6489 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:57.169958  6221 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.188774  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.205379  1799 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:57.074448  6572 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:23:57.182699  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 73124.3us
I20260709 12:23:57.193329  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.195055  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:23:57.183769  6561 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
I20260709 12:23:57.151640  6472 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:57.152582  6574 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:57.196014  6505 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:57.225812  6574 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.226361  6505 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.228775  6574 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:57.204649  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:23:57.232175  6544 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.867s	user 0.002s	sys 0.000s
I20260709 12:23:57.235739  6472 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:57.237700  6557 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.416s	user 0.001s	sys 0.000s
I20260709 12:23:57.239084  6429 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:57.239459  6468 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.598s	user 0.012s	sys 0.002s
I20260709 12:23:57.240978  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2088) took 1701 ms. Trace:
I20260709 12:23:57.240942  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.241629  1806 rpcz_store.cc:276] 0709 12:23:55.539888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.105967 (+566079us) service_pool.cc:224] Handling call
0709 12:23:56.106334 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.107978 (+  1644us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.108022 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.108547 (+   525us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.338680 (+230133us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:56.338714 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.338763 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.231480 (+892717us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:57.240653 (+  9173us) raft_consensus.cc:1685] finished
0709 12:23:57.240678 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.240899 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2378,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"spinlock_wait_cycles":2417920,"thread_start_us":229341,"threads_started":1,"wal-append.queue_time_us":256520}
I20260709 12:23:57.196995  1803 rpcz_store.cc:276] 0709 12:23:55.788441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.158323 (+369882us) service_pool.cc:224] Handling call
0709 12:23:56.158800 (+   477us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:56.175276 (+ 16476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.175319 (+    43us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:56.176224 (+   905us) log.cc:844] Serialized 78 byte log entry
0709 12:23:56.377247 (+201023us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:56.390981 (+ 13734us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.391401 (+   420us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.421649 (+ 30248us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.421938 (+   289us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.422019 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.522486 (+100467us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b180048cee0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:23:57.173088 (+650602us) raft_consensus.cc:1685] finished
0709 12:23:57.173122 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.173356 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31449,"prepare.queue_time_us":14093,"prepare.run_cpu_time_us":153,"prepare.run_wall_time_us":948,"raft.queue_time_us":43160,"raft.run_cpu_time_us":435,"raft.run_wall_time_us":31609,"spinlock_wait_cycles":38369792,"thread_start_us":100978,"threads_started":1,"wal-append.queue_time_us":239070}
I20260709 12:23:57.212711  1801 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:57.215356  1807 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:57.234349  1832 heartbeater.cc:646] Failed to heartbeat to 127.0.140.126:43371 (2 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.140.126:43371 timed out after 15.000s (SENT)
I20260709 12:23:57.213670  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:57.223579  1804 raft_consensus.cc:2410] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
W20260709 12:23:57.229278  6570 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
W20260709 12:23:57.251277  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 90452us
I20260709 12:23:57.252141  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.252415  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:57.120857  6488 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.144759  6043 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:57.266389  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52848.9us
I20260709 12:23:57.267315  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.269284  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.271415  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:57.272531  1873 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:57.253093  1798 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:23:57.247530  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.278501  1726 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:23:57.280639  1726 pending_rounds.cc:85] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 7
I20260709 12:23:57.280926  1726 pending_rounds.cc:107] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:23:57.281220  1726 raft_consensus.cc:2889] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:57.281682  1726 pending_rounds.cc:107] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:23:57.281850  1807 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
W20260709 12:23:57.278630  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.281972  1726 raft_consensus.cc:2889] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:57.285848  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.288543  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:57.194195  6496 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.289335  1875 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:57.244899  6472 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:57.297614  1801 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:57.214620  6564 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260709 12:23:57.291944  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.292351  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.301946  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:57.303112  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.305701  6589 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:57.306440  6589 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:57.307439  6589 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.312289  6043 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:57.312582  1728 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:57.317874  6135 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.293578  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.257290  6451 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.296712  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.322398  2414 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:57.341043  1844 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:57.322561  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.293803  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:57.342587  1879 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:23:57.304823  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.329416  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:57.334057  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:57.247298  6588 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:57.346398  1798 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:23:57.217330  6489 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.351698  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.197532  6579 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.384907  1730 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:23:57.391219  1726 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:57.247970  6042 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.092s
W20260709 12:23:57.395627  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66312.2us
I20260709 12:23:57.396524  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:57.273818  6583 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:57.390373  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:57.297130  6575 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:57.398124  1725 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:57.402987  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:57.405202  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:57.383242  6221 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:57.387801  6573 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:57.222743  6492 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.383374  6348 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:57.339864  6505 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:57.266813  6485 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.439731  6592 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.439421  6576 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:57.440411  6563 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:57.439778  6590 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:57.431267  6472 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:57.382406  6574 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:23:57.495805  6590 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:57.504405  6601 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:57.505841  6579 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:57.312362  6560 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.507s	user 0.003s	sys 0.000s
I20260709 12:23:57.451150  6573 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:57.508004  6595 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:23:57.510061  6489 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:57.511333  6586 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:23:57.408980  6213 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:23:57.517889  6135 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.462102  6569 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:57.433854  6467 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:23:57.432890  6432 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:57.384341  6451 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:57.498587  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 217328us
I20260709 12:23:57.447477  6433 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.449080  6505 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.503010  6572 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:23:57.542738  6174 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.034s	user 0.003s	sys 0.012s
I20260709 12:23:57.523216  1799 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:23:57.523993  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.392807  6324 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.588042  1725 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:57.589669  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:57.592607  6599 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.070s	user 0.001s	sys 0.001s
I20260709 12:23:57.593180  6594 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:57.593662  6580 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:57.597427  1807 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
W20260709 12:23:57.598886  6521 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.233s	user 0.002s	sys 0.018s
I20260709 12:23:57.492523  6468 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:57.498687  6221 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:23:57.601778  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 196630us
I20260709 12:23:57.662107  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:57.663565  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:57.664819  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:57.599982  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 347592us
W20260709 12:23:57.587862  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.605427  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:57.601975  6505 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:57.602530  6433 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.667759  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:57.611433  1799 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:57.616717  1731 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
W20260709 12:23:57.668385  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.037s	sys 0.045s
W20260709 12:23:57.668915  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.037s	sys 0.045s
W20260709 12:23:57.669209  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 507880us
I20260709 12:23:57.491854  6306 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.110s
I20260709 12:23:57.388818  6516 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 12:23:57.495893  6472 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:57.654837  1807 pending_rounds.cc:85] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 9
I20260709 12:23:57.675269  1807 pending_rounds.cc:107] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:23:57.675643  1807 raft_consensus.cc:2889] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:57.683004  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:57.684072  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:57.685757  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:57.654178  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64674.4us
I20260709 12:23:57.665668  2414 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:57.662246  6521 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.294s	user 0.003s	sys 0.019s
W20260709 12:23:57.660744  6427 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.828s	user 0.002s	sys 0.021s
I20260709 12:23:57.658618  6174 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:57.691052  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:57.654206  2413 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
W20260709 12:23:57.699090  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.654188  6594 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:57.668597  6468 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.496387  6592 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:57.708349  6427 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.893s	user 0.003s	sys 0.021s
I20260709 12:23:57.726701  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:57.668807  6580 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:57.728147  1807 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:57.736742  1807 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:57.737802  6603 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:57.656709  6348 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:23:57.739161  6598 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:57.739569  6598 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:57.739655  6594 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.740156  6042 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:57.681674  6472 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:57.671904  6432 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:57.727051  6484 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:57.671999  6221 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:57.737684  6043 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:57.740476  6598 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.673357  6428 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:57.713069  6467 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:57.729159  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 386148us
W20260709 12:23:57.776274  6556 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.964s	user 0.004s	sys 0.000s
I20260709 12:23:57.776876  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:57.743639  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:57.778335  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 906) took 1226 ms. Trace:
I20260709 12:23:57.778647  6521 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:57.779238  6521 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:57.780259  6592 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.780198  6521 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.748296  6504 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:57.781854  6521 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:57.733681  6489 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
W20260709 12:23:57.784264  6597 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:23:57.784960  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.761179  6484 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.785640  6484 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:57.763337  6391 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.542s	user 0.007s	sys 0.003s
I20260709 12:23:57.786741  6391 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.788800  6391 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:57.790020  6433 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.791749  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:57.718056  6574 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:57.792102  6573 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.757793  1726 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:57.748570  6564 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:23:57.795396  6221 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.796233  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:57.797703  6042 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:57.740499  6580 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.798220  6042 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:57.798022  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:23:57.800962  6027 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.829s	user 0.011s	sys 0.003s
I20260709 12:23:57.801674  6391 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:57.802350  6391 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:57.801589  6027 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.803058  6391 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:57.804246  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.723832  2414 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:57.760402  6602 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:57.775312  6598 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:57.791939  6609 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:57.800748  6042 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:57.772069  6526 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.881s	user 0.002s	sys 0.002s
I20260709 12:23:57.808111  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:57.799124  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:57.765146  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 79480.9us
I20260709 12:23:57.809026  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.799393  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:57.799082  6432 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:57.831736  6432 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.834160  6043 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.763284  6496 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:57.834808  6043 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.835258  6602 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:57.835650  6602 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:57.835495  6468 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:57.836849  6468 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:57.837147  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.837558  6468 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:57.838505  6523 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.856s	user 0.002s	sys 0.010s
I20260709 12:23:57.838693  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:57.839465  1723 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:23:57.839401  6602 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:57.779079  2419 rpcz_store.cc:276] 0709 12:23:56.552026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.552231 (+   205us) service_pool.cc:224] Handling call
0709 12:23:56.561029 (+  8798us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.561433 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:56.561775 (+   342us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.663851 (+102076us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.673594 (+  9743us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.673756 (+   162us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:56.673790 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.673881 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.775381 (+1101500us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:57.777910 (+  2529us) raft_consensus.cc:1685] finished
0709 12:23:57.778058 (+   148us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.778252 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2095,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":150871,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":204,"spinlock_wait_cycles":24724224,"thread_start_us":93796,"threads_started":1,"wal-append.queue_time_us":208468}
I20260709 12:23:57.841959  6615 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:57.842873  6615 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:57.843371  6615 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:57.844623  6615 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.828289  6596 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.859246  6596 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.823598  6594 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:23:57.807662  6174 consensus_queue.cc:1175] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" responder_term: 10 status { last_received { term: 9 index: 6 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:23:57.849536  6464 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.757s	user 0.012s	sys 0.001s
I20260709 12:23:57.805207  6504 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:57.863587  6504 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:57.864166  6504 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:57.866533  6043 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.869113  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:57.871361  6505 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.825941  1807 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:57.840569  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:57.839622  1729 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:57.877615  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:57.875443  1723 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:57.884397  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.863341  6464 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.893744  6464 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:57.894963  6569 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:57.895095  6596 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.895488  6569 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:57.897588  6569 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:57.826805  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:57.899799  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.825445  6027 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:57.902587  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.911077  1800 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:57.915167  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:57.916440  1807 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:57.916532  1799 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:57.919132  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 175580us
I20260709 12:23:57.898911  6609 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:57.930158  1723 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:23:57.934540  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:57.934700  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:57.935230  1723 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:57.938076  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:57.941195  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:23:57.935760  6027 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:57.955102  6027 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:57.955262  1803 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:23:57.936710  6609 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.957250  1798 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: [10.6-13.7]
I20260709 12:23:57.959345  6472 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.960107  6478 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:57.960629  6478 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:57.961712  6478 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.962474  1726 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:23:57.962801  6505 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:57.963783  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:57.964653  6550 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.965366  6550 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.964519  6042 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.966673  6568 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:23:57.967283  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.915292  6615 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:57.953998  6569 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.961419  6602 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.991865  1723 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:57.971586  6568 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:57.994180  1799 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:57.995074  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:57.936515  6468 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.003362  6521 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.004557  6042 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.004416  6521 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:57.988716  6622 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:58.008563  6622 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:58.009467  6622 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.013995  6391 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:57.993549  6472 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:58.021203  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 80080.2us
I20260709 12:23:57.956498  6027 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.022213  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:58.024399  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:58.028115  1799 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:58.025851  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:58.029341  1723 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:23:58.029580  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:57.999226  6505 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:58.041815  6569 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:57.993865  6484 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:58.021417  6042 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:58.047756  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.017060  6609 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.025125  6594 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:58.025098  6488 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.067338  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:58.068073  6484 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:58.068792  6484 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:58.017580  6590 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.069419  6484 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.070281  6590 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:58.072803  1799 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:58.089706  6550 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:58.095371  6602 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:58.050400  6505 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:58.096372  6555 raft_consensus.cc:2764] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:23:58.096819  6505 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:58.097182  6521 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.116503  6391 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:58.123785  6627 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
W20260709 12:23:58.071631  6043 consensus_queue.cc:1175] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "123095127f334445889c21a42ba1d41e" responder_term: 10 status { last_received { term: 10 index: 8 } last_committed_idx: 7 last_received_current_leader { term: 10 index: 8 } }
I20260709 12:23:58.136377  6504 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.140250  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.072347  6042 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.143199  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 615) took 1030 ms. Trace:
I20260709 12:23:58.143491  6042 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.143656  1802 rpcz_store.cc:276] 0709 12:23:57.112877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.113030 (+   153us) service_pool.cc:224] Handling call
0709 12:23:58.143013 (+1029983us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b5400067054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:58.143144 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":378233088}
I20260709 12:23:58.091292  6594 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.145732  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:58.146880  6477 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:58.147315  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.148150  6477 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:58.149101  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:58.069428  6620 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:58.152318  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:58.153046  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.097196  6555 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:23:58.148104  2417 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:58.155427  6569 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:58.155831  6569 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:58.156701  6569 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.159534  6594 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.159818  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:58.183765  1881 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:58.160691  6219 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:23:58.161430  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.193550  6478 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:58.193994  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:58.162230  1879 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:23:58.194996  6429 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:58.195626  6429 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.195772  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3008) took 1761 ms. Trace:
I20260709 12:23:58.165030  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.196301  1731 rpcz_store.cc:276] 0709 12:23:56.434593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.444234 (+  9641us) service_pool.cc:224] Handling call
0709 12:23:58.195543 (+1751309us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54002b70d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:58.195676 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2693472000}
I20260709 12:23:58.166249  6469 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:58.197404  6469 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:58.197872  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:58.198674  6469 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:58.200503  6469 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:58.195148  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:58.201973  6515 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:58.168503  6393 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:58.196308  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:58.205020  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.168649  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:58.207036  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:58.207787  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.208750  1876 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:23:58.209569  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:58.207928  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:58.209388  6417 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.212618  6417 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.215265  6609 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:58.216302  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:58.216912  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:58.217551  1805 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:58.218283  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:58.218587  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:58.219318  1802 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:58.219918  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:58.220585  1723 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:58.223428  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:58.225720  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.226227  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 1980) took 1034 ms. Trace:
I20260709 12:23:58.226503  2417 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:23:58.226832  1801 rpcz_store.cc:276] 0709 12:23:57.192240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.269051 (+ 76811us) service_pool.cc:224] Handling call
0709 12:23:58.226046 (+956995us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b5400166cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:58.226162 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":63336704}
I20260709 12:23:58.228116  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:58.228873  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:58.231202  6488 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:23:58.231631  6488 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.232008  6488 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:23:58.210709  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.233098  1876 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:58.234074  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:58.236421  6609 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:58.237138  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:58.237690  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.238180  6043 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:58.238643  6043 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.239485  6043 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:58.171191  6326 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:58.241357  6326 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:58.171562  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2090) took 2494 ms (client timeout 1566 ms). Trace:
I20260709 12:23:58.194900  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:23:58.243031  1804 rpcz_store.cc:269] 0709 12:23:55.676921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.736420 (+ 59499us) service_pool.cc:224] Handling call
0709 12:23:58.171271 (+2434851us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b5400066154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:58.171422 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3394746752}
I20260709 12:23:58.171938  1878 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:58.244364  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.244958  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.245417  6587 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.172313  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.173774  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2056) took 1057 ms. Trace:
I20260709 12:23:58.246271  6587 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:58.246973  1728 rpcz_store.cc:276] 0709 12:23:57.116634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.133824 (+ 17190us) service_pool.cc:224] Handling call
0709 12:23:58.173572 (+1039748us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b540030af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:58.173694 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":329688448}
I20260709 12:23:58.248461  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:58.249125  1728 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:58.250181  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:58.172298  1877 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:23:58.252681  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.196096  6429 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:58.166672  6472 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.255093  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:58.255226  6472 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.168958  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.258152  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:23:58.263578  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1964) took 1135 ms. Trace:
I20260709 12:23:58.264086  1800 rpcz_store.cc:276] 0709 12:23:57.128031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.128218 (+   187us) service_pool.cc:224] Handling call
0709 12:23:57.128557 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.915938 (+787381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.263268 (+347330us) spinlock_profiling.cc:243] Waited 782 ms on lock 0x7b540035f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:58.263499 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1798859136}
I20260709 12:23:58.196724  1875 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:58.273377  1877 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:23:58.279950  1874 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:58.262506  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1969) took 1124 ms. Trace:
I20260709 12:23:58.285241  1806 rpcz_store.cc:276] 0709 12:23:57.138353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.387191 (+248838us) service_pool.cc:224] Handling call
0709 12:23:57.387574 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.433081 (+ 45507us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.433137 (+    56us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.433212 (+    75us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:57.433278 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.262189 (+828911us) spinlock_profiling.cc:243] Waited 45.1 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:58.262245 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.262429 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103804160}
I20260709 12:23:58.277180  1877 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
W20260709 12:23:58.291173  6625 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.202s	user 0.005s	sys 0.000s
W20260709 12:23:58.301893  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.303637  1728 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:58.318310  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:23:58.328104  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.238497  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.329316  2417 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:58.344892  1728 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:58.349128  6135 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:58.358141  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:58.361153  6598 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:58.369985  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3011) took 1263 ms. Trace:
I20260709 12:23:58.433269  1729 rpcz_store.cc:276] 0709 12:23:57.106402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.107873 (+  1471us) service_pool.cc:224] Handling call
0709 12:23:58.369673 (+1261800us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b54002b6bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:58.369882 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1629847808}
I20260709 12:23:58.371248  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 955) took 1732 ms. Trace:
W20260709 12:23:58.373116  6605 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.706s	user 0.001s	sys 0.002s
I20260709 12:23:58.439248  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3014) took 1331 ms. Trace:
I20260709 12:23:58.439824  1732 rpcz_store.cc:276] 0709 12:23:57.107388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.109452 (+  2064us) service_pool.cc:224] Handling call
0709 12:23:57.109807 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.194141 (+ 84334us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.194198 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:57.196661 (+  2463us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.386710 (+190049us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:57.386752 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.386865 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:58.372112 (+985247us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:23:58.438771 (+ 66659us) raft_consensus.cc:1685] finished
0709 12:23:58.438830 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.439159 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":136681,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":7024,"spinlock_wait_cycles":4440960,"thread_start_us":207380,"threads_started":2,"wal-append.queue_time_us":310775}
I20260709 12:23:58.442719  6523 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:58.169956  6504 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:58.408967  6477 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:58.243496  1880 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:58.423174  6588 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:58.444451  6588 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.443408  6523 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.443846  6477 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.446264  6523 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:58.455771  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 97745.2us
W20260709 12:23:58.478425  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.458292  6588 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:58.461862  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 254013us
I20260709 12:23:58.499541  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:58.500412  6495 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:58.462345  6135 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.462558  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.503584  1877 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:23:58.506640  6135 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.464851  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.507773  2417 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:58.464148  6477 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.465509  6609 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.509598  6609 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.513526  1806 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:23:58.516479  2417 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:23:58.518167  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:58.519727  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:58.520833  6495 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.521124  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.521457  6495 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:58.521740  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:58.468003  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.522392  1801 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:58.522846  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.468096  6587 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:58.523761  6495 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:58.469907  1802 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:23:58.470746  6472 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.468214  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.524823  1874 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:23:58.473385  6451 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:58.527112  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:58.528048  1801 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:23:58.245512  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:58.478739  1725 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:58.535562  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:58.481077  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.538301  2419 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:58.539327  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:58.506814  1879 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:58.482894  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.484125  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:23:58.484912  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.555363  2418 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:58.485280  2410 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:23:58.486411  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:58.560047  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.560782  1877 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:23:58.562213  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:23:58.458258  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 190368us
I20260709 12:23:58.494923  1880 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:58.246292  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:58.563328  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:58.512264  6609 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.595573  1874 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:58.514097  6043 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.568215  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:58.575690  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:58.434902  2416 rpcz_store.cc:276] 0709 12:23:56.638831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.659594 (+ 20763us) service_pool.cc:224] Handling call
0709 12:23:56.659970 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:56.666409 (+  6439us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:56.705246 (+ 38837us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.839844 (+134598us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:56.839920 (+    76us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:56.840001 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.370875 (+1530874us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x7b54005b0254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:58.370956 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.371168 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":38582,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":89,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":19984,"spinlock_wait_cycles":14158592,"thread_start_us":134149,"threads_started":1,"wal-append.queue_time_us":232744,"wal-append.run_cpu_time_us":2250,"wal-append.run_wall_time_us":1099676}
I20260709 12:23:58.577904  1723 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:58.579351  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.626509  2417 raft_consensus.cc:2410] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 12:23:58.627290  1728 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:58.563941  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
W20260709 12:23:58.596139  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 109781us
W20260709 12:23:58.465960  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 237926us
I20260709 12:23:58.596797  1877 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:23:58.596752  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 351316us
I20260709 12:23:58.597879  2413 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:23:58.632090  1731 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:23:58.633911  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:58.625619  1802 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:58.604650  1881 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:23:58.634136  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:23:58.629012  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:58.641693  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66111.7us
I20260709 12:23:58.639258  1731 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:23:58.645676  1802 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:23:58.647096  1879 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:23:58.643091  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:58.647360  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.647701  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:58.648676  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:23:58.650287  6549 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.844s	user 0.007s	sys 0.018s
I20260709 12:23:58.595587  6326 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.640342  6135 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:58.640522  6602 raft_consensus.cc:2764] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:23:58.681739  1729 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:58.651793  6505 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.017s	sys 0.013s
I20260709 12:23:58.657210  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:58.687069  1723 raft_consensus.cc:2410] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 12 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 11.
I20260709 12:23:58.688032  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:58.664487  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:58.640198  6495 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:58.461444  6484 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.658245  6590 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:58.677479  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3012) took 1570 ms. Trace:
I20260709 12:23:58.466187  6504 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.193s
W20260709 12:23:58.679278  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.000s	sys 0.028s
I20260709 12:23:58.699705  1724 rpcz_store.cc:276] 0709 12:23:57.106702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.108157 (+  1455us) service_pool.cc:224] Handling call
0709 12:23:57.108426 (+   269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.108632 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:57.108882 (+   250us) log.cc:844] Serialized 14 byte log entry
0709 12:23:58.191138 (+1082256us) spinlock_profiling.cc:243] Waited 2.56 ms on lock 0x7b18001b4b40. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:23:58.668176 (+477038us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.668337 (+   161us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:58.668540 (+   203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.676588 (+  8048us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.677148 (+   560us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16740,"prepare.queue_time_us":134,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"spinlock_wait_cycles":5893504,"thread_start_us":459674,"threads_started":1}
W20260709 12:23:58.701949  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.001s	sys 0.028s
W20260709 12:23:58.702299  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 674432us
W20260709 12:23:58.683216  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.704407  6504 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:58.704866  6504 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:58.705420  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:58.705703  6504 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.706676  6321 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:58.706962  6568 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:58.707727  6626 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:58.708305  6568 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:23:58.666527  6549 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.861s	user 0.007s	sys 0.019s
I20260709 12:23:58.654412  6588 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:58.666203  6417 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.668262  6174 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:58.644304  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:58.699893  1800 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:58.655300  6628 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:58.707031  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:58.748256  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:58.693709  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:58.709364  6626 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:58.678889  6564 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:58.714115  1726 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:23:58.687167  6135 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:58.669190  6468 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:23:58.691335  6495 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:23:58.670006  6536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.856s	user 0.007s	sys 0.023s
W20260709 12:23:58.739130  6213 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.012s	sys 0.049s
I20260709 12:23:58.750118  1729 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
W20260709 12:23:58.755817  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.760787  6568 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:58.749563  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:23:58.758944  6643 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
W20260709 12:23:58.826153  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:58.830643  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.740651  6587 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:58.753931  6504 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:58.837324  1728 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:23:58.755501  6549 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:58.840293  1726 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:58.829679  6609 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:23:58.843146  1723 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:23:58.760987  6306 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:58.699456  6492 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:58.826946  6628 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:23:58.848299  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 345539us
I20260709 12:23:58.756634  6588 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:58.755560  6626 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:58.710224  6590 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.853776  1729 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:23:58.839825  6536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.026s	user 0.007s	sys 0.024s
I20260709 12:23:58.739436  6326 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.097s
W20260709 12:23:58.875288  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 125777us
I20260709 12:23:58.750648  6572 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260709 12:23:58.835709  6135 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:58.841914  6213 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.012s	sys 0.049s
I20260709 12:23:58.840067  6495 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:58.953091  1731 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:23:58.824643  6483 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:23:58.945420  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3024) took 1162 ms. Trace:
I20260709 12:23:58.901826  6492 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.877204  6321 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:23:58.879597  6549 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:58.927007  6642 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:58.946350  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:23:58.954012  1729 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:23:58.902262  6306 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:58.960208  1725 rpcz_store.cc:276] 0709 12:23:57.783451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.784685 (+  1234us) service_pool.cc:224] Handling call
0709 12:23:58.945212 (+1160527us) spinlock_profiling.cc:243] Waited 693 ms on lock 0x7b540015cf54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:58.945341 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1593938304}
W20260709 12:23:58.935356  6219 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.744s	user 0.016s	sys 0.001s
I20260709 12:23:58.968683  1732 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260709 12:23:58.978636  2414 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:23:58.876751  6587 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.959255  6568 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:58.962769  6492 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:58.972289  6549 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:58.960086  6043 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:58.973802  6306 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:58.980628  6559 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.993340  6626 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:58.996336  6632 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:23:58.923650  6609 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.978303  6642 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:58.991513  6027 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.012s	sys 0.028s
I20260709 12:23:59.005039  6628 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 12.12, Last appended by leader: 12, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:59.003669  6637 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
W20260709 12:23:59.019582  6027 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.012s	sys 0.028s
I20260709 12:23:59.010931  6559 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.994391  6568 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:58.994032  6588 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.031548  6602 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:23:59.056631  6638 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
W20260709 12:23:59.064127  6523 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.617s	user 0.009s	sys 0.004s
I20260709 12:23:59.066084  6642 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.029573  6536 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:59.079421  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:23:59.081292  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.086216  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.121699  2418 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:23:59.090647  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.100437  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:59.123194  2412 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
W20260709 12:23:59.123893  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 482734us
W20260709 12:23:59.131352  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 488283us
W20260709 12:23:59.144913  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 480485us
I20260709 12:23:59.145987  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:59.045728  6451 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.089s
I20260709 12:23:59.147486  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:59.150617  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:23:59.153986  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.158385  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.009s	sys 0.013s
W20260709 12:23:59.158913  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.009s	sys 0.013s
W20260709 12:23:59.159324  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 529652us
I20260709 12:23:59.165978  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:59.167418  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:59.170467  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:59.170761  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:59.172437  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:59.172828  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:59.143007  6590 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.123025  6523 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.183596  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:59.135167  6536 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.142201  6515 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260709 12:23:59.196938  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:59.199275  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:59.197929  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2072) took 1014 ms. Trace:
I20260709 12:23:59.201689  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:59.175769  6468 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:59.177516  6642 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:59.175618  5903 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:59.151968  6648 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
W20260709 12:23:59.215482  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.207465  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:23:59.196667  6478 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.002s	user 0.011s	sys 0.008s
W20260709 12:23:59.221823  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 251513us
I20260709 12:23:59.181128  6654 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:59.229262  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:59.231419  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70724.9us
I20260709 12:23:59.152582  6645 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:23:59.233094  1725 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:23:59.210624  6429 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.957s	user 0.016s	sys 0.010s
I20260709 12:23:59.201614  6602 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.194449  6652 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:23:59.200932  6590 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:59.200228  6587 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.210901  6468 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.251758  6659 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:23:59.253185  6667 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:59.254055  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:23:59.212026  6043 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:59.232697  6469 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.032s	user 0.015s	sys 0.002s
I20260709 12:23:59.227881  5903 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:59.268898  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:23:59.273947  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66587.1us
I20260709 12:23:59.232270  6654 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:59.277606  1873 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:23:59.242013  6174 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.279784  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:59.250479  6588 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:59.243028  6477 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:59.243328  6642 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:59.265909  6468 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:59.249039  6655 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:59.293444  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 155032us
I20260709 12:23:59.296380  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:59.249559  6213 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.043s
I20260709 12:23:59.276364  6587 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:59.366147  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.256026  6429 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.273754  6653 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:59.276346  5903 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:59.279302  6654 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:59.342067  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 112900us
I20260709 12:23:59.290972  6651 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:23:59.266486  6602 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:59.380796  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 84507.7us
I20260709 12:23:59.385983  1800 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: [12.11-13.12]
I20260709 12:23:59.299103  6667 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:59.389061  1802 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
W20260709 12:23:59.388278  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 92134us
I20260709 12:23:59.400559  6666 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:59.401086  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:23:59.393671  1804 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:23:59.336362  6635 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:59.403764  6635 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:59.404743  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.404860  6635 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.338539  6027 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.406865  6027 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.409102  6027 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.295020  6478 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260709 12:23:59.290875  6664 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.393585  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:23:59.336302  6588 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:23:59.278842  6646 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.111s
I20260709 12:23:59.391969  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:23:59.382349  6393 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:59.434675  6393 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:59.435741  6213 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:59.435890  6393 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.436394  6213 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.397009  6515 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.438639  6515 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.422241  6670 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.381855  6653 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.429720  6609 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.420862  6667 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.431051  6659 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:59.431946  6588 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.433719  6646 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:59.386667  6651 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:23:59.292830  6174 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:59.440433  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46954.5us
I20260709 12:23:59.390838  6645 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:23:59.438874  6417 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:59.440059  6429 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.401741  6666 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.398190  6568 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:59.455462  6670 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.503911  6666 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:59.505570  6647 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.262s	user 0.003s	sys 0.000s
I20260709 12:23:59.507524  6670 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:59.452839  6027 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:59.508555  6027 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.508901  6666 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.509094  6027 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:59.512414  6492 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.507s	user 0.009s	sys 0.003s
W20260709 12:23:59.443727  6644 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.308s	user 0.000s	sys 0.002s
I20260709 12:23:59.457420  6663 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:59.513002  6492 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:59.443526  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51616.9us
I20260709 12:23:59.446725  2418 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:23:59.464720  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:23:59.567195  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:23:59.568351  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 103696us
I20260709 12:23:59.461341  6609 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.339082  6477 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.449780  6653 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:59.476395  6649 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260709 12:23:59.431095  6664 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.481630  6306 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.484313  6667 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.485273  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:59.487458  6661 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:23:59.489651  1730 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:59.594993  1730 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:59.494102  6622 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.389s	user 0.014s	sys 0.053s
I20260709 12:23:59.462281  6651 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:59.201277  1728 rpcz_store.cc:276] 0709 12:23:58.183922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.939224 (+755302us) service_pool.cc:224] Handling call
0709 12:23:58.966045 (+ 26821us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.966244 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:58.981459 (+ 15215us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.197532 (+216073us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.197588 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:59.197660 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.197733 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.197863 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5971,"prepare.queue_time_us":15350,"prepare.run_cpu_time_us":143,"prepare.run_wall_time_us":2151,"raft.queue_time_us":9614,"thread_start_us":209542,"threads_started":1}
I20260709 12:23:59.440649  6652 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:23:59.450351  6483 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.031s
I20260709 12:23:59.514237  6536 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:23:59.515008  6135 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.553s	user 0.009s	sys 0.003s
I20260709 12:23:59.515982  6469 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:23:59.516757  6495 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.552s	user 0.008s	sys 0.003s
I20260709 12:23:59.512456  6655 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:59.519088  6590 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:59.520581  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 646) took 1311 ms. Trace:
W20260709 12:23:59.530651  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.537945  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 147008us
I20260709 12:23:59.462635  6642 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.572053  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:59.575311  6520 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:59.576596  2410 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:59.583007  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 350523us
W20260709 12:23:59.583537  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 307754us
W20260709 12:23:59.687887  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.570676  6488 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.338s	user 0.012s	sys 0.009s
I20260709 12:23:59.470273  6417 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:59.489959  6515 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:59.607517  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:59.701646  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3037) took 1201 ms. Trace:
I20260709 12:23:59.701911  1727 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
W20260709 12:23:59.609315  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.569870  6569 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:59.606884  1730 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.536157  6663 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:59.577319  6650 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:59.585664  6478 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.526623  6321 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.080s
I20260709 12:23:59.585637  6043 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.623497  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 968) took 1479 ms. Trace:
I20260709 12:23:59.755914  2414 rpcz_store.cc:276] 0709 12:23:58.144308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.145017 (+   709us) service_pool.cc:224] Handling call
0709 12:23:58.145308 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.979118 (+833810us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.623204 (+644086us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b54005438d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:23:59.623423 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1915640576}
I20260709 12:23:59.691565  6589 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.629607  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:59.634621  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2074) took 1068 ms. Trace:
I20260709 12:23:59.643174  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:59.438773  6213 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.643391  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:23:59.611449  6393 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.641579  1732 rpcz_store.cc:276] 0709 12:23:58.209386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.975824 (+766438us) service_pool.cc:224] Handling call
0709 12:23:59.086304 (+110480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.086607 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.086827 (+   220us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.086920 (+    93us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:59.086991 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.087085 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.520350 (+433265us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b5400119ed0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:59.520500 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":252950784}
I20260709 12:23:59.643913  6495 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:59.654253  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82288.9us
I20260709 12:23:59.687556  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:23:59.685730  6628 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.627504  6536 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.625682  6469 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:59.641803  6655 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:59.591459  6609 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.702240  1726 rpcz_store.cc:276] 0709 12:23:58.499917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.067136 (+567219us) service_pool.cc:224] Handling call
0709 12:23:59.067747 (+   611us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.068274 (+   527us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.068318 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.068978 (+   660us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.152837 (+ 83859us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:59.154002 (+  1165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.154094 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.701364 (+547270us) raft_consensus.cc:1685] finished
0709 12:23:59.701387 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.701557 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28910,"raft.queue_time_us":8807,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":17293,"thread_start_us":82941,"threads_started":1,"wal-append.queue_time_us":143444}
I20260709 12:23:59.736923  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:59.813206  1727 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
W20260709 12:23:59.702868  6549 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.687s	user 0.008s	sys 0.006s
W20260709 12:23:59.605001  6622 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.500s	user 0.014s	sys 0.054s
I20260709 12:23:59.690752  6326 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:59.759259  1730 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:59.755283  6668 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.626667  6135 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.772820  6650 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:59.760566  1723 rpcz_store.cc:276] 0709 12:23:58.566071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.687723 (+121652us) service_pool.cc:224] Handling call
0709 12:23:59.634366 (+946643us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b540035d754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:23:59.634545 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":354914688}
I20260709 12:23:59.764406  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:59.700680  6520 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:23:59.772326  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 128998us
W20260709 12:23:59.774878  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87443.5us
I20260709 12:23:59.807572  6587 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:23:59.838415  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.790925  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.007s	sys 0.015s
W20260709 12:23:59.841364  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.008s	sys 0.016s
I20260709 12:23:59.839105  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:59.841799  1723 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 12:23:59.817524  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46712us
I20260709 12:23:59.817662  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:23:59.842657  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2077) took 1132 ms. Trace:
I20260709 12:23:59.826577  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:23:59.765621  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 122517us
I20260709 12:23:59.845722  1772 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:23:59.768250  6602 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.778033  6645 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:23:59.778173  6667 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:23:59.837553  1732 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:23:59.811650  6536 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.848758  1732 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:59.849228  1732 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:59.850040  6536 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.850299  6492 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.780534   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:59.768471  6043 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.854164  6043 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:23:59.849952  6681 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:23:59.694305  6488 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.845882  6587 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:23:59.857950  6587 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.852249  6622 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.859196  6622 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.808842  6213 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:23:59.764382  6478 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.852249  6135 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.831296  1727 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:59.865572  6662 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:23:59.866176  6662 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:59.865631  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 647) took 1342 ms. Trace:
I20260709 12:23:59.771166  6589 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.867295  6662 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:59.867996  6636 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.162s	user 0.000s	sys 0.005s
I20260709 12:23:59.811203  6417 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.868070  1727 rpcz_store.cc:276] 0709 12:23:58.523678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.827587 (+1303909us) service_pool.cc:224] Handling call
0709 12:23:59.827995 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.863829 (+ 35834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.865556 (+  1727us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.814323  6027 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.869936  6662 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:23:59.843191  1723 rpcz_store.cc:276] 0709 12:23:58.710524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.841149 (+1130625us) service_pool.cc:224] Handling call
0709 12:23:59.841483 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.842329 (+   846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.842586 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:59.872463  6027 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.849947  1732 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.860924  6213 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.872624  1723 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:59.875665  6477 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.831009  6549 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.878002  6549 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.879077  1804 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:23:59.879077  1728 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:59.881289  6676 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:59.882082  5903 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.882999  6654 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.884421  5903 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.885347  6654 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.873507  1732 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:59.843546  1726 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:23:59.890051  6602 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:23:59.890668  6602 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.891953  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:59.892957  6602 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.893422  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:59.892724  1802 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:23:59.773293  6393 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.895587  6213 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:59.896786  6393 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.895926  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:59.897454  6664 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:23:59.898572  1800 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:59.898772  2414 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:59.899271  2414 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:23:59.899859  6609 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.900156  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:59.900223  2414 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.807004  6628 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:23:59.901402  2414 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:59.838176  6673 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.071s
I20260709 12:23:59.903077  6478 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.833285  6668 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.903077  6628 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.903987  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:59.895994  6488 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.905429  6668 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.905843  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:59.906562  6520 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.907224  6306 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.908978  6622 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.831661  6326 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:23:59.912537  2415 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:23:59.832839  6663 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:23:59.916235  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 77283.5us
I20260709 12:23:59.984222  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:59.916524  6609 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:59.989100  6609 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.989573  6609 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:59.917800  6663 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:23:59.924886  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.925650  6664 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.916445  1728 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:23:59.997212  6664 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.926927  6213 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:00.001238  6213 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:23:59.928386  6468 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.589s	user 0.008s	sys 0.004s
I20260709 12:24:00.004536  6468 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:23:59.929121  6589 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:23:59.929594  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:59.931367  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:00.011585  2411 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:00.013713  6213 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:23:59.936151  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:00.016846  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:00.017338  6664 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:00.010206  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:00.017604  2419 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:00.018985  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:00.020435  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:23:59.948149  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:00.022226  2416 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.022325  6468 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:00.024363  6569 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:59.929970  1727 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:59.954294  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.029922  1881 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
W20260709 12:23:59.965518  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.974068  1877 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:23:59.979931  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:59.981856  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:23:59.982039  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:00.037307  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:59.929953  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:00.039304  1726 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.032379  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:23:59.952786  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.041146  1798 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:00.038372  1807 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:00.021378  2413 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:23:59.936853  1730 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 12:24:00.043437  1730 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 7
I20260709 12:24:00.043763  1730 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:24:00.044023  1730 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:00.050386  1875 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:00.051067  1877 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:00.051852  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.052508  1879 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:00.053634  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.053741  1872 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:00.053867  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.054306  1879 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:24:00.056494  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:00.058727  1876 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:00.059408  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.889767  6417 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:00.078930  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94305.4us
I20260709 12:24:00.080664  2413 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:24:00.060417  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2062) took 2606 ms. Trace:
I20260709 12:24:00.084036  1730 rpcz_store.cc:276] 0709 12:23:57.438905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.439104 (+   199us) service_pool.cc:224] Handling call
0709 12:23:57.439447 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.489541 (+2050094us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b5400178520. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:24:00.044502 (+554961us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.044763 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2999809920}
I20260709 12:24:00.055737  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.026255  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2079) took 1174 ms. Trace:
I20260709 12:24:00.085606  1727 rpcz_store.cc:276] 0709 12:23:58.851897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.887939 (+1036042us) service_pool.cc:224] Handling call
0709 12:23:59.888287 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.025935 (+137648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.026190 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.086869  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:23:59.981071  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:00.087524  1732 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:24:00.075675  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:00.061957  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:59.915750  6495 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:00.077656  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.090029  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.090590  1876 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 7.
I20260709 12:24:00.088626  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2078) took 1350 ms. Trace:
I20260709 12:24:00.090981  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:23:59.961081  6477 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.087597  1727 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:24:00.099443  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:00.090673  1872 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:00.091198  1732 rpcz_store.cc:276] 0709 12:23:58.738632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.848196 (+1109564us) service_pool.cc:224] Handling call
0709 12:23:59.848608 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.088328 (+239720us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.088563 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.074846  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:00.093103  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.058357  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:00.091727  1881 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
W20260709 12:24:00.060161  6466 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:24:00.087517  2414 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 12:24:00.105676  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:00.099598  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:00.100800  1723 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:00.107127  1727 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:00.100894  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.108326  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:00.101447  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:00.085455  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:00.091550  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:00.088938  1875 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:00.049194  6622 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.100538  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:00.215476  1880 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:00.101488  1877 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:24:00.217278  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:00.218040  1877 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.112689  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:00.109371  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:00.219419  1872 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:00.101917  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.114427  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.220135  1881 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:23:59.914744  6676 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:59.950765  6635 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.103147  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:00.221276  1879 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:00.111544  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.222090  1726 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
W20260709 12:24:00.128997  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.106390  2414 pending_rounds.cc:85] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 11
I20260709 12:24:00.101780  1802 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.110507  1732 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:00.222811  2414 pending_rounds.cc:107] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 12:24:00.198843  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:00.110553  1730 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:00.049083  6429 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:00.110816  1876 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.106748  1799 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.259480  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:00.206558  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 105671us
I20260709 12:24:00.263538  1879 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:00.264452  1880 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:00.264631  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:00.265030  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:00.265477  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.206503  2411 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:00.210697  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:23:59.912494  6326 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.110980  6468 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:00.291007  6626 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.291429  6468 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.291885  6468 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.100132  6682 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:00.293618  6682 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:00.294057  6682 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:59.948982  6651 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.312390  1802 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:00.227656  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:00.334358  1730 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:00.226333  2414 raft_consensus.cc:2889] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:00.229722  6688 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:24:00.231441  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:00.337456  1732 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
W20260709 12:24:00.337666  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 225035us
I20260709 12:24:00.240917  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:00.243552  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:00.339016  1728 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.339529  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:00.251026  6683 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.253887  6646 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:00.261735  1726 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:00.267095  1880 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:24:00.271510  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60916us
I20260709 12:24:00.272241  2416 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:00.268599  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:00.283471  1877 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:00.287907  1799 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:00.270253  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:00.289474  6326 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.259516  6306 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:00.206192  6645 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20260709 12:24:00.290287  6564 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:00.400307  6651 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:00.290601  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:00.290750  6628 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.257527  6635 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.401693  6628 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.308694  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3042) took 1176 ms. Trace:
I20260709 12:24:00.335395  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:00.403141  6655 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.573s	user 0.010s	sys 0.000s
I20260709 12:24:00.104609  6477 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.395233  6677 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:00.404091  6677 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.403596  6655 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:00.404500  6677 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.249552  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:00.403483  1730 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 7.
I20260709 12:24:00.406337  1881 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:00.406694  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:00.408972  1803 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:24:00.402846  1723 rpcz_store.cc:276] 0709 12:23:59.132849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.889595 (+756746us) service_pool.cc:224] Handling call
0709 12:23:59.889876 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.107435 (+217559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.308345 (+200910us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b5400327dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:00.308611 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":484601216}
I20260709 12:24:00.412125  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3043) took 1279 ms. Trace:
I20260709 12:24:00.412695  1723 rpcz_store.cc:276] 0709 12:23:59.133143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.410919 (+1277776us) service_pool.cc:224] Handling call
0709 12:24:00.411387 (+   468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.411615 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.411661 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.411726 (+    65us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:00.411789 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.411867 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.412048 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.025521  6393 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
I20260709 12:24:00.356745  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:00.354696  6684 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:24:00.417232  1732 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:00.418215  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 649) took 1284 ms. Trace:
I20260709 12:24:00.418710  1732 rpcz_store.cc:276] 0709 12:23:59.133978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.416520 (+1282542us) service_pool.cc:224] Handling call
0709 12:24:00.416923 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.417882 (+   959us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.418144 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.424817  1732 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:00.424540  6469 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.425734  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 650) took 1291 ms. Trace:
I20260709 12:24:00.395293  6472 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:00.427749  1732 rpcz_store.cc:276] 0709 12:23:59.134574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.424153 (+1289579us) service_pool.cc:224] Handling call
0709 12:24:00.424607 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.425441 (+   834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.425672 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.428707  6472 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:00.429793  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 645) took 2239 ms. Trace:
I20260709 12:24:00.370405  6646 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:00.432713  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:00.433630  2412 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:00.432781  6646 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.436802  6646 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.369796  1799 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:24:00.372080  1880 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:00.438270  1727 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:00.256950  6495 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.128s
I20260709 12:24:00.440094  6628 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:00.356796  6483 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:00.440593  1727 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:00.375272  6683 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.352025  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:00.441402  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2081) took 1059 ms. Trace:
I20260709 12:24:00.300069  6429 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:00.442196  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 90296.1us
I20260709 12:24:00.350189  6662 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:00.443200  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:00.430294  1729 rpcz_store.cc:276] 0709 12:23:58.190235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.971747 (+781512us) service_pool.cc:224] Handling call
0709 12:23:59.635237 (+663490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.635706 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.635746 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.642547 (+  6801us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.951672 (+309125us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:59.951719 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.951797 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:00.296832 (+345035us) raft_consensus.cc:1685] finished
0709 12:24:00.296869 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.429440 (+132571us) spinlock_profiling.cc:243] Waited 660 ms on lock 0x7b540035d750. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:00.429718 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":583,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":8222,"spinlock_wait_cycles":1518891904,"thread_start_us":193826,"threads_started":1,"wal-append.queue_time_us":504242}
I20260709 12:24:00.429822  1732 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:24:00.445242  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:24:00.394444  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 124262us
I20260709 12:24:00.446833  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:00.441843  1727 rpcz_store.cc:276] 0709 12:23:59.381731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.439765 (+1058034us) service_pool.cc:224] Handling call
0709 12:24:00.440364 (+   599us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.441124 (+   760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.441339 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.454144  1723 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:24:00.384459  1876 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:00.388877  1804 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:00.302027  6432 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:00.295974  6668 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:24:00.391250  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 122744us
I20260709 12:24:00.430172  6472 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.246539  6622 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.394431  6675 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.022s
I20260709 12:24:00.302675  6559 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:00.400002  6504 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:00.400574  6564 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:00.292037  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3034) took 2034 ms. Trace:
W20260709 12:24:00.451858  6567 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 12:24:00.445689  6642 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.452492  6671 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:00.446769  6662 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:24:00.453157  6650 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.615s	user 0.006s	sys 0.006s
I20260709 12:24:00.448693  6645 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:24:00.459825  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3045) took 1312 ms. Trace:
I20260709 12:24:00.460638  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3044) took 1327 ms. Trace:
I20260709 12:24:00.461514  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:00.473096  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:00.496383  1732 rpcz_store.cc:276] 0709 12:23:59.147049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.429069 (+1282020us) service_pool.cc:224] Handling call
0709 12:24:00.429435 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.459397 (+ 29962us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.459732 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.428562  6469 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:24:00.476130  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.448863  6477 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:00.475864  1725 rpcz_store.cc:276] 0709 12:23:58.258079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.988356 (+730277us) service_pool.cc:224] Handling call
0709 12:23:59.005542 (+ 17186us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.237848 (+232306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.291762 (+1053914us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b5400435dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:00.291972 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522143488}
I20260709 12:24:00.477371  6225 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.464816  6432 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:00.466805  6664 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:00.468616  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:00.468988  1799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:24:00.511780  1799 raft_consensus.cc:606] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:24:00.465128  6668 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:00.463871  1728 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
W20260709 12:24:00.473706  6653 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.867s	user 0.012s	sys 0.000s
I20260709 12:24:00.386025  1872 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:00.521236  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:00.514269  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45792.8us
W20260709 12:24:00.490317  6685 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.278s	user 0.002s	sys 0.000s
I20260709 12:24:00.492533  1726 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:24:00.522163  1872 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:24:00.498623  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:00.474701  6651 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.500674  1844 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:00.496501  1723 rpcz_store.cc:276] 0709 12:23:59.133561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.414064 (+1280503us) service_pool.cc:224] Handling call
0709 12:24:00.414500 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.458967 (+ 44467us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.460547 (+  1580us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.477797  6564 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:00.495826  6675 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.532629  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:00.476863  6468 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.517620  6219 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 12:24:00.537395  6174 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:00.518987  6213 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:00.615350  6213 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.615967  6213 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.533874  6653 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.533243  6505 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 12:24:00.533262  6683 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:00.533368  6635 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:00.545027  2415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1044) took 2079 ms (client timeout 2039 ms). Trace:
I20260709 12:24:00.620807  6653 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:00.620718  2415 rpcz_store.cc:269] 0709 12:23:58.465173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.465288 (+   115us) service_pool.cc:224] Handling call
0709 12:23:58.705345 (+240057us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b5c00380a8c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16af742161 00007f16af8e3a79 00007f16b0d9d9b6 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:00.544786 (+1839441us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540046a354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:00.544940 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2926185472}
I20260709 12:24:00.546880  6669 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:00.552250  1732 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:24:00.560215  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2866) took 2366 ms. Trace:
I20260709 12:24:00.498502  6650 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:24:00.478055  6504 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.563882  6651 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.566870  6477 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:00.659132  6651 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:00.659386  6504 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:00.660040  1807 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:24:00.567993  1798 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
W20260709 12:24:00.568572  6695 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:24:00.493557  6642 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.664710  6622 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:00.518092  6569 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.142s
I20260709 12:24:00.665237  6642 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:00.665884  6683 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:00.666483  6683 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.666738  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 620) took 1276 ms. Trace:
I20260709 12:24:00.573836  6483 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:00.574357  6645 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:00.667460  1806 rpcz_store.cc:276] 0709 12:23:59.390244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.390362 (+   118us) service_pool.cc:224] Handling call
0709 12:23:59.390677 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.812601 (+421924us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:59.812643 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.824200 (+ 11557us) log.cc:844] Serialized 26 byte log entry
0709 12:24:00.090092 (+265892us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:00.090132 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.090199 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:00.567903 (+477704us) spinlock_profiling.cc:243] Waited 3.6 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:00.666291 (+ 98388us) raft_consensus.cc:1685] finished
0709 12:24:00.666324 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.666652 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7883,"raft.queue_time_us":39727,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":8286464,"thread_start_us":618679,"threads_started":2,"wal-append.queue_time_us":593983}
I20260709 12:24:00.576591  1725 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:24:00.669412  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2088) took 1058 ms. Trace:
I20260709 12:24:00.669597  1728 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 12:24:00.570006  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3036) took 2070 ms. Trace:
I20260709 12:24:00.669785  1725 rpcz_store.cc:276] 0709 12:23:59.610683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.510155 (+899472us) service_pool.cc:224] Handling call
0709 12:24:00.576299 (+ 66144us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.669141 (+ 92842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.669360 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:00.670264  1731 rpcz_store.cc:276] 0709 12:23:58.499650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.056498 (+556848us) service_pool.cc:224] Handling call
0709 12:23:59.056937 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.217882 (+160945us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:23:59.582540 (+364658us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.045456 (+462916us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.045557 (+   101us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:00.045642 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.569742 (+524100us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540030be54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:00.569790 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.569941 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":37190,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":3107,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":30476,"spinlock_wait_cycles":369576832,"thread_start_us":801082,"threads_started":2,"wal-append.queue_time_us":669660}
I20260709 12:24:00.499907  6671 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:00.691608  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.607669  6432 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.609557  6682 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.537581  6704 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:00.513659  6668 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:24:00.626495  1727 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [12.6-14.7]
I20260709 12:24:00.648787  1873 rpcz_store.cc:276] 0709 12:23:58.193676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.272086 (+ 78410us) service_pool.cc:224] Handling call
0709 12:23:58.272486 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.278013 (+1005527us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.559852 (+1281839us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b5400117e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:00.560104 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2310688512}
I20260709 12:24:00.514161  6664 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.666950  6683 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:00.514252  6225 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:00.583874  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.673946  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:00.578106  1723 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 15 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260709 12:24:00.620762  6468 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:00.530936  6451 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:24:00.532254  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:00.727573  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 195445us
I20260709 12:24:00.722494  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:00.728806  1723 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.728902  1872 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:00.723894  1873 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:00.726800  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:00.690632  6559 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.692730  6555 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:00.703971  6642 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.701177  6635 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.754812  1725 raft_consensus.cc:2368] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16.
I20260709 12:24:00.717139  6704 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:00.719765  6664 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:00.628376  6675 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.755899  6432 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:00.763322  2382 leader_election.cc:400] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16.
I20260709 12:24:00.769841  2382 leader_election.cc:403] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:00.734766  6669 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:00.772368  1875 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:24:00.755785  6591 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.020s	sys 0.024s
I20260709 12:24:00.754276  6555 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:00.763288  6429 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:00.727146  6451 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:00.802971  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.777861  6569 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 16
W20260709 12:24:00.787147  6609 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.797s	user 0.014s	sys 0.001s
I20260709 12:24:00.748359  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1051) took 1598 ms. Trace:
I20260709 12:24:00.787007  1725 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:24:00.756711  6642 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:00.760041  6559 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:00.776229  6591 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.021s	sys 0.024s
I20260709 12:24:00.823946  2410 rpcz_store.cc:276] 0709 12:23:59.150127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.150235 (+   108us) service_pool.cc:224] Handling call
0709 12:23:59.568092 (+417857us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.686692 (+118600us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.747984 (+1061292us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b5400525150. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:00.748249 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":960161024}
I20260709 12:24:00.782951  6225 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:00.828661  1725 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:00.832429  1725 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:00.735704  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:00.816401  6451 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:00.840432  1723 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
W20260709 12:24:00.829313  6663 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.838s	user 0.014s	sys 0.000s
I20260709 12:24:00.786185  6671 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.879539  1877 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:24:00.879942  1877 pending_rounds.cc:85] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 5
I20260709 12:24:00.880187  1877 pending_rounds.cc:107] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:24:00.880450  1877 raft_consensus.cc:2889] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:00.882675  1723 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:00.883342  1731 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:00.769452  6635 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:00.908161  2417 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.959s	user 0.033s	sys 0.032s
W20260709 12:24:00.856809  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 121312us
I20260709 12:24:00.877920  6225 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:00.912132  6225 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.912736  6225 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:00.825999  6609 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.917446  6609 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:00.784085  6669 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:00.911713  2417 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.033s	sys 0.032s
I20260709 12:24:00.927573  6669 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:00.956133  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 418027us
W20260709 12:24:00.963524  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.644s (SENT) [suppressed 6 similar messages]
I20260709 12:24:00.949293  6659 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.989658  1803 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:24:00.829634  6704 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.997145  1803 raft_consensus.cc:606] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:24:00.997624  1803 raft_consensus.cc:614] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:00.971201  6682 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.015976  6653 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:01.016525  6653 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:01.020160  6677 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.615s	user 0.012s	sys 0.000s
I20260709 12:24:01.001760  6659 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.020697  6677 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.024250  2410 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:00.999310  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:01.032085  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:01.036360  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:01.037469  2415 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:24:01.040902  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:01.041550  1875 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:00.892894  6555 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:00.997236  6504 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:01.052232  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:01.053434  1725 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:01.053648  6655 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:01.054828  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:01.055783  1880 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:01.056126  6642 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.056797  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.058117  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.057922  6213 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.060194  1880 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:01.019397  6653 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.040733  6704 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:01.073899  6703 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:24:01.090654  1875 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
W20260709 12:24:01.052279  6701 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.175s	user 0.000s	sys 0.001s
I20260709 12:24:01.051095  6682 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.045979  6659 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.098292  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:01.100010  1725 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:01.100881  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:01.105098  1875 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:01.078171  6675 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.075409  6477 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:01.112087  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2881) took 1036 ms. Trace:
W20260709 12:24:01.074471  6321 consensus_queue.cc:1175] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" responder_term: 12 status { last_received { term: 12 index: 10 } last_committed_idx: 10 last_received_current_leader { term: 12 index: 10 } }
W20260709 12:24:01.102298  2412 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.007s	sys 0.019s
I20260709 12:24:01.114140  1725 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
W20260709 12:24:01.117952  2412 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.567s	user 0.010s	sys 0.019s
I20260709 12:24:01.061268  6225 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.100862  6653 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:01.108530  6417 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.018s	user 0.011s	sys 0.004s
W20260709 12:24:01.121387  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 588786us
I20260709 12:24:01.122079  1803 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40096
W20260709 12:24:01.123113  1772 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.013s	sys 0.023s
I20260709 12:24:01.072211  6504 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:01.128855  1772 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.013s	sys 0.023s
W20260709 12:24:01.144001  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 570159us
W20260709 12:24:01.145246  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.146082  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2226) took 1190 ms. Trace:
I20260709 12:24:01.150231  1878 rpcz_store.cc:276] 0709 12:23:59.955710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.955876 (+   166us) service_pool.cc:224] Handling call
0709 12:23:59.956291 (+   415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.956789 (+   498us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.956827 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.957374 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 12:24:00.476122 (+518748us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:00.476171 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.476244 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.145743 (+669499us) raft_consensus.cc:1685] finished
0709 12:24:01.145776 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.145993 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65613,"raft.queue_time_us":870,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":15725,"thread_start_us":417316,"threads_started":1,"wal-append.queue_time_us":771408}
I20260709 12:24:01.153019  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.160794  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:01.104321  6682 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.125669  6628 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:24:01.168118  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:01.195171  1729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.006s	sys 0.039s
I20260709 12:24:01.195425  1878 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:01.195608  1729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.006s	sys 0.039s
I20260709 12:24:01.198377  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:01.199208  1878 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:01.147394  6523 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.145128  6705 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:01.117558  6477 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.226266  1803 raft_consensus.cc:606] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:24:01.226758  1803 raft_consensus.cc:614] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:01.206863  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.205050  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.228519  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:01.193362  6469 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:24:01.203446  6691 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:24:01.243356  1875 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
W20260709 12:24:01.265944  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.250502  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:01.197361  6713 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:01.269173  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3046) took 1382 ms. Trace:
I20260709 12:24:01.164058  6483 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.066s
W20260709 12:24:01.244482  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 245252us
W20260709 12:24:01.284755  2383 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.239s (SENT)
I20260709 12:24:01.167317  6429 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.068s
I20260709 12:24:01.196642  6671 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:24:01.275640  1726 rpcz_store.cc:276] 0709 12:23:59.887272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.541408 (+654136us) service_pool.cc:224] Handling call
0709 12:24:00.541788 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.542064 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:00.967766 (+425702us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.268309 (+300543us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:01.268396 (+    87us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:01.268501 (+   105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.268589 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.269056 (+   467us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":606040,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":4604,"raft.queue_time_us":3710,"thread_start_us":725088,"threads_started":2}
I20260709 12:24:01.272663  1875 raft_consensus.cc:2368] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13.
I20260709 12:24:01.299517  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.301059  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.285609  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:01.304695  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.307722  1875 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:01.236637  6635 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:01.314288  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 159354us
I20260709 12:24:01.316959  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:01.318439  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:01.322588  2416 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:01.333041  1878 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:24:01.334528  1878 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:01.336537  1878 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:01.306938  6716 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.076s	user 0.001s	sys 0.001s
W20260709 12:24:01.337447  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.344185  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 189197us
I20260709 12:24:01.346212  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:01.339316  1878 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:01.350879  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.270488  6650 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 12:24:01.351702  1878 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:24:01.234896  6469 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:01.353250  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:01.355531  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:01.333340  6665 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 2.057s	user 0.034s	sys 0.035s
I20260709 12:24:01.357788  2419 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:01.356606  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.358994  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:01.355562  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:01.368872  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.230144  6705 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.390175  1878 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 9.
I20260709 12:24:01.407017  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:01.407980  2418 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:01.359257  6469 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:01.414137  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:01.410969  1878 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: [12.9-12.9]
W20260709 12:24:01.412673  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 127947us
I20260709 12:24:01.424790  2418 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:24:01.350426  6659 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
W20260709 12:24:01.392011  6706 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.295s	user 0.002s	sys 0.000s
I20260709 12:24:01.401299  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:01.394650  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77630.2us
I20260709 12:24:01.426976  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.405283  6213 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.433771  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:01.434263  2414 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:24:01.437229  1878 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:24:01.439239  6694 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.042s	user 0.006s	sys 0.006s
I20260709 12:24:01.414117  6523 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.441788  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2082) took 2014 ms. Trace:
I20260709 12:24:01.442238  1729 rpcz_store.cc:276] 0709 12:23:59.427372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.447469 (+1020097us) service_pool.cc:224] Handling call
0709 12:24:00.449575 (+  2106us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.450050 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:24:00.450111 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:00.450847 (+   736us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.196292 (+745445us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:01.196361 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.196484 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.441592 (+245108us) raft_consensus.cc:1685] finished
0709 12:24:01.441610 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.441732 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1278,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":7298,"thread_start_us":745006,"threads_started":1,"wal-append.queue_time_us":865628}
I20260709 12:24:01.443650  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:01.444422  1731 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:01.408964  6727 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:01.453135  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:01.456789  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2066) took 3279 ms. Trace:
I20260709 12:24:01.457417  1724 rpcz_store.cc:276] 0709 12:23:58.177464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.711757 (+534293us) service_pool.cc:224] Handling call
0709 12:23:58.712251 (+   494us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.126481 (+414230us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.126512 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.127061 (+   549us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.241481 (+114420us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:59.241520 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.241587 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.120613 (+1879026us) spinlock_profiling.cc:243] Waited 3.33 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:01.451885 (+331272us) raft_consensus.cc:1685] finished
0709 12:24:01.451926 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.456691 (+  4765us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":48209,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":7073,"spinlock_wait_cycles":7668992,"thread_start_us":489247,"threads_started":2,"wal-append.queue_time_us":89681}
I20260709 12:24:01.460944  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:01.444156  6225 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.445621  6504 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:01.468621  6722 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:24:01.457243  6727 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.455905  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:01.462677  6714 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:24:01.465443  1697 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:01.449203  6734 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:01.473870  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 119300us
I20260709 12:24:01.447196  6306 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:01.474436  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:01.481906  1878 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:01.483042  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.483693  1878 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:01.484397  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:01.484923  1878 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:01.487079  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:01.477955  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1056) took 1536 ms. Trace:
I20260709 12:24:01.432402  6713 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:01.457002  6682 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.485600  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.488286  2417 rpcz_store.cc:276] 0709 12:23:59.942205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.942388 (+   183us) service_pool.cc:224] Handling call
0709 12:23:59.942857 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.943340 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.943387 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.948942 (+  5555us) log.cc:844] Serialized 26 byte log entry
0709 12:24:00.926289 (+977347us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:00.926360 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.926477 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.477683 (+551206us) raft_consensus.cc:1685] finished
0709 12:24:01.477711 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.477881 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":133,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":1841,"thread_start_us":976771,"threads_started":1,"wal-append.queue_time_us":1059111}
I20260709 12:24:01.476961  6225 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:01.449594  6523 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.497699  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.494398  2410 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:24:01.470167  6735 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:01.475350  6635 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 12:24:01.486649  6642 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:01.491523  6417 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
W20260709 12:24:01.518610  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.545730  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:01.493419  6668 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.758s	user 0.006s	sys 0.009s
I20260709 12:24:01.547271  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 628) took 1449 ms. Trace:
I20260709 12:24:01.509657  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:01.474128  6650 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:24:01.499567  6225 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.584767  6225 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:01.491950  6477 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.492254  6651 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:01.559003  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 231499us
I20260709 12:24:01.542042  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.498816  6738 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:01.587513  6651 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.588496  1878 raft_consensus.cc:2368] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16.
I20260709 12:24:01.589478  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.590190  1878 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:01.498046  6466 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:24:01.563831  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2877) took 1506 ms. Trace:
I20260709 12:24:01.495600  6655 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:24:01.492223  6734 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.563215  1807 rpcz_store.cc:276] 0709 12:24:00.098007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.102192 (+  4185us) service_pool.cc:224] Handling call
0709 12:24:01.547073 (+1444881us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b540026f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:01.547208 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":995508992}
I20260709 12:24:01.569408  6523 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:01.556763  6719 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260709 12:24:01.577756  6705 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:01.557796  6741 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
W20260709 12:24:01.581059  6432 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.805s	user 0.010s	sys 0.001s
I20260709 12:24:01.576944  6727 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.653529  6733 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:24:01.660353  6734 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.661162  6734 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:24:01.558327  6683 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.821s	user 0.010s	sys 0.004s
I20260709 12:24:01.591837  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.633817  1879 rpcz_store.cc:276] 0709 12:24:00.057923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.401778 (+343855us) service_pool.cc:224] Handling call
0709 12:24:00.402277 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.402712 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:00.771101 (+368389us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.101814 (+330713us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.110409 (+  8595us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.110590 (+   181us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:01.110625 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.110719 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.563616 (+452897us) raft_consensus.cc:1685] finished
0709 12:24:01.563635 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.563767 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4447,"prepare.queue_time_us":571619,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":96,"raft.queue_time_us":9392,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":38674,"thread_start_us":643907,"threads_started":2,"wal-append.queue_time_us":479737}
W20260709 12:24:01.667575  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.647104  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 59147.7us
I20260709 12:24:01.557267  6642 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.615834  6740 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:01.590653  6651 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:01.614077  6664 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.835s	user 0.010s	sys 0.001s
W20260709 12:24:01.653810  6451 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.765s	user 0.010s	sys 0.003s
W20260709 12:24:01.668499  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.595638  6739 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:24:01.668531  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:01.663726  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 170790us
I20260709 12:24:01.643819  6417 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:01.702045  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.671140  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.669431  6735 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:01.667537  6734 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:01.710897  6664 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.712924  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.655169  6477 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.086s
I20260709 12:24:01.716333  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.664794  6741 consensus_queue.cc:1243] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 12.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:01.736627  1804 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:24:01.738643  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.741060  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.706449  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:01.667826  6731 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.099s	user 0.000s	sys 0.004s
I20260709 12:24:01.721428  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.617182  6668 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.716754  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:01.751998  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45650.9us
I20260709 12:24:01.753156  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.756856  1806 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
W20260709 12:24:01.759200  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.757517  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:01.759390  1804 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
W20260709 12:24:01.736261  6564 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.081s	user 0.006s	sys 0.008s
I20260709 12:24:01.734189  6642 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:01.760763  1879 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 13 index: 13. (index mismatch)
I20260709 12:24:01.752873  1807 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:01.766364  2383 leader_election.cc:400] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16.
I20260709 12:24:01.773447  1801 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:24:01.744334  1805 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:24:01.737555  6523 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.749234  1878 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
W20260709 12:24:01.753751  6728 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:24:01.773604  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:01.738076  6743 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:24:01.739357  6664 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:01.767779  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.777850  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.758306  6741 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:01.779119  1802 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 8: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:01.780261  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.780951  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 623) took 1726 ms. Trace:
I20260709 12:24:01.787526  1798 rpcz_store.cc:276] 0709 12:24:00.054851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.054992 (+   141us) service_pool.cc:224] Handling call
0709 12:24:00.567795 (+512803us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.661353 (+ 93558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.780669 (+1119316us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b54002f2e50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:01.780868 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1178610432}
I20260709 12:24:01.774859  6738 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:01.789003  6738 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.789470  6738 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:24:01.792694  6736 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20260709 12:24:01.707105  6451 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.795347  6677 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.796993  6451 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.797708  6747 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.798069  6682 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:01.703451  6746 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.104s
I20260709 12:24:01.747664  6417 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.751209  6727 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.037s
W20260709 12:24:01.802184  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 300167us
I20260709 12:24:01.670630  6683 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.798327  6747 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.828558  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2888) took 1559 ms. Trace:
I20260709 12:24:01.830705  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.831113  1872 rpcz_store.cc:276] 0709 12:24:00.269597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.524188 (+254591us) service_pool.cc:224] Handling call
0709 12:24:00.524626 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.729336 (+204710us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.828268 (+1098932us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b5400205dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:01.828470 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":468939008}
I20260709 12:24:01.832053  1806 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:01.832839  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.807237  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:01.810308  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.804636  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.834328  1807 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:01.822388  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:01.799173  6700 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.311s	user 0.003s	sys 0.011s
I20260709 12:24:01.838462  1878 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:01.771420  6705 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:24:01.798743  6682 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:01.799994  6741 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.770063  6564 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.844623  6650 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 12:24:01.737324  6734 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.849367  1873 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:24:01.850135  6740 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:24:01.850656  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.850205  6741 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:01.851161  6628 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:01.853174  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.875066  1801 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:01.876245  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.877144  1801 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:01.878026  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:01.857816  6417 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.846473  6730 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:01.879375  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:01.880971  1807 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:24:01.882148  6559 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.882287  1807 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:24:01.854288  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2887) took 1585 ms. Trace:
I20260709 12:24:01.883095  1876 rpcz_store.cc:276] 0709 12:24:00.269218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.471631 (+202413us) service_pool.cc:224] Handling call
0709 12:24:00.478922 (+  7291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.509002 (+ 30080us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.509055 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.509120 (+    65us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:00.509179 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.854028 (+1344849us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b5400074254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:01.854076 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.854224 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68520192}
I20260709 12:24:01.854758  1805 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:01.844704  6732 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 12:24:01.884637  6732 consensus_queue.cc:1243] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 12.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:24:01.855998  6683 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:01.855779  6682 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.885951  6682 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:01.886384  6669 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:01.886813  6669 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:01.886869  6645 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:01.887220  6569 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.064s	user 0.007s	sys 0.007s
I20260709 12:24:01.887470  6645 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.887746  6569 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16.
I20260709 12:24:01.735703  6735 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.897357  6752 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.857884  1806 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260709 12:24:01.898015  6752 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.858412  6469 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.899647  1806 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
W20260709 12:24:01.866719  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 701730us
I20260709 12:24:01.901203  6469 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:01.901693  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:01.862294  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 983) took 1411 ms. Trace:
I20260709 12:24:01.903075  2412 rpcz_store.cc:276] 0709 12:24:00.451039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.451238 (+   199us) service_pool.cc:224] Handling call
0709 12:24:00.451740 (+   502us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.452101 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.452132 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:00.458098 (+  5966us) log.cc:844] Serialized 26 byte log entry
0709 12:24:00.550643 (+ 92545us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b180044bb40. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:01.120261 (+569618us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:01.120330 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.120769 (+   439us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.856634 (+735865us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b2400305318. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:24:01.861929 (+  5295us) raft_consensus.cc:1685] finished
0709 12:24:01.861967 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.862200 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5923,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":1887,"spinlock_wait_cycles":17825792,"thread_start_us":569351,"threads_started":1,"wal-append.queue_time_us":814230}
I20260709 12:24:01.831604  6743 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.904446  6738 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.905405  6743 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.907747  6495 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:01.908061  6738 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.878717  6219 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:01.909493  6219 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:01.910462  6747 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.844873  6451 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.066s
I20260709 12:24:01.911885  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3048) took 1991 ms. Trace:
I20260709 12:24:01.912145  6451 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:24:01.912595  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:01.913486  6650 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:01.848531  6504 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 12:24:01.913877  6650 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:01.912372  1732 rpcz_store.cc:276] 0709 12:23:59.920854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.648433 (+727579us) service_pool.cc:224] Handling call
0709 12:24:00.702540 (+ 54107us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:00.703091 (+   551us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:24:00.972426 (+269335us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.313419 (+340993us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:01.319891 (+  6472us) log.cc:844] Serialized 52 byte log entry
0709 12:24:01.320295 (+   404us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:01.320353 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.320463 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.911629 (+591166us) raft_consensus.cc:1685] finished
0709 12:24:01.911655 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.911813 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4608,"prepare.queue_time_us":525667,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":11820,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":25098,"thread_start_us":515060,"threads_started":2,"wal-append.queue_time_us":545987}
I20260709 12:24:01.914325  6650 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:01.908198  6743 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.783910  6523 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.890692  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:01.921694  2411 raft_consensus.cc:2410] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:24:01.924635  1803 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:24:01.925343  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:01.925798  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:01.892971  6724 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.534s	user 0.001s	sys 0.001s
I20260709 12:24:01.893492  6626 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:01.894866  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.932524  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:01.916566  6564 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:01.917120  1730 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:01.755662  6713 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.917587  6628 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:24:01.917896  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:01.920895  6523 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.927049  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:01.927824  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:01.887822  6669 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.929021  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:01.949548  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:01.930088  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80222us
I20260709 12:24:01.933391  2411 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:01.934211  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:01.933107  1805 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:01.928345  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:01.945659  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.949716  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2230) took 1897 ms. Trace:
I20260709 12:24:01.972172  1874 rpcz_store.cc:276] 0709 12:24:00.051806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.051935 (+   129us) service_pool.cc:224] Handling call
0709 12:24:00.052202 (+   267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.442858 (+390656us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:00.783809 (+340951us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.127648 (+343839us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.147174 (+ 19526us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:01.147224 (+    50us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:01.147276 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.853676 (+706400us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b1800326500. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16afcac58f 00007f16a8ceb571 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd
0709 12:24:01.949443 (+ 95767us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54001f8454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:01.949505 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.949651 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":687967,"prepare.queue_time_us":573961,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":100,"raft.queue_time_us":37325,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":23031,"spinlock_wait_cycles":979766400,"thread_start_us":683551,"threads_started":2,"wal-append.queue_time_us":514914}
I20260709 12:24:01.116271  1881 rpcz_store.cc:276] 0709 12:24:00.071998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.407294 (+335296us) service_pool.cc:224] Handling call
0709 12:24:00.407658 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.426244 (+ 18586us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.426287 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:00.426917 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 12:24:00.781652 (+354735us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:01.107201 (+325549us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.107576 (+   375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.107686 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.107750 (+    64us) raft_consensus.cc:1685] finished
0709 12:24:01.107770 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.107974 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":241922,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":1187,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":16662,"spinlock_wait_cycles":9681152,"thread_start_us":642982,"threads_started":2,"wal-append.queue_time_us":559081}
I20260709 12:24:01.977479  6683 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.978128  6683 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.981076  6683 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:01.946967  6504 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.985245  6504 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:01.985935  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:01.987144  6483 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.988054  6504 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:01.887907  6645 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:01.988901  2413 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:01.908288  6495 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:01.924039  6668 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:01.990528  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63563.2us
I20260709 12:24:01.987848  6483 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:01.931811  6664 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:24:02.010883  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:01.935992  6704 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.843s	user 0.008s	sys 0.005s
I20260709 12:24:01.935811  6626 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:01.987890  6472 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.931839  6609 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 12:24:02.020427  2415 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:02.009452  6495 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:01.999588  6432 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:01.910619  6219 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.031301  6669 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:02.034971  6653 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.891s	user 0.004s	sys 0.010s
I20260709 12:24:02.047664  6752 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:02.056210  6483 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:02.062836  6626 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.064723  6626 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:02.099566  6642 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.078205  6650 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.121097  6650 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:02.094192  6741 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.114100  6730 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:02.126375  6741 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:02.188181  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.189841  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.989941  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:02.195691  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 267278us
I20260709 12:24:02.196523  1801 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
W20260709 12:24:02.199357  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 248292us
I20260709 12:24:02.200973  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:02.202169  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:02.204257  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 204889us
I20260709 12:24:02.204291  1807 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:02.194661  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:02.215827  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.218464  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:02.219558  1876 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:24:02.220649  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 648) took 3087 ms. Trace:
I20260709 12:24:02.221398  1730 rpcz_store.cc:276] 0709 12:23:59.133118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.408111 (+1274993us) service_pool.cc:224] Handling call
0709 12:24:00.408465 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.936442 (+1527977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.219795 (+283353us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54003ccdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:02.220271 (+   476us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3469008768}
I20260709 12:24:02.223714  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:02.213507  1807 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8c92163822f64868a647b688599f49bd"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:24:02.231763  1807 raft_consensus.cc:606] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:24:02.232473  1807 raft_consensus.cc:614] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:02.232785  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:02.234704  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:02.235870  1724 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:02.237732  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:02.209546  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:02.226625  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:02.224705  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2083) took 2788 ms. Trace:
I20260709 12:24:02.228323  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:02.246532  1727 rpcz_store.cc:276] 0709 12:23:59.436383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.454127 (+1017744us) service_pool.cc:224] Handling call
0709 12:24:00.454689 (+   562us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:00.718943 (+264254us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.223833 (+1504890us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54003cd7d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:02.224490 (+   657us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":394037504}
I20260709 12:24:02.247471  1880 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
W20260709 12:24:02.248117  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.229508  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2872) took 2172 ms. Trace:
I20260709 12:24:02.230561  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2892) took 1203 ms. Trace:
I20260709 12:24:02.250167  1877 rpcz_store.cc:276] 0709 12:24:00.056470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.369751 (+313281us) service_pool.cc:224] Handling call
0709 12:24:00.370051 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.880922 (+510871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.228787 (+1347865us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b54000f0854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:02.229241 (+   454us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1170768128}
I20260709 12:24:02.236004  1731 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:24:02.251139  1873 rpcz_store.cc:276] 0709 12:24:01.026855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.027019 (+   164us) service_pool.cc:224] Handling call
0709 12:24:01.027454 (+   435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.865957 (+838503us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.229811 (+363854us) spinlock_profiling.cc:243] Waited 822 ms on lock 0x7b540021b1d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:02.230331 (+   520us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1889596288}
I20260709 12:24:02.253057  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:02.228653  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:02.230422  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 652) took 2581 ms. Trace:
I20260709 12:24:02.258474  1728 rpcz_store.cc:276] 0709 12:23:59.648459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.513660 (+865201us) service_pool.cc:224] Handling call
0709 12:24:00.514038 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.669971 (+155933us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.229678 (+1559707us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54003cc3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:02.230169 (+   491us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":356937600}
I20260709 12:24:02.265277  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:02.267370  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:02.268137  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:02.305049  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 52465.3us
I20260709 12:24:02.305688  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:02.306216  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:02.306767  1805 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:24:02.308140  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:02.309939  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:02.310595  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:02.267725  1730 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:24:02.314232  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57790.3us
I20260709 12:24:02.315181  6766 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:02.316455  6766 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.328516  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:02.329974  6650 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:02.328660  2418 raft_consensus.cc:2410] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260709 12:24:02.325711  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63983.4us
I20260709 12:24:02.339190  2413 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.345505  6663 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.005s
I20260709 12:24:02.353220  6609 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:24:02.354125  6609 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:02.354717  6609 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:02.028457  6472 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.356253  6664 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:02.356698  6659 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:02.357514  6659 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:02.359418  6668 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:02.362035  6668 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:02.363772  6682 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:02.363205  6645 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.366016  6677 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:02.367101  6432 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.364293  6682 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.368322  6682 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:02.369745  6477 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.370651  6477 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.371505  1802 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:24:02.372473  6466 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:02.373721  6477 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:02.118575  6642 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:02.375084  6677 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:02.376972  6225 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.791s	user 0.008s	sys 0.004s
I20260709 12:24:02.377771  6225 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.379515  6321 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.378580  6655 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:02.358297  6472 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:02.398751  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.024s	sys 0.028s
W20260709 12:24:02.400867  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.555s	user 0.024s	sys 0.029s
W20260709 12:24:02.402937  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.405649  1878 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:02.413833  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:02.354405  6451 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:02.380425  6321 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:02.417358  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 101782us
I20260709 12:24:02.418155  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:02.416574  6451 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.426571  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:02.415735  6766 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:02.440935  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.439101  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:02.441689  2411 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:02.444166  6651 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.444754  6720 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:24:02.445365  2410 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.008s	sys 0.016s
I20260709 12:24:02.447043  6747 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:02.376878  6677 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.475291  6469 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:02.509670  6469 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:02.511139  6469 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.512867  6469 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:02.515120  1804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.016s	sys 0.010s
I20260709 12:24:02.519333  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 932) took 1206 ms. Trace:
I20260709 12:24:02.519836  2415 rpcz_store.cc:276] 0709 12:24:01.313058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.318088 (+  5030us) service_pool.cc:224] Handling call
0709 12:24:02.518849 (+1200761us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b54005230d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:02.519135 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1437923072}
W20260709 12:24:02.521046  2410 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.009s	sys 0.017s
W20260709 12:24:02.523124  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 126803us
I20260709 12:24:02.368055  6432 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.463006  1799 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:02.420019  6655 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:02.463651  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 48528.2us
I20260709 12:24:02.464151  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:02.466547  6225 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:02.467324  6645 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:02.469563  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:02.470052  6321 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:02.470989  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:02.421165  6659 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.472913  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:02.425071  6219 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:02.463042  6734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.036s	sys 0.061s
I20260709 12:24:02.475227  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:02.452481  6451 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.470082  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:02.477385  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 134590us
W20260709 12:24:02.478991  6735 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.017s	sys 0.045s
I20260709 12:24:02.469579  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:02.488123  2413 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
W20260709 12:24:02.493206  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.497505  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:02.418383  6650 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.503850  6609 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.504129  6747 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.451047  1798 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
W20260709 12:24:02.514498  6628 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.521s	user 0.011s	sys 0.009s
I20260709 12:24:02.526998  6748 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:02.542765  6655 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.475656  6717 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.548795  6664 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:24:02.588382  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 321387us
I20260709 12:24:02.605805  6628 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.607316  1724 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:02.611196  6750 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
W20260709 12:24:02.611611  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114518us
I20260709 12:24:02.611229  1726 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:24:02.583750  6735 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.018s	sys 0.045s
I20260709 12:24:02.633747  1802 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:02.646262  6740 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:24:02.648468  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:02.650622  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:02.653591  6628 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:02.654996  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:02.653667  6748 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:02.657214  6748 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:02.659701  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 184781us
I20260709 12:24:02.661546  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:02.663210  6645 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.632658  1725 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:02.662436  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:02.641234  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:02.670456  2412 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:02.671231  6752 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:24:02.671855  1724 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:02.673919  6555 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:02.674273  6717 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.555495  6651 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.679327  6774 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:02.679982  6774 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:02.680405  6774 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:02.682852  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.674365  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:02.701300  2415 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:24:02.705976  2413 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
W20260709 12:24:02.713826  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 126444us
W20260709 12:24:02.714704  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.723276  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 75012.6us
I20260709 12:24:02.724782  1800 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 8. Current term is 9. Ops: []
I20260709 12:24:02.727757  6472 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:24:02.729269  6472 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:02.731025  6663 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:24:02.731547  6663 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:02.732329  6650 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:02.732785  6609 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:02.733507  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78791.1us
I20260709 12:24:02.733085  6663 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:02.734831  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.735519  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:02.735214  6645 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:02.744827  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.664970  6682 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.689491  6717 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:02.752743  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:02.730619  6472 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:02.734690  6432 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:02.763327  6564 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:02.764627  6451 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.768348  6466 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:02.770622  6645 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:02.771327  6769 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:02.772007  6769 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:02.772538  6769 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:02.758528  6472 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:02.777383  6472 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.778364  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:02.778927  6663 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:02.780150  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:02.783303  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:02.784991  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:02.786990  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 112719us
I20260709 12:24:02.789649  6771 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:02.790971  6771 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:02.791706  6771 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:02.792970  6771 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.794864  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1995) took 1082 ms. Trace:
I20260709 12:24:02.795439  1799 rpcz_store.cc:276] 0709 12:24:01.712856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.713056 (+   200us) service_pool.cc:224] Handling call
0709 12:24:01.713659 (+   603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.584076 (+870417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.794469 (+210393us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b54001d4f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:02.794738 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1603833088}
I20260709 12:24:02.796733  6682 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:02.799561  6495 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.764s	user 0.020s	sys 0.000s
I20260709 12:24:02.800092  6495 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.802433  6495 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:02.758733  6664 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:02.804085  6664 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.804658  6664 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:02.807451  6483 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:02.808104  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 1997) took 1093 ms. Trace:
I20260709 12:24:02.808641  6635 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.094s
I20260709 12:24:02.808816  1800 rpcz_store.cc:276] 0709 12:24:01.714637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.714834 (+   197us) service_pool.cc:224] Handling call
0709 12:24:01.715212 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.725601 (+1010389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.807659 (+ 82058us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b5400132c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:02.808013 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2188200320}
W20260709 12:24:02.811017  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 93452.5us
I20260709 12:24:02.811373  6757 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:02.813678  6757 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:02.814333  6504 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:02.815088  6757 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:02.815764  6757 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:02.817261  6757 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.819247  6757 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:02.821111  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2008) took 1015 ms. Trace:
I20260709 12:24:02.821656  1798 rpcz_store.cc:276] 0709 12:24:01.805333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.806922 (+  1589us) service_pool.cc:224] Handling call
0709 12:24:02.820788 (+1013866us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b540009fbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:02.821027 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":816145280}
I20260709 12:24:02.822917  6609 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:02.825640  6523 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:02.829779  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:02.835222  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:02.842051  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 102075us
I20260709 12:24:02.789747  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:02.835301  6675 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:02.838595  6771 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.851482  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:02.852867  6720 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:24:02.854875  2416 raft_consensus.cc:2368] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11.
I20260709 12:24:02.856456  2416 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:24:02.858057  2416 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:24:02.860178  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 81453.5us
I20260709 12:24:02.860163  6451 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:02.863019  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:02.874383  2413 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:24:02.786701  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:02.875866  2415 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:02.758528  6735 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:02.876371  6732 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:02.885162  1846 leader_election.cc:400] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11.
I20260709 12:24:02.896256  1846 leader_election.cc:403] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:24:02.826464  6651 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:02.792975  6737 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:02.886756  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:02.903192  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 113679us
I20260709 12:24:02.896165  6732 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.882584  6483 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:02.883993  6472 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:02.846302  6609 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:24:02.815351  6504 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:02.902484  6682 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:24:02.929347  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.903730  6737 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:02.933519  6659 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:24:02.967865  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:02.934218  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:02.934657  6752 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:02.895694  6466 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:02.931371  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 79953.8us
I20260709 12:24:02.968815  1725 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:02.844686  6717 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.071s
I20260709 12:24:02.913067  6483 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.909780  6732 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:02.877856  6523 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:02.995551  2410 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:02.906291  6720 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260709 12:24:02.961858  6675 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:02.965533  6635 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:03.013648  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 275612us
I20260709 12:24:02.996183  6483 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:02.967865  6770 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.035856  6635 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:03.037286  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 166408us
I20260709 12:24:03.038410  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:03.028875  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94755.3us
I20260709 12:24:02.980701  6750 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:03.044996  2412 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:03.047034  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.060415  1727 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:24:02.981765  6609 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
W20260709 12:24:03.066879  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.067559  1877 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:03.030689  6432 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.047922  6747 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:03.080639  2417 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
I20260709 12:24:03.081103  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1068) took 1146 ms. Trace:
I20260709 12:24:03.065892  6659 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:03.073453  1724 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:03.082674  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:03.056749  6750 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.089207  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:03.120728  1799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "908089442ca44109875257bfbc9d06c2"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:24:03.079186  6466 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.121671  1878 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:03.120407  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.137246  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.121430  1799 raft_consensus.cc:606] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:24:03.145112  1732 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:03.148689  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:03.150215  1881 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:24:03.092514  6626 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.027s	user 0.010s	sys 0.010s
I20260709 12:24:03.074097  6469 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:03.063987  6495 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260709 12:24:03.184463  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:03.094830  2418 rpcz_store.cc:276] 0709 12:24:01.934694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.467848 (+533154us) service_pool.cc:224] Handling call
0709 12:24:02.468406 (+   558us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.531807 (+ 63401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.531844 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.542353 (+ 10509us) log.cc:844] Serialized 26 byte log entry
0709 12:24:02.694723 (+152370us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:02.694932 (+   209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.695289 (+   357us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.080865 (+385576us) raft_consensus.cc:1685] finished
0709 12:24:03.080891 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.081029 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33372,"raft.queue_time_us":74119,"raft.run_cpu_time_us":415,"raft.run_wall_time_us":44271,"thread_start_us":143300,"threads_started":1,"wal-append.queue_time_us":389649}
I20260709 12:24:03.190907  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:03.193204  6790 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:24:03.092468  6609 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:03.124738  6750 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:03.203733  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.204327  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:03.193491  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.125088  6645 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:24:03.194027  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:03.201614  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.199148  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:03.205888  1798 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:03.207798  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:03.066870  6770 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.160422  6659 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:03.214581  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.224102  2418 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:03.233376  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46466.5us
I20260709 12:24:03.182184  6432 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.237226  1729 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:03.223548  6768 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:24:03.255107  1724 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:03.256189  6683 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 12:24:03.258039  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:03.259295  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:03.260061  6213 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:03.267199  6792 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:03.267838  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:03.267907  1874 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:03.228739  6609 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.180503  6737 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.280383  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.283077  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:03.237751  6682 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.225914  6469 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:03.306527  6555 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.632s	user 0.007s	sys 0.004s
I20260709 12:24:03.284273  1727 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:03.290376  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:03.275717  6783 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:24:03.292677  6779 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
I20260709 12:24:03.293274  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:03.293650  6795 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.295454  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:03.303794  6787 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:24:03.305982  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:03.315999  2417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 987) took 1385 ms (client timeout 1757 ms). Trace:
I20260709 12:24:03.277324  6768 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:03.275784  6635 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:03.344455  1728 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:03.340425  1798 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:03.347949  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:03.280997  6213 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:03.343420  2417 rpcz_store.cc:269] 0709 12:24:01.930739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.985605 (+ 54866us) service_pool.cc:224] Handling call
0709 12:24:03.315629 (+1330024us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b54005609d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:03.315909 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2244732672}
I20260709 12:24:03.361595  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:03.368530  6523 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:03.369675  1731 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:03.370086  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:03.328840  6682 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.345168  6766 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:03.372025  6766 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.373416  6682 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:03.374368  6766 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:03.371135  1727 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:03.382529  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.325305  6504 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:03.388056  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:03.388823  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 142663us
I20260709 12:24:03.387105  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1072) took 1393 ms. Trace:
I20260709 12:24:03.394013  1800 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:03.397727  1878 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:03.386240  2412 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:24:03.377936  6213 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:03.356124  6768 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:03.389029  6737 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:03.397507  6564 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.631s	user 0.007s	sys 0.007s
W20260709 12:24:03.435798  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 345478us
I20260709 12:24:03.391136  2411 rpcz_store.cc:276] 0709 12:24:01.993842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.541045 (+547203us) service_pool.cc:224] Handling call
0709 12:24:02.636986 (+ 95941us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.835723 (+198737us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:02.835758 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.836410 (+   652us) log.cc:844] Serialized 26 byte log entry
0709 12:24:02.950055 (+113645us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:02.950107 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.950184 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.380738 (+430554us) raft_consensus.cc:1685] finished
0709 12:24:03.380763 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.387022 (+  6259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52489,"raft.queue_time_us":75610,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":14541,"thread_start_us":113084,"threads_started":1,"wal-append.queue_time_us":209811}
I20260709 12:24:03.395242  6555 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:03.450202  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.403322  6219 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.453684  6219 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.454470  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:24:03.455660  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54193.4us
I20260709 12:24:03.456600  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:03.457722  6794 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:24:03.452765  6564 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:24:03.451764  6555 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:03.480371  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2104) took 1134 ms. Trace:
I20260709 12:24:03.493736  2412 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:03.399660  6659 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.511299  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:03.492007  6651 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.525517  6801 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:24:03.525900  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69399.6us
I20260709 12:24:03.526129  6801 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:03.526863  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:03.529397  6806 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:24:03.530230  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 84098.3us
I20260709 12:24:03.535112  1732 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:24:03.503445  6555 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:03.515910  6219 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:03.554755  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.492281  1730 rpcz_store.cc:276] 0709 12:24:02.391784 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.391957 (+   173us) service_pool.cc:224] Handling call
0709 12:24:02.410774 (+ 18817us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.451342 (+ 40568us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.451391 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.457336 (+  5945us) log.cc:844] Serialized 26 byte log entry
0709 12:24:02.682652 (+225316us) spinlock_profiling.cc:243] Waited 21.6 ms on lock 0x7b1800458d40. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:02.939722 (+257070us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:02.940538 (+   816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.941924 (+  1386us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.475831 (+533907us) raft_consensus.cc:1685] finished
0709 12:24:03.475861 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.480283 (+  4422us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26220,"raft.queue_time_us":3666,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":158,"spinlock_wait_cycles":49598336,"thread_start_us":254389,"threads_started":1,"wal-append.queue_time_us":629663}
I20260709 12:24:03.546021  1872 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:24:03.548066  1879 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:24:03.545011  1876 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:24:03.605795  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.433060  6795 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.506512  6504 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.607908  1798 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:03.609043  1724 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:03.540414  1728 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
W20260709 12:24:03.619277  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 65516.2us
I20260709 12:24:03.606850  2418 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:03.615458  1879 pending_rounds.cc:85] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 4
I20260709 12:24:03.541842  6472 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:03.635207  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:03.555986  6796 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
W20260709 12:24:03.617175  6664 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.812s	user 0.009s	sys 0.008s
I20260709 12:24:03.637544  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.629026  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:03.640971  2410 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 7. Current term is 8. Ops: []
I20260709 12:24:03.641820  1729 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:03.644917  6806 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:03.645457  6777 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:24:03.633486  1879 pending_rounds.cc:107] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:24:03.527777  6801 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:03.675890  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 149159us
W20260709 12:24:03.625512  6675 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.595s	user 0.016s	sys 0.000s
I20260709 12:24:03.647500  6795 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:03.660617  1879 raft_consensus.cc:2889] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:03.676407  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.620908  6651 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.627414  6771 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:24:03.674839  6752 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:24:03.684355  1874 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:24:03.654034  1878 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
W20260709 12:24:03.684985  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 58702.7us
I20260709 12:24:03.687443  6806 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.689407  6801 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:24:03.690403  6732 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.652s	user 0.006s	sys 0.004s
I20260709 12:24:03.693490  6750 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:03.694015  6483 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.633s	user 0.010s	sys 0.000s
I20260709 12:24:03.695099  6469 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:03.634727  6786 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:03.639000  6770 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:03.694512  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.696228  6645 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:24:03.696475  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:03.699335  1728 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
W20260709 12:24:03.704836  1726 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.982s	user 0.003s	sys 0.010s
I20260709 12:24:03.743645  1726 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:03.708238  6748 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.708583  6795 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:03.710145  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:03.710983  6809 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:03.641607  6609 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:03.727532  1725 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:03.702879  6675 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11.
I20260709 12:24:03.642256  6504 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.711822  6766 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:03.783444  6809 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:24:03.784111  6809 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:03.786703  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 76674us
I20260709 12:24:03.787818  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:03.795181  6609 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:03.795639  6735 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:03.795851  6609 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.796506  6735 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:03.798449  6609 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:03.684721  6664 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.721002  6628 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:03.747308  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2107) took 1276 ms. Trace:
I20260709 12:24:03.747637  2419 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:03.720592  6219 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:24:03.808151  1726 rpcz_store.cc:276] 0709 12:24:02.467955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.469122 (+  1167us) service_pool.cc:224] Handling call
0709 12:24:03.744284 (+1275162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11008}
I20260709 12:24:03.749872  6748 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.816188  1725 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:24:03.752923  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:03.751104  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 210313us
W20260709 12:24:03.756135  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59795.8us
W20260709 12:24:03.759622  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 228306us
I20260709 12:24:03.817127  1728 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:03.764487  6768 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.767161  6795 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:03.768637  6504 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:03.769378  6769 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.996s	user 0.015s	sys 0.000s
I20260709 12:24:03.769819  6469 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:03.770426  6786 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:03.735332  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:03.812412  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:03.732861  6732 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.738844  6483 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:03.818449  6789 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.498s	user 0.001s	sys 0.003s
I20260709 12:24:03.848297  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.852967  6730 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.855296  1844 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:03.861326  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:03.798053  6735 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.811890  6752 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.858501  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:03.862324  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.785326  6809 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.827935  6737 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.827562  6795 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:03.851176  6803 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.214s	user 0.000s	sys 0.001s
I20260709 12:24:03.847064  6469 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:03.930032  1877 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:24:03.942458  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:03.944337  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 212748us
I20260709 12:24:03.866130  6810 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:03.952468  6213 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:03.956826  6213 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:03.828794  6628 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.850751  6750 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:03.872206  6774 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.191s	user 0.013s	sys 0.005s
I20260709 12:24:03.947499  6795 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:03.987870  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:03.868157  6504 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.930755  6645 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:04.000648  2410 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:24:04.005606  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 153707us
I20260709 12:24:03.873910  6466 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:24:03.867074  6734 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.123s
I20260709 12:24:04.009689  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:03.979001  6735 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:04.011631  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:03.970948  6810 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:04.014518  1725 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:03.852104  6769 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.868929  6555 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.018759  2418 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:24:03.928907  6737 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:03.875255  6752 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.056622  1725 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:24:04.056650  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:04.057047  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:04.056879  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.019798  6809 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:04.087342  1731 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:04.061252  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 248952us
I20260709 12:24:03.996537  6774 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.004522  6213 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.975587  6609 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:24:04.039537  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 177953us
I20260709 12:24:03.870533  6730 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.088337  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.051769  6806 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.020927  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:04.036402  6451 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:04.095572  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 945) took 2164 ms. Trace:
I20260709 12:24:04.060511  6645 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.100797  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:04.060257  6219 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:24:04.041538  6720 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:04.103575  2417 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
W20260709 12:24:04.112707  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 282397us
I20260709 12:24:04.136658  1726 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:24:04.112352  2414 rpcz_store.cc:276] 0709 12:24:01.768715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.768908 (+   193us) service_pool.cc:224] Handling call
0709 12:24:01.769301 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.771339 (+  2038us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:01.771373 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.771938 (+   565us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.889700 (+117762us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b48000cd2d8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:24:02.831279 (+941579us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b1800043ba0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:03.081621 (+250342us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:03.081680 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.081779 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.793367 (+711588us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:03.866556 (+ 73189us) raft_consensus.cc:1685] finished
0709 12:24:03.866578 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.932906 (+ 66328us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1596,"raft.queue_time_us":1899,"raft.run_cpu_time_us":2031,"raft.run_wall_time_us":114375,"spinlock_wait_cycles":51947008,"thread_start_us":208260,"threads_started":1,"wal-append.queue_time_us":1441980}
I20260709 12:24:04.138988  6645 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.060762  6812 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:04.110751  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.017980  6810 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:04.133013  6219 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:04.017797  6504 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.132543  6821 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:04.102267  6466 raft_consensus.cc:2764] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:24:04.163748  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:04.012673  6750 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.113s
I20260709 12:24:04.101193  6651 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:04.149659  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.157366  6734 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.167388  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:04.096246  6786 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:04.175626  6813 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.138s	user 0.001s	sys 0.000s
I20260709 12:24:04.268677  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 670) took 1077 ms. Trace:
I20260709 12:24:04.269114  1724 rpcz_store.cc:276] 0709 12:24:03.191712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.661639 (+469927us) service_pool.cc:224] Handling call
0709 12:24:03.662070 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.662502 (+   432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.662601 (+    99us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.663320 (+   719us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.738670 (+ 75350us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:03.738722 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.738938 (+   216us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:04.268404 (+529466us) raft_consensus.cc:1685] finished
0709 12:24:04.268429 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:04.268602 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15253,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":14370,"thread_start_us":68518,"threads_started":1,"wal-append.queue_time_us":276397}
W20260709 12:24:04.270773  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:04.272512  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 105222us
I20260709 12:24:04.273886  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:04.274948  6659 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:04.275261  6740 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:04.275796  6740 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:04.275952  6659 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:04.277331  6659 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:04.177508  6827 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:04.277987  6827 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:04.278422  6827 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:04.278323  6734 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.281337  6734 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:04.281903  6824 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:04.242059  6821 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:04.282506  6824 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:04.282929  6821 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:04.283957  6659 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.283694  6824 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.284677  6659 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.179417  6769 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:04.180397  6737 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:04.287483  6737 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:04.287947  6737 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:04.288368  6737 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:04.175966  6555 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.289489  6750 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.163055  6720 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.291080  6651 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:04.291821  6651 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:04.293059  6720 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.181303  6732 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:04.293681  6766 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.294328  6766 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.182502  6822 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.295967  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.295897  6822 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.296972  6766 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:04.297528  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.145937  6806 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.298296  1724 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:04.182694  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3071) took 1001 ms. Trace:
I20260709 12:24:04.299695  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1075) took 1600 ms. Trace:
I20260709 12:24:04.298997  1729 rpcz_store.cc:276] 0709 12:24:03.181145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.684055 (+502910us) service_pool.cc:224] Handling call
0709 12:24:03.715890 (+ 31835us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.750902 (+ 35012us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.750939 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.760733 (+  9794us) log.cc:844] Serialized 26 byte log entry
0709 12:24:04.071058 (+310325us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:04.071121 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.071218 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:04.182292 (+111074us) raft_consensus.cc:1685] finished
0709 12:24:04.182329 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:04.182592 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35227,"raft.queue_time_us":10175,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":16326,"thread_start_us":210293,"threads_started":1,"wal-append.queue_time_us":313687}
I20260709 12:24:04.300163  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:04.301319  1729 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:24:04.182973  6826 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:04.303031  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:04.303392  6826 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:04.302997  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:04.304930  6682 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:04.185446  1878 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:24:04.305815  6682 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:04.305888  6771 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:04.185890  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:04.186698  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:04.266664  6801 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.532s	user 0.010s	sys 0.000s
I20260709 12:24:04.307703  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:04.186178  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:04.309051  1874 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:04.309894  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:04.186925  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:04.311215  1874 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:24:04.311878  1872 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:24:04.312325  1874 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 9. Current term is 10. Ops: []
I20260709 12:24:04.314381  6806 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:04.203150  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 152157us
I20260709 12:24:04.315487  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:04.317135  6770 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.185065  1880 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
W20260709 12:24:04.318122  2419 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 938) took 2928 ms (client timeout 1998 ms). Trace:
I20260709 12:24:04.203831  6797 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.318449  1880 raft_consensus.cc:606] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:24:04.238549  6483 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:04.231024  6748 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:04.318650  2419 rpcz_store.cc:269] 0709 12:24:01.389749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.398147 (+  8398us) service_pool.cc:224] Handling call
0709 12:24:04.317841 (+2919694us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b5400489754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:04.318006 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4456770304}
I20260709 12:24:04.318449  6642 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.320237  6642 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.323083  6797 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.326933  1729 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:24:04.327414  1724 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:04.328444  1729 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:04.328547  6432 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:04.306461  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:04.330082  1878 raft_consensus.cc:2376] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10. Re-sending same reply.
I20260709 12:24:04.332690  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:04.332829   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:24:04.164486  6664 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.298550  6822 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:04.252288  6645 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.252986  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:04.178510  6504 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:04.260214  6628 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.262544  6752 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:04.264127  6814 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.284s	user 0.001s	sys 0.000s
I20260709 12:24:04.265892  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 665) took 1167 ms. Trace:
I20260709 12:24:04.338420  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2917) took 1262 ms. Trace:
W20260709 12:24:04.242605  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.338965  1877 rpcz_store.cc:276] 0709 12:24:03.075648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.075831 (+   183us) service_pool.cc:224] Handling call
0709 12:24:03.119353 (+ 43522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.934950 (+815597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.338159 (+403209us) spinlock_profiling.cc:243] Waited 810 ms on lock 0x7b5400073d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:04.338355 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1863243136}
I20260709 12:24:04.173516  6768 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:04.142870  6213 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.290148  6750 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.145613  1726 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:04.294433  6651 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:24:04.170631  6609 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:04.304520  6826 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.306461  6771 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:04.190281  6800 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:04.317631  6770 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.332955  6642 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:04.240365  6774 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.195183  6219 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.342329  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.276830  6740 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.367630  6752 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:04.368414  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.370476  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:04.370990  6659 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:04.338125  1732 rpcz_store.cc:276] 0709 12:24:03.098166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.146646 (+ 48480us) service_pool.cc:224] Handling call
0709 12:24:03.147468 (+   822us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.535698 (+388230us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.265529 (+729831us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b5400119ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:04.265807 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":845550336}
I20260709 12:24:04.371307  6821 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.300072  2416 rpcz_store.cc:276] 0709 12:24:02.699219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.858754 (+159535us) service_pool.cc:224] Handling call
0709 12:24:03.346291 (+487537us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.358741 (+ 12450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.358770 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:03.358877 (+   107us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:03.372385 (+ 13508us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.372451 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:04.299440 (+926989us) spinlock_profiling.cc:243] Waited 464 ms on lock 0x7b5400487450. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:04.299586 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1067916672}
W20260709 12:24:04.391906  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.417452  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 669) took 1243 ms. Trace:
I20260709 12:24:04.398468  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:04.419328  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:04.417953  1731 rpcz_store.cc:276] 0709 12:24:03.174358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.371824 (+197466us) service_pool.cc:224] Handling call
0709 12:24:03.372182 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:04.093327 (+721145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.417207 (+323880us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54000d2ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:04.417387 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1582025984}
W20260709 12:24:04.389171  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 288423us
I20260709 12:24:04.401464  6609 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:04.397518  6466 raft_consensus.cc:2764] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:24:04.400326  6628 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:04.423240  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52868.8us
I20260709 12:24:04.426705  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:04.422798  6822 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:04.429996  1727 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:04.461941  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:04.424965  6609 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:04.459182  1876 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
W20260709 12:24:04.437382  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.000s	sys 0.009s
W20260709 12:24:04.469123  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 152316us
W20260709 12:24:04.470516  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.000s	sys 0.013s
I20260709 12:24:04.444435  6771 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.472467  6771 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:04.474893  6651 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:04.476050  6713 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:04.476505  6713 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:04.476879  6713 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:04.478048  1879 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:04.414664  6748 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:24:04.480880  6797 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:04.428380  6659 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:04.481665  6659 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:04.482048  6659 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:04.485929  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.486164  6213 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.429761  6800 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.488737  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.489285  6800 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:04.491120  6821 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.492609  6750 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:04.495281  6770 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:04.466634  6752 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:24:04.502449  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:04.506845  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:04.460428  6822 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:04.516101  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.516301  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:04.499648  6682 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:04.521214  1803 raft_consensus.cc:2368] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11.
I20260709 12:24:04.521237  1802 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:24:04.528507  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.529824  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.532454  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:04.510637  6651 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:04.544253  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.548560  2419 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:24:04.544631  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:04.550971  1800 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
W20260709 12:24:04.551046  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 494021us
I20260709 12:24:04.514263  6822 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:04.463717  6737 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.540926  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.552814  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:04.500612  6628 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:04.559376  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 395694us
I20260709 12:24:04.513015  6752 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:24:04.553985  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.571165  1801 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
W20260709 12:24:04.563019  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 542187us
I20260709 12:24:04.571906  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:04.575217  1802 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
W20260709 12:24:04.581135  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.551455  6750 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.587457  1804 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:04.598819  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83521.4us
W20260709 12:24:04.603715  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 205871us
I20260709 12:24:04.531423  6827 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.581779  6819 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:04.581811  6770 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:04.606195  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:04.609442  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.598049  6750 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.618175  6750 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:04.624596  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 343281us
I20260709 12:24:04.603820  6800 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:04.632324  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:04.633239  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.633616  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.635130  1874 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:24:04.638895  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.638422  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:04.641244  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.643147  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:04.643893  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:04.643893  1881 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 12 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:04.645756  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:04.648048  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.648679  1726 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:24:04.648770  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:04.649628  1880 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:04.646279  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:04.650306  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:04.652007  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.652583  1728 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:04.654671  1723 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:04.629226  6213 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.660013  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.658761  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.658245  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.661405  1724 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:04.661545  2412 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:24:04.649183  6800 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:04.663046  6800 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.663774  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:04.664985  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:04.665846  1798 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:04.667160  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.668373  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:04.668921  1799 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:04.669406  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:04.669966  1803 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:04.666728  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.670382  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.671005  1798 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:04.672667  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:04.634117  6819 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.655557  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:04.685484  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48608.2us
I20260709 12:24:04.661019  1728 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:04.634773  6770 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:04.689313  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.690109  2412 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:04.686630  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:04.719568  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 95090.1us
I20260709 12:24:04.720644  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.721570  1728 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:04.719599  2418 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:24:04.742467  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.743479  2412 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:04.748176  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.754302  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:04.755125  1798 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:04.752974  2410 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
W20260709 12:24:04.690903  6813 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:24:04.655355  6833 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.659111  6768 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:24:04.758217  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66982.7us
I20260709 12:24:04.758698  1726 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:24:04.767843  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 91859.5us
I20260709 12:24:04.768662  1845 leader_election.cc:400] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 10. Current term is 11.
I20260709 12:24:04.638739  6825 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.081s
W20260709 12:24:04.760737  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.763983  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:04.759621  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 73340.7us
I20260709 12:24:04.653342  6826 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.083s
W20260709 12:24:04.771695  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.760231  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.691243  6664 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:04.776712  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.720062  6809 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.675218  6774 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.103s
I20260709 12:24:04.797384  1728 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:24:04.798178  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:04.799381  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:04.800747  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:04.751070  6837 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.082s
I20260709 12:24:04.803329  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:04.824990  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50341.7us
I20260709 12:24:04.820401  1878 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:04.829331  1798 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:24:04.803683  6834 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.172s	user 0.000s	sys 0.000s
I20260709 12:24:04.804368  6730 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.849442  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:04.804659  6827 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:04.773145  6768 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:04.719923  6819 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.810106  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:04.831084  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50877us
W20260709 12:24:04.830942  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 148140us
I20260709 12:24:04.833060  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:04.834228  6835 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.191s	user 0.000s	sys 0.001s
I20260709 12:24:04.834622  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:04.835176  6836 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 12:24:04.834868  6469 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.797564  6663 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.760413  6432 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.760073  6675 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:04.778330  6451 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.643s	user 0.008s	sys 0.004s
I20260709 12:24:04.843902  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:04.683966  6213 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:04.773779  6766 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:04.826996  6795 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.817s	user 0.012s	sys 0.001s
I20260709 12:24:04.819666  1880 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:24:04.818849  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:05.124471  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 280639us
I20260709 12:24:05.125236  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:04.853071  6810 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.610s	user 0.012s	sys 0.000s
I20260709 12:24:04.852932  6664 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:04.942003  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:04.916591  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 118542us
I20260709 12:24:04.883342  6768 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:04.835577  6833 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:04.827234  6809 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:04.917721  6845 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:24:04.989032  6827 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.767187  6682 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260709 12:24:04.845389  6815 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.102s
I20260709 12:24:05.065294  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:05.066409  1876 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
W20260709 12:24:05.063879  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61049.2us
W20260709 12:24:05.067145  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 257069us
I20260709 12:24:05.174070  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:05.069514  1723 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:05.175832  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:05.070320  1801 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:24:05.075565  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:05.070864  1804 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
W20260709 12:24:05.084791  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.089157  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:05.090760  1803 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:05.063277   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:24:05.095675  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.105896  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3072) took 1881 ms. Trace:
I20260709 12:24:05.107250  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:04.943270  6770 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:05.179358  2412 raft_consensus.cc:2376] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 13. Re-sending same reply.
W20260709 12:24:05.131435  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.104737  6713 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.627s	user 0.019s	sys 0.000s
I20260709 12:24:05.140342  2413 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:05.106494  6663 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:05.185250  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.101815  6628 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:05.147851  6837 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:05.166165  1877 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:05.115346  6451 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:05.133313  6469 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:05.173174  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:05.126672  6766 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:05.063277  6786 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.170707  6757 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:05.191452  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:05.139631  6737 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:05.195158  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:05.195535  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 130214us
I20260709 12:24:05.176570  6821 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:05.200886  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45426.2us
I20260709 12:24:05.202091  1728 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:05.204375  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:05.140439  6810 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.179389  1730 rpcz_store.cc:276] 0709 12:24:03.225007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.755044 (+530037us) service_pool.cc:224] Handling call
0709 12:24:03.766439 (+ 11395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.801591 (+ 35152us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.801638 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.820888 (+ 19250us) log.cc:844] Serialized 26 byte log entry
0709 12:24:04.517408 (+696520us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:04.517438 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.517544 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.105508 (+587964us) raft_consensus.cc:1685] finished
0709 12:24:05.105547 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.105795 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":292,"raft.queue_time_us":34797,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":8456,"thread_start_us":606143,"threads_started":1,"wal-append.queue_time_us":724064}
I20260709 12:24:05.210871  1874 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:24:05.116770  6432 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:05.215556  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3077) took 1740 ms. Trace:
I20260709 12:24:05.140012  6795 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:05.126927  6675 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:05.230589  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 57543.6us
I20260709 12:24:05.240823  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:05.238936  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 175643us
W20260709 12:24:05.237783  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 103087us
I20260709 12:24:05.254717  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:05.255483  1726 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:05.196638  6837 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:05.205467  6797 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:05.204210  6642 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:24:05.176226  6827 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.261859  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:05.268491  1723 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:05.212512  6815 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:05.140203  6861 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.111s
I20260709 12:24:05.203814  6766 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:05.278072  1799 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.609s	user 0.009s	sys 0.000s
I20260709 12:24:05.285907  1799 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:05.286257  2416 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:05.245064  1725 rpcz_store.cc:276] 0709 12:24:03.475251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.090917 (+615666us) service_pool.cc:224] Handling call
0709 12:24:04.101923 (+ 11006us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:04.149030 (+ 47107us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:04.149067 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:04.166584 (+ 17517us) log.cc:844] Serialized 26 byte log entry
0709 12:24:04.514807 (+348223us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:04.514853 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.514954 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.215355 (+700401us) raft_consensus.cc:1685] finished
0709 12:24:05.215373 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.215497 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5600,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":4632,"thread_start_us":331627,"threads_started":1,"wal-append.queue_time_us":420961}
I20260709 12:24:05.282457  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:05.292320  1728 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:05.273663  6852 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:05.190631  6713 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.335103  1730 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:05.293895  6832 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:24:05.294479  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63305.2us
I20260709 12:24:05.236907  6821 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.308984  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 660) took 1123 ms. Trace:
W20260709 12:24:05.315115  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 99715.6us
I20260709 12:24:05.263154  6837 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:05.261457  6797 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:05.293133  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:05.293694  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:05.332585  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:05.340297  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:05.277587  6815 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:05.334375  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 147911us
I20260709 12:24:05.229885  6757 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:05.330526  6852 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.253796  6786 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.356969  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:05.338243  6642 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:05.375408  2414 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:24:05.338915  6832 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.380510  2413 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:05.339588  1873 rpcz_store.cc:276] 0709 12:24:04.185399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.185539 (+   140us) service_pool.cc:224] Handling call
0709 12:24:04.336400 (+150861us) spinlock_profiling.cc:243] Waited 56.6 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:24:04.336627 (+   227us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:04.337082 (+   455us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:04.337453 (+   371us) log.cc:844] Serialized 14 byte log entry
0709 12:24:04.613454 (+276001us) log.cc:844] Serialized 14 byte log entry
0709 12:24:04.613573 (+   119us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:04.648551 (+ 34978us) log.cc:844] Serialized 52 byte log entry
0709 12:24:04.648734 (+   183us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:04.648987 (+   253us) log.cc:844] Serialized 14 byte log entry
0709 12:24:04.655745 (+  6758us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.666196 (+ 10451us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.308717 (+642521us) raft_consensus.cc:1685] finished
0709 12:24:05.308747 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.308909 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60658,"prepare.queue_time_us":4074,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":154,"raft.queue_time_us":7704,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":61597,"spinlock_wait_cycles":179934720,"thread_start_us":269583,"threads_started":1,"wal-append.queue_time_us":416889}
I20260709 12:24:05.345607  6797 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:05.410712  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53139.2us
W20260709 12:24:05.370512  6735 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.334s	user 0.016s	sys 0.006s
I20260709 12:24:05.348948  6815 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:05.410125  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 128821us
I20260709 12:24:05.367697  6852 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:05.404253  6849 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.320s	user 0.000s	sys 0.002s
I20260709 12:24:05.422533  6809 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:05.346765  6682 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.065s
I20260709 12:24:05.403417  6860 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:05.317138  6664 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:05.405804  6832 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:05.443960  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:05.426510  6869 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.431823  6861 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:24:05.471587  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 177553us
I20260709 12:24:05.423682  6472 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:24:05.427929  6757 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.482576  6735 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:05.482290  6839 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:24:05.445564  6213 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:05.447930  6860 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:05.497138  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 140296us
I20260709 12:24:05.520565  1726 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:24:05.530992  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:05.478457  6848 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.265s	user 0.002s	sys 0.000s
I20260709 12:24:05.543674  6469 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.544683  6469 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:05.535411  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 123458us
I20260709 12:24:05.535918  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:05.471975  6861 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:05.498229  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54358.9us
I20260709 12:24:05.556421  2417 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:05.471321  6847 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:05.524307  6833 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:05.577131  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1087) took 1090 ms. Trace:
W20260709 12:24:05.571853  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.524497  6663 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:05.466238  6873 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:05.533025  6504 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.557468  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:05.615149  1726 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:24:05.577791  2419 rpcz_store.cc:276] 0709 12:24:04.486718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.488463 (+  1745us) service_pool.cc:224] Handling call
0709 12:24:05.576955 (+1088492us) spinlock_profiling.cc:243] Waited 59 ms on lock 0x7b5400572e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:05.577060 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":135605120}
I20260709 12:24:05.538331  6869 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:05.583256  6871 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:05.551249  6825 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:05.584851  6861 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:05.585755  6771 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.113s	user 0.009s	sys 0.004s
I20260709 12:24:05.586550  6713 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:05.586711  6853 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:24:05.587141  6875 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.099s
W20260709 12:24:05.588004  6851 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.336s	user 0.001s	sys 0.000s
I20260709 12:24:05.587586  6472 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:05.529564  6860 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:05.661545  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:05.537551  6213 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:24:05.591193  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 181821us
I20260709 12:24:05.550949  6795 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:05.590679  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:05.608001  1877 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:05.577186  6847 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:05.614461  1732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.012s	sys 0.010s
I20260709 12:24:05.588960  6810 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:05.577521  6675 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:05.631212  6663 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:24:05.670941  1732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.013s	sys 0.010s
I20260709 12:24:05.588510  6859 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:24:05.623196  6871 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:05.690351  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.692572  1800 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.014s	sys 0.007s
W20260709 12:24:05.693046  1800 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.014s	sys 0.007s
W20260709 12:24:05.693216  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 102634us
I20260709 12:24:05.693816  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2038) took 1031 ms. Trace:
I20260709 12:24:05.694113  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:05.636176  6504 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:05.672919  1802 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.003s	sys 0.006s
W20260709 12:24:05.709167  1802 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.003s	sys 0.006s
W20260709 12:24:05.682617  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 155923us
I20260709 12:24:05.686710  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:05.667263  6659 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.185s	user 0.015s	sys 0.000s
I20260709 12:24:05.673877  6847 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:05.685736  6861 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:05.672245  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 136389us
I20260709 12:24:05.694293  1800 rpcz_store.cc:276] 0709 12:24:04.662240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.666677 (+  4437us) service_pool.cc:224] Handling call
0709 12:24:04.676539 (+  9862us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:04.676741 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:05.057635 (+380894us) log.cc:844] Serialized 14 byte log entry
0709 12:24:05.693402 (+635767us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:05.693461 (+    59us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:05.693508 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.693572 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.693763 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63929,"prepare.queue_time_us":634436,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":117052,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":15877,"thread_start_us":920731,"threads_started":3,"wal-append.queue_time_us":351937}
I20260709 12:24:05.715548  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:05.663456  6213 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:05.718767  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:05.690762  6859 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:05.722163  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:05.689836  6821 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:05.690625  6871 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:05.706972  6663 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:05.721963  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3100) took 1076 ms. Trace:
I20260709 12:24:05.719679  6810 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:24:05.733145  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1000) took 1132 ms. Trace:
I20260709 12:24:05.757225  2411 rpcz_store.cc:276] 0709 12:24:04.600899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.601216 (+   317us) service_pool.cc:224] Handling call
0709 12:24:04.612971 (+ 11755us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:04.631994 (+ 19023us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:04.632041 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:04.642195 (+ 10154us) log.cc:844] Serialized 26 byte log entry
0709 12:24:05.130932 (+488737us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:05.131003 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.131129 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.732880 (+601751us) raft_consensus.cc:1685] finished
0709 12:24:05.732906 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.733073 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11990,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":11615,"thread_start_us":455646,"threads_started":1,"wal-append.queue_time_us":553858}
I20260709 12:24:05.727035  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:05.711732  6748 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:24:05.732487  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.034s	sys 0.038s
W20260709 12:24:05.763044  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.034s	sys 0.038s
I20260709 12:24:05.727069  6213 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:05.726444  6786 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:05.770758  6472 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:05.771473  6472 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:05.770170  6663 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:05.772450  6472 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:05.752619  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:05.726969  6825 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:05.765741  6821 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:05.767022  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2035) took 1172 ms. Trace:
I20260709 12:24:05.767448  6504 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:05.768293  6609 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.262s	user 0.005s	sys 0.010s
I20260709 12:24:05.726199  6659 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.778174  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2127) took 1108 ms. Trace:
I20260709 12:24:05.786747  1880 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:05.756923  6451 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
I20260709 12:24:05.796754  6890 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:05.797073  6880 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.797802  6880 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.798763  1879 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:05.731463  6877 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:05.787276  1805 rpcz_store.cc:276] 0709 12:24:04.595111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.595385 (+   274us) service_pool.cc:224] Handling call
0709 12:24:04.608259 (+ 12874us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:04.625745 (+ 17486us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:04.625792 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:04.643193 (+ 17401us) log.cc:844] Serialized 52 byte log entry
0709 12:24:04.720801 (+ 77608us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:04.720878 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.721011 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.766732 (+1045721us) raft_consensus.cc:1685] finished
0709 12:24:05.766758 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.766952 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16995,"raft.queue_time_us":17798,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":13378,"thread_start_us":46541,"threads_started":1,"wal-append.queue_time_us":442915}
I20260709 12:24:05.795627  1723 rpcz_store.cc:276] 0709 12:24:04.669629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.281242 (+611613us) service_pool.cc:224] Handling call
0709 12:24:05.311151 (+ 29909us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.311482 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:05.311822 (+   340us) log.cc:844] Serialized 14 byte log entry
0709 12:24:05.777748 (+465926us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:05.777814 (+    66us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:05.777870 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.777945 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.778089 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6161,"prepare.queue_time_us":136,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":273631,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":4644,"thread_start_us":454562,"threads_started":2,"wal-append.queue_time_us":222516}
I20260709 12:24:05.753372  6642 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:05.779224  6801 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:05.808585  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89974.4us
I20260709 12:24:05.817775  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:05.780747  6825 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:05.865654  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.887410  1807 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:05.888263  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:05.808458  6850 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.321s	user 0.001s	sys 0.000s
W20260709 12:24:05.902068  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84399.3us
I20260709 12:24:05.799607  6609 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.908710  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:05.909722  1879 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:24:05.928640  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:05.808223  6469 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:05.907399  6886 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:05.891250  6825 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:05.882562  6730 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:05.934476  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:05.936451  6881 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:05.937423  6894 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
W20260709 12:24:05.937777  6896 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:24:05.968505  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:05.938531  6882 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:24:05.941618  6854 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:05.931663  6821 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
W20260709 12:24:05.908205  6867 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.326s	user 0.001s	sys 0.000s
I20260709 12:24:05.947229  1875 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:05.893836  6451 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.890857  6801 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:05.955361  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:05.992503  1723 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:05.958360  6877 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:05.957397  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1088) took 1465 ms. Trace:
I20260709 12:24:05.903599  6879 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:05.994848  2414 rpcz_store.cc:276] 0709 12:24:04.491869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.492465 (+   596us) service_pool.cc:224] Handling call
0709 12:24:04.492833 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:05.375960 (+883127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.957129 (+581169us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b54006009d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:05.957336 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2014513408}
W20260709 12:24:05.996248  6878 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.285s	user 0.001s	sys 0.000s
I20260709 12:24:05.952947  6682 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:05.976188  6797 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.544s	user 0.012s	sys 0.000s
I20260709 12:24:05.960395  6881 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:06.008322  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.018990  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:05.957094  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2132) took 1184 ms. Trace:
I20260709 12:24:05.986615  6730 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:24:06.032668  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:06.040167  1880 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:06.041283  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:06.048291  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.050246  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.051841  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:06.053275  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84944.6us
I20260709 12:24:06.054265  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:06.025223  2382 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.684s (SENT) [suppressed 4 similar messages]
I20260709 12:24:06.056568  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.058311  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.064671  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:06.068370  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.071571  2411 raft_consensus.cc:2410] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:24:06.072296  1800 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:06.082007  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2119) took 1414 ms. Trace:
I20260709 12:24:06.082587  1724 rpcz_store.cc:276] 0709 12:24:04.667405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.678358 (+ 10953us) service_pool.cc:224] Handling call
0709 12:24:04.725483 (+ 47125us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:04.836977 (+111494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:04.837035 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:04.839361 (+  2326us) log.cc:844] Serialized 26 byte log entry
0709 12:24:05.183855 (+344494us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:05.183930 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.184098 (+   168us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:06.081700 (+897602us) raft_consensus.cc:1685] finished
0709 12:24:06.081732 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:06.081929 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":47805,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":10387,"thread_start_us":213483,"threads_started":2,"wal-append.queue_time_us":347197}
I20260709 12:24:06.026638  6865 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.069242  1801 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:24:06.047535  6730 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:05.994338  6451 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.093760  2414 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:06.028519  6881 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.095039  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.103394  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.104228  1803 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:24:06.043057  6504 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:06.027947  6797 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.028335  6682 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.045034  6734 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 12:24:06.093688  6713 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:24:06.129277  6713 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:06.130463  6713 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.144717  6451 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:06.146013  6822 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.589s	user 0.010s	sys 0.006s
I20260709 12:24:06.146497  6822 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:06.147518  6893 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:24:06.148350  6888 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:06.148792  6822 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:05.997090  6854 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.149327  6888 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.025559  6642 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.153770  6888 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:06.123075  6504 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.156867  6504 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:06.060039  6859 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
I20260709 12:24:06.034809  6879 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:06.159696  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.159322  6730 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.160172  6801 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:06.164392  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.162629  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 137449us
I20260709 12:24:06.099666  6865 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.180840  6865 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:06.173257  6837 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.817s	user 0.000s	sys 0.012s
I20260709 12:24:06.224613  6883 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
W20260709 12:24:06.204095  6900 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:24:06.196915  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:06.159336  6879 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.210855  1727 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:06.270550  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.271241  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:06.271706  6740 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.271993  1876 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:06.275417  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.276347  1881 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:24:06.277127  6839 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.277887  6839 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.211261  6833 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:06.274204  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.280563  6833 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:06.281114  1879 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:24:06.281600  6833 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.164273  6877 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.217700  6797 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:06.220657  6832 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.747s	user 0.008s	sys 0.001s
W20260709 12:24:06.221294  6768 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.045s	user 0.009s	sys 0.004s
I20260709 12:24:06.191042  6824 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:06.223331  6682 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:06.222894  6213 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.224331  6837 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.228334  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2043) took 1564 ms. Trace:
I20260709 12:24:06.228658  6869 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:06.289417  6771 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.321200  6771 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.324206  6771 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.325022  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.325757  1804 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:06.329115  6837 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:06.331107  6213 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:06.332010  2413 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:06.326668  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.230532  6609 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.207404  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:06.209870  6825 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.239450  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:06.283659  6839 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:06.245841  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2931) took 2700 ms. Trace:
I20260709 12:24:06.247196  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:06.248265  6883 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:24:06.248823  6847 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.519s	user 0.003s	sys 0.007s
I20260709 12:24:06.346731  6847 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.346763  6466 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.348127  6466 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.350034  6847 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:06.350287  6466 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:06.251367  6815 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.803s	user 0.009s	sys 0.000s
I20260709 12:24:06.351174  6815 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.354164  6815 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:06.254087  6886 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:06.295826  6832 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:06.356688  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2029) took 1842 ms (client timeout 1704 ms). Trace:
W20260709 12:24:06.357174  1807 rpcz_store.cc:269] 0709 12:24:04.514652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.516584 (+  1932us) service_pool.cc:224] Handling call
0709 12:24:06.356383 (+1839799us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54003b7ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:06.356567 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3111668480}
I20260709 12:24:06.255642  6432 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:06.360355  6825 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.255887  6871 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.262120  6819 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:06.262259  6852 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.798s	user 0.010s	sys 0.001s
I20260709 12:24:06.361690  6852 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.363051  6774 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:06.363708  6774 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:06.364078  6852 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.266517  6881 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.344870  6883 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:06.365758  1799 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
W20260709 12:24:06.268606  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.366102  6883 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.369306  1804 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:06.358534  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:06.300122  1723 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:06.371554  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:06.029474  1725 rpcz_store.cc:276] 0709 12:24:04.772512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.290546 (+518034us) service_pool.cc:224] Handling call
0709 12:24:05.315656 (+ 25110us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.315921 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:05.348053 (+ 32132us) log.cc:844] Serialized 14 byte log entry
0709 12:24:05.940969 (+592916us) spinlock_profiling.cc:243] Waited 9.5 ms on lock 0x7b180030c880. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:24:05.956678 (+ 15709us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:05.956746 (+    68us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:05.956803 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.956880 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.957028 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56083,"prepare.queue_time_us":32205,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":4610,"raft.queue_time_us":20579,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":11943,"spinlock_wait_cycles":21852544,"thread_start_us":86375,"threads_started":1,"wal-append.queue_time_us":268828}
I20260709 12:24:06.315701  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:06.231091  6766 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.919s	user 0.012s	sys 0.003s
I20260709 12:24:06.226178  2415 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:06.240306  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2944) took 1432 ms. Trace:
I20260709 12:24:06.334132  2411 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:06.375298  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:06.378127  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:06.295466  6865 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.332017  6659 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.296736  6833 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:06.318327  1802 rpcz_store.cc:276] 0709 12:24:04.663932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.746505 (+ 82573us) service_pool.cc:224] Handling call
0709 12:24:04.788144 (+ 41639us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:05.712865 (+924721us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:05.712895 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:05.728684 (+ 15789us) log.cc:844] Serialized 52 byte log entry
0709 12:24:05.927776 (+199092us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:05.934125 (+  6349us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.034220 (+100095us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.034309 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:06.228068 (+193759us) raft_consensus.cc:1685] finished
0709 12:24:06.228090 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:06.228267 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":253471,"prepare.queue_time_us":13028,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":383706,"raft.run_cpu_time_us":409,"raft.run_wall_time_us":26649,"thread_start_us":877111,"threads_started":3,"wal-append.queue_time_us":240846}
I20260709 12:24:06.340174  1875 rpcz_store.cc:276] 0709 12:24:03.545783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.555098 (+  9315us) service_pool.cc:224] Handling call
0709 12:24:04.268396 (+713298us) spinlock_profiling.cc:243] Waited 2.49 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:24:04.268686 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.983949 (+1715263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.245494 (+261545us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54002058d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:06.245728 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3865363072}
I20260709 12:24:06.360754  6651 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:24:06.337038  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.374359  1874 rpcz_store.cc:276] 0709 12:24:04.808254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.833736 (+ 25482us) service_pool.cc:224] Handling call
0709 12:24:04.834110 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.231588 (+397478us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.240064 (+1008476us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b540001efd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:06.240248 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":865511296}
I20260709 12:24:06.358238  6832 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.336424  6856 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:06.307852  6768 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.362970  6871 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:06.411664  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:06.412901  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.010s	sys 0.002s
W20260709 12:24:06.481755  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.010s	sys 0.004s
W20260709 12:24:06.411594  6862 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.984s	user 0.019s	sys 0.013s
I20260709 12:24:06.439009  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:06.366750  6881 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.451211  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.456966  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.497208  1803 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:06.406610  6837 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:06.405851  6504 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:05.345620   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:06.406283  6875 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.467314  2411 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:06.412943  6883 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:06.472288  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:06.514955  2416 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:06.476408  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.465771  6651 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:06.484563  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 117296us
I20260709 12:24:06.462311  6865 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.410212  6859 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 12:24:06.407193  6766 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.364575  6774 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.500735  1807 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:06.545922  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:06.485854  6862 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.075s	user 0.019s	sys 0.014s
I20260709 12:24:06.525859  1875 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:24:06.488040  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:06.554139  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:06.554709  2412 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.530483  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:06.498653  6659 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.532424  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:06.563943  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:06.534567  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:06.535579  6735 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:06.514818  6504 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:06.538086  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.506592  6837 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.547114  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.550735  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.550951  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.639770  1728 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:06.551488  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.643147  1727 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:06.556864  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:06.644487  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 87973us
I20260709 12:24:06.645686  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:06.557376  1724 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:06.559074  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:06.576145  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.647197  1723 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.601024  6847 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:06.603042  6774 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:06.531996  6871 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.617609  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.558481  6822 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.607596  6651 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.575037  6609 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:06.580884  1731 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:06.526158  6819 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:06.629699  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59505.2us
I20260709 12:24:06.538307  6472 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.544020  6832 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:06.631491  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 57639.9us
I20260709 12:24:06.531383  6815 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.612378  6659 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.660315  6469 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:24:06.696805  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.697429  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.646628  2411 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.505640  6856 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.653218  1874 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:06.678113  6861 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.951s	user 0.014s	sys 0.000s
I20260709 12:24:06.657923  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:06.660874  1725 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:06.660962  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.665266  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:06.539700  6881 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.093s
W20260709 12:24:06.679647  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 165557us
I20260709 12:24:06.654332  6847 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.652459  6837 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.687702  1723 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:06.694751  2412 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:06.524622  6875 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:06.717311  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.701735  2410 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:06.664731  6432 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:06.708756  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:06.716325  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 58556.1us
I20260709 12:24:06.719337  1874 raft_consensus.cc:2410] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 14 because replica has last-logged OpId of term: 13 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 12.
W20260709 12:24:06.709185  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.675094  6877 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.712013  1875 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
W20260709 12:24:06.720183  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.720108  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:06.652863  6642 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:24:06.722328  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.727625  2414 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:06.724181  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.723541  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:06.725786  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.736193  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.737066  1876 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:24:06.737326  1723 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:06.731541  1730 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:06.716754  6651 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:06.739938  1874 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:06.676983  6822 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.741685  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.778716  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.718348  6847 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:06.738268  1731 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:06.779235  1875 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:06.783200  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.748831  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:06.758255  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:06.696398  6472 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.792128  6822 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:06.793789  6472 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:06.782413  6432 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 12:24:06.794909  6865 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:06.795755  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:06.795846  6432 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:06.796977  6766 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.797289  6879 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:24:06.797708  6879 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:06.791605  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:06.798089  6879 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:06.798372  6856 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:06.798691  1875 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:06.743314  6914 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 12:24:06.799726  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:06.800585  1875 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:06.800959  6912 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.164s	user 0.000s	sys 0.000s
I20260709 12:24:06.801491  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:06.802305  1875 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:06.802603  6913 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
I20260709 12:24:06.681180  6819 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:06.804214  6920 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
W20260709 12:24:06.805608  6905 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.266s	user 0.001s	sys 0.000s
I20260709 12:24:06.806012  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.733321  6469 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:06.702581  6815 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.806986  6469 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:06.807257  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:06.807503  6469 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:06.808552  6815 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:06.808640  6879 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.725919  6877 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.721882  6909 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:06.811018  6879 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:06.811621  6909 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:06.812628  6819 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:06.813249  6910 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.258s	user 0.001s	sys 0.000s
I20260709 12:24:06.812817  6909 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:06.794306  6870 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:06.795033  6797 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:06.817492  6659 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.818337  2415 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:06.820130  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:06.704684  6871 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.823454  1874 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:06.824672  6871 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.825105  6875 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:06.793922  6919 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:24:06.701632  6832 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.828222  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.830682  1730 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:24:06.836953  2411 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:06.839388  1801 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:06.828954  1725 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:06.845319  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.846578  1723 raft_consensus.cc:2410] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:24:06.854660  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.854918  2415 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:06.849700  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:06.857425  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.858479  1723 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:06.856922  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:06.810922  6768 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.863587  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:06.864456  1725 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 15
W20260709 12:24:06.865664  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.864640  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:06.867777  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:06.874239  6871 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:06.875471  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.879004  6870 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:06.879513  6870 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.879842  6870 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:06.880560  6213 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.881129  6213 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.883833  6831 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:06.884405  6831 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:06.885531  6831 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.889372  2416 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:06.873539  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:06.878475  6877 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:06.895399  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.864957  6825 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:06.900053  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:06.900131  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:06.903493  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:06.901114  1723 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:06.907899  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:06.906811  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:06.916348  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48665.5us
I20260709 12:24:06.756245  6504 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:06.891099  6766 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:06.897825  6740 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.625s	user 0.015s	sys 0.000s
I20260709 12:24:06.930035  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:06.923743  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:06.920286  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:06.927603  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:06.928298  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:06.933756  1799 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:06.933849  2411 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:06.937806  2416 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:06.916198  6734 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:06.907874  6797 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:24:06.923017  6825 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:06.953853  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:06.923606  6659 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:06.920382  6822 raft_consensus.cc:2764] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
W20260709 12:24:06.934921  6860 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.250s	user 0.011s	sys 0.006s
I20260709 12:24:06.929585  6831 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:06.966316  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 169458us
I20260709 12:24:06.981606  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:06.928786  6713 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:06.949218  6865 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:06.958774  1874 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:06.955457  6797 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.974615  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:06.993518  1799 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:06.964604  6824 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.656s	user 0.012s	sys 0.002s
I20260709 12:24:06.951249  6734 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.926467  6768 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:24:06.973323  6659 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.999773  1807 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:07.000046  6645 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:07.007830  6645 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:07.008083  6832 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:07.001827  6871 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:06.990859  6860 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.009047  6645 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:07.010390  6645 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:07.011149  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 57393.3us
I20260709 12:24:07.012331  6860 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.013471  6877 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:06.981475  6213 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:06.992013  6713 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:06.995944  6921 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.133s	user 0.001s	sys 0.001s
I20260709 12:24:07.022186  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.027035  1807 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:06.995056  6797 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:07.001313  6734 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:07.006036  6659 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:07.003404  6748 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.006632  6865 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:07.045106  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.027755  6713 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:06.996291  6847 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:07.053085  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.037705  6883 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.514s	user 0.008s	sys 0.004s
I20260709 12:24:07.068665  1799 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
W20260709 12:24:07.070413  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78738.9us
W20260709 12:24:07.056416  6833 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.583s	user 0.010s	sys 0.001s
I20260709 12:24:07.077006  1802 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
W20260709 12:24:07.079960  1772 service_pool.cc:136] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33790 dropped due to backpressure: service queue is full with 50 items
I20260709 12:24:07.019567  6825 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.085253  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2945) took 1301 ms. Trace:
I20260709 12:24:07.085708  1873 rpcz_store.cc:276] 0709 12:24:05.783713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.783981 (+   268us) service_pool.cc:224] Handling call
0709 12:24:05.799653 (+ 15672us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:06.289246 (+489593us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:24:06.289287 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:06.319482 (+ 30195us) log.cc:844] Serialized 26 byte log entry
0709 12:24:06.477584 (+158102us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:24:06.477626 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.477699 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.085022 (+607323us) raft_consensus.cc:1685] finished
0709 12:24:07.085044 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.085188 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1535,"raft.queue_time_us":157469,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":15137,"thread_start_us":605575,"threads_started":2,"wal-append.queue_time_us":364821}
I20260709 12:24:07.086746  6865 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.087543  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:07.047528  6748 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.091373  6870 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:07.100418  6916 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.557s	user 0.000s	sys 0.002s
I20260709 12:24:07.103014  1730 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:07.080339  1772 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:33790 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 649)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 647)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 654)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2069)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2075)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2072)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2076)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2155)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2066)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2158)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2169)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2070)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2149)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 660)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2160)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2159)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2166)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2167)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2077)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2074)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2065)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 651)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2165)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2161)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2152)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2060)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2153)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 656)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2151)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 644)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2142)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 657)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2144)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2059)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2061)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2062)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2150)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2156)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2157)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2162)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2163)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2071)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2078)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2170)
I20260709 12:24:07.105759  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.107026  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:07.109589  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:07.109998  1730 raft_consensus.cc:2410] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:07.111096  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.111907  1807 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:07.116181  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.116802  1807 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:07.122069  1806 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:07.124043  6469 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.126888  6866 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.127557  6866 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.111955  6883 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:07.135133  6469 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:07.102676  6929 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.139530  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.140036  6929 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.142190  6847 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:07.140707  6930 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:07.143452  6930 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.145511  6837 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.149847  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.151320  6929 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:07.152192  6930 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.152360  6219 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:07.153048  1799 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:07.158399  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.138293  1807 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:07.159097  1805 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:07.159698  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.160296  1807 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:07.161156  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.161338  6609 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:07.161741  1807 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:07.153911  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.163857  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.150601  1881 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 8.
I20260709 12:24:07.166172  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.166965  6862 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:07.167402  6862 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:07.168239  6862 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.169554  6862 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:07.169974  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:07.170815  1728 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:07.173009  6837 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:07.173233  2419 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:24:05.753839  1732 rpcz_store.cc:276] 0709 12:24:04.645562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.676487 (+ 30925us) service_pool.cc:224] Handling call
0709 12:24:04.725135 (+ 48648us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:04.725353 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:04.824047 (+ 98694us) log.cc:844] Serialized 14 byte log entry
0709 12:24:05.717535 (+893488us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:05.717653 (+   118us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:05.721655 (+  4002us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.721747 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.721893 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":456376,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":71882,"thread_start_us":983907,"threads_started":3,"wal-append.queue_time_us":888759}
I20260709 12:24:07.175068  6628 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.175777  6628 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.176280  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:07.166926  1881 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:07.177731  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:07.178578  1732 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:24:07.191720  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:07.192587  1732 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:07.192634  6895 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:07.193327  6895 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:07.193825  1732 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:07.197687  6451 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:07.198227  6451 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:07.199287  6451 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.200688  1725 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:07.180614  6827 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:07.201785  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2952) took 1301 ms. Trace:
I20260709 12:24:07.202373  1880 rpcz_store.cc:276] 0709 12:24:05.901031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.907736 (+  6705us) service_pool.cc:224] Handling call
0709 12:24:05.908031 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.040623 (+132592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.201455 (+1160832us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b5400206cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:07.201702 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":291925888}
I20260709 12:24:07.204210  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.205796  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.150441  1772 service_pool.cc:137] RequestConsensusVote request on kudu.consensus.ConsensusService from 127.0.0.1:33790 dropped due to backpressure: service queue is full with 50 items Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2171)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2066)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2158)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2169)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2070)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2168)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2149)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 660)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2160)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2159)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2166)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2167)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2164)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2172)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2077)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2074)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2065)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2082)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 651)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2165)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2161)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2152)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2073)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2060)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2153)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 656)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2151)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 644)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2142)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 655)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 657)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2144)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2059)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2061)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2062)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2150)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2156)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2157)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2162)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2163)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2071)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2078)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2170)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2080)
I20260709 12:24:07.162674  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.207477  1807 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:07.207744  1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3097) took 2572 ms (client timeout 1824 ms). Trace:
I20260709 12:24:07.179512  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.219905  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1010) took 1011 ms. Trace:
I20260709 12:24:07.209353  1875 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:07.311219  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:07.254284  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.270239  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:07.312984  1730 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:07.208890  1801 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
W20260709 12:24:07.208159  1729 rpcz_store.cc:269] 0709 12:24:04.635686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.635795 (+   109us) service_pool.cc:224] Handling call
0709 12:24:07.207490 (+2571695us) spinlock_profiling.cc:243] Waited 1.73 s on lock 0x7b540044aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:07.207641 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3971978624}
W20260709 12:24:07.316134  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46024us
W20260709 12:24:07.262001  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 271871us
I20260709 12:24:07.208359  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.213615  1728 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:07.206590  1880 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:07.310431  2413 rpcz_store.cc:276] 0709 12:24:06.208219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.210701 (+  2482us) service_pool.cc:224] Handling call
0709 12:24:06.211066 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.333331 (+122265us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.219446 (+886115us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54005f28d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:07.219800 (+   354us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277651200}
I20260709 12:24:07.322914  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:07.326169  1881 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:07.327224  1805 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:07.164405  1799 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
W20260709 12:24:07.313311  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.334234  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.321483  6898 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:24:07.336427  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2934) took 2693 ms (client timeout 1788 ms). Trace:
W20260709 12:24:07.336859  1877 rpcz_store.cc:269] 0709 12:24:04.629306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.629416 (+   110us) service_pool.cc:224] Handling call
0709 12:24:07.321939 (+2692523us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b540021acd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:07.322158 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1205074176}
I20260709 12:24:07.340268  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.341403  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.342085  1805 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:07.343585  1880 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:07.200692  6451 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:07.358570  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.194408  6895 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.364650  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.336150  6219 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:07.350291  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.260706  6797 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.356231  1727 pending_rounds.cc:85] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 6
I20260709 12:24:07.376610  1727 pending_rounds.cc:107] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:24:07.376951  1727 raft_consensus.cc:2889] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:07.348243  1729 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:24:07.359576  1807 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:07.288247  6821 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:07.365536  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:07.380481  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.380951  1730 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:07.381670  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:07.349967  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.158679  6713 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.268415  6651 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.336321  6898 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:07.391180  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:07.368642  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.367691  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:07.372213  6219 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.395078  6219 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:07.401654  1799 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:07.406630  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.409616  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.388602  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.411113  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.411716  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.411871  6854 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.412381  1881 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:07.392554  1877 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:07.414636  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.415323  6934 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:07.379223  1805 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:07.411921  1875 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:07.415863  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:07.416404  6934 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:07.417117  1877 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:07.413496  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.434091  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.434415  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.435076  1881 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:07.435494  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.436007  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:07.436283  1873 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:07.436743  1881 raft_consensus.cc:2368] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12.
I20260709 12:24:07.417629  6934 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.437136  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.437750  1873 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:07.430485  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.418197  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.418653  6898 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:07.419104  1729 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:07.419533  6862 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:07.419978  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.421136  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:07.393671  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.446019  1807 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:07.421430  6797 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:07.394124  6825 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:07.424328  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.425387  6626 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:24:07.418123  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:07.427747  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.428526  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.476150  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 645) took 1171 ms. Trace:
I20260709 12:24:07.491875  1802 rpcz_store.cc:276] 0709 12:24:06.304749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.617192 (+312443us) service_pool.cc:224] Handling call
0709 12:24:07.468363 (+851171us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400066654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:07.476072 (+  7709us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":939090688}
I20260709 12:24:07.492986  1877 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:07.493398  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.389864  6847 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:07.442967  2413 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:07.443755  1873 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:07.447979  1875 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:07.448601  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:07.503192  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54666.4us
I20260709 12:24:07.452536  6870 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:07.449889  6219 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.455827  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:07.508994  1881 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 15 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:24:07.464628  1874 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:07.512193  1874 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:07.470062  1879 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:07.513833  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.514530  6825 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:07.485713  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.497394  6626 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:07.517264  6626 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.497905  1880 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:24:07.519249  1876 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:07.520418  6866 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.521014  1876 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:24:07.429389  6713 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.510229  6642 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:07.522260  6642 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:07.522385  1876 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:07.522713  6642 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:07.514508  1879 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:24:07.520606  1880 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:07.527658  1874 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:07.525008  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.515146  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:07.532728  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:07.534435  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:07.536141  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:07.538591  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:07.450521  6880 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:07.518651  6626 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:24:07.512926  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.542908  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1097) took 1510 ms. Trace:
I20260709 12:24:07.528486  1800 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:07.550433  2419 rpcz_store.cc:276] 0709 12:24:06.032390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.032575 (+   185us) service_pool.cc:224] Handling call
0709 12:24:07.542669 (+1510094us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400591ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:07.542843 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2547383168}
I20260709 12:24:07.475287  6768 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:07.544456  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:07.546020  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:07.390172  6821 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.546329  1873 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:07.547506  1802 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:07.567234  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.552172  1807 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:07.569173  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.558884  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:07.440233  6651 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.561892  1877 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:24:07.518201  6748 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:07.547187  6713 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:24:07.562655  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.564669  1881 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:07.568087  1802 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:07.567531  1731 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:07.554577  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:07.582612  2410 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:07.524972  6219 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.540063  6847 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:07.574262  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:07.543347  6504 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:07.539641  6659 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.577262  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 2950) took 1783 ms. Trace:
W20260709 12:24:07.593921  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 682) took 1980 ms (client timeout 1501 ms). Trace:
W20260709 12:24:07.594729  1726 rpcz_store.cc:269] 0709 12:24:05.613395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.624794 (+ 11399us) service_pool.cc:224] Handling call
0709 12:24:07.591662 (+1966868us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b54000d1bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:07.593815 (+  2153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1771357056}
I20260709 12:24:07.594192  1878 rpcz_store.cc:276] 0709 12:24:05.795535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:05.805442 (+  9907us) service_pool.cc:224] Handling call
0709 12:24:05.893699 (+ 88257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:06.158327 (+264628us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:06.158378 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:06.158442 (+    64us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:06.158504 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.576909 (+1418405us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54002df5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:07.576982 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.577177 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":607817472}
I20260709 12:24:07.581086  6883 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:07.555846  6854 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:24:07.583143  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.584332  1730 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:24:07.588546  2412 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
I20260709 12:24:07.572082  6837 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:07.476927  6862 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.583264  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.584908  6713 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:07.565172  6933 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
W20260709 12:24:07.617086  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 223031us
I20260709 12:24:07.639015  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.621337  6713 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:07.646335  1805 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:07.649639  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.650177  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2065) took 1015 ms. Trace:
I20260709 12:24:07.650836  1807 rpcz_store.cc:276] 0709 12:24:06.635245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.606492 (+971247us) service_pool.cc:224] Handling call
0709 12:24:07.650069 (+ 43577us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.634248  6504 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:07.633586  6766 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:07.659875  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.621655  6883 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:07.682493  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:07.663359  6940 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:24:07.663515  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.675360  6847 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.661701  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.656749  6837 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:07.601397  6825 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.682710  6768 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:07.694362  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 652) took 1387 ms (client timeout 1578 ms). Trace:
I20260709 12:24:07.682940  6504 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:07.695171  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 651) took 1388 ms. Trace:
W20260709 12:24:07.702176  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.697834  1804 rpcz_store.cc:269] 0709 12:24:06.306654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.326376 (+ 19722us) service_pool.cc:224] Handling call
0709 12:24:07.694131 (+1367755us) spinlock_profiling.cc:243] Waited 34.2 ms on lock 0x7b5400166cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:07.694287 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":78546688}
I20260709 12:24:07.706107  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.707017  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1104) took 1198 ms. Trace:
I20260709 12:24:07.706046  1802 rpcz_store.cc:276] 0709 12:24:06.306378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.681520 (+1375142us) service_pool.cc:224] Handling call
0709 12:24:07.695092 (+ 13572us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.707706  2415 rpcz_store.cc:276] 0709 12:24:06.509075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.509219 (+   144us) service_pool.cc:224] Handling call
0709 12:24:07.706732 (+1197513us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400572e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:07.706926 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":584879872}
I20260709 12:24:07.713487  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:07.711410  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.715402  1803 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260709 12:24:07.727329  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.728127  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:07.695262  6734 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.657s	user 0.012s	sys 0.002s
I20260709 12:24:07.729157  1807 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:07.740973  1800 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:24:07.684635  6941 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:24:07.745260  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:07.746716  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2060) took 1221 ms. Trace:
I20260709 12:24:07.747166  1804 rpcz_store.cc:276] 0709 12:24:06.525378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.726421 (+1201043us) service_pool.cc:224] Handling call
0709 12:24:07.746609 (+ 20188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.758599  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.718959  6809 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:07.769200  1807 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:07.732841  6651 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:07.773545  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.775034  1804 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:07.776044  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 656) took 1455 ms. Trace:
I20260709 12:24:07.776587  1804 rpcz_store.cc:276] 0709 12:24:06.320128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.749835 (+1429707us) service_pool.cc:224] Handling call
0709 12:24:07.775957 (+ 26122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.781316  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:07.782557  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2151) took 1068 ms. Trace:
I20260709 12:24:07.784235  1803 rpcz_store.cc:276] 0709 12:24:06.714230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.763589 (+1049359us) service_pool.cc:224] Handling call
0709 12:24:07.782464 (+ 18875us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.786414  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3104) took 1150 ms. Trace:
I20260709 12:24:07.790935  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:07.787206  1732 rpcz_store.cc:276] 0709 12:24:06.655869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.194723 (+538854us) service_pool.cc:224] Handling call
0709 12:24:07.195139 (+   416us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:07.195555 (+   416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.195584 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:07.211593 (+ 16009us) log.cc:844] Serialized 52 byte log entry
0709 12:24:07.532599 (+321006us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:07.532657 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.532774 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.786090 (+253316us) raft_consensus.cc:1685] finished
0709 12:24:07.786110 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.786325 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5358,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":129,"thread_start_us":195326,"threads_started":1,"wal-append.queue_time_us":235360}
I20260709 12:24:07.797204  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:07.801013  1800 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:24:07.809505  2410 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:07.810220  1804 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:07.744303  6734 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.824880  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:07.758842  6945 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:24:07.812680  1803 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 7. Current term is 8. Ops: []
I20260709 12:24:07.814210  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 657) took 1471 ms. Trace:
W20260709 12:24:07.815438  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.819334  6870 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:07.821498  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:07.823228  1807 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:07.830536  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:07.827529  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:07.798141  6954 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
W20260709 12:24:07.833180  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.859125  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 302454us
I20260709 12:24:07.859581  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 655) took 1539 ms. Trace:
W20260709 12:24:07.859803  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.859923  1803 rpcz_store.cc:276] 0709 12:24:06.319787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.787178 (+1467391us) service_pool.cc:224] Handling call
0709 12:24:07.812402 (+ 25224us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.859302 (+ 46900us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.859521 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.861770  1803 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:07.862469  1800 rpcz_store.cc:276] 0709 12:24:06.337422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.800271 (+1462849us) service_pool.cc:224] Handling call
0709 12:24:07.800643 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.808829 (+  8186us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.809093 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.864797  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:07.866108  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:07.804054  6664 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:07.835233  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:07.811162  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2142) took 1492 ms. Trace:
I20260709 12:24:07.870893  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:07.871709  1804 rpcz_store.cc:276] 0709 12:24:06.319021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.784961 (+1465940us) service_pool.cc:224] Handling call
0709 12:24:07.810021 (+ 25060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.810738 (+   717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.811085 (+   347us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.835969  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:07.828893  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 193647us
I20260709 12:24:07.839210  6872 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:07.834331  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:07.866552  1799 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:07.775312  6809 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.865020  6219 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:07.885627  6472 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:07.885466  6219 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.886258  6472 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:07.886494  6809 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.887748  6219 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.887398  6472 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.892983  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:07.893663  2414 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 8.
I20260709 12:24:07.890544  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3105) took 1234 ms. Trace:
I20260709 12:24:07.772619  6768 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.861843  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 644) took 1557 ms. Trace:
I20260709 12:24:07.895722  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:07.895982  1807 rpcz_store.cc:276] 0709 12:24:06.304457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.780159 (+1475702us) service_pool.cc:224] Handling call
0709 12:24:07.822744 (+ 42585us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.861385 (+ 38641us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.861758 (+   373us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.896466  2414 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:07.897614  1807 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:07.798552  6862 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.875131  2418 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:07.898275  6909 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 12:24:07.900071  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:07.900923  2418 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:07.902355  2418 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:24:07.761894  6883 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.876261  6866 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:24:07.794319  6837 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.902978  6766 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:07.903123  6866 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.903751  6866 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:07.905562  6895 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:07.876937  6865 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:07.877907  6819 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:07.878425  6748 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:07.880437  6628 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:07.794112  6504 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:07.881166  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2257) took 1089 ms. Trace:
I20260709 12:24:07.882264  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2149) took 1245 ms. Trace:
I20260709 12:24:07.882925  6879 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:07.883394  6872 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:07.884793  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:07.934393  2415 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:07.893806  2412 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:07.836110  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:07.877043  6942 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:24:07.906481  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2144) took 1494 ms. Trace:
I20260709 12:24:07.940335  1803 rpcz_store.cc:276] 0709 12:24:06.368209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.861329 (+1493120us) service_pool.cc:224] Handling call
0709 12:24:07.861614 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.862167 (+   553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.862344 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.942229  1803 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:07.906848  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.919786  6819 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.868798  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:07.951378  2419 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:07.894721  1725 rpcz_store.cc:276] 0709 12:24:06.656384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.202213 (+545829us) service_pool.cc:224] Handling call
0709 12:24:07.202567 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.210065 (+  7498us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.210108 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:07.212347 (+  2239us) log.cc:844] Serialized 26 byte log entry
0709 12:24:07.485256 (+272909us) spinlock_profiling.cc:243] Waited 47.8 ms on lock 0x7b1800038200. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:07.696493 (+211237us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:24:07.696536 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.696611 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.890173 (+193562us) raft_consensus.cc:1685] finished
0709 12:24:07.890199 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.890470 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7990,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":124,"spinlock_wait_cycles":109886720,"thread_start_us":211056,"threads_started":1,"wal-append.queue_time_us":615491}
I20260709 12:24:07.827867  6651 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:07.916308  1800 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:07.955503  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2059) took 1439 ms. Trace:
I20260709 12:24:07.955977  1800 rpcz_store.cc:276] 0709 12:24:06.515864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.863220 (+1347356us) service_pool.cc:224] Handling call
0709 12:24:07.916057 (+ 52837us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.955145 (+ 39088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.955427 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.803534  6797 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.072s
I20260709 12:24:07.863969  6821 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:07.863883  6870 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:07.920668  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49899.9us
I20260709 12:24:07.921504  1804 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:24:07.988117  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2061) took 1454 ms. Trace:
I20260709 12:24:07.870973  6664 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:07.877925  6877 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:07.903584  6766 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:07.928701  1805 rpcz_store.cc:276] 0709 12:24:06.637185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.427254 (+790069us) service_pool.cc:224] Handling call
0709 12:24:07.882073 (+454819us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b540013a6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:07.882204 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":352469376}
I20260709 12:24:07.954178  6819 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:07.928458  1875 rpcz_store.cc:276] 0709 12:24:06.791579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.501202 (+709623us) service_pool.cc:224] Handling call
0709 12:24:07.501665 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.502124 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.502176 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:07.524722 (+ 22546us) log.cc:844] Serialized 26 byte log entry
0709 12:24:07.695920 (+171198us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:07.695961 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.696075 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:07.880958 (+184883us) raft_consensus.cc:1685] finished
0709 12:24:07.880980 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.881109 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25707,"raft.queue_time_us":3816,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":5421,"thread_start_us":170729,"threads_started":1,"wal-append.queue_time_us":351432}
I20260709 12:24:07.935910  2410 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:07.935011  2414 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:24:07.898360  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2062) took 1355 ms. Trace:
I20260709 12:24:07.921269  6933 consensus_queue.cc:1243] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:07.948596  1798 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:07.923132  6903 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:07.952857  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2150) took 1273 ms. Trace:
I20260709 12:24:08.050081  1803 rpcz_store.cc:276] 0709 12:24:06.679641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.941508 (+1261867us) service_pool.cc:224] Handling call
0709 12:24:07.941978 (+   470us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.952412 (+ 10434us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.952744 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:07.954177  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2033) took 3431 ms (client timeout 2058 ms). Trace:
I20260709 12:24:07.935858  6713 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:07.977237  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.979959  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.988720  1804 rpcz_store.cc:276] 0709 12:24:06.534155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.872673 (+1338518us) service_pool.cc:224] Handling call
0709 12:24:07.921255 (+ 48582us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.987631 (+ 66376us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.988022 (+   391us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:08.003511  6664 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:07.989082  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:08.067656  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 78679.2us
I20260709 12:24:08.069209  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:08.001046  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:08.001853  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2168) took 1011 ms. Trace:
I20260709 12:24:07.954663  6862 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:07.949733  6879 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:08.010347  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:07.928050  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 679) took 1191 ms. Trace:
I20260709 12:24:08.033294  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:08.091109  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3107) took 1434 ms. Trace:
I20260709 12:24:08.035605  1807 rpcz_store.cc:276] 0709 12:24:06.543270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.897005 (+1353735us) service_pool.cc:224] Handling call
0709 12:24:07.897414 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.898062 (+   648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.898273 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:07.936340  6854 consensus_queue.cc:1243] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:24:07.992336  6870 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:08.050972  1806 rpcz_store.cc:269] 0709 12:24:04.517048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.563166 (+ 46118us) service_pool.cc:224] Handling call
0709 12:24:07.947752 (+3384586us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b54001680d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:07.947913 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5203848576}
I20260709 12:24:07.931110  6219 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
W20260709 12:24:08.061700  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.129142  1806 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:08.061395  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:08.134825  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2162) took 1316 ms. Trace:
I20260709 12:24:08.135375  1806 rpcz_store.cc:276] 0709 12:24:06.817912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.128385 (+1310473us) service_pool.cc:224] Handling call
0709 12:24:08.128883 (+   498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.134398 (+  5515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.134720 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:08.136039  1803 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
W20260709 12:24:08.048254  6642 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.525s	user 0.008s	sys 0.006s
I20260709 12:24:08.091383  1802 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:08.005321  6877 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:08.078701  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 186358us
I20260709 12:24:08.090986  1872 rpcz_store.cc:276] 0709 12:24:06.730754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.445224 (+714470us) service_pool.cc:224] Handling call
0709 12:24:07.445556 (+   332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.445788 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:07.598079 (+152291us) log.cc:844] Serialized 14 byte log entry
0709 12:24:07.921694 (+323615us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:07.921751 (+    57us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:07.921807 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.921880 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:07.922021 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":75677,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":97,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":5734,"thread_start_us":475047,"threads_started":2,"wal-append.queue_time_us":387305}
I20260709 12:24:08.090870  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:08.144048  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:08.144630  1805 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:08.074726  1772 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:08.053207  6865 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
W20260709 12:24:08.126914  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 116715us
I20260709 12:24:08.150966  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.124275  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:08.070744  1799 rpcz_store.cc:276] 0709 12:24:06.990730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.402593 (+411863us) service_pool.cc:224] Handling call
0709 12:24:08.001524 (+598931us) spinlock_profiling.cc:243] Waited 459 ms on lock 0x7b54001d4a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:08.001748 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1055366400}
I20260709 12:24:08.071882  6432 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:08.141296  1807 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: [10.7-12.8]
W20260709 12:24:08.145653  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 71022.5us
I20260709 12:24:08.145531  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:08.091651  1728 rpcz_store.cc:276] 0709 12:24:06.657008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.318972 (+661964us) service_pool.cc:224] Handling call
0709 12:24:07.319463 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.340131 (+ 20668us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.340169 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:07.364817 (+ 24648us) log.cc:844] Serialized 26 byte log entry
0709 12:24:07.618159 (+253342us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:07.618220 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.618340 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.090797 (+472457us) raft_consensus.cc:1685] finished
0709 12:24:08.090821 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.091021 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15850,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":16373,"thread_start_us":231495,"threads_started":1,"wal-append.queue_time_us":353150}
I20260709 12:24:08.146690  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2163) took 1328 ms. Trace:
I20260709 12:24:08.157644  1803 rpcz_store.cc:276] 0709 12:24:06.818187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.135350 (+1317163us) service_pool.cc:224] Handling call
0709 12:24:08.135828 (+   478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.146282 (+ 10454us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.146602 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:08.072319  6832 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:08.147753  1802 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:24:08.059787  6933 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.082s
W20260709 12:24:08.164554  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 288765us
I20260709 12:24:08.153313  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.167570  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:08.169385  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2071) took 1261 ms. Trace:
I20260709 12:24:08.169868  1802 rpcz_store.cc:276] 0709 12:24:06.908433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.139989 (+1231556us) service_pool.cc:224] Handling call
0709 12:24:08.140561 (+   572us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.168997 (+ 28436us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.169229 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:08.065503  6651 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.160566  1725 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:24:08.165148  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2157) took 1348 ms. Trace:
I20260709 12:24:08.174562  1807 rpcz_store.cc:276] 0709 12:24:06.816549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.092170 (+1275621us) service_pool.cc:224] Handling call
0709 12:24:08.140874 (+ 48704us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:08.161970 (+ 21096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.165073 (+  3103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:08.174897  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:08.091526  6880 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:08.076705  6862 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:08.156868  6837 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:08.173285  2383 leader_election.cc:400] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12.
I20260709 12:24:08.209071  2383 leader_election.cc:403] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:24:08.159322  6827 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:08.123003  6890 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:08.192859  1805 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:08.215813  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2078) took 1208 ms. Trace:
I20260709 12:24:08.216279  1805 rpcz_store.cc:276] 0709 12:24:07.019128 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.156571 (+1137443us) service_pool.cc:224] Handling call
0709 12:24:08.192633 (+ 36062us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.215440 (+ 22807us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.215733 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:08.163586  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 303377us
I20260709 12:24:08.202749  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.073601  6953 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.089s
I20260709 12:24:08.222995  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.175587  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2056) took 2142 ms. Trace:
I20260709 12:24:08.224622  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.224809  1798 rpcz_store.cc:276] 0709 12:24:06.020169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:06.020370 (+   201us) service_pool.cc:224] Handling call
0709 12:24:07.948132 (+1927762us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.046823 (+ 98691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.160986 (+114163us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b54001680d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:08.161266 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4386189184}
I20260709 12:24:08.209942  1800 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 12:24:08.226662  1800 pending_rounds.cc:85] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 8
I20260709 12:24:08.227047  1800 pending_rounds.cc:107] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:24:08.227367  1800 raft_consensus.cc:2889] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:08.228008  1800 pending_rounds.cc:107] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:24:08.228685  1800 raft_consensus.cc:2889] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:08.213792  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:08.221557  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.207960  6945 consensus_queue.cc:1243] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:08.156360  6962 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:08.217348  1806 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:08.218863  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:08.161692  6821 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:08.156834  6432 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:08.214406  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3114) took 1552 ms. Trace:
I20260709 12:24:08.236467  1732 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:08.142668  6483 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:24:08.249203  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.250047  1726 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:08.192013  6628 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:08.172591  6734 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:08.175283  6809 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:08.253782  1807 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
W20260709 12:24:08.220441  6898 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.744s	user 0.005s	sys 0.009s
I20260709 12:24:08.149724  6642 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:08.303493  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.262562  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.304908  1730 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
W20260709 12:24:08.251768  6955 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.332s	user 0.001s	sys 0.001s
I20260709 12:24:08.208739  6837 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:08.262082  6968 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:08.162813  6934 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:08.268635  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2170) took 1238 ms. Trace:
I20260709 12:24:08.322420  1806 rpcz_store.cc:276] 0709 12:24:07.029846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.174035 (+1144189us) service_pool.cc:224] Handling call
0709 12:24:08.217072 (+ 43037us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.268207 (+ 51135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.268551 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:08.173667  6832 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.272324  6860 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 12:24:08.249419  6880 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:08.291064  1726 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:08.291702  2414 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
W20260709 12:24:08.263231  6956 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260709 12:24:08.229986  6883 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12.
I20260709 12:24:08.254057  6890 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:08.205152  6651 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.266862  6909 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:08.308214  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89534.9us
W20260709 12:24:08.309137  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 175420us
I20260709 12:24:08.283448  6219 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 12:24:08.230926  6856 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:24:08.272107  6962 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:08.292182  6872 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:08.319679  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.325142  6483 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:08.329903  6472 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.328900  1728 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:08.300853  6827 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:08.272912  6877 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:08.333143  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 343260us
I20260709 12:24:08.315864  6837 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:08.343421  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 168619us
I20260709 12:24:08.367086  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:08.345546  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 187553us
I20260709 12:24:08.347416  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:08.351965  6909 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.349932  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:08.322933  6974 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:08.260617  6879 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.354367  1732 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:08.290315  6628 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:08.343346  1726 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:08.320869  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70848.1us
I20260709 12:24:08.380463  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.308311  6898 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.333372  6880 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.290930  6862 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.290027  1724 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:08.390101  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.343609  6933 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:24:08.366034  1805 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 12. (term mismatch)
I20260709 12:24:08.317757  6968 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:08.369432  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:08.353164  6713 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:08.373747  1872 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:24:08.331959  6934 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.355908  6483 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:08.374450  6909 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:08.374414  6941 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.009s	sys 0.018s
I20260709 12:24:08.390445  1726 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:08.391057  1805 pending_rounds.cc:85] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 11
W20260709 12:24:08.377980  6961 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.229s	user 0.001s	sys 0.001s
I20260709 12:24:08.383461  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:08.395368  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.402689  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:08.354176  6432 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.375455  6974 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:08.367264  6827 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.425776  1729 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 8. Current term is 9. Ops: []
I20260709 12:24:08.443329  1805 pending_rounds.cc:107] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 14.12
I20260709 12:24:08.529037  1805 raft_consensus.cc:2889] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:08.451267  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.408676  1731 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
W20260709 12:24:08.375789  6957 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.244s	user 0.002s	sys 0.000s
I20260709 12:24:08.362607  6472 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:24:08.514851  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 143842us
I20260709 12:24:08.403931  6821 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.407269  6898 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:08.528206  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:08.539708  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:08.540136  1730 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:08.428833  6651 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:08.380780  6958 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 12:24:08.530180  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.530074  1732 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:08.389607  6880 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:08.401742  6980 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.401957  6734 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:08.523969  6933 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:08.404520  6973 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:08.381006  6964 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.143s
W20260709 12:24:08.523280  6983 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.103s	user 0.008s	sys 0.000s
I20260709 12:24:08.384749  6628 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:08.524516  6960 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.252s	user 0.004s	sys 0.000s
W20260709 12:24:08.524271  6971 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.258s	user 0.002s	sys 0.003s
I20260709 12:24:08.537367  2414 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:08.443955  6941 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.009s	sys 0.019s
I20260709 12:24:08.388020  6981 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:24:08.569725  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 219889us
I20260709 12:24:08.414156  6659 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:08.403122  6877 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:08.543179  1724 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:08.400270  6968 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:08.424427  6954 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:08.398386  6809 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.146s
I20260709 12:24:08.416271  6483 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.613969  6483 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:08.542693  6821 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.430907  6890 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.616649  6821 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:08.596235  6974 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.618189  6974 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:08.424347  6832 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:08.619275  6832 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:08.620857  6832 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:24:08.621426  6832 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:08.411696  6642 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:08.541618  6432 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:08.405019  6969 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:24:08.624918  6986 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.625593  6986 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.628096  6986 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:08.398921  6975 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.162s
I20260709 12:24:08.562399  1726 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:24:08.631361  1726 pending_rounds.cc:85] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 7
I20260709 12:24:08.631728  1726 pending_rounds.cc:107] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:24:08.632014  1726 raft_consensus.cc:2889] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:08.633764  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:08.634027  6819 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.580s	user 0.011s	sys 0.000s
I20260709 12:24:08.634714  1726 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:08.580804  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 207470us
I20260709 12:24:08.546892  6734 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.523542  6982 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.088s
I20260709 12:24:08.600628  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:08.612968  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 73357.3us
I20260709 12:24:08.391772  6883 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.140s
I20260709 12:24:08.569268  6898 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:08.400420  6962 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.617605  6933 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.648574  6962 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:08.649737  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.559898  2410 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:08.569429  6987 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.089s
I20260709 12:24:08.651247  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.651962  1878 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:08.526003  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.654654  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.655447  1881 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:08.622550  6832 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.637458  6941 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.606756  1730 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:08.636323  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3109) took 1978 ms. Trace:
I20260709 12:24:08.641953  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:08.661513  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.662508  1872 raft_consensus.cc:2410] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:24:08.663908  1798 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:24:08.660661  1727 rpcz_store.cc:276] 0709 12:24:06.657634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.330932 (+673298us) service_pool.cc:224] Handling call
0709 12:24:07.355992 (+ 25060us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:07.567340 (+211348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.567373 (+    33us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:07.579081 (+ 11708us) log.cc:844] Serialized 78 byte log entry
0709 12:24:07.810678 (+231597us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:24:07.810717 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:07.810808 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.635992 (+825184us) raft_consensus.cc:1685] finished
0709 12:24:08.636027 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.636253 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15086,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":143,"thread_start_us":379108,"threads_started":2,"wal-append.queue_time_us":407850}
I20260709 12:24:08.597173  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:24:08.645433  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:08.665349  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.665241  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:08.647396  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.643230  6898 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:08.668668  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.650558  1877 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:08.657037  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.666472  1880 raft_consensus.cc:2376] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16. Re-sending same reply.
W20260709 12:24:08.670755  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 136338us
W20260709 12:24:08.636252  6958 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.084s	user 0.008s	sys 0.004s
I20260709 12:24:08.612332  6954 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:08.672991  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.704941  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:08.672660  1771 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.788s (SENT)
I20260709 12:24:08.675629  6941 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.685470  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.686012  1807 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:08.686599  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:08.688675  6870 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.000s
I20260709 12:24:08.689713  6976 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:08.689920  6987 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:08.691810  6898 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.612108  6934 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:08.694169  6472 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:08.695427  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:08.598585  6768 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.100s
I20260709 12:24:08.714040  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:08.715106  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 118094us
I20260709 12:24:08.719473  1881 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:08.741792  6933 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:08.749197  6768 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:08.749634  6768 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:08.751116  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 78486.7us
I20260709 12:24:08.753918  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:08.756536  1877 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
W20260709 12:24:08.758136  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53390us
I20260709 12:24:08.733201  6859 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.759891  1875 raft_consensus.cc:2410] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:08.760461  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.761461  1727 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:08.764503  6768 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.767683  6954 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.769449  6954 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:08.774572  6933 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:24:08.776733  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.776324  6909 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.779716  6651 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:24:08.780844  6664 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.689s	user 0.013s	sys 0.001s
W20260709 12:24:08.781390  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.781330  6664 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.727802  1878 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:08.645736  6734 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:08.788384  6941 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:08.738304  6992 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:08.789307  6992 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.789737  6992 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:08.740271  6987 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:08.551054  6980 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.791195  6987 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.792884  6987 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:08.751719  6976 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.793058  6980 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:08.795379  6976 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:08.797230  1872 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:08.734607  1880 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: []
I20260709 12:24:08.737541  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1107) took 1206 ms. Trace:
W20260709 12:24:08.669567  6866 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.765s	user 0.011s	sys 0.003s
I20260709 12:24:08.643034  6827 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:08.634613  6819 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.734753  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.803253  1726 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:08.798056  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:08.800546  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.803814  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.734059  6972 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:08.808105  1880 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:08.798389  2413 rpcz_store.cc:276] 0709 12:24:07.531022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.545614 (+ 14592us) service_pool.cc:224] Handling call
0709 12:24:07.567543 (+ 21929us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:07.567938 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:07.567983 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:07.584533 (+ 16550us) log.cc:844] Serialized 26 byte log entry
0709 12:24:08.002958 (+418425us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:08.003016 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.003108 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:08.737234 (+734126us) raft_consensus.cc:1685] finished
0709 12:24:08.737263 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.737448 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":560,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":9388,"thread_start_us":368127,"threads_started":1,"wal-append.queue_time_us":562332}
I20260709 12:24:08.809916  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:08.811698  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:08.809773  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2080) took 1660 ms. Trace:
I20260709 12:24:08.813133  1802 rpcz_store.cc:276] 0709 12:24:07.149008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.496815 (+1347807us) service_pool.cc:224] Handling call
0709 12:24:08.497388 (+   573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.497708 (+   320us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:08.662107 (+164399us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.809361 (+147254us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:08.809423 (+    62us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:08.809471 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.809560 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.809706 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":177112,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":86,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"thread_start_us":303553,"threads_started":2,"wal-append.queue_time_us":151407}
W20260709 12:24:08.817377  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.817646  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.819470  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:08.738933  6870 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.827986  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:08.821835  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.828984  1877 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:08.821120  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.830196  1726 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 15 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:24:08.828035  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.831043  1880 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:08.821755  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:08.760484  6981 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:08.835269  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.838022  2417 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:24:08.820175  6766 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.770s	user 0.013s	sys 0.001s
I20260709 12:24:08.808602  6941 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:08.808555  6996 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:08.810582  6827 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:08.846228  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.848937  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:08.786672  6880 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.825548  6432 raft_consensus.cc:2764] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:24:08.849480  1726 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:08.847121  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.849728  2413 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:08.849972  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.852836  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.856771  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:08.805658  6866 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:08.858811  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.856588  1880 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:08.833371  6472 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:08.805656  6859 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.838932  6989 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:08.816699  6972 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.869447  1806 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:08.867043  6819 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:08.844352  6974 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:08.878901  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.853791  6996 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.884655  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:08.851146  6909 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:08.892928  1726 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:08.894446  1726 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:24:08.840664  6986 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
W20260709 12:24:08.895949  6769 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 12:24:08.846485  6766 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:08.896596  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.897475  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.868165  6933 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:08.910754  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:08.844646  6628 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:08.915889  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68905.3us
I20260709 12:24:08.921353  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.922959  6941 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:08.923467  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:08.924612  6941 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:08.925123  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.925283  6941 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:08.852928  6837 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.896850  6996 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:08.926002  6999 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:08.909543  6974 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:08.955682  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.957546  1874 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:08.943069  6628 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:08.905531  6432 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:08.937260  6982 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.947526  1881 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:08.909335  6986 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.955996  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.974257  1879 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:08.948663  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.949561  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.950294  6880 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:08.975984  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 119292us
I20260709 12:24:08.955646  1873 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:08.958580  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.957791  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:08.979336  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:08.936139  1877 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:24:08.897225  6980 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.057s
I20260709 12:24:08.984653  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.986917  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.987049  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:08.988294  1874 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:08.988816  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:08.989497  2413 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:24:08.990338  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:08.991137  1881 raft_consensus.cc:2376] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14. Re-sending same reply.
I20260709 12:24:08.991918  2417 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:08.992339  1880 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:08.973274  6866 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:08.995235  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:08.974459  6664 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:08.999997  1877 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:24:08.999788  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.976541  6984 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:09.003463  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.977831  6993 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:08.986491  6769 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:08.988235  6933 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:09.019151  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:08.994230  6880 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.973385  6968 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:09.027545  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 209140us
W20260709 12:24:09.001012  6994 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
W20260709 12:24:09.005626  7001 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
W20260709 12:24:09.032505  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.038648  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.031963  6993 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.984521  6986 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:09.039600  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.938887  6981 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.045344  2413 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:09.037333  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:09.008666  6934 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:09.062938  6837 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.030290  6895 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:08.983886  6982 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.038342  6933 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.042016  6870 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:09.051110  6945 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:09.051389  6974 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.056046  6993 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.057854  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:09.009753  6984 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:09.060984  1772 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:09.005976  7004 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.030020  6769 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.066970  1876 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:09.050711  6986 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.030573  6999 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:09.047796  6880 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.249522  2413 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
W20260709 12:24:09.255194  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48564.5us
I20260709 12:24:09.205724  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:09.205559  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:09.219974  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:09.266085  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:09.071587  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:09.262267  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 201413us
W20260709 12:24:09.267467  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 195965us
I20260709 12:24:09.195815  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2155) took 1036 ms. Trace:
I20260709 12:24:09.179198  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.268826  1728 rpcz_store.cc:276] 0709 12:24:08.159182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.159303 (+   121us) service_pool.cc:224] Handling call
0709 12:24:09.195580 (+1036277us) spinlock_profiling.cc:243] Waited 111 ms on lock 0x7b54004362d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:09.195729 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":254405504}
I20260709 12:24:09.220026  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:09.077277  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74052.8us
I20260709 12:24:09.198704  1879 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:09.267619  2382 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:09.171021  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.309379  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.268427  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:09.183559  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.312134  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.269326  1730 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:09.312419  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44879.5us
I20260709 12:24:09.146090  6968 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:09.313110  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44795.3us
I20260709 12:24:09.313782  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:09.224305  6962 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.575s	user 0.014s	sys 0.000s
I20260709 12:24:09.316778  6986 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:09.316891  6954 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:09.317281  6986 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.317689  6986 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.317433  6954 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.319657  6954 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.320809  2415 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:09.321851  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:09.274466  6769 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:09.323153  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:09.324750  2418 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:09.270331  6875 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:09.325791  6875 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:09.159978  6734 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.138933  6972 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:09.066568  6997 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:24:09.272100  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2156) took 2455 ms. Trace:
W20260709 12:24:09.273877  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 229465us
I20260709 12:24:09.333607  1800 rpcz_store.cc:276] 0709 12:24:06.816232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.080694 (+1264462us) service_pool.cc:224] Handling call
0709 12:24:08.142225 (+ 61531us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.270138 (+127913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.271844 (+1001706us) spinlock_profiling.cc:243] Waited 67.2 ms on lock 0x7b54001eb9d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:09.272040 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":154625408}
I20260709 12:24:09.308386  2411 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:09.196099  6982 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.315066  6837 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:09.315593  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.311292  6748 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.066344  6941 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:09.326758  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.327593  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:09.141194  6766 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.249259  6626 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.275123  6987 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:09.331516  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:09.079347  6933 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:09.069397  6827 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:09.121282  6859 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.336894  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.374109  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:09.340574  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.069236  7005 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:24:09.273213  6984 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.275707  6659 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:09.348644  6959 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:09.348325  6968 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:09.349092  6797 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:09.275656  6996 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.349843  6963 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:09.275790  7004 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.350559  6748 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.351821  6993 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:24:09.353896  6879 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:24:09.282965  6945 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:09.354715  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.359274  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:09.364058  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.364739  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.370347  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:09.150410  6992 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.331843  6815 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:09.374511  1728 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:09.382651  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.385362  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.381632  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.385499  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.428052  1877 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:09.392017  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:09.392930  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:09.428876  1802 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:09.430225  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.333079  6642 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:24:09.338420  6870 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:24:09.393463  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:09.432590  1800 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:09.413882  6832 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.397539  1878 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:09.343588  6954 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:24:09.387248  6959 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:09.416751  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.418493  1874 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:09.420509  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 61458.7us
I20260709 12:24:09.334228  6875 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.394033  6963 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:09.450223  6875 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:09.424919  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54658.9us
I20260709 12:24:09.455816  1881 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:09.456542  6992 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.366573  6626 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.434168  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.459739  1800 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:09.352082  6989 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:09.374821  7002 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:09.463474  7002 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.463872  7002 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.435348  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.436362  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.431730  6879 consensus_queue.cc:1243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:09.446100  6880 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.446256  6734 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.446576  7004 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.434051  6859 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:09.446954  6996 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:09.462109  6769 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.449368  1807 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:24:09.487671  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:09.450286  7005 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:09.394198  6659 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:09.469885  1800 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:09.465975  1873 raft_consensus.cc:2376] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17. Re-sending same reply.
I20260709 12:24:09.422639  6766 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:09.476502  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 129060us
I20260709 12:24:09.477644  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:09.478843  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.480460  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:09.483026  1728 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:09.447674  6959 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.488651  2419 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:09.496387  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:09.496671  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:09.476527  6642 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:09.501562  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:09.501617  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.504393  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.505559  1800 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.506266  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:09.507059  2416 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.466888  6832 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.487859  6859 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:09.509759  6859 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:09.511230  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.485466  6713 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:09.501363  6945 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.512516  6945 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.512949  6626 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:09.504856  6642 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:09.514204  6642 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.511896  1806 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:09.515632  6959 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:09.502638  6975 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:09.498711  6954 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.517935  6954 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.491186  7005 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:09.542454  7005 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.543429  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:09.543879  7005 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:09.552023  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50533.5us
I20260709 12:24:09.553241  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:09.555797  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:09.545159  7006 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:09.557674  7006 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.558068  7006 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:09.559957  6832 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:09.550045  6642 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:09.561769  6978 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.160s	user 0.002s	sys 0.002s
I20260709 12:24:09.563277  6815 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:24:09.564167  6664 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:09.563156  6989 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.003s
I20260709 12:24:09.476614  6827 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.502295  6837 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.567866  6837 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.502205  6879 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:09.570262  6879 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.570393  6837 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.570756  6879 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:09.571663  7016 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:09.572736  6945 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.573833  6769 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.550923  6968 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:09.576301  2419 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:24:09.576964  6659 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.547794  1772 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:09.583165  6748 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.573801  1872 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:09.588125  6713 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:09.589003  6954 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.590655  2416 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:09.592147  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.592293  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:09.594300  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:09.563156  1800 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:09.414587  6999 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:09.611088  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63429.9us
I20260709 12:24:09.586352  6933 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.614401  2412 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:09.615281  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:09.620549  1806 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:09.620927  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.621089  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.621542  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.621587  1873 raft_consensus.cc:2376] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15. Re-sending same reply.
I20260709 12:24:09.622972  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.623528  1873 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.626464  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:09.627461  2412 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:09.600498  6832 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:09.616632  7004 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:09.628243  6933 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.628750  7004 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.616798  6713 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:09.629268  7004 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:09.616479  6993 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:24:09.616998  6989 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:09.635251  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:09.635896  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 195620us
I20260709 12:24:09.636605  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.637290  1732 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:09.637784  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:09.638191  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.638988  6933 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:09.640878  7004 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.642220  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.642905  1724 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:09.643836  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.644788  1724 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:09.618640  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:09.649333  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:24:09.652112  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.652601  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:09.654275  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:09.655223  6989 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:09.655876  6989 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:09.656821  6989 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.657867  6986 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.659191  6664 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:09.660257  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.659719  6933 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260709 12:24:09.650460  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:09.655390  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:09.663442  6993 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:09.664003  6993 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:09.665190  6769 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:09.666026  6945 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:09.667098  6766 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:09.667176  6945 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:09.668053  6766 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:09.668428  6766 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:09.668471  6859 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.669430  6766 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.672003  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.672816  1800 raft_consensus.cc:2368] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16.
I20260709 12:24:09.672935  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:09.673604  2411 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.674875  6748 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:09.675683  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.664820  6993 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.609740  6968 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.679108  6968 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:09.681429  6768 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.683961  6986 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.685017  6880 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:09.685482  6880 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.685865  6880 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:09.686368  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.687219  1806 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:09.687088  6821 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:09.687578  6821 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:09.688192  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.688511  6821 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.685633  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.690586  1800 raft_consensus.cc:2376] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 15. Re-sending same reply.
W20260709 12:24:09.627461  7014 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
I20260709 12:24:09.692333  1879 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:24:09.692719  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:09.711757  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.704241  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:09.705927  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.653169  1732 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:09.713249  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:09.702239  2411 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:09.695693  2382 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:09.698650  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:09.716485  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:09.718709  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:09.719085  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.719581  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.719960  1725 raft_consensus.cc:2376] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14. Re-sending same reply.
I20260709 12:24:09.723165  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.717713  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.728298  1881 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:24:09.726675  6626 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:09.732899  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:09.720301  6821 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:09.670768  6766 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:09.648603  6832 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:09.737031  6832 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.739998  6859 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.743017  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.713879  6748 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:09.746899  6748 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.747995  7004 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:09.733721  1802 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:09.749922  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.750305  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:09.751322  2411 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:09.752933  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54382.7us
I20260709 12:24:09.756069  1879 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 8: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:09.742055  7002 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.761643  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.764816  1728 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:09.781549  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.762511  1879 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:09.766907  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.775992  2411 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:09.783478  1881 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:09.749012  6872 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:09.785699  1772 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:09.791038  2419 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:09.794922  1803 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:24:09.796955  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.797714  1879 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:09.758549  6986 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:24:09.812625  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.772123  6945 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.840610  6945 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:08.283815  1723 rpcz_store.cc:276] 0709 12:24:06.662372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.497897 (+835525us) service_pool.cc:224] Handling call
0709 12:24:07.502513 (+  4616us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:07.502770 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:24:07.777974 (+275204us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.213708 (+435734us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:08.213846 (+   138us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:08.213966 (+   120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.214108 (+   142us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:08.214322 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23061,"prepare.queue_time_us":291578,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":1808,"raft.queue_time_us":68205,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":20230,"thread_start_us":658887,"threads_started":2}
I20260709 12:24:09.844110  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.845114  1723 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:09.805150  6856 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:09.817308  1732 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:09.847441  6856 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:09.849304  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2161) took 1670 ms (client timeout 2213 ms). Trace:
I20260709 12:24:09.849813  6856 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:09.851102  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3113) took 3192 ms. Trace:
I20260709 12:24:09.851524  1729 rpcz_store.cc:276] 0709 12:24:06.662056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.444789 (+782733us) service_pool.cc:224] Handling call
0709 12:24:07.447797 (+  3008us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.528621 (+1080824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.850732 (+1322111us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b54002b7fd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:09.851007 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2248618752}
I20260709 12:24:09.852733  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.853895  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.854444  1729 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:09.855017  6769 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:09.855501  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:09.856262  1802 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:09.856482  6879 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:09.857285  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.857862  1802 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:09.859546  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:09.860448  6827 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:09.860980  6827 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.861341  6827 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:09.865314  6968 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:09.866400  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.817641  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:09.820289  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 662) took 1876 ms. Trace:
I20260709 12:24:09.821208  6981 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:09.787918  6642 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.800940  6859 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:09.823832  6987 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:09.798966  6713 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:09.832655  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.886647  1874 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:09.833959  1879 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:09.835799  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:09.836786  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2967) took 1168 ms. Trace:
I20260709 12:24:09.889379  1875 rpcz_store.cc:276] 0709 12:24:08.668154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.668324 (+   170us) service_pool.cc:224] Handling call
0709 12:24:09.836534 (+1168210us) spinlock_profiling.cc:243] Waited 90 ms on lock 0x7b5400260a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:09.836692 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":206924544}
I20260709 12:24:09.866709  2419 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:09.870584  1881 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:09.868175  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.849830  1731 rpcz_store.cc:269] 0709 12:24:08.178937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.179090 (+   153us) service_pool.cc:224] Handling call
0709 12:24:09.848934 (+1669844us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b54000d0cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:09.849091 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":410560896}
I20260709 12:24:09.868673  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:09.869683  1805 rpcz_store.cc:276] 0709 12:24:07.943848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.217296 (+273448us) service_pool.cc:224] Handling call
0709 12:24:08.256683 (+ 39387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.536944 (+280261us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.819902 (+1282958us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b5400066654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:09.820201 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":239390592}
I20260709 12:24:09.846508  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.875913  1873 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:09.871799  6981 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:09.876729  6642 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:09.879009  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.909759  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:09.885143  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 972) took 1688 ms. Trace:
I20260709 12:24:09.910396  1724 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
W20260709 12:24:09.898077  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 248345us
I20260709 12:24:09.903872  1879 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:09.888736  1877 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:09.906265  1874 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: [15.7-15.7]
W20260709 12:24:09.915446  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.910574  2410 rpcz_store.cc:276] 0709 12:24:08.196962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.197121 (+   159us) service_pool.cc:224] Handling call
0709 12:24:08.197434 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.651085 (+453651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.884848 (+1233763us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400489754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:09.885075 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":832803072}
I20260709 12:24:09.918135  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.923681  1731 raft_consensus.cc:2368] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12.
W20260709 12:24:09.924254  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.924921  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.928747  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.929724  1725 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
W20260709 12:24:09.931020  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.803807  6984 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:09.893220  6898 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:09.873183  6986 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:09.932767  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.880577  6879 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:09.902967  6856 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.946751  1800 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:09.960163  1807 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:24:09.962695  1725 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:09.904649  6626 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:09.976230  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:09.977105  1805 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:09.980919  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.982935  1725 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:09.990723  1807 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:24:09.993386  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:09.911584  6996 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:09.994002  2382 leader_election.cc:400] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12.
I20260709 12:24:09.994436  2382 leader_election.cc:403] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:09.912232  6859 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:09.998077  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:09.998893  1731 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:09.935621  6974 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:09.977036  6986 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:10.021968  1725 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:10.031210  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:10.030673  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:10.027768  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.923573  6959 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.086s
I20260709 12:24:10.031450  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:10.033420  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:10.041590  1723 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 12: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:10.043087  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:10.000608  6996 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:09.944002  6987 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.083s
I20260709 12:24:10.051592  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:10.025318  1805 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:10.049934  1731 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 7.
I20260709 12:24:10.054354  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2183) took 1001 ms. Trace:
I20260709 12:24:10.055119  1723 rpcz_store.cc:276] 0709 12:24:09.052634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.042674 (+990040us) service_pool.cc:224] Handling call
0709 12:24:10.054285 (+ 11611us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.054307  1725 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.013921  6879 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.009375  6974 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:10.081302  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 281310us
I20260709 12:24:10.081449  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:10.082624  1731 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:24:09.992894  6856 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.095582  6862 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:10.100696  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.060578  6945 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:10.111507  2416 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 12:24:10.078282  6866 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:10.125130  2416 pending_rounds.cc:85] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 8
I20260709 12:24:10.125723  2416 pending_rounds.cc:107] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:24:10.080791  6968 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:10.126024  2416 raft_consensus.cc:2889] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:10.129472  1727 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:24:10.105846  6933 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.140420  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:10.028360  6981 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.077672  6879 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:10.133716  7022 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.189s	user 0.000s	sys 0.005s
W20260709 12:24:10.148716  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 239031us
I20260709 12:24:10.150491  1729 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:10.132947  6821 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12.
I20260709 12:24:10.126364  6866 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.166550  1725 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:10.147513  6945 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:24:10.153757  6875 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.703s	user 0.010s	sys 0.003s
I20260709 12:24:10.150348  6768 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:10.180339  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:10.182135  1873 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:24:10.114603  6974 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.185537  1802 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:10.104744  6859 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.188627  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:10.153605  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:10.192597  1725 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:10.150734  6933 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.198310  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:10.199182  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 205326us
I20260709 12:24:10.183527  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:10.202907  1877 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:10.204461  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:10.217679  1873 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:10.216848  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:10.207969  1731 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:24:10.171784  6713 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:10.211388  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2171) took 2029 ms. Trace:
I20260709 12:24:10.160775  6981 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:10.212307  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58798us
W20260709 12:24:10.213459  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.213855  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.220580  1725 rpcz_store.cc:276] 0709 12:24:08.181452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.192007 (+2010555us) service_pool.cc:224] Handling call
0709 12:24:10.192363 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.211050 (+ 18687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.211308 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.147859  6996 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.223412  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:10.189049  6825 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:24:10.224502  6941 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:24:10.225818  1802 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:24:10.222210  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:10.237648  2410 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:10.202239  6768 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:10.227526  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:10.227599  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:10.243376  1800 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:10.228513  1805 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: [15.7-15.7]
I20260709 12:24:10.196575  6642 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:10.235507  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.238998  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:10.253510  1877 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:10.193851  6866 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:10.210665  6984 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:10.240626  6626 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:10.247512  2410 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:10.247923  6825 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:24:10.249729  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:10.218067  6875 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:10.254968  1807 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:10.256748  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:10.257139  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2179) took 1347 ms (client timeout 1684 ms). Trace:
W20260709 12:24:10.261852  1726 rpcz_store.cc:269] 0709 12:24:08.910028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.910564 (+   536us) service_pool.cc:224] Handling call
0709 12:24:10.256918 (+1346354us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b54003cd7d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:10.257074 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1175259392}
I20260709 12:24:10.263118  1726 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:10.263674  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 688) took 2081 ms. Trace:
I20260709 12:24:10.263949  1726 rpcz_store.cc:276] 0709 12:24:08.182361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.262616 (+2080255us) service_pool.cc:224] Handling call
0709 12:24:10.262977 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.263486 (+   509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.263630 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.265277  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3131) took 2041 ms. Trace:
I20260709 12:24:10.265604  1726 rpcz_store.cc:276] 0709 12:24:08.223682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.264689 (+2041007us) service_pool.cc:224] Handling call
0709 12:24:10.264914 (+   225us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.265062 (+   148us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.265077 (+    15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.265107 (+    30us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:10.265137 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.265170 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.265237 (+    67us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.266206  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:10.187965  2383 connection.cc:256] RPC call timeout handler was delayed by -1.339725972999986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
I20260709 12:24:10.280627  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:10.240477  6974 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:10.251451  6768 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:10.286512  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.260023  6875 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.261073  6996 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.275978  6941 consensus_queue.cc:1243] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 16.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:10.256192  6825 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:10.308221  1727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 700) took 1743 ms (client timeout 2091 ms). Trace:
W20260709 12:24:10.318972  1727 rpcz_store.cc:269] 0709 12:24:08.564835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.762488 (+197653us) service_pool.cc:224] Handling call
0709 12:24:10.308029 (+1545541us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b540035d754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:10.308164 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3028412160}
I20260709 12:24:10.309796  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:10.287919  7023 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.220s	user 0.000s	sys 0.002s
I20260709 12:24:10.315784  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.287662  6713 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:10.288707  6987 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:10.316640  6875 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:10.336016  1727 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:10.342139  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 694) took 2022 ms. Trace:
I20260709 12:24:10.342620  1727 rpcz_store.cc:276] 0709 12:24:08.319698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.320316 (+2000618us) service_pool.cc:224] Handling call
0709 12:24:10.335731 (+ 15415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.341790 (+  6059us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.342063 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:10.347774  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.350246  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:10.287847  6821 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
W20260709 12:24:10.356925  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 168340us
I20260709 12:24:10.361785  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:10.367244  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:10.331301  6856 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:10.328841  7036 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:10.376470  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:10.351442  6837 raft_consensus.cc:2764] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:24:10.365742  6832 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:10.323531  7043 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:24:10.324465  6626 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:10.387295  7028 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
W20260709 12:24:10.387805  7035 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:24:10.401928  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98321us
I20260709 12:24:10.388199  7002 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:10.375759  7025 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.221s	user 0.000s	sys 0.002s
I20260709 12:24:10.354338  6713 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:10.391213  1727 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:24:10.393426  1807 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:10.367627  7024 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.217s	user 0.002s	sys 0.000s
I20260709 12:24:10.416610  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 699) took 1878 ms. Trace:
I20260709 12:24:10.417272  1727 rpcz_store.cc:276] 0709 12:24:08.538372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.362611 (+1824239us) service_pool.cc:224] Handling call
0709 12:24:10.379817 (+ 17206us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.416228 (+ 36411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.416532 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:10.396158  1806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.005s	sys 0.014s
I20260709 12:24:10.371284  6825 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.379830  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:10.405225  1802 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:10.373517  7037 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.373730  6879 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:10.454005  1806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.011s	sys 0.019s
W20260709 12:24:10.455119  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 116339us
I20260709 12:24:10.461913  6968 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:10.462725  6968 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:10.463956  6968 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.469470  1802 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:10.476930  1802 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:24:10.474951  6968 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:10.478292  6968 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:10.478698  6968 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:10.466727  6993 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.000s	sys 0.012s
W20260709 12:24:10.488651  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 108908us
W20260709 12:24:10.492105  6993 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.000s	sys 0.015s
I20260709 12:24:10.493435  6821 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:10.499076  6832 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:10.501770  6832 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.499212  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:10.509364  6968 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:10.418810  7032 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
W20260709 12:24:10.516104  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.523695  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 116556us
W20260709 12:24:10.523152  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 219188us
I20260709 12:24:10.533422  1727 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:24:10.534416  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2175) took 1719 ms. Trace:
I20260709 12:24:10.542452  1727 rpcz_store.cc:276] 0709 12:24:08.814921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.451452 (+1636531us) service_pool.cc:224] Handling call
0709 12:24:10.533049 (+ 81597us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.534095 (+  1046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.534348 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:10.524364  7033 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.293s	user 0.001s	sys 0.000s
I20260709 12:24:10.489362  6933 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:10.554646  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.556720  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.555419  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 689) took 1191 ms. Trace:
I20260709 12:24:10.558840  2412 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:10.559278  2412 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:10.559772  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.558525  1872 rpcz_store.cc:276] 0709 12:24:09.363648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.363780 (+   132us) service_pool.cc:224] Handling call
0709 12:24:10.555214 (+1191434us) spinlock_profiling.cc:243] Waited 49.7 ms on lock 0x7b5400223654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:10.555351 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":114349952}
I20260709 12:24:10.562837  7057 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:24:10.563747  7057 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:10.564093  6968 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:10.565126  7057 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:10.565835  7057 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:24:10.566159  7057 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:10.567476  7057 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.568840  7057 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:10.576362  1807 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:10.560078  2412 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.576362  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2170) took 2395 ms. Trace:
I20260709 12:24:10.578009  1729 rpcz_store.cc:276] 0709 12:24:08.181240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.171440 (+1990200us) service_pool.cc:224] Handling call
0709 12:24:10.171791 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.172213 (+   422us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:10.334582 (+162369us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.576020 (+241438us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.576074 (+    54us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:10.576124 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.576185 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.576307 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":232497,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":1818,"raft.queue_time_us":122,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":105,"thread_start_us":385518,"threads_started":2}
I20260709 12:24:10.551301  6993 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.584522  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:10.578387  6748 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.526772  6879 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:24:10.589067  1727 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:10.590682  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:10.590919  2412 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.587493  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:10.593331  1807 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:10.597028  1802 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:10.514684  7037 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.609400  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:10.614326  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:10.627491  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2160) took 3810 ms (client timeout 2097 ms). Trace:
W20260709 12:24:10.628127  1801 rpcz_store.cc:269] 0709 12:24:06.817372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:07.492243 (+674871us) service_pool.cc:224] Handling call
0709 12:24:10.627268 (+3135025us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b540035f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:10.627404 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6091917568}
I20260709 12:24:10.587426  7046 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:24:10.627352  1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 706) took 1429 ms (client timeout 1627 ms). Trace:
W20260709 12:24:10.633479  1728 rpcz_store.cc:269] 0709 12:24:09.197955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.505540 (+307585us) service_pool.cc:224] Handling call
0709 12:24:10.627098 (+1121558us) spinlock_profiling.cc:243] Waited 77.6 ms on lock 0x7b54001905d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:10.627264 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":178385792}
I20260709 12:24:10.635460  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:10.636602  1728 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:24:10.598263  6748 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.598395  7071 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:24:10.594548  7005 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.050s	user 0.016s	sys 0.001s
I20260709 12:24:10.639026  7071 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:10.639460  7071 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:10.639562  7005 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:10.640129  7005 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.642282  7005 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:10.594244  6856 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.643139  6856 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:10.643024  6898 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.651s	user 0.007s	sys 0.007s
I20260709 12:24:10.645841  6856 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:10.630051  6999 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:24:10.647655  6999 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:10.648036  6999 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:10.648871  6999 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:10.682170  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.690328  7037 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:10.743916  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 349886us
I20260709 12:24:10.745520  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:10.750061  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.692070  1727 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
W20260709 12:24:10.694362  7050 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.205s	user 0.000s	sys 0.001s
W20260709 12:24:10.697232  6880 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.011s	user 0.007s	sys 0.004s
I20260709 12:24:10.698449  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2172) took 2516 ms. Trace:
I20260709 12:24:10.764564  1725 rpcz_store.cc:276] 0709 12:24:08.181751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.230215 (+2048464us) service_pool.cc:224] Handling call
0709 12:24:10.230573 (+   358us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:10.234474 (+  3901us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.234501 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:10.242650 (+  8149us) log.cc:844] Serialized 52 byte log entry
0709 12:24:10.391708 (+149058us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:10.391749 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.391816 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.698201 (+306385us) raft_consensus.cc:1685] finished
0709 12:24:10.698220 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.698385 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3403,"raft.queue_time_us":10802,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":9107,"thread_start_us":84366,"threads_started":1,"wal-append.queue_time_us":371583}
W20260709 12:24:10.705444  6766 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.970s	user 0.005s	sys 0.007s
W20260709 12:24:10.706288  6827 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.844s	user 0.010s	sys 0.001s
W20260709 12:24:10.742996  6981 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.512s	user 0.009s	sys 0.005s
I20260709 12:24:10.595286  6993 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.707998  6768 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:10.708518  6986 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.665s	user 0.007s	sys 0.005s
I20260709 12:24:10.656388  6713 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:10.709218  7041 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
W20260709 12:24:10.655354  7039 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
I20260709 12:24:10.650003  6999 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:10.710597  6996 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.712100  7036 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:10.712973  6903 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:10.713721  6865 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:10.714433  6875 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:10.718067  6945 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.520s	user 0.009s	sys 0.003s
I20260709 12:24:10.721451  6659 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:10.722512  7046 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:10.723101  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:10.807853  1726 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:10.723716  7057 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:24:10.728466  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.733801  1696 leader_election.cc:400] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16.
W20260709 12:24:10.811668  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 178161us
I20260709 12:24:10.812186  1696 leader_election.cc:403] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:24:10.782596  6933 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:10.761024  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:10.713555  1629 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 4 to 5, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 5 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:24:10.768714  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 280257us
W20260709 12:24:10.706517  7006 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.148s	user 0.004s	sys 0.010s
W20260709 12:24:10.775004  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.783259  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:10.737612  7071 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.763226  7063 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:10.805615  6872 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.004s	user 0.011s	sys 0.006s
I20260709 12:24:10.787601  7064 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:10.781376  6959 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.738078  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:10.811789  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 88764.2us
I20260709 12:24:10.832861  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.834062  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.835319  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.763531  7053 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 12:24:10.810962  6977 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:10.815990  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:10.819346  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58444.7us
I20260709 12:24:10.848676  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:10.849080  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:10.772329  6880 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.777252  1730 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
W20260709 12:24:10.821960  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.824707  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 101059us
I20260709 12:24:10.825635  1872 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:10.826447  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:10.823127  6981 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.779284  6827 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.817983  1728 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:10.838027  1732 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:10.839968  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:10.850286  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.880316  1879 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:10.851063  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:10.795265  6986 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.811298  6945 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:24:10.850365  1877 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:10.887820  1729 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:24:10.887580  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 703) took 1690 ms. Trace:
I20260709 12:24:10.820091  6651 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:10.893702  6651 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:10.894006  1724 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:24:10.894865  6651 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:10.841975  6866 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.562s	user 0.014s	sys 0.000s
I20260709 12:24:10.896394  6880 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:10.896796  6866 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.876120  6981 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.899645  6866 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:10.900238  6981 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:10.901084  6981 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:10.896385  6651 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.902355  6981 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:10.876857  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2181) took 1923 ms. Trace:
I20260709 12:24:10.837136  6898 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16.
I20260709 12:24:10.877997  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:10.830742  6903 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:10.842190  6872 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:10.881639  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:10.882483  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:10.917999  1873 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:24:10.840992  7057 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:10.923416  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3140) took 1457 ms. Trace:
W20260709 12:24:10.911175  7039 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:24:10.836830  7058 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:24:10.828214  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94562.8us
I20260709 12:24:10.929018  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.931013  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.933909  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1032) took 1240 ms. Trace:
I20260709 12:24:10.934031  1801 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: [13.10-14.11]
I20260709 12:24:10.934581  2416 rpcz_store.cc:276] 0709 12:24:09.693178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.693989 (+   811us) service_pool.cc:224] Handling call
0709 12:24:09.751108 (+ 57119us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.126863 (+375755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.933531 (+806668us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b5400560c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:10.933749 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":815974656}
I20260709 12:24:10.888653  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3137) took 1423 ms. Trace:
I20260709 12:24:10.888464  1732 rpcz_store.cc:276] 0709 12:24:09.197084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.812611 (+1615527us) service_pool.cc:224] Handling call
0709 12:24:10.837641 (+ 25030us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.879779 (+ 42138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.887479 (+  7700us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:10.805466  6996 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.942742  6996 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:10.847851  6734 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:24:10.890844  7059 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:24:10.943490  1806 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:24:10.943992  6827 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:10.944603  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:10.945255  7047 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.369s	user 0.002s	sys 0.002s
I20260709 12:24:10.877461  7055 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20260709 12:24:10.946936  7075 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:10.947702  7075 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:10.948660  7075 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.950022  7075 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:10.925861  7046 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:10.952091  7065 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:24:10.954032  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2102) took 1115 ms. Trace:
I20260709 12:24:10.925458  7044 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:10.955291  7044 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.955935  7044 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:10.956543  6875 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:10.954473  1803 rpcz_store.cc:276] 0709 12:24:09.838411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.878733 (+ 40322us) service_pool.cc:224] Handling call
0709 12:24:09.879211 (+   478us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:09.894145 (+ 14934us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.894195 (+    50us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:09.920204 (+ 26009us) log.cc:844] Serialized 78 byte log entry
0709 12:24:10.487767 (+567563us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:10.863818 (+376051us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.864240 (+   422us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.870452 (+  6212us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.953735 (+ 83283us) raft_consensus.cc:1685] finished
0709 12:24:10.953760 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.953950 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":432482,"prepare.queue_time_us":61498,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":56957,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":427832,"thread_start_us":825397,"threads_started":2,"wal-append.queue_time_us":820215}
I20260709 12:24:10.890937  6768 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:10.892915  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:10.969223  1881 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:10.912147  6626 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:10.912341  6981 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.893198  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:10.970930  6981 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:10.971402  1877 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:10.972507  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 691) took 2684 ms. Trace:
I20260709 12:24:10.823949  6865 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:10.850050  7064 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:10.905856  7053 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:10.972546  6993 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:10.973170  6865 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:10.907166  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:10.973644  6865 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:10.974290  7064 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:10.970945  1881 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:24:10.978013  7058 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:10.849279  7036 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:10.852078  7042 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:10.847254  6977 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:10.856896  6856 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.823714  7006 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.916007  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:10.916525  1872 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:24:10.829530  6933 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.992018  7006 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:10.993077  6933 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:10.992941  1880 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:10.993381  6977 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:10.840610  7063 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.994608  6977 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:10.996366  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2169) took 2815 ms. Trace:
I20260709 12:24:10.997015  1731 rpcz_store.cc:276] 0709 12:24:08.181054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.114271 (+1933217us) service_pool.cc:224] Handling call
0709 12:24:10.139684 (+ 25413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.219211 (+ 79527us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.995905 (+776694us) spinlock_profiling.cc:243] Waited 67.8 ms on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:10.996293 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":155899776}
I20260709 12:24:10.998260  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:10.864861  7060 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:10.921144  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.000638  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:11.000640  7060 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:11.004446  7030 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.753s	user 0.001s	sys 0.001s
I20260709 12:24:11.004554  7060 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:10.923513  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:11.007087  6976 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:11.007611  6976 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.007982  6976 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:11.009568  7063 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.009487  7040 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:10.887557  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2285) took 1178 ms. Trace:
I20260709 12:24:10.986292  6766 raft_consensus.cc:2764] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election decision vote started in defunct term 15: won
I20260709 12:24:11.012125  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:11.011060  1876 rpcz_store.cc:276] 0709 12:24:09.709474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.887229 (+177755us) service_pool.cc:224] Handling call
0709 12:24:09.887690 (+   461us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:09.903435 (+ 15745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.903486 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:09.904306 (+   820us) log.cc:844] Serialized 52 byte log entry
0709 12:24:10.189683 (+285377us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:10.405435 (+215752us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.478525 (+ 73090us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.478752 (+   227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.479230 (+   478us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.887296 (+408066us) raft_consensus.cc:1685] finished
0709 12:24:10.887319 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.887482 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49356,"prepare.queue_time_us":161455,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":2530,"raft.queue_time_us":255779,"raft.run_cpu_time_us":630,"raft.run_wall_time_us":36936,"thread_start_us":472730,"threads_started":3,"wal-append.queue_time_us":408575}
I20260709 12:24:11.012882  1731 raft_consensus.cc:2376] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14. Re-sending same reply.
I20260709 12:24:10.848219  6959 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.924379  1728 rpcz_store.cc:276] 0709 12:24:09.466059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.883226 (+1417167us) service_pool.cc:224] Handling call
0709 12:24:10.916844 (+ 33618us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.922925 (+  6081us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:10.922979 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.923076 (+    97us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:10.923122 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.923205 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.923347 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.015592  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:11.016266  1728 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:10.922284  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:10.932714  6898 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:24:10.945326  1732 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:10.912930  6903 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:10.891336  7071 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:10.892083  6986 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:10.978437  1881 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:10.972988  1726 rpcz_store.cc:276] 0709 12:24:08.288483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.266398 (+1977915us) service_pool.cc:224] Handling call
0709 12:24:10.266677 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.808364 (+541687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.972275 (+163911us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b54003cc3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:10.972445 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1238219648}
W20260709 12:24:10.980711  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49822.6us
I20260709 12:24:11.049290  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:10.989404  6856 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:10.974356  1879 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
W20260709 12:24:11.006569  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.909291  6974 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.620s	user 0.013s	sys 0.000s
I20260709 12:24:10.937667  1729 rpcz_store.cc:276] 0709 12:24:09.465190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.845060 (+1379870us) service_pool.cc:224] Handling call
0709 12:24:10.878208 (+ 33148us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.888352 (+ 10144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.888596 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.050065  1728 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
W20260709 12:24:11.021956  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 99780.2us
I20260709 12:24:11.026048  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:11.051445  1731 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
W20260709 12:24:11.023896  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.049302  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:11.004102  7021 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.141s	user 0.000s	sys 0.003s
I20260709 12:24:11.059216  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:11.063987  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 198898us
W20260709 12:24:11.062945  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.052006  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:11.076512  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:11.077246  1726 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:11.074204  2419 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:11.082787  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:11.083647  1729 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:11.043627  6626 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:11.080130  6989 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.235s	user 0.008s	sys 0.011s
W20260709 12:24:11.097128  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47918us
I20260709 12:24:11.101886  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3143) took 1634 ms. Trace:
I20260709 12:24:11.070135  6898 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:11.052778  7042 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:11.101883  6989 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.262s	user 0.009s	sys 0.012s
I20260709 12:24:11.020124  7009 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 12:24:11.102654  1728 rpcz_store.cc:276] 0709 12:24:09.466985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.083684 (+1616699us) service_pool.cc:224] Handling call
0709 12:24:11.100934 (+ 17250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.101508 (+   574us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.101533 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:11.101571 (+    38us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:11.101608 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.101716 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.101824 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.102344  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:11.107671  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.103173  1726 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:11.121030  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.121351  7088 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:11.123008  6974 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:11.081198  7043 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:11.119800  7042 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.129608  7055 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.130697  7055 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.134160  7055 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.100350  7061 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:24:11.135084  7043 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.123268  7088 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:11.140805  7042 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.141649  6766 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:11.143940  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.121601  1726 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:11.147670  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3145) took 1673 ms. Trace:
W20260709 12:24:11.108385  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.170312  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 67740.9us
I20260709 12:24:11.148550  1726 rpcz_store.cc:276] 0709 12:24:09.474509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.121110 (+1646601us) service_pool.cc:224] Handling call
0709 12:24:11.121437 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.147224 (+ 25787us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.147576 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.173580  1726 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:11.175069  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1123) took 2121 ms. Trace:
I20260709 12:24:11.175314  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:11.173846  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.182029  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:11.089742  6986 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.064s
I20260709 12:24:11.089301  6847 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.086s
I20260709 12:24:11.177872  2417 rpcz_store.cc:276] 0709 12:24:09.053763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.053942 (+   179us) service_pool.cc:224] Handling call
0709 12:24:09.054280 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:09.112986 (+ 58706us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:09.113041 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.116931 (+  3890us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.352436 (+235505us) spinlock_profiling.cc:243] Waited 2.03 ms on lock 0x7b180021e020. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:09.503076 (+150640us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:09.503123 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.640937 (+137814us) spinlock_profiling.cc:243] Waited 57.9 ms on lock 0x7b5400561b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:09.640993 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:11.003361 (+1362368us) spinlock_profiling.cc:243] Waited 47.6 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:11.174769 (+171408us) raft_consensus.cc:1685] finished
0709 12:24:11.174840 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.175003 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1769,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":1940,"spinlock_wait_cycles":247326080,"thread_start_us":150399,"threads_started":1,"wal-append.queue_time_us":652126}
I20260709 12:24:11.187270  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.189441  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3146) took 1704 ms. Trace:
I20260709 12:24:11.147486  6959 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.191911  1726 rpcz_store.cc:276] 0709 12:24:09.474825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.172981 (+1698156us) service_pool.cc:224] Handling call
0709 12:24:11.173370 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.177827 (+  4457us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.179105 (+  1278us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.192836  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:11.149750  7028 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:24:11.116441  6626 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.194015  1879 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:24:11.209718  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:11.174515  6898 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:11.197114  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:11.196985  1726 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:11.234861  1771 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.590s (SENT) [suppressed 35 similar messages]
I20260709 12:24:11.200963  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:11.203392  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.148320  7043 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:11.119383  6865 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.203771  6768 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:11.184069  7068 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.237344  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3149) took 1743 ms. Trace:
I20260709 12:24:11.237782  1726 rpcz_store.cc:276] 0709 12:24:09.494399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.196033 (+1701634us) service_pool.cc:224] Handling call
0709 12:24:11.196792 (+   759us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.235236 (+ 38444us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.237281 (+  2045us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.185384  6766 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:11.185451  7051 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.232542  1881 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:11.189932  7082 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.190055  6933 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:11.204563  6945 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:11.208280  6875 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:24:11.265367  7068 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.233434  6626 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.269182  7068 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.246608  6748 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.273479  6974 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:24:11.251037  7055 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:11.252453  7042 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:11.275096  7042 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:11.230661  1877 raft_consensus.cc:2376] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15. Re-sending same reply.
W20260709 12:24:11.276437  7074 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.379s	user 0.000s	sys 0.002s
W20260709 12:24:11.247373  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46514.6us
I20260709 12:24:11.278308  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.109560  6903 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.248809  6768 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.276134  7042 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.262403  1732 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:11.097193  7058 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.9, Last appended by leader: 9, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.280133  1726 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 9. Current term is 10. Ops: []
I20260709 12:24:11.289959  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:11.291347  1729 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:11.294648  1732 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 12:24:11.294234  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.292646  6768 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.304075  7090 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:24:11.306128  7089 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:11.306998  6999 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:11.307539  7042 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:11.308096  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:11.309048  1723 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:11.310308  6825 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:24:11.310827  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.265599  7043 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:24:11.295146  1732 pending_rounds.cc:85] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 10
I20260709 12:24:11.313223  1732 pending_rounds.cc:107] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 13.11
W20260709 12:24:11.312951  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.313583  1732 raft_consensus.cc:2889] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:11.315150  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 709) took 1675 ms. Trace:
I20260709 12:24:11.315603  1732 rpcz_store.cc:276] 0709 12:24:09.640055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.291151 (+1651096us) service_pool.cc:224] Handling call
0709 12:24:11.294320 (+  3169us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.314757 (+ 20437us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.315070 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.280062  6959 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:11.296868  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:11.295279  7068 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.026s
I20260709 12:24:11.294953  6903 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:11.313792  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3154) took 1446 ms. Trace:
I20260709 12:24:11.328336  1723 rpcz_store.cc:276] 0709 12:24:09.867270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.309907 (+1442637us) service_pool.cc:224] Handling call
0709 12:24:11.310325 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.313356 (+  3031us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:24:11.313394 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:11.313442 (+    48us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:11.313483 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.313572 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.313720 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2638,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":2534}
W20260709 12:24:11.331528  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.306572  6768 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.329488  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:11.316781  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.266152  7051 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.327628  7075 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:11.335887  7064 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:11.336434  7064 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:11.336992  7051 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:11.337328  7064 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:11.299067  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.324445  6977 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:11.341985  1732 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 14. (index mismatch)
I20260709 12:24:11.324851  6959 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.342878  1723 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:11.343214  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 712) took 1123 ms. Trace:
I20260709 12:24:11.343569  1732 rpcz_store.cc:276] 0709 12:24:10.220020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.324237 (+1104217us) service_pool.cc:224] Handling call
0709 12:24:11.341548 (+ 17311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.342889 (+  1341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.343148 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.317777  6626 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.344818  1732 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
W20260709 12:24:11.348678  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.348760  1732 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:11.238629  7044 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:11.350823  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 153773us
I20260709 12:24:11.352489  6768 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:11.353641  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.356096  1732 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:11.356849  1732 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:11.357645  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.908252  1725 rpcz_store.cc:276] 0709 12:24:08.953484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.767122 (+1813638us) service_pool.cc:224] Handling call
0709 12:24:10.767600 (+   478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.767965 (+   365us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:10.783267 (+ 15302us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.876470 (+ 93203us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.876522 (+    52us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:10.876572 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.876641 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.876788 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":20077,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":63,"raft.queue_time_us":177,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":28835,"thread_start_us":92589,"threads_started":1}
I20260709 12:24:11.347182  6977 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:11.346566  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.367213  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.357774  1732 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 15.6, Last appended by leader: 6, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.368999  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.368999  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:11.359714  6768 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:11.360972  7091 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:24:11.347460  6959 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:11.361773  6976 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.363330  1877 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:24:11.318809  6981 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.370476  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:11.371723  1879 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:11.359773  6865 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:11.403271  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.369755  1798 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:11.334510  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:11.406080  1725 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:11.407420  1875 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:11.376948  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:11.408324  1803 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:11.408732  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.379484  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:11.271330  7082 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.381335  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:11.373566  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:11.375399  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:11.415212  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.410691  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 76273.2us
I20260709 12:24:11.417672  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:11.407589  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.413327  1805 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:11.334014  7068 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:24:11.421725  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:11.414197  1881 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:11.415580  1876 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:11.425280  1876 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
I20260709 12:24:11.374369  7075 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.433493  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:11.434103  1881 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:11.434968  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.429225  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:11.371783  7042 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:11.376314  6815 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:11.440752  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:11.431291  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:11.374630  6977 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:11.430737  1725 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:11.374519  7044 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.437516  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:11.450174  1803 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:11.374086  6933 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.056s
I20260709 12:24:11.410619  6879 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:11.427136  6748 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.427423  6987 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:11.421041  6862 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:24:11.431506  7068 consensus_queue.cc:1243] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:24:11.456096  6879 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.453177  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:11.455466  1881 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:11.432559  7092 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:11.437913  6768 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:11.477276  1732 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.483218  1875 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:11.461980  6875 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:11.502612  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49529.9us
I20260709 12:24:11.452960  6815 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.438421  6996 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:24:11.457834  6993 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:11.450994  7058 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:11.445029  6974 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 12:24:11.444826  6945 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260709 12:24:11.444424  7036 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:24:11.440083  7075 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.466895  6987 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.476413  6981 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:11.504555  6875 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.523865  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:11.503988  6768 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:11.506639  6898 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:11.522228  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.525767  6875 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:11.529076  1732 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 6. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:24:11.521050  6664 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:11.531383  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:11.533090  6664 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:11.526503  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89070.5us
I20260709 12:24:11.533646  6815 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.534075  6664 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.535620  6664 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:11.536181  7094 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:24:11.521440  6993 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:11.537166  7073 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:11.537539  6993 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.537761  7068 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:11.522583  7058 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:11.540735  7058 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.543936  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2180) took 2627 ms. Trace:
I20260709 12:24:11.545320  7075 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:11.529277  7029 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.395s	user 0.002s	sys 0.004s
I20260709 12:24:11.546959  7082 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:11.547982  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2105) took 1708 ms. Trace:
W20260709 12:24:11.549116  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.527400  6898 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.550822  6898 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.552564  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 692) took 3255 ms. Trace:
I20260709 12:24:11.520773  6865 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:11.553449  6734 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.554207  6734 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:11.556494  6734 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:24:11.529932  7036 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:11.557968  7036 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:11.470407  6748 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.553057  1724 rpcz_store.cc:276] 0709 12:24:08.297090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.307231 (+2010141us) service_pool.cc:224] Handling call
0709 12:24:10.307564 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.894516 (+586952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.552221 (+657705us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b54000d0cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:11.552463 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1347665152}
I20260709 12:24:11.523536  6987 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:11.530851  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 299488us
I20260709 12:24:11.548583  1799 rpcz_store.cc:276] 0709 12:24:09.839466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.864002 (+ 24536us) service_pool.cc:224] Handling call
0709 12:24:09.864394 (+   392us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:09.935365 (+ 70971us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.935404 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:09.961007 (+ 25603us) log.cc:844] Serialized 52 byte log entry
0709 12:24:10.231577 (+270570us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:10.231626 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.231692 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:11.544816 (+1313124us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x7b18003c5020. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:11.547550 (+  2734us) raft_consensus.cc:1685] finished
0709 12:24:11.547598 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.547907 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":476881,"raft.queue_time_us":78736,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":406193,"spinlock_wait_cycles":8840704,"thread_start_us":195286,"threads_started":1,"wal-append.queue_time_us":355764}
I20260709 12:24:11.544400  1727 rpcz_store.cc:276] 0709 12:24:08.916869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.752847 (+1835978us) service_pool.cc:224] Handling call
0709 12:24:10.753566 (+   719us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.759910 (+  6344us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:24:10.759979 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:10.760975 (+   996us) log.cc:844] Serialized 26 byte log entry
0709 12:24:11.308681 (+547706us) spinlock_profiling.cc:243] Waited 6.55 ms on lock 0x7b18004860a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:11.379547 (+ 70866us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:11.379615 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.379723 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:11.542663 (+162940us) raft_consensus.cc:1685] finished
0709 12:24:11.542695 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.543862 (+  1167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10268,"raft.queue_time_us":5744,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":8468,"spinlock_wait_cycles":15055232,"thread_start_us":70420,"threads_started":1,"wal-append.queue_time_us":605267}
I20260709 12:24:11.566426  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.526746  6974 consensus_queue.cc:1243] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:11.526157  6768 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:11.581542  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.582441  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.583099  1798 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:24:11.583803  1803 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 16 index: 8. (index mismatch)
I20260709 12:24:11.583879  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:11.585003  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.586853  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.587031  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:11.587914  1725 raft_consensus.cc:2376] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12. Re-sending same reply.
I20260709 12:24:11.592546  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:11.594753  7097 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:24:11.569792  6865 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.599346  1807 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:24:11.617504  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70540.1us
I20260709 12:24:11.576385  6890 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:11.619879  6890 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.621747  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:11.622330  6890 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.622488  1803 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:11.596851  6974 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:11.624939  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 2997) took 1974 ms (client timeout 1590 ms). Trace:
I20260709 12:24:11.624843  6974 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:11.625478  1880 rpcz_store.cc:269] 0709 12:24:09.650173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.650909 (+   736us) service_pool.cc:224] Handling call
0709 12:24:11.624689 (+1973780us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54000f0854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:11.624841 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3062890240}
I20260709 12:24:11.626543  7073 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.627164  6974 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.627830  6974 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:11.628192  6974 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.628537  6963 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.7, Last appended by leader: 7, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.627173  7073 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.630177  6963 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.603984  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:11.631232  1801 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:11.631400  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:11.632198  1727 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:11.590224  7068 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:24:11.596447  6832 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 12:24:11.604833  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:11.644610  1724 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:11.646899  1798 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:11.655090  1801 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:11.698159  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:11.864850  7090 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.008s	sys 0.011s
I20260709 12:24:11.788623  6987 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.606419  6748 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.609628  6875 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.667618  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.879611  1798 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
W20260709 12:24:11.797003  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.711128  6976 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:11.878422  7090 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.008s	sys 0.011s
I20260709 12:24:11.882045  6987 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:11.885386  7043 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:11.885893  7043 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:11.886917  7043 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:11.888146  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.894881  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 702) took 2698 ms (client timeout 1611 ms). Trace:
I20260709 12:24:11.896888  6977 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.898514  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3144) took 2424 ms. Trace:
I20260709 12:24:11.898988  1729 rpcz_store.cc:276] 0709 12:24:09.474162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.088485 (+1614323us) service_pool.cc:224] Handling call
0709 12:24:11.291117 (+202632us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.295065 (+  3948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.898267 (+603202us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b5400435dd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:11.898440 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":458982784}
I20260709 12:24:11.900295  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:11.901016  1729 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:11.902311  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:11.903544  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3150) took 2396 ms. Trace:
I20260709 12:24:11.904150  1726 rpcz_store.cc:276] 0709 12:24:09.506616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.238892 (+1732276us) service_pool.cc:224] Handling call
0709 12:24:11.265473 (+ 26581us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.293634 (+ 28161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.903161 (+609527us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b54002b70d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:11.903433 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33245056}
I20260709 12:24:11.905730  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:11.900485  6875 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.906424  1726 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.907639  6977 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.878827  7081 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:11.910219  7081 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:11.911161  7081 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.912160  7081 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:11.912564  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.917228  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:11.919996  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:11.920746  2419 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
W20260709 12:24:11.921200  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.921752  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.921940  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:11.922529  6875 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:11.923223  2417 raft_consensus.cc:2410] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:24:11.923408  6875 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.925338  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:11.925738  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:11.926736  6875 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.927673  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:11.929646  2419 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:11.930320  7092 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:24:11.895397  1730 rpcz_store.cc:269] 0709 12:24:09.196687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.315288 (+118601us) service_pool.cc:224] Handling call
0709 12:24:11.894582 (+2579294us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400327dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:11.894741 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3030560128}
I20260709 12:24:11.930771  7092 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:11.941891  7092 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.598207  6968 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.132s
I20260709 12:24:11.983186  7092 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:11.987259  1729 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:24:11.988711  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:11.989501  1729 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:11.990919  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:11.991549  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:11.966244  6898 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.000134  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:12.002099  2412 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:12.005028  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:12.007745  7088 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:12.007822  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:12.008749  7088 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.010226  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:12.011226  2412 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:12.011498  7088 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.992050  1729 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:12.014957  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:12.015740  1729 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:12.019485  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:12.023577  2416 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:11.684563  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:12.025698  1729 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
W20260709 12:24:11.671478  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.617113  6865 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:11.656482  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:12.034389  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:11.783852  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 252522us
I20260709 12:24:12.035986  1729 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:12.036415  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.794221  7089 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:11.736969  7103 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.662161  7057 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:11.700165  7073 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:11.699110  7044 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:11.849612  7070 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.012s	sys 0.005s
I20260709 12:24:11.586431  7093 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.109s
I20260709 12:24:11.846284  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.659029  6832 consensus_queue.cc:1243] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 15.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:24:11.628547  6974 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:12.051496  1627 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 256) took 28519 ms (client timeout 15000 ms). Trace:
W20260709 12:24:12.044872  7070 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.013s	sys 0.005s
I20260709 12:24:11.845448  6959 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.740767  1803 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:12.034436  6898 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.873507  7061 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:24:12.057480  7061 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:12.039342  6875 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:11.851330  7111 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:11.876220  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:11.877511  1802 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:12.059206  7089 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.059234  7111 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.060345  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:12.061051  1802 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
W20260709 12:24:11.873180  7060 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.868s	user 0.012s	sys 0.003s
I20260709 12:24:12.062060  7060 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.575397  7036 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:11.936932  6879 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.938694  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:11.948859  1727 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:11.959195  1874 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:24:11.938910  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:11.964994  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 665) took 4020 ms (client timeout 1869 ms). Trace:
W20260709 12:24:11.965376  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.966856  1629 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 3 to 9. New cstate: current_term: 9 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:11.967996  6768 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:11.968725  1807 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
W20260709 12:24:12.081741  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.969607  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47170.9us
W20260709 12:24:12.082866  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.083869  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:11.969975  7043 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:10.939312   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:11.981602  1723 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:11.707119  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:12.087474  1800 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:11.992039  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:12.025969  1725 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:11.660336  6837 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:12.089109  1724 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:12.033581  7107 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:12.034854  1732 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:11.666512  6856 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:11.705792  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:12.037340  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:12.041630  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:12.100773  2414 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:12.100944  1729 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:12.038424  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:12.042333  1846 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:12.047755  1726 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:12.051271  7093 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:11.849061  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:12.106096  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 257130us
W20260709 12:24:12.106734  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 64518.2us
I20260709 12:24:12.053426  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:12.108783  1803 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:12.110949  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:12.111593  1881 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:24:12.112138  1872 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:12.112481  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:12.054278  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:12.043401  6963 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.014s
I20260709 12:24:11.700238  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:12.062598  1802 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:12.074631  1801 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
W20260709 12:24:12.078598  1804 rpcz_store.cc:269] 0709 12:24:07.944663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.067150 (+122487us) service_pool.cc:224] Handling call
0709 12:24:11.964736 (+3897586us) spinlock_profiling.cc:243] Waited 3.38 s on lock 0x7b54001d5954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:11.964911 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7766328576}
I20260709 12:24:12.131695  1877 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:12.133430  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:12.062705  7111 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:12.135869  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:12.085431  1728 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:11.596438  6626 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:12.081969  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:12.091104  1727 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:12.045397  7057 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:12.087257  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:12.095767  1629 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 5 to 6, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 6 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:12.094507  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:12.103888  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65540us
I20260709 12:24:12.114809  1805 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:12.117661  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:12.150847  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:12.151266  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:12.151917  1881 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:12.152837  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:12.153822  1881 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:12.119725  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:12.155875  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:12.156661  1872 raft_consensus.cc:2368] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14.
I20260709 12:24:12.143646  1879 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:24:12.158033  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:12.158033  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:12.153388  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:12.134222  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2203) took 1213 ms. Trace:
I20260709 12:24:12.159636  1731 rpcz_store.cc:276] 0709 12:24:10.920555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.052436 (+131881us) service_pool.cc:224] Handling call
0709 12:24:11.705677 (+653241us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3eee 000000000036eb39 00007f16b0d271e0 00007f16b0df00c4 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:24:12.134143 (+428466us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2539648}
I20260709 12:24:12.163777  1801 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:12.085856  6879 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.183454  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:12.181880  1881 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
W20260709 12:24:12.171981  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77571.6us
W20260709 12:24:12.172098  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 84934.3us
I20260709 12:24:12.085856  6865 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.189137  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 694) took 2012 ms. Trace:
I20260709 12:24:12.189798  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:12.189687  1874 rpcz_store.cc:276] 0709 12:24:10.177181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.177354 (+   173us) service_pool.cc:224] Handling call
0709 12:24:10.177799 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.077935 (+1900136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.188878 (+110943us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54001f7a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:12.189076 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4095822720}
I20260709 12:24:12.093626  6837 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:12.177747  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2194) took 1651 ms. Trace:
I20260709 12:24:12.190563  1724 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:12.177868  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:12.191162  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:12.178928  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:12.192013  1874 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:12.192914  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:12.193554  2412 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:24:12.193662  1771 leader_election.cc:400] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14.
I20260709 12:24:12.190858  1723 rpcz_store.cc:276] 0709 12:24:10.526821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.343954 (+817133us) service_pool.cc:224] Handling call
0709 12:24:11.359221 (+ 15267us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.086958 (+727737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.177461 (+ 90503us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b540044af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:12.177678 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1430246656}
I20260709 12:24:12.194123  1771 leader_election.cc:403] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:24:12.179726  1880 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:12.098168  6766 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.047366  7044 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.187492  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:12.202584  1881 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:12.202823  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:12.203074  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:12.204684  1803 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:12.188421  6860 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:24:12.193454  1874 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:12.104803  7107 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.105206  7093 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:12.176645  6963 consensus_queue.cc:1243] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 14.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:12.126999  7088 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:12.205862  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:12.237600  1726 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:12.205060  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:12.197386  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:12.187434  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:12.146489  7040 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:12.211864  1881 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:12.046048  7103 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.152778  6862 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:12.218432  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:12.143224  7064 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:12.153668  6890 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:12.153448  6933 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.131067  6875 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:12.189067  6825 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:24:12.224721  7104 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:12.201268  6827 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:24:12.204991  6879 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.208616  1804 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:12.218762  6766 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.261250  6963 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.244093  1729 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:24:12.209990  7002 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14.
I20260709 12:24:12.247241  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:12.480453  1803 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:12.201764  6898 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.482774  1629 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:12.252009  1800 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:12.058636  7061 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.182910  7111 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
I20260709 12:24:12.280992  7070 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:12.277859  6827 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:12.228706  7053 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:12.530655  6827 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:12.154265  7114 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.095s
I20260709 12:24:12.531294  2415 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:24:12.223800  6860 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:12.532425  6860 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:12.533061  2415 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 12:24:12.534507  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.268287  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80968.3us
I20260709 12:24:12.536621  2415 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:12.165671  7110 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.111s
I20260709 12:24:12.539541  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:12.159672  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:12.546700  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 387071us
I20260709 12:24:12.537262  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:12.547583  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:12.214290  6865 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.283030  1730 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:12.550233  1629 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 3 to 4, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 4 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:12.282967  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:12.292413  1726 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 13. (term mismatch)
I20260709 12:24:12.552841  1726 pending_rounds.cc:85] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 12
I20260709 12:24:12.553108  1726 pending_rounds.cc:107] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 12:24:12.553328  1726 raft_consensus.cc:2889] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:12.262542  6862 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:12.275262  6933 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.240463  1732 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:24:12.448740  6879 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:12.283666  6875 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:12.455551  6959 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:12.455915  6974 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.457734  1728 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 17 because replica has last-logged OpId of term: 15 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:12.471642  7060 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:12.474242  7044 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:12.476320  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:12.616796  1807 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:12.479185  6768 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:12.549167  7070 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:12.563264  6890 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:12.254292  7046 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:24:12.225039  6626 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:12.262960  1874 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:12.582499  6832 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.525436  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:12.632439  1807 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:12.633240  2412 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:12.527405  7061 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:12.528770  2416 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:24:12.529502  7117 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.241s	user 0.000s	sys 0.000s
I20260709 12:24:12.529968  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:12.282936  7104 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.530751  7053 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.228986  6903 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:12.534992  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:12.653901  2413 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:12.534526  2418 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:24:12.627983  6890 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.528867  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:12.264163  7064 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.247646  7057 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.667073  7064 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:12.667613  7057 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:12.509502  7082 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.503599  6963 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.670905  6875 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.671232  7082 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.557881  6862 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:12.268813  7040 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:12.675951  6974 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:12.675701  7040 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.676934  7070 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:12.677966  7043 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.583s	user 0.005s	sys 0.003s
I20260709 12:24:12.618517  6879 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:12.680064  6879 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:12.680621  6879 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:12.681934  7104 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:12.525203  6898 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.682848  6890 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:12.685007  6898 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:12.235241  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47769.9us
I20260709 12:24:12.656483  1874 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:12.653687  7053 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:12.674441  7082 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:12.693271  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:12.694434  7046 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:12.695035  7067 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:12.696503  7067 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:12.696974  7067 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:12.693974  7082 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.005s
I20260709 12:24:12.694976  7046 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.704582  7046 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:12.632025  6626 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:12.705416  6626 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:12.700273  7103 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:12.703256  6963 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:12.698081  7067 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.713433  7067 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:12.649483  6832 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:12.714344  6903 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:12.715683  6903 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.716516  6832 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:12.719373  7107 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:12.720767  6865 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:12.702881  6827 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:12.052201  1627 rpcz_store.cc:269] 0709 12:23:43.531804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.532191 (+   387us) service_pool.cc:224] Handling call
0709 12:23:44.090885 (+558694us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd42090 00007f16abde0f3a 00007f16b1b96e9c 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:24:12.051240 (+27960355us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:54.918272 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:54.919236 (+   964us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:54.919346 (+   110us) write_op.cc:435] Acquired schema lock
0709 12:23:54.919373 (+    27us) tablet.cc:662] Decoding operations
0709 12:23:54.921776 (+  2403us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:54.921903 (+   127us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:54.921942 (+    39us) tablet.cc:685] Acquiring locks for 25 operations
0709 12:23:55.962248 (+1040306us) tablet.cc:701] Row locks acquired
0709 12:23:55.991575 (+ 29327us) write_op.cc:260] PREPARE: finished
0709 12:23:56.048103 (+ 56528us) write_op.cc:270] Start()
0709 12:23:56.122920 (+ 74817us) write_op.cc:276] Timestamp: P: 1783599836019799 usec, L: 0
0709 12:23:56.122990 (+    70us) op_driver.cc:348] REPLICATION: starting
0709 12:23:56.139857 (+ 16867us) log.cc:844] Serialized 9743 byte log entry
0709 12:23:57.049248 (+909391us) op_driver.cc:464] REPLICATION: finished
0709 12:23:57.387981 (+338733us) write_op.cc:301] APPLY: starting
0709 12:23:57.400468 (+ 12487us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:58.516846 (+1116378us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:58.516982 (+   136us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:00.049635 (+1532653us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:00.247448 (+197813us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=50,mrs_lookups=0
0709 12:24:00.247526 (+    78us) write_op.cc:312] APPLY: finished
0709 12:24:00.248362 (+   836us) log.cc:844] Serialized 220 byte log entry
0709 12:24:00.884566 (+636204us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:00.902411 (+ 17845us) write_op.cc:454] Released schema lock
0709 12:24:00.940592 (+ 38181us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:01.024131 (+ 83539us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:01.028476 (+  4345us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":995614,"reported_tablets":50,"spinlock_wait_cycles":564180736,"child_traces":[["op",{"apply.queue_time_us":218661,"apply.run_cpu_time_us":109109,"apply.run_wall_time_us":3687910,"cfile_cache_hit":102,"cfile_cache_hit_bytes":108727,"num_ops":25,"prepare.queue_time_us":276998,"prepare.run_cpu_time_us":53545,"prepare.run_wall_time_us":1422459,"raft.queue_time_us":318060,"raft.run_cpu_time_us":22194,"raft.run_wall_time_us":226693,"replication_time_us":876615,"thread_start_us":1239542,"threads_started":5,"wal-append.queue_time_us":963757,"wal-append.run_cpu_time_us":27342,"wal-append.run_wall_time_us":2643614}]]}
I20260709 12:24:12.584566  1806 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:24:12.535265  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:12.266839  1724 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:12.630221  1727 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:12.562044  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3155) took 1031 ms. Trace:
I20260709 12:24:12.551501  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:12.590875  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2119) took 1007 ms. Trace:
I20260709 12:24:12.507278  1800 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:12.586395  1803 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:12.726938  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:12.864730  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 137862us
W20260709 12:24:12.844950  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 309591us
I20260709 12:24:12.870015  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:12.846928  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2216) took 1284 ms. Trace:
I20260709 12:24:12.733439  2419 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:12.882076  1724 rpcz_store.cc:276] 0709 12:24:11.561994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.239574 (+677580us) service_pool.cc:224] Handling call
0709 12:24:12.266352 (+ 26778us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.845937 (+579585us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.846534 (+   597us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.885535  1874 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:12.885941  1724 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:12.848325  1732 rpcz_store.cc:276] 0709 12:24:11.531821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.573025 (+ 41204us) service_pool.cc:224] Handling call
0709 12:24:12.561947 (+988922us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.889161  1732 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:24:12.854410  1801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.008s	sys 0.008s
W20260709 12:24:12.890087  1801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.008s	sys 0.008s
W20260709 12:24:12.841143  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.002s	sys 0.011s
W20260709 12:24:12.891922  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.003s	sys 0.011s
W20260709 12:24:12.892797  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.878927  1873 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
W20260709 12:24:12.737774  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.055s	sys 0.093s
W20260709 12:24:12.893982  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.759s	user 0.055s	sys 0.093s
W20260709 12:24:12.894542  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 780336us
I20260709 12:24:12.895658  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 718) took 1253 ms. Trace:
I20260709 12:24:12.896342  1732 rpcz_store.cc:276] 0709 12:24:11.641914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.887815 (+1245901us) service_pool.cc:224] Handling call
0709 12:24:12.888441 (+   626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.895159 (+  6718us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.895531 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.734843  2413 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:12.889155  1874 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:24:12.730355  6832 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.174s
W20260709 12:24:12.906636  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.892666  1875 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 8. Current term is 9. Ops: []
W20260709 12:24:12.907683  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.900490  6766 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:12.902060  6856 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:12.909520  6856 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:12.914029  6856 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:12.848527  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2126) took 1169 ms. Trace:
I20260709 12:24:12.915020  1802 rpcz_store.cc:276] 0709 12:24:11.679333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.153682 (+474349us) service_pool.cc:224] Handling call
0709 12:24:12.154008 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.154244 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:12.502774 (+348530us) log.cc:844] Serialized 14 byte log entry
0709 12:24:12.847639 (+344865us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:12.847776 (+   137us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:12.847895 (+   119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.848052 (+   157us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.848275 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":63493,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":4342,"thread_start_us":692315,"threads_started":2}
I20260709 12:24:12.916797  1732 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:12.850595  1798 rpcz_store.cc:276] 0709 12:24:11.583862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.882970 (+299108us) service_pool.cc:224] Handling call
0709 12:24:11.883301 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:11.883959 (+   658us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.883992 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:11.884603 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.213208 (+328605us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:12.561089 (+347881us) log.cc:844] Serialized 14 byte log entry
0709 12:24:12.590428 (+ 29339us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.590523 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:12.590573 (+    50us) raft_consensus.cc:1685] finished
0709 12:24:12.590586 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.590768 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36930,"prepare.queue_time_us":263608,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":29135,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":142,"spinlock_wait_cycles":10240768,"thread_start_us":647647,"threads_started":2,"wal-append.queue_time_us":358266}
I20260709 12:24:12.919822  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:12.918376  1873 pending_rounds.cc:85] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 10
I20260709 12:24:12.923156  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2220) took 1359 ms. Trace:
I20260709 12:24:12.927615  1724 rpcz_store.cc:276] 0709 12:24:11.563258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.884236 (+1320978us) service_pool.cc:224] Handling call
0709 12:24:12.885351 (+  1115us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.886675 (+  1324us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.923027 (+ 36352us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.926831  1873 pending_rounds.cc:107] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:24:12.925089  6832 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:12.980659  1873 raft_consensus.cc:2889] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:12.981186  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2223) took 1326 ms. Trace:
I20260709 12:24:13.023267  1732 rpcz_store.cc:276] 0709 12:24:11.642581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.897751 (+1255170us) service_pool.cc:224] Handling call
0709 12:24:12.916186 (+ 18435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.964347 (+ 48161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.968772 (+  4425us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.964178  6827 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:12.910616  6860 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.021253  6832 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:13.041625  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.694906  6977 consensus_queue.cc:1175] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 14 status { last_received { term: 13 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:13.043355  6832 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:13.057210  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 137634us
I20260709 12:24:12.980530  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:13.064015  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.105259  7044 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.105661  7002 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:13.105336  1878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.011s	sys 0.042s
I20260709 12:24:13.103722  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.116175  1724 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:13.120515  6987 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.077s
I20260709 12:24:13.123322  7082 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.124336  7082 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:13.126157  7036 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.022s	sys 0.020s
W20260709 12:24:13.126670  7036 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.022s	sys 0.021s
I20260709 12:24:13.126915  7110 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:13.122987  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.119115  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:13.127877  7110 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:13.131296  7092 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.042s	sys 0.080s
W20260709 12:24:13.131819  7092 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.042s	sys 0.081s
I20260709 12:24:13.132591  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3161) took 1570 ms. Trace:
I20260709 12:24:13.132715  6860 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.133265  1727 rpcz_store.cc:276] 0709 12:24:11.561613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.146411 (+584798us) service_pool.cc:224] Handling call
0709 12:24:13.132053 (+985642us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54002d14d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:13.132312 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1088018816}
W20260709 12:24:13.134246  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.017s	sys 0.021s
W20260709 12:24:13.134856  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.017s	sys 0.021s
W20260709 12:24:13.135617  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 598395us
I20260709 12:24:13.136507  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3168) took 1216 ms. Trace:
I20260709 12:24:13.137094  1728 rpcz_store.cc:276] 0709 12:24:11.920312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.178620 (+258308us) service_pool.cc:224] Handling call
0709 12:24:13.135980 (+957360us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b540044af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:13.136264 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":627328768}
I20260709 12:24:13.137707  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:13.143186  6713 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:13.144196  7120 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.144367  6713 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:13.145219  7120 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.151674  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:13.152979  1874 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:12.783022  7064 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.128604  7063 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:13.159256  1872 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:13.163305  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 293753us
I20260709 12:24:12.729417  7093 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.168356  7093 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:13.121865  1878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.011s	sys 0.042s
I20260709 12:24:13.171717  1873 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:13.173213  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2224) took 1479 ms. Trace:
I20260709 12:24:13.176775  1724 rpcz_store.cc:276] 0709 12:24:11.642837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.015325 (+1372488us) service_pool.cc:224] Handling call
0709 12:24:13.062306 (+ 46981us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.119575 (+ 57269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.122368 (+  2793us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:13.192047  6862 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.518s	user 0.016s	sys 0.008s
W20260709 12:24:13.192433  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 103630us
I20260709 12:24:13.106657  7044 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:13.195122  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.192778  6862 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.210099  7042 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:13.210876  7151 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.212790  6856 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:13.213863  6963 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.177057  7110 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.231379  6987 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:24:13.240465  7036 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:13.241214  6837 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.243347  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2307) took 1069 ms. Trace:
W20260709 12:24:13.244354  6626 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.538s	user 0.008s	sys 0.015s
I20260709 12:24:13.223758  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 990) took 1292 ms. Trace:
I20260709 12:24:13.246142  2419 rpcz_store.cc:276] 0709 12:24:11.931249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.999742 (+ 68493us) service_pool.cc:224] Handling call
0709 12:24:13.223292 (+1223550us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b540049eb54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:13.223543 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1557646208}
I20260709 12:24:13.243741  6862 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.214681  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:13.251673  6963 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.257355  7151 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.259053  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3173) took 1337 ms. Trace:
I20260709 12:24:13.259626  1724 rpcz_store.cc:276] 0709 12:24:11.921630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.206455 (+1284825us) service_pool.cc:224] Handling call
0709 12:24:13.258115 (+ 51660us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.258554 (+   439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.258587 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:13.258655 (+    68us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:13.258723 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.258809 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.258951 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.260232  6963 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.220327  7044 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:13.223387  7120 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.251577  7042 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:13.263813  1725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.006s	sys 0.009s
W20260709 12:24:13.265075  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.000s	sys 0.013s
W20260709 12:24:13.271819  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 134377us
I20260709 12:24:13.145224  6713 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:13.267020  1731 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.881s	user 0.024s	sys 0.015s
I20260709 12:24:13.271943  1873 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:13.248638  7036 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:13.280344  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.000s	sys 0.014s
W20260709 12:24:13.281543  1731 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.064s	user 0.025s	sys 0.016s
W20260709 12:24:13.273593  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 107553us
I20260709 12:24:13.287470  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:13.287599  6875 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:13.289211  6875 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:13.290269  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:13.295789  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:13.297138  7140 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.299288  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:13.300567  6856 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.266419  7082 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:13.266975  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 560089us
W20260709 12:24:13.303771  1725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.782s	user 0.007s	sys 0.010s
I20260709 12:24:13.243909  1880 rpcz_store.cc:276] 0709 12:24:12.173312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.236317 (+ 63005us) service_pool.cc:224] Handling call
0709 12:24:12.267792 (+ 31475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.279751 (+ 11959us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.279835 (+    84us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.280468 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.866888 (+586420us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:12.867053 (+   165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.867274 (+   221us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.242811 (+375537us) raft_consensus.cc:1685] finished
0709 12:24:13.242836 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.243071 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71133,"raft.queue_time_us":12486,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":6817,"thread_start_us":360602,"threads_started":1,"wal-append.queue_time_us":953547}
W20260709 12:24:13.306835  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70050.6us
I20260709 12:24:13.245047  6626 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:13.318464  1881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.004s	sys 0.010s
W20260709 12:24:13.325232  1771 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.590s (SENT)
I20260709 12:24:13.324838  7042 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.325510  7082 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:24:13.326658  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 57033.8us
I20260709 12:24:13.325273  6963 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:13.326433  7082 consensus_queue.cc:1243] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 15.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:13.328636  7141 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.329458  7141 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.324585  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.332145  7120 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:13.332484  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.333546  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:13.334048  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3015) took 1224 ms. Trace:
I20260709 12:24:13.334342  7120 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:13.334931  1878 rpcz_store.cc:276] 0709 12:24:12.109176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.109476 (+   300us) service_pool.cc:224] Handling call
0709 12:24:12.164792 (+ 55316us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:12.246481 (+ 81689us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.246517 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:12.262027 (+ 15510us) log.cc:844] Serialized 52 byte log entry
0709 12:24:13.170346 (+908319us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:13.170500 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.170700 (+   200us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.333708 (+163008us) raft_consensus.cc:1685] finished
0709 12:24:13.333737 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.333964 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70779,"raft.queue_time_us":81407,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":9313,"thread_start_us":665065,"threads_started":1,"wal-append.queue_time_us":1023842}
I20260709 12:24:13.336794  7120 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:13.337406  7120 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.337894  7120 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:13.340210  7141 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:13.341626  7067 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.624s	user 0.009s	sys 0.006s
I20260709 12:24:13.337651  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.333586  1880 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:13.344496  7151 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.345460  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.345799  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2133) took 1609 ms. Trace:
I20260709 12:24:13.347036  6626 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.346231  1805 rpcz_store.cc:276] 0709 12:24:11.736841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.157970 (+421129us) service_pool.cc:224] Handling call
0709 12:24:12.168682 (+ 10712us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.168926 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:12.511855 (+342929us) log.cc:844] Serialized 14 byte log entry
0709 12:24:13.237417 (+725562us) log.cc:844] Serialized 14 byte log entry
0709 12:24:13.261521 (+ 24104us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:13.265296 (+  3775us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:13.279366 (+ 14070us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.311928 (+ 32562us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.345553 (+ 33625us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3407,"prepare.queue_time_us":146691,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":134,"raft.queue_time_us":223278,"raft.run_cpu_time_us":573,"raft.run_wall_time_us":35045,"thread_start_us":922127,"threads_started":3,"wal-append.queue_time_us":664370}
I20260709 12:24:13.349277  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:13.343528  7130 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:24:13.352545  6879 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.671s	user 0.012s	sys 0.009s
I20260709 12:24:13.353298  6879 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.343783  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:13.351135  6875 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:13.351619  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 137190us
I20260709 12:24:13.324734  1727 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:24:13.369539  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.373260  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3171) took 1452 ms. Trace:
W20260709 12:24:13.342586  7046 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.637s	user 0.010s	sys 0.005s
I20260709 12:24:13.374770  1727 rpcz_store.cc:276] 0709 12:24:11.921108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.180580 (+1259472us) service_pool.cc:224] Handling call
0709 12:24:13.282712 (+102132us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.369449 (+ 86737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.373175 (+  3726us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.374993  7151 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.322594  6837 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:13.357853  6865 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:13.298305  7140 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.383433  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:13.356863  6626 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:24:13.386936  7111 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:24:13.387125  7036 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.388242  6713 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.388391  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2215) took 1826 ms. Trace:
I20260709 12:24:13.388929  6766 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:13.429634  6996 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.419070  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.430402  6996 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.389955  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2227) took 1412 ms. Trace:
W20260709 12:24:13.391124  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2127) took 1694 ms (client timeout 1926 ms). Trace:
I20260709 12:24:13.392545  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:13.393810  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.395957  7063 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:13.430455  7151 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.387171  1723 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:13.399785  6856 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.451280  6856 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.451639  7130 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.451972  6856 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.454245  7002 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:13.454944  6856 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.401259  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:13.402228  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68880.6us
I20260709 12:24:13.441758  1723 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:13.403640  2413 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:13.404367  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:13.383985  7046 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:13.407450  7053 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.715s	user 0.004s	sys 0.018s
I20260709 12:24:13.408429  7100 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:13.409382  6825 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:13.409842  7057 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.742s	user 0.012s	sys 0.009s
W20260709 12:24:13.473666  2417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 991) took 1542 ms (client timeout 1841 ms). Trace:
W20260709 12:24:13.474141  2417 rpcz_store.cc:269] 0709 12:24:11.931608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.991129 (+ 59521us) service_pool.cc:224] Handling call
0709 12:24:13.473274 (+1482145us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b54005609d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:13.473491 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2044078336}
I20260709 12:24:13.414990  7081 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.415275  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:13.467854  1877 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:13.368352  6875 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.419349  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:13.419054  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3019) took 1307 ms. Trace:
W20260709 12:24:13.426867  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77905.3us
W20260709 12:24:13.427250  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45005.5us
I20260709 12:24:13.481726  1723 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:13.482532  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 102070us
I20260709 12:24:13.483585  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:13.427678  7111 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:13.484493  7111 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:13.485275  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:13.425557  1726 rpcz_store.cc:276] 0709 12:24:11.561743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.238539 (+676796us) service_pool.cc:224] Handling call
0709 12:24:12.240758 (+  2219us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.553760 (+313002us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.387869 (+834109us) spinlock_profiling.cc:243] Waited 51.2 ms on lock 0x7b54000d2ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:13.388174 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":117653376}
I20260709 12:24:13.397195  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3028) took 1255 ms. Trace:
I20260709 12:24:13.435856  1727 rpcz_store.cc:276] 0709 12:24:11.977680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.388721 (+1411041us) service_pool.cc:224] Handling call
0709 12:24:13.389250 (+   529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.389609 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.389640 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:13.389700 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:13.389739 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.389792 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.389895 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:13.435150  1806 rpcz_store.cc:269] 0709 12:24:11.696231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.699836 (+  3605us) service_pool.cc:224] Handling call
0709 12:24:13.390611 (+1690775us) spinlock_profiling.cc:243] Waited 456 ms on lock 0x7b5400132c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:13.390845 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1049534208}
I20260709 12:24:13.443290  1875 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
W20260709 12:24:13.447589  7147 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.204s	user 0.000s	sys 0.002s
I20260709 12:24:13.438877  7063 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:13.510919  7140 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.480952  1876 rpcz_store.cc:276] 0709 12:24:12.111747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.114171 (+  2424us) service_pool.cc:224] Handling call
0709 12:24:12.114534 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.114830 (+   296us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.114866 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.185274 (+ 70408us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b2c00009c38. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:24:12.196531 (+ 11257us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.654256 (+457725us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:12.654300 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.654368 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.418203 (+763835us) raft_consensus.cc:1685] finished
0709 12:24:13.418265 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.418732 (+   467us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57832704,"thread_start_us":405391,"threads_started":1,"wal-append.queue_time_us":1024904}
W20260709 12:24:13.513984  6890 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.830s	user 0.017s	sys 0.006s
I20260709 12:24:13.514722  6890 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:13.515679  1729 pending_rounds.cc:85] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 7
I20260709 12:24:13.516242  1729 pending_rounds.cc:107] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:24:13.516538  1729 raft_consensus.cc:2889] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:13.518324  6890 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:24:13.403679  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.470122  7130 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.471150  1804 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
W20260709 12:24:13.471818  7115 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.759s	user 0.003s	sys 0.002s
W20260709 12:24:13.471957  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.471868  2382 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:13.472831  1800 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 9. Current term is 10. Ops: []
I20260709 12:24:13.472400  7053 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.459074  7002 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:13.414916  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:13.478281  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 43711.6us
I20260709 12:24:13.389505  6865 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:13.486836  6875 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:13.477523  7151 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.487130  1723 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:13.485787  7111 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.488998  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:13.495272  7100 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.004s
I20260709 12:24:13.501744  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 683) took 1513 ms. Trace:
I20260709 12:24:13.479858  6825 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:13.502110  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3170) took 1581 ms (client timeout 1543 ms). Trace:
I20260709 12:24:13.502707  1805 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:13.486127  7081 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.488741  7046 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:13.507263  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3030) took 1012 ms. Trace:
I20260709 12:24:13.580881  1873 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:13.493970  6933 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.450155  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.519081  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3165) took 1956 ms. Trace:
W20260709 12:24:13.576289  1730 rpcz_store.cc:269] 0709 12:24:11.920850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.938377 (+ 17527us) service_pool.cc:224] Handling call
0709 12:24:13.501624 (+1563247us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:13.501866 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":475692928}
I20260709 12:24:13.450510  6879 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.519009  6832 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.520191  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.521066  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:13.509271  1872 rpcz_store.cc:276] 0709 12:24:12.149230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.162102 (+ 12872us) service_pool.cc:224] Handling call
0709 12:24:12.162355 (+   253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.159937 (+997582us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.396882 (+236945us) spinlock_profiling.cc:243] Waited 996 ms on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:13.397133 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2291234688}
I20260709 12:24:13.524596  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.526664  7125 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.526153  7063 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.528834  6766 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:13.497905  7138 raft_consensus.cc:2764] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:24:13.533972  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.543361  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3167) took 1623 ms. Trace:
I20260709 12:24:13.539161  7120 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.544718  6996 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:13.546113  6963 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:24:13.546352  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61227.6us
I20260709 12:24:13.506578  7092 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.534523  7064 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:13.591159  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 136605us
I20260709 12:24:13.573297  1801 rpcz_store.cc:276] 0709 12:24:11.988395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.164804 (+176409us) service_pool.cc:224] Handling call
0709 12:24:12.165114 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.165453 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.165488 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.189182 (+ 23694us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.891169 (+701987us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:12.891285 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.891418 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.501355 (+609937us) raft_consensus.cc:1685] finished
0709 12:24:13.501389 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.501650 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":517,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":23747,"thread_start_us":644307,"threads_started":1,"wal-append.queue_time_us":1270633}
I20260709 12:24:13.581153  1881 rpcz_store.cc:276] 0709 12:24:12.494421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.494660 (+   239us) service_pool.cc:224] Handling call
0709 12:24:12.495151 (+   491us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:12.537499 (+ 42348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.537553 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:12.538256 (+   703us) log.cc:844] Serialized 52 byte log entry
0709 12:24:13.423237 (+884981us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:13.438089 (+ 14852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.465972 (+ 27883us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.466052 (+    80us) raft_consensus.cc:1685] finished
0709 12:24:13.466071 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.507172 (+ 41101us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66027,"raft.queue_time_us":50107,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":7477,"thread_start_us":755031,"threads_started":1,"wal-append.queue_time_us":776184}
I20260709 12:24:13.596820  1803 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:13.599440  1726 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 9: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:13.580297  6856 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:13.597771  1732 rpcz_store.cc:276] 0709 12:24:11.919992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.044647 (+1124655us) service_pool.cc:224] Handling call
0709 12:24:13.086208 (+ 41561us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.114745 (+ 28537us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:13.114804 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.180054 (+ 65250us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.364351 (+184297us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:13.364389 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.364461 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.543044 (+178583us) raft_consensus.cc:1685] finished
0709 12:24:13.543071 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.543270 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":116101,"threads_started":1,"wal-append.queue_time_us":292994}
I20260709 12:24:13.603434  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:13.585238  7093 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:13.583918  1725 rpcz_store.cc:276] 0709 12:24:11.562608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.519480 (+956872us) service_pool.cc:224] Handling call
0709 12:24:12.519967 (+   487us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.520362 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.520398 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.520997 (+   599us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.327055 (+806058us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:13.327509 (+   454us) log.cc:844] Serialized 14 byte log entry
0709 12:24:13.327693 (+   184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.327828 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.518844 (+191016us) raft_consensus.cc:1685] finished
0709 12:24:13.518864 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.519019 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16810,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":107452,"raft.run_cpu_time_us":436,"raft.run_wall_time_us":6314,"thread_start_us":786540,"threads_started":1,"wal-append.queue_time_us":721971}
I20260709 12:24:13.596993  7111 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:13.599201  6766 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:13.605180  1798 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:13.616616  1878 raft_consensus.cc:2368] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13.
W20260709 12:24:13.617995  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 146348us
I20260709 12:24:13.619372  7005 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.604210  6933 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.623713  7151 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:13.612155  6713 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.643261  6713 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.644865  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2213) took 2083 ms. Trace:
I20260709 12:24:13.645545  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 679) took 2065 ms. Trace:
I20260709 12:24:13.626196  1727 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:13.615896  6856 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:13.629370  6976 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:13.631043  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 142154us
I20260709 12:24:13.631608  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.631947  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:13.618355  6963 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:13.632513  7125 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.634056  7130 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.633945  7073 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 12:24:13.687347  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:13.634809  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:13.634861  6865 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.637298  7053 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:13.623487  7092 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.624078  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:13.625581  6825 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.614717  7093 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:13.678429  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54740.7us
W20260709 12:24:13.676771  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 218100us
I20260709 12:24:13.686017  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:13.686964  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:13.645443  1731 rpcz_store.cc:276] 0709 12:24:11.561242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.207753 (+646511us) service_pool.cc:224] Handling call
0709 12:24:12.216959 (+  9206us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.283769 (+1066810us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.283840 (+    71us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.284609 (+   769us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.375517 (+ 90908us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:13.375554 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.383999 (+  8445us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.644583 (+260584us) raft_consensus.cc:1685] finished
0709 12:24:13.644606 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.644791 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":262444,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":20623,"thread_start_us":1156575,"threads_started":2,"wal-append.queue_time_us":286425}
I20260709 12:24:13.646371  6713 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:13.719574  6963 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.721102  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.721947  1731 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:13.685979  6879 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:13.719619  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.723233  6879 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.724272  1732 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:13.719841  7046 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:13.725562  7046 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.689525  2417 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
W20260709 12:24:13.726536  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:13.704026  7157 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:24:13.710592  7093 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:13.711104  6976 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.713564  7063 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.714085  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2138) took 1635 ms. Trace:
I20260709 12:24:13.677167  7005 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.717343  7064 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:13.718838  1725 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:13.719832  7043 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:24:13.709343  7125 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.652011  6996 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.081s
I20260709 12:24:13.725965  7046 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:13.732003  7081 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.735399  2419 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:13.736334  2418 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
W20260709 12:24:13.729516  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94848.2us
W20260709 12:24:13.785764  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:13.740689  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101595us
I20260709 12:24:13.734323  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:13.745585  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.730023  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.685580  7151 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:13.640787  6933 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.759280  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:13.760048  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:13.761098  6903 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.761402  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2218) took 2198 ms. Trace:
I20260709 12:24:13.761766  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:13.765344  7145 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:24:13.765743  7064 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.766040  2416 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
W20260709 12:24:13.767431  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71080us
W20260709 12:24:13.768801  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.729543  6832 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:13.730595  7073 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:24:13.770219  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83017.4us
W20260709 12:24:13.773137  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.730921  7120 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.646448  1799 rpcz_store.cc:276] 0709 12:24:11.580291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.584661 (+  4370us) service_pool.cc:224] Handling call
0709 12:24:11.585222 (+   561us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:24:11.672107 (+ 86885us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:11.708078 (+ 35971us) log.cc:844] Serialized 14 byte log entry
0709 12:24:12.027827 (+319749us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:24:12.106168 (+ 78341us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b2c0005c178. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:24:12.152943 (+ 46775us) log.cc:844] Serialized 104 byte log entry
0709 12:24:12.153141 (+   198us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:12.185931 (+ 32790us) log.cc:844] Serialized 14 byte log entry
0709 12:24:12.190006 (+  4075us) log.cc:844] Serialized 14 byte log entry
0709 12:24:12.190229 (+   223us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.208223 (+ 17994us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:12.712486 (+504263us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x7b18004002a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:13.470620 (+758134us) spinlock_profiling.cc:243] Waited 6.89 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:13.487606 (+ 16986us) spinlock_profiling.cc:243] Waited 8.14 ms on lock 0x7b18004002a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f
0709 12:24:13.488318 (+   712us) spinlock_profiling.cc:243] Waited 8.51 ms on lock 0x7b48001c0158. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:24:13.496130 (+  7812us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b18004002a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:13.645086 (+148956us) raft_consensus.cc:1685] finished
0709 12:24:13.645107 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.645349 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":89497,"prepare.queue_time_us":151936,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":125,"raft.queue_time_us":96336,"raft.run_cpu_time_us":14813,"raft.run_wall_time_us":1329768,"spinlock_wait_cycles":196813824,"thread_start_us":319104,"threads_started":1,"wal-append.queue_time_us":483092}
I20260709 12:24:13.774562  1723 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 12:24:13.777313  6977 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:13.731884  7092 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.774622  6890 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.753286  7100 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:13.799432  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.803633  2417 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:24:13.790031  6875 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.748585  1807 rpcz_store.cc:276] 0709 12:24:12.044998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.633571 (+588573us) service_pool.cc:224] Handling call
0709 12:24:12.840152 (+206581us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:12.841770 (+  1618us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:12.842639 (+   869us) log.cc:844] Serialized 14 byte log entry
0709 12:24:13.103349 (+260710us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:13.270905 (+167556us) log.cc:844] Serialized 52 byte log entry
0709 12:24:13.275097 (+  4192us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:13.275445 (+   348us) log.cc:844] Serialized 14 byte log entry
0709 12:24:13.275651 (+   206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.275740 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.357151 (+ 81411us) spinlock_profiling.cc:243] Waited 3.84 ms on lock 0x7b1800232ae0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:13.673415 (+316264us) raft_consensus.cc:1685] finished
0709 12:24:13.673449 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.680345 (+  6896us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20562,"prepare.queue_time_us":136963,"prepare.run_cpu_time_us":171,"prepare.run_wall_time_us":7176,"raft.queue_time_us":215395,"raft.run_cpu_time_us":671,"raft.run_wall_time_us":37938,"spinlock_wait_cycles":8833920,"thread_start_us":259966,"threads_started":1,"wal-append.queue_time_us":736507}
I20260709 12:24:13.808872  2414 pending_rounds.cc:85] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 10
I20260709 12:24:13.820930  2414 pending_rounds.cc:107] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 12:24:13.821256  2414 raft_consensus.cc:2889] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:13.788182  7057 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.821905  7057 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.823662  6875 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.824126  7057 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.815392  6977 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.821928  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:13.825040  6977 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:13.832423  6903 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.835058  2410 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:13.835755  6976 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.719602  6856 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.842625  1731 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:13.813753  2412 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:13.843703  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:13.846189  7158 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.208s	user 0.003s	sys 0.000s
I20260709 12:24:13.847366  7002 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.849709  7002 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.854607  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2226) took 1877 ms. Trace:
I20260709 12:24:13.854888  6996 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:13.855099  1724 rpcz_store.cc:276] 0709 12:24:11.977416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.260995 (+1283579us) service_pool.cc:224] Handling call
0709 12:24:13.269563 (+  8568us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.426821 (+157258us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.426852 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.427567 (+   715us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.578822 (+151255us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:13.578886 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.578989 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.782273 (+203284us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:13.854277 (+ 72004us) raft_consensus.cc:1685] finished
0709 12:24:13.854300 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.854544 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37734,"raft.queue_time_us":46686,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"spinlock_wait_cycles":68630272,"thread_start_us":150692,"threads_started":1,"wal-append.queue_time_us":184773}
I20260709 12:24:13.857014  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.857548  6933 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.857625  1724 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 8.
I20260709 12:24:13.842753  7005 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.864529  6903 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.794562  1725 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:13.866007  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.810961  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3172) took 1889 ms. Trace:
I20260709 12:24:13.791561  1729 rpcz_store.cc:276] 0709 12:24:11.562516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.293824 (+731308us) service_pool.cc:224] Handling call
0709 12:24:13.515067 (+1221243us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:13.517531 (+  2464us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.517567 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:13.518272 (+   705us) log.cc:844] Serialized 52 byte log entry
0709 12:24:13.597412 (+ 79140us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:13.597476 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.602771 (+  5295us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.720244 (+117473us) raft_consensus.cc:1685] finished
0709 12:24:13.720270 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.761045 (+ 40775us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002d1ed0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:13.761333 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2174,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"spinlock_wait_cycles":2784004480,"thread_start_us":78545,"threads_started":1,"wal-append.queue_time_us":155118}
I20260709 12:24:13.866920  7151 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.867429  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:13.868157  1729 raft_consensus.cc:606] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:24:13.869393  7151 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.845090  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:13.812160  1723 pending_rounds.cc:85] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 11
I20260709 12:24:13.816542  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:13.872184  1723 pending_rounds.cc:107] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 12:24:13.872550  1723 raft_consensus.cc:2889] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:13.866647  1725 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:13.858625  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:13.874678  1727 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:13.875849  1730 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:13.819917  1726 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:13.866947  1728 rpcz_store.cc:276] 0709 12:24:11.921379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.138286 (+1216907us) service_pool.cc:224] Handling call
0709 12:24:13.224990 (+ 86704us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.282248 (+ 57258us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:24:13.282289 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.373399 (+ 91110us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.583367 (+209968us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:13.583402 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.583568 (+   166us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.810708 (+227140us) raft_consensus.cc:1685] finished
0709 12:24:13.810732 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.810898 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":180009,"threads_started":1,"wal-append.queue_time_us":248221}
W20260709 12:24:13.879212  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65129.3us
I20260709 12:24:13.879359  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3160) took 2318 ms. Trace:
I20260709 12:24:13.881278  1731 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:13.881070  1723 rpcz_store.cc:276] 0709 12:24:11.561354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.226459 (+665105us) service_pool.cc:224] Handling call
0709 12:24:12.226863 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.386876 (+1160013us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b54004b2420. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:24:13.873171 (+486295us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.879303 (+  6132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13654912}
I20260709 12:24:13.800323  6879 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:13.885227  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 86061.7us
I20260709 12:24:13.886512  1724 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:13.888497  6832 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:13.888752  7100 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:13.889412  6945 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 12:24:13.891234  7100 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.886515  6626 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.892867  6626 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.894912  7150 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 12:24:13.891155  6875 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:13.895854  6875 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.896291  6875 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:13.896526  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:13.898278  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:13.895310  2416 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:24:13.888944  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:13.899262  2418 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:13.903095  7092 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:13.903892  7092 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:13.904402  7064 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:13.905078  7092 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.905526  1798 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:24:13.905434  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.905884  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.906427  1805 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:13.909276  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.909744  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.910208  1874 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 9: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:13.910568  1873 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:24:13.911209  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.911513  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.911643  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.912529  1872 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:13.896004  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:13.914337  2411 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:13.891273  1728 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 12:24:13.915895  7161 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:24:13.876231  6890 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.815649  7073 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:13.920105  7073 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.914167  7043 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.883044  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:13.925864  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.935460  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.938264  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.940697  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.895383  2412 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:13.940188  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:13.934996  7063 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.939909  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.937402  7042 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:24:13.937134  6890 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.932107  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.945665  1728 raft_consensus.cc:1217] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Deduplicated request from leader. Original: 7.4->[10.5-16.8]   Dedup: 10.5->[12.6-16.8]
I20260709 12:24:13.946983  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:13.970657  1799 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:13.971634  7043 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.974324  1801 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:24:13.974993  6976 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:13.936517  6626 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.953531  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.974567  2413 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:13.977824  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:13.959641  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 73582.7us
I20260709 12:24:13.942687  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:13.963282  1803 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:24:13.959602  6766 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.979789  1726 raft_consensus.cc:606] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:24:13.981331  1726 raft_consensus.cc:614] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:13.981395  6837 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.982069  6837 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:13.982689  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.983500  1726 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:13.984128  1730 pending_rounds.cc:85] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 6
I20260709 12:24:13.984227  6837 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.984567  1730 pending_rounds.cc:107] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 15.7
I20260709 12:24:13.963292  1805 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:13.964411  7057 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20260709 12:24:13.985746  7057 consensus_queue.cc:1243] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:24:13.984867  1730 raft_consensus.cc:2889] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:13.986258  1726 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:24:13.963722  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.965557  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.965680  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.961481  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:13.964764  2419 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:13.982414  6766 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:13.965088  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:13.940099  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:14.008100  1798 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:13.976658  7043 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:13.963181  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:13.993005  1732 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:14.019315  1723 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:13.969393  1728 pending_rounds.cc:85] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 5
I20260709 12:24:14.027905  1728 pending_rounds.cc:107] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 15.6
I20260709 12:24:14.028228  1728 raft_consensus.cc:2889] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:14.030222  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 212614us
I20260709 12:24:14.029549  2418 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:14.036144  6875 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:14.037274  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86541.1us
I20260709 12:24:13.966344  7063 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:14.040526  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:14.042030  7063 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:14.038466  1872 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:24:14.038519  7138 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:14.043890  1725 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:14.049036  1873 raft_consensus.cc:2410] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:24:13.959473  6832 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.997720  6977 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:13.984880  7150 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.070683  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:14.075392  1801 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:14.076068  1879 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:14.070374  6875 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:14.027680  7100 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:24:14.030006  6890 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:14.088229  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:14.111650  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 71196us
W20260709 12:24:14.136545  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50729.6us
W20260709 12:24:14.138250  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56579.8us
I20260709 12:24:14.100832  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:14.142411  1724 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:14.019063  7151 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.068s
I20260709 12:24:14.174881  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:14.143800  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.000s	sys 0.028s
W20260709 12:24:14.176342  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.000s	sys 0.028s
I20260709 12:24:14.096033  7043 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:14.096984  6976 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.226486  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:14.094131  7100 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.222918  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.093922  6890 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:14.108318  6875 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.172847  7063 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:14.179876  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:14.229547  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.174115  7081 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:14.110380  7125 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:14.082455  6832 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:14.245810  1874 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:14.265918  1727 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:14.267841  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.255018  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:14.250190  7100 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:24:14.273712  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.144572  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:14.178936  6626 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:14.253234  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:14.258414  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 83664.4us
I20260709 12:24:14.252359  7043 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.265125  6856 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:14.272617  6963 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.102841  7036 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.164s
W20260709 12:24:14.281543  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 55178.2us
I20260709 12:24:14.265944  7081 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.295835  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.300388  7036 consensus_queue.cc:1243] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 12.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:14.301029  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.301414  7090 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:14.302009  7090 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:14.302207  1877 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:14.222985  6766 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.052s
W20260709 12:24:14.299414  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:14.300302  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 155820us
I20260709 12:24:14.267951  7111 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.226167  6825 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:24:14.314718  1731 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:14.315894  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:14.328955  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:14.329921  1727 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 8.
I20260709 12:24:14.281006  7002 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:14.276458  6626 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.338492  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:14.292896  7043 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:14.268388  6875 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:14.279740  6837 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.346133  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.422504  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:14.422962  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:14.425737  2418 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:24:14.351361  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.351605  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:14.350391  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:14.309778  7081 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:14.423976  1878 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:14.298359  7053 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:14.356702  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:14.342267  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.268548  6976 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:14.301055  7036 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:24:14.429297  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:14.431490  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70940.1us
I20260709 12:24:14.429800  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:14.094342  7150 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:14.431421  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.507961  1879 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:14.437067  1875 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:14.433219  1876 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:14.355041  6626 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:14.518013  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:14.519088  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:14.519912  2418 raft_consensus.cc:2410] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 9 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:24:14.521539  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:14.522490  1727 raft_consensus.cc:606] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:14.523007  1727 raft_consensus.cc:614] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:14.518054  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:14.459657  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.459137  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:14.505254  1731 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
W20260709 12:24:14.508661  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 165003us
I20260709 12:24:14.288930  7063 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.510195  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:14.458877  6933 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:14.513492  1771 leader_election.cc:400] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13.
I20260709 12:24:14.307102  6856 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:14.528529  1878 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:24:14.529703  2411 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:14.530129  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:14.428141  6865 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:14.342846  7111 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.536638  2416 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:14.454212  7053 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.510965  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:14.549141  2419 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:14.543058  1771 leader_election.cc:403] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:24:14.544883  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.546247  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:14.528429  2412 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:14.553157  1731 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:14.554158  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.554348  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.513876  7120 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.565663  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:14.567740  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:14.572219  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 252201us
W20260709 12:24:14.582103  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 142363us
I20260709 12:24:14.427866  6963 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.514500  7151 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:24:14.509786  6837 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.592144  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:14.587073  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:14.588666  1731 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:14.547645  6933 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:14.593575  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:14.595685  1879 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:14.421725  7090 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.536134  6903 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:24:14.617175  2418 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:14.566017  7036 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:14.624107  1724 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
W20260709 12:24:14.630059  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.559954  7053 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:14.599890  1731 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:24:14.608732  1723 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:14.595739  2410 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:14.617043  1879 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:14.635098  1723 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: [15.14-15.14]
I20260709 12:24:14.636914  6968 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:14.638406  6968 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:14.636264  7073 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:24:14.640985  1723 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:24:14.642114  7073 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:14.580037  6865 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.642733  7073 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:14.648882  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:14.650038  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 372742us
W20260709 12:24:14.653647  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.654822  1875 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:14.655274  2417 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 12: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:14.655495  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58071us
I20260709 12:24:14.596410  7063 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:14.605360  6933 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:14.656726  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:14.622246  7137 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:24:14.622140  7057 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:24:14.630270  7046 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.851s	user 0.009s	sys 0.003s
I20260709 12:24:14.538655  6890 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.623535  6875 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:14.666194  1879 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 13. (index mismatch)
I20260709 12:24:14.661952  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:14.646752  6879 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:14.648793  6832 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:14.675025  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:14.676323  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:14.656883  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:14.605237  7168 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:14.667541  7093 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.914s	user 0.008s	sys 0.004s
I20260709 12:24:14.689391  2418 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:14.607227  7120 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.599514  7005 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.082s
I20260709 12:24:14.673744  7063 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:14.719539  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44718us
I20260709 12:24:14.646194  6945 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.041s
W20260709 12:24:14.783594  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 236897us
W20260709 12:24:14.782714  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59508us
I20260709 12:24:14.702519  7168 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:14.813650  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:14.821139  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 144881us
I20260709 12:24:14.777177  7090 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:14.684675  7046 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:14.776374  6875 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.819995  6865 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:14.832377  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 295985us
I20260709 12:24:14.835040  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:14.845212  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 415445us
W20260709 12:24:14.826392  7137 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:24:14.709802  7093 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.778518  6879 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:14.851320  7173 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.358s	user 0.001s	sys 0.001s
I20260709 12:24:14.851085  6875 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:14.853112  6879 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:14.853576  6875 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:14.826900  7191 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:14.854614  7191 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:14.855079  7191 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:14.858927  7175 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.328s	user 0.002s	sys 0.000s
I20260709 12:24:14.824210  7168 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:14.694139  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:14.831581  6890 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:14.868085  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 233209us
I20260709 12:24:14.875741  1880 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:14.889966  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 195903us
I20260709 12:24:14.844573  6856 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:14.923188  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:14.844457  7193 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:24:14.931463  6945 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:14.923020  6879 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
W20260709 12:24:14.949575  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64918.7us
W20260709 12:24:14.944222  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 69864.1us
W20260709 12:24:14.960646  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 125643us
I20260709 12:24:14.919867  6875 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:14.886209  6832 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:24:14.943151  7182 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:24:14.961213  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:14.940301  7170 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
W20260709 12:24:14.935047  7171 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.372s	user 0.000s	sys 0.002s
W20260709 12:24:15.009009  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47883.8us
I20260709 12:24:14.946067  7198 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:24:14.939292  7172 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
I20260709 12:24:14.970374  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:14.934979  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:14.939740  7180 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.317s	user 0.001s	sys 0.000s
W20260709 12:24:14.944600  7179 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.379s	user 0.000s	sys 0.001s
W20260709 12:24:14.968941  7067 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.689s	user 0.006s	sys 0.011s
W20260709 12:24:15.023106  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 54287us
W20260709 12:24:15.030524  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 60308.7us
I20260709 12:24:14.925169  7150 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:14.951292  7193 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.829770  7073 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:14.942294  7093 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:14.943615  6963 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:14.946322  6879 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:15.031384  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:15.044705  1879 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:14.965848  6856 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:24:14.966766  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 103610us
I20260709 12:24:15.047361  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:14.968286  6875 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:15.049618  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:15.047448  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 81571.3us
W20260709 12:24:15.050832  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 427086us
I20260709 12:24:15.054417  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:15.063702  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.020689  6832 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:15.039937  7067 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.006s	sys 0.011s
I20260709 12:24:15.019733  6837 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:15.071794  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:15.075845  1873 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:24:14.966219  7057 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.084s
I20260709 12:24:14.935489  6945 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.077040  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.077039  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.087154  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:15.079082  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 58783us
I20260709 12:24:14.932763  7200 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.125s
W20260709 12:24:15.088640  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.055817  6875 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:15.045897  7193 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:15.086598  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1048) took 1303 ms. Trace:
I20260709 12:24:15.088009  1801 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:15.049325  6856 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.055253  6963 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:15.110886  7177 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.429s	user 0.000s	sys 0.001s
I20260709 12:24:15.110917  1803 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:15.111166  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:15.113472  6945 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:15.115494  7187 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
W20260709 12:24:15.115573  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68336.6us
I20260709 12:24:15.116928  7190 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:15.107393  7203 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:15.124361  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.125777  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.127317  7207 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:15.078518  7150 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.103557  2414 rpcz_store.cc:276] 0709 12:24:13.774420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.789708 (+ 15288us) service_pool.cc:224] Handling call
0709 12:24:13.808362 (+ 18654us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.821963 (+ 13601us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.821996 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.822449 (+   453us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.112183 (+289734us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:14.112240 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.112340 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.077788 (+965448us) raft_consensus.cc:1685] finished
0709 12:24:15.077814 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.078013 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":288024,"threads_started":1,"wal-append.queue_time_us":454334}
I20260709 12:24:15.077318  7111 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.115322  7188 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.086256  7073 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:15.145535  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:15.151716  7198 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:15.165594  1879 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:15.174592  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:15.198220  1804 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: [15.12-15.12]
W20260709 12:24:15.175614  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 130039us
I20260709 12:24:15.092046  6837 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:15.118717  7181 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:24:15.191989  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.087152  7093 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:15.228875  7093 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.129007  7120 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.200657  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.229471  7093 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:15.229882  7120 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:15.200877  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:15.230636  7120 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.231133  7120 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.140710  6856 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:15.205732  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:15.206395  7043 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.782s	user 0.007s	sys 0.007s
I20260709 12:24:15.233285  7120 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.233826  7120 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:15.115403  7058 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.241s
I20260709 12:24:15.235100  7190 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:15.235853  7190 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:15.207831  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.237138  1802 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:15.215027  7201 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 12:24:15.242496  7205 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:15.243142  7205 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.193813  7067 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:15.216786  7188 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.250460  7188 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:15.250746  7063 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13.
W20260709 12:24:15.251296  7178 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.550s	user 0.000s	sys 0.004s
W20260709 12:24:15.219493  7002 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.864s	user 0.013s	sys 0.000s
I20260709 12:24:15.252419  7002 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:15.115331  6832 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.172387  7046 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:15.221807  7036 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.591s	user 0.006s	sys 0.006s
I20260709 12:24:15.222496  7207 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.144218  6963 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.178071  6945 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.225236  7198 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.202299  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2232) took 1718 ms. Trace:
W20260709 12:24:15.226212  6626 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.671s	user 0.007s	sys 0.004s
I20260709 12:24:15.226446  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.226621  6837 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.227864  7218 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.233456  7043 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.206552  7150 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:15.235229  1770 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.513s (SENT)
I20260709 12:24:15.303591  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:15.247081  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72580us
I20260709 12:24:15.237018  7190 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.196913  7203 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:15.299116  6968 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.269948  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2242) took 1683 ms. Trace:
I20260709 12:24:15.271279  1801 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:15.289007  1799 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:24:15.279124  1728 rpcz_store.cc:276] 0709 12:24:13.484176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.944884 (+460708us) service_pool.cc:224] Handling call
0709 12:24:13.945401 (+   517us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:24:14.103261 (+157860us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:24:14.103307 (+    46us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:14.104317 (+  1010us) log.cc:844] Serialized 78 byte log entry
0709 12:24:14.256079 (+151762us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:24:14.256136 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.256229 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.201964 (+945735us) raft_consensus.cc:1685] finished
0709 12:24:15.202006 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.202220 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68653,"raft.queue_time_us":68819,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":150,"thread_start_us":151218,"threads_started":1,"wal-append.queue_time_us":580348}
W20260709 12:24:15.317603  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.247769  7120 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:15.296892  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2251) took 1489 ms. Trace:
I20260709 12:24:15.256767  7067 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:15.276531  7036 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:15.310700  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:15.313206  2412 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:24:15.314292  6837 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.346727  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:15.315555  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:15.347996  2419 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 9.
I20260709 12:24:15.318039  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:15.303764  7218 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.324877  6977 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.325568  7150 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.279886  7201 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:15.340065  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:15.336405  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:15.334170  1725 rpcz_store.cc:276] 0709 12:24:13.807508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.533004 (+725496us) service_pool.cc:224] Handling call
0709 12:24:14.533469 (+   465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.590098 (+ 56629us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:14.658265 (+ 68167us) log.cc:844] Serialized 14 byte log entry
0709 12:24:15.169524 (+511259us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.170170 (+   646us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.170332 (+   162us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:15.170367 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.170445 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.296553 (+126108us) raft_consensus.cc:1685] finished
0709 12:24:15.296577 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.296821 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20319,"prepare.queue_time_us":52764,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":153600,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":257,"thread_start_us":415899,"threads_started":3,"wal-append.queue_time_us":530621}
I20260709 12:24:15.342614  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:15.345584  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:15.362303  1728 raft_consensus.cc:2368] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16.
I20260709 12:24:15.316996  7191 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.303071  6945 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.366102  2416 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
W20260709 12:24:15.349917  7176 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.565s	user 0.002s	sys 0.001s
W20260709 12:24:15.357751  7183 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.931s	user 0.011s	sys 0.017s
I20260709 12:24:15.327327  1726 rpcz_store.cc:276] 0709 12:24:13.586582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.987385 (+400803us) service_pool.cc:224] Handling call
0709 12:24:13.987734 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.028351 (+ 40617us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:24:14.028385 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.039168 (+ 10783us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.301011 (+261843us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:14.301058 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.301138 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.250505 (+949367us) spinlock_profiling.cc:243] Waited 6.15 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:15.256722 (+  6217us) raft_consensus.cc:1685] finished
0709 12:24:15.256744 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.269861 (+ 13117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34996,"raft.queue_time_us":41815,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":126,"spinlock_wait_cycles":14133504,"thread_start_us":261364,"threads_started":1,"wal-append.queue_time_us":583957}
I20260709 12:24:15.327183  7205 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:15.326628  6968 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.296068  6626 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.367996  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.350443  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:15.326745  7203 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.371042  6890 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:15.385406  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:15.389966  1725 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:15.389961  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:15.304112  7207 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.393267  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:15.421463  7207 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.422981  7125 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:24:15.387756  6879 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:15.424026  7125 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:15.424762  6879 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:15.425936  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.427356  6862 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:15.402654  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.407362  6945 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.407979  7218 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.428000  6862 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:15.410169  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 727) took 1459 ms. Trace:
I20260709 12:24:15.411927  2416 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20260709 12:24:15.401482  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:15.413074  6963 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.429706  7228 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.429297  6862 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.428324  1725 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:15.430253  7228 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.431454  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:15.432166  1725 raft_consensus.cc:2368] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13.
I20260709 12:24:15.432806  7228 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.432993  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:15.433403  6976 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.434034  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.429368  1728 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:15.435619  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.436187  1728 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
W20260709 12:24:15.418259  7090 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.589s	user 0.010s	sys 0.002s
I20260709 12:24:15.422712  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:15.394001  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 158788us
I20260709 12:24:15.413825  7191 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.444614  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.437158  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:15.521162  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 60479.6us
W20260709 12:24:15.697820  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 402532us
I20260709 12:24:15.685024  1799 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:24:15.460248  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.428944  2416 pending_rounds.cc:85] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 11
I20260709 12:24:15.512591  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.434675  1725 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:24:15.515767  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:15.428807  1731 rpcz_store.cc:276] 0709 12:24:13.950302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.632840 (+682538us) service_pool.cc:224] Handling call
0709 12:24:14.633280 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.633740 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.633770 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.634352 (+   582us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.832251 (+197899us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:14.832320 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.832440 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.409891 (+577451us) raft_consensus.cc:1685] finished
0709 12:24:15.409919 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.410085 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":52003,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":15671,"thread_start_us":197305,"threads_started":1,"wal-append.queue_time_us":736833}
W20260709 12:24:15.640805  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.705989  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.704334  2416 pending_rounds.cc:107] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260709 12:24:15.518908  1803 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:24:15.510924  1804 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:15.496227  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2229) took 2013 ms. Trace:
I20260709 12:24:15.439973  1878 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
W20260709 12:24:15.518465  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 43957us
I20260709 12:24:15.511602  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.710474  1807 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:15.706733  2416 raft_consensus.cc:2889] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:15.713838  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2239) took 2227 ms. Trace:
W20260709 12:24:15.713876  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 198183us
I20260709 12:24:15.372915  7201 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:15.604907  7168 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.663s	user 0.006s	sys 0.008s
I20260709 12:24:15.698724  1728 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
W20260709 12:24:15.437068  7186 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.619s	user 0.002s	sys 0.000s
I20260709 12:24:15.750438  1802 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
W20260709 12:24:15.708390  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.715271  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3042) took 1332 ms. Trace:
I20260709 12:24:15.719110  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.716797  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:15.381527  7150 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:15.747711  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:15.849368  1724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 730) took 1810 ms (client timeout 2095 ms). Trace:
I20260709 12:24:15.832399  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:15.687194  7216 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.707265  1729 rpcz_store.cc:276] 0709 12:24:13.483259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.869317 (+386058us) service_pool.cc:224] Handling call
0709 12:24:13.869676 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.880998 (+ 11322us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:13.881028 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.967682 (+ 86654us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.275589 (+307907us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:14.275629 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.278298 (+  2669us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.333445 (+1055147us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:15.444050 (+110605us) raft_consensus.cc:1685] finished
0709 12:24:15.444073 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.496151 (+ 52078us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19172,"raft.queue_time_us":733,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":7205,"spinlock_wait_cycles":8668928,"thread_start_us":307059,"threads_started":1,"wal-append.queue_time_us":697676}
W20260709 12:24:15.517112  6933 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.831s	user 0.009s	sys 0.002s
W20260709 12:24:15.513764  7081 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.999s	user 0.007s	sys 0.003s
I20260709 12:24:15.756364  6963 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:15.806133  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:15.870069  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 153354us
I20260709 12:24:15.615481  6766 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.830651  2419 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:24:15.812722  1802 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:24:15.820088  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.441045  7220 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:15.440929  7226 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:15.814285  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:15.808786  1880 rpcz_store.cc:276] 0709 12:24:14.358182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.536556 (+178374us) service_pool.cc:224] Handling call
0709 12:24:15.690084 (+1153528us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b5400072e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:15.690314 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":680475392}
I20260709 12:24:15.824492  1846 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:15.839767  1725 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:15.811018  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:15.844434  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.831867  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:15.809973  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.849522  1727 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:24:15.687003  7193 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:15.872888  7053 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.204s	user 0.015s	sys 0.000s
I20260709 12:24:15.557626  6862 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:15.849961  1724 rpcz_store.cc:269] 0709 12:24:14.038874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.321460 (+282586us) service_pool.cc:224] Handling call
0709 12:24:15.849117 (+1527657us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b540015cf54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:15.849293 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":457879936}
I20260709 12:24:15.857496  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 732) took 1817 ms. Trace:
I20260709 12:24:15.517956  6626 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:15.516089  7218 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.714566  1730 rpcz_store.cc:276] 0709 12:24:13.486314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.983432 (+497118us) service_pool.cc:224] Handling call
0709 12:24:13.983855 (+   423us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:14.012309 (+ 28454us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.012358 (+    49us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:14.027043 (+ 14685us) log.cc:844] Serialized 78 byte log entry
0709 12:24:14.330977 (+303934us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:14.331021 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.331107 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.253754 (+922647us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:15.511025 (+257271us) raft_consensus.cc:1685] finished
0709 12:24:15.511044 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.713768 (+202724us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":24940,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":12275,"spinlock_wait_cycles":8424704,"thread_start_us":189648,"threads_started":1,"wal-append.queue_time_us":278387}
I20260709 12:24:15.517976  7043 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.441854  6968 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:15.391943  6875 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.824512  7064 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.087s
I20260709 12:24:15.873375  1803 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:15.875437  7220 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.875327  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.884696  7226 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.833979  7208 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:15.793195  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 77534.5us
I20260709 12:24:15.905742  1807 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
W20260709 12:24:15.854801  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47977.9us
I20260709 12:24:15.916802  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:15.924499  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 100073us
I20260709 12:24:15.862674  6933 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.942454  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:15.923054  1723 rpcz_store.cc:276] 0709 12:24:14.039594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.681989 (+642395us) service_pool.cc:224] Handling call
0709 12:24:14.701676 (+ 19687us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.031007 (+329331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.038004 (+  6997us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.079173 (+ 41169us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.359679 (+280506us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:15.359715 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.359781 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.857170 (+497389us) raft_consensus.cc:1685] finished
0709 12:24:15.857202 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.857407 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":148991,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":502655,"threads_started":2,"wal-append.queue_time_us":595280}
I20260709 12:24:15.943348  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:15.918085  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:15.944029  1730 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:15.926272  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:15.944789  1802 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:15.927350  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.945670  1725 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
W20260709 12:24:15.922744  7100 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.592s	user 0.006s	sys 0.010s
I20260709 12:24:15.952046  7100 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.952957  6875 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:15.953822  7219 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:15.954525  7211 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:15.955122  7092 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:15.955219  7211 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.701326  7120 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.14, Last appended by leader: 14, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.930147  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.967897  1729 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:15.874172  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:15.440645  7067 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.962729  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:15.977530  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:15.932071  1805 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.015s	sys 0.036s
I20260709 12:24:15.399420  6856 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:15.978507  1805 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.016s	sys 0.036s
I20260709 12:24:15.964884  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:15.808784  7168 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.818874  7204 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.546s
I20260709 12:24:15.980755  7201 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:15.938745  7226 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:15.981550  7210 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:15.982040  7210 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:15.751997  7125 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.983460  7125 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:15.984071  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:15.950274  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:15.872066  7081 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.961704  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2247) took 2204 ms. Trace:
I20260709 12:24:15.868503  7216 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:15.861166  7150 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:15.965705  1770 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.014s	sys 0.039s
I20260709 12:24:15.878679  6766 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.985267  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.986454  1726 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:15.987679  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.981675  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.994431  1732 rpcz_store.cc:276] 0709 12:24:13.757122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.032795 (+275673us) service_pool.cc:224] Handling call
0709 12:24:14.033255 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.097817 (+ 64562us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.097865 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.226869 (+129004us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.353928 (+127059us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:14.353978 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.354050 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.436404 (+1082354us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b2400186c38. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d2a09d 00007f16ab95c10e 00007f16ab97388c 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221
0709 12:24:15.944732 (+508328us) raft_consensus.cc:1685] finished
0709 12:24:15.944756 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.961608 (+ 16852us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10017,"raft.queue_time_us":21996,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":31628,"spinlock_wait_cycles":7750528,"thread_start_us":76199,"threads_started":1,"wal-append.queue_time_us":394016}
I20260709 12:24:15.962544  1877 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:16.015790  1731 raft_consensus.cc:2410] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 9.
I20260709 12:24:15.928395  7053 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:15.999961  1730 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
W20260709 12:24:16.000238  1845 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.049s	sys 0.023s
I20260709 12:24:16.001521  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:15.935889  7220 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:16.004016  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61632us
W20260709 12:24:16.008419  1770 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.014s	sys 0.039s
W20260709 12:24:16.011204  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.954607  7100 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:16.039297  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 589590us
I20260709 12:24:16.014341  1728 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:16.040781  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:16.041108  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:16.042057  1803 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
W20260709 12:24:16.042656  1845 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.050s	sys 0.028s
I20260709 12:24:15.933697  7218 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:16.020334  1724 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:16.009720  6626 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:16.020318  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:16.050810  6626 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.050527  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:15.978607  7219 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:16.024561  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.047526  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:16.057176  1729 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 17
W20260709 12:24:16.047055  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.014398  7201 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:16.039616  7198 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.011s	sys 0.012s
I20260709 12:24:16.051240  6626 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:16.051889  1724 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:16.043190  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 612926us
I20260709 12:24:16.002714  7092 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:16.066982  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:16.088719  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:16.059417  1770 leader_election.cc:336] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.543s (SENT)
I20260709 12:24:16.089393  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:16.089954  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:16.075265  1800 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:16.077178  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 93199.4us
I20260709 12:24:16.048961  1801 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 9.
I20260709 12:24:16.089550  1807 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:16.090277  1730 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:16.073529  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:16.093915  1732 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:16.094179  1728 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:15.998853  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:16.090817  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.014508  6856 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:16.096721  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 97985.3us
I20260709 12:24:16.046914  6963 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:16.043637  7231 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.010449  6875 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.069s
W20260709 12:24:16.081503  7093 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.848s	user 0.009s	sys 0.005s
I20260709 12:24:16.063714  6890 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.960307  7208 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:16.104875  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:16.113306  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:16.115090  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 74385us
I20260709 12:24:16.116521  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:16.126541  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:16.121124  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.127207  1872 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:24:16.121431  1724 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 18 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:24:16.127823  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.121935  1730 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:16.123376  1731 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 16 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:24:16.129962  7221 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:16.130589  7221 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:16.131145  7221 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:16.115994  7093 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:16.132784  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:16.133535  1880 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:16.134158  7093 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:16.136207  1724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:16.134580  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:16.136919  1724 raft_consensus.cc:606] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:16.137936  1724 raft_consensus.cc:614] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:16.107090  7228 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:16.146235  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40200 (request call id 40) took 1189 ms. Trace:
I20260709 12:24:16.146860  1724 rpcz_store.cc:276] 0709 12:24:14.949196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.136087 (+1186891us) service_pool.cc:224] Handling call
0709 12:24:16.138443 (+  2356us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:16.107084  6825 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:16.148820  7222 consensus_queue.cc:1175] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 9 status { last_received { term: 9 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 9 index: 7 } }
I20260709 12:24:16.150017  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.152221  7196 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:16.155845  2415 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:16.158813  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 720) took 2248 ms. Trace:
I20260709 12:24:16.159304  1727 rpcz_store.cc:276] 0709 12:24:13.910374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.558606 (+648232us) service_pool.cc:224] Handling call
0709 12:24:15.849275 (+1290669us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.883313 (+ 34038us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.158561 (+275248us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b540015cf50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:16.158713 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2900128512}
I20260709 12:24:16.161294  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2254) took 1126 ms. Trace:
I20260709 12:24:16.161681  1727 rpcz_store.cc:276] 0709 12:24:15.034542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.160314 (+1125772us) service_pool.cc:224] Handling call
0709 12:24:16.160695 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.160907 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.160941 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:16.160989 (+    48us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:16.161025 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.161086 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.161197 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:16.103621  7064 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:16.182039  7073 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.980s	user 0.007s	sys 0.007s
I20260709 12:24:16.182848  1879 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:16.183617  7046 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.184227  7046 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:16.186221  7208 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:16.171197  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 48867.3us
I20260709 12:24:16.190634  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:16.172309  7228 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:16.177493  1730 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:16.177736  7053 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:16.179033  7168 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:16.179327  6933 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:16.179646  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2225) took 1125 ms. Trace:
I20260709 12:24:16.180047  6766 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:16.115319  7230 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:16.181361  7081 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:16.044778  7210 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:16.240123  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 723) took 1943 ms. Trace:
I20260709 12:24:16.213403  1807 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:16.241883  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:16.216045  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.216867  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:24:16.219173  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 159795us
I20260709 12:24:16.240598  1876 rpcz_store.cc:276] 0709 12:24:14.296438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.605991 (+309553us) service_pool.cc:224] Handling call
0709 12:24:14.645325 (+ 39334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.657310 (+ 11985us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.657343 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.661798 (+  4455us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.579525 (+917727us) spinlock_profiling.cc:243] Waited 64.5 ms on lock 0x7b18003b5c60. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:16.049779 (+470254us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:16.049842 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.050046 (+   204us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.233747 (+183701us) raft_consensus.cc:1685] finished
0709 12:24:16.233776 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.240050 (+  6274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33612,"raft.queue_time_us":52665,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":17363,"spinlock_wait_cycles":148237184,"thread_start_us":469970,"threads_started":1,"wal-append.queue_time_us":1539195}
W20260709 12:24:16.144696  6832 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.889s	user 0.013s	sys 0.021s
I20260709 12:24:16.225813  1805 rpcz_store.cc:276] 0709 12:24:15.054577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.054770 (+   193us) service_pool.cc:224] Handling call
0709 12:24:15.087723 (+ 32953us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:15.106693 (+ 18970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.106747 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:15.178592 (+ 71845us) log.cc:844] Serialized 52 byte log entry
0709 12:24:15.979086 (+800494us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:15.979139 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.979229 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.179392 (+200163us) raft_consensus.cc:1685] finished
0709 12:24:16.179416 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.179576 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":169,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":9652,"thread_start_us":688760,"threads_started":1,"wal-append.queue_time_us":836413}
I20260709 12:24:16.245610  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.227854  7201 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:16.246541  1801 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:16.116693  7231 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:16.244944  1770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.868s (SENT) [suppressed 26 similar messages]
I20260709 12:24:16.239527  7210 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:16.140049  6890 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:16.247423  1807 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:16.242697  7193 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:16.248342  7201 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:16.263163  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:16.249248  6832 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.993s	user 0.013s	sys 0.021s
I20260709 12:24:16.263950  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:16.280341  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.260174  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:16.285038  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.285629  1799 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:16.291455  7230 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:24:16.297549  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.299187  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.299386  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:16.271044  7043 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:16.273895  7168 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:16.335409  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:16.339354  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 295359us
W20260709 12:24:16.342267  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.340893  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:16.353353  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:16.354478  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:16.355314  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:16.360335  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:16.288602  7201 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:16.362373  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:16.372270  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:16.385232  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.419144  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.440536  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.385540  1872 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:16.477876  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:16.483713  1874 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:16.474488  1729 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:16.486413  1729 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:16.339646  7230 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:16.482465  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:16.479653  1798 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:16.541410  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:16.541757  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.012s	sys 0.026s
I20260709 12:24:16.482486  1873 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
W20260709 12:24:16.491897  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.478703  1805 raft_consensus.cc:2376] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11. Re-sending same reply.
W20260709 12:24:16.507339  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 359903us
W20260709 12:24:16.508348  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 83455us
I20260709 12:24:16.370126  6766 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:16.488499  7204 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:16.551158  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:16.419682  7240 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:24:16.287444  7228 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:16.528887  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3175) took 1435 ms. Trace:
W20260709 12:24:16.542403  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.012s	sys 0.026s
I20260709 12:24:16.544355  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.373317  7168 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:16.418912  7093 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:16.442536  7217 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:16.547351  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64999.2us
I20260709 12:24:16.283967  7219 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:16.419236  7207 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.130s
I20260709 12:24:16.457008  7218 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:16.550212  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:16.508741  7150 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:16.565474  1872 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:16.528602  7230 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:16.560839  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 598109us
W20260709 12:24:16.454522  7238 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.172s	user 0.001s	sys 0.001s
I20260709 12:24:16.564285  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:16.551810  6766 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.564065  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:16.579736  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:16.567201  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 195046us
I20260709 12:24:16.569588  1805 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:16.559718  1729 rpcz_store.cc:276] 0709 12:24:15.051742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.485817 (+1434075us) service_pool.cc:224] Handling call
0709 12:24:16.486180 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.486949 (+   769us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.487157 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:16.570153  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.552026  6963 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:24:16.600605  1802 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.639s	user 0.007s	sys 0.007s
I20260709 12:24:16.574935  6856 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:16.565105  7093 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.564922  7217 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.551005  6945 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:16.559432  7203 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.010s	sys 0.034s
W20260709 12:24:16.591267  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:16.606251  7203 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.011s	sys 0.035s
I20260709 12:24:16.579370  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:16.587858  1807 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:16.601238  1802 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:16.583891  1697 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:16.570636  7230 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:16.578011  6862 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.652s	user 0.004s	sys 0.006s
I20260709 12:24:16.581924  6766 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:16.601024  6963 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.570185  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:16.607162  7092 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:24:16.605129  7217 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:16.647255  1799 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:16.605201  7093 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:16.610246  7190 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:16.686463  1728 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.580s	user 0.010s	sys 0.006s
I20260709 12:24:16.643055  6963 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:16.689059  1728 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:24:16.634677  7203 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:16.609017  6945 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:16.694001  1805 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:24:16.696113  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3183) took 1458 ms. Trace:
W20260709 12:24:16.696333  7247 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:24:16.704967  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3176) took 1625 ms. Trace:
I20260709 12:24:16.688505  7168 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:16.708228  1724 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 12:24:16.719805  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.730666  1728 rpcz_store.cc:276] 0709 12:24:15.237225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.044298 (+807073us) service_pool.cc:224] Handling call
0709 12:24:16.696044 (+651746us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:16.736518  1732 rpcz_store.cc:276] 0709 12:24:15.079587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.703948 (+1624361us) service_pool.cc:224] Handling call
0709 12:24:16.704351 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.704613 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.704639 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:16.704681 (+    42us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:16.704726 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.704783 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.704901 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:16.723929  7203 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.769642  1732 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:16.759799  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2256) took 1724 ms. Trace:
I20260709 12:24:16.728904  7046 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:16.770557  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3191) took 1418 ms. Trace:
I20260709 12:24:16.774055  1732 rpcz_store.cc:276] 0709 12:24:15.352436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.741480 (+1389044us) service_pool.cc:224] Handling call
0709 12:24:16.758936 (+ 17456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.770239 (+ 11303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.770489 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:16.756582  7245 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.216s	user 0.002s	sys 0.000s
I20260709 12:24:16.784248  1804 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:24:16.770359  1727 rpcz_store.cc:276] 0709 12:24:15.035243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.162675 (+1127432us) service_pool.cc:224] Handling call
0709 12:24:16.178300 (+ 15625us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.178493 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:16.485796 (+307303us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.759430 (+273634us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:16.759493 (+    63us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:16.759538 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.759608 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.759727 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":393119,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":105813,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":111,"thread_start_us":580139,"threads_started":3,"wal-append.queue_time_us":263107}
I20260709 12:24:16.770723  7218 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:16.771320  7203 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:16.803959  6626 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.697s	user 0.011s	sys 0.003s
I20260709 12:24:16.789458  7216 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:16.837031  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1165) took 1007 ms. Trace:
I20260709 12:24:16.839805  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2257) took 1804 ms. Trace:
I20260709 12:24:16.840636  1724 rpcz_store.cc:276] 0709 12:24:15.035551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.170250 (+1134699us) service_pool.cc:224] Handling call
0709 12:24:16.707870 (+537620us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.737129 (+ 29259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.839387 (+102258us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b5400435dd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:16.839685 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1226130176}
I20260709 12:24:16.811565  7211 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:16.870292  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3195) took 1081 ms. Trace:
I20260709 12:24:16.879752  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:16.880100  1724 rpcz_store.cc:276] 0709 12:24:15.789118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.842287 (+1053169us) service_pool.cc:224] Handling call
0709 12:24:16.869452 (+ 27165us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.869740 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.869782 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:16.869853 (+    71us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:16.869908 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.869989 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.870199 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:16.881459  1804 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:16.888116  7120 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:16.892884  7251 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.263s	user 0.001s	sys 0.001s
I20260709 12:24:16.895979  6963 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:16.899320  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2143) took 1454 ms. Trace:
I20260709 12:24:16.899993  1806 rpcz_store.cc:276] 0709 12:24:15.444950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.445078 (+   128us) service_pool.cc:224] Handling call
0709 12:24:15.593388 (+148310us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:24:15.834725 (+241337us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.834771 (+    46us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:24:15.868825 (+ 34054us) log.cc:844] Serialized 104 byte log entry
0709 12:24:16.277922 (+409097us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:24:16.277963 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.278045 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.898945 (+620900us) raft_consensus.cc:1685] finished
0709 12:24:16.898976 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.899221 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85422,"raft.queue_time_us":103929,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":136,"thread_start_us":339667,"threads_started":1,"wal-append.queue_time_us":460754}
W20260709 12:24:16.908809  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149917us
I20260709 12:24:16.838212  2410 rpcz_store.cc:276] 0709 12:24:15.829209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.830920 (+  1711us) service_pool.cc:224] Handling call
0709 12:24:15.831535 (+   615us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.832031 (+   496us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.832071 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.832642 (+   571us) log.cc:844] Serialized 26 byte log entry
0709 12:24:16.258383 (+425741us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:16.258490 (+   107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.258588 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.836784 (+578196us) raft_consensus.cc:1685] finished
0709 12:24:16.836807 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.836965 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":165,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"thread_start_us":407256,"threads_started":1,"wal-append.queue_time_us":718084}
W20260709 12:24:16.914938  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.002s	sys 0.026s
W20260709 12:24:16.915318  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.003s	sys 0.026s
I20260709 12:24:16.910817  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:16.920825  1872 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:24:16.840338  6626 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:16.929400  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.935004  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2259) took 1898 ms. Trace:
I20260709 12:24:16.935386  1730 rpcz_store.cc:276] 0709 12:24:15.036134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.196396 (+1160262us) service_pool.cc:224] Handling call
0709 12:24:16.196714 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.196941 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:16.368614 (+171673us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.925915 (+557301us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.934579 (+  8664us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:16.934631 (+    52us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:16.934707 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.934808 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.934939 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18006,"prepare.queue_time_us":386918,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":105,"raft.run_wall_time_us":7755,"thread_start_us":686166,"threads_started":2}
I20260709 12:24:16.922434  6933 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:16.944366  7138 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:24:16.945322  7274 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:24:16.947719  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.948421  1730 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:24:16.948786  2418 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:16.949772  6875 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:16.951138  6626 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:16.937927  7221 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.806s	user 0.015s	sys 0.000s
I20260709 12:24:16.960700  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:16.960922  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.961788  1802 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:16.936645  7273 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:16.964251  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:16.973529  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:16.974153  1802 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:16.981181  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:16.984148  1803 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.843s	user 0.003s	sys 0.006s
I20260709 12:24:16.984573  1803 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:16.987919  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.988639  1805 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:16.990350  7196 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:16.990993  7196 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:16.999652  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 728) took 2650 ms (client timeout 1959 ms). Trace:
I20260709 12:24:17.016242  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:16.988768  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:17.019488  1806 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:24:16.956348  6963 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:17.017954  1877 rpcz_store.cc:269] 0709 12:24:14.330402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.351120 (+ 20718us) service_pool.cc:224] Handling call
0709 12:24:16.980166 (+2629046us) spinlock_profiling.cc:243] Waited 1.53 s on lock 0x7b5400025b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:16.980353 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3517533952}
I20260709 12:24:17.022084  1802 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
W20260709 12:24:16.986630  7275 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.104s	user 0.002s	sys 0.000s
I20260709 12:24:17.026358  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:17.027320  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3198) took 1065 ms. Trace:
I20260709 12:24:17.031687  1724 rpcz_store.cc:276] 0709 12:24:15.962431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.904921 (+942490us) service_pool.cc:224] Handling call
0709 12:24:16.906650 (+  1729us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.910100 (+  3450us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:16.925551 (+ 15451us) log.cc:844] Serialized 14 byte log entry
0709 12:24:17.026745 (+101194us) log.cc:844] Serialized 14 byte log entry
0709 12:24:17.026959 (+   214us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:17.027007 (+    48us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:17.027053 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.027137 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.027254 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":17357,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":2388,"thread_start_us":100742,"threads_started":1}
W20260709 12:24:17.020583  7220 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.966s	user 0.013s	sys 0.003s
I20260709 12:24:17.035128  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:17.025966  7125 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.042s	user 0.011s	sys 0.004s
I20260709 12:24:17.015784  6875 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:17.040499  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 471441us
I20260709 12:24:17.040657  1724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:17.041960  1724 raft_consensus.cc:606] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:24:17.028321  7138 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:24:17.025291  7273 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:16.962450  7221 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:17.032433  7201 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.611s	user 0.014s	sys 0.000s
I20260709 12:24:17.063035  1807 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:24:17.042424  1724 raft_consensus.cc:614] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:17.047916  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 464078us
I20260709 12:24:17.028970  6945 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:17.054404  1731 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 15 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:17.043301  6875 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.067688  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:17.069527  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.046747  6963 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:17.071466  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.059221  7273 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.072979  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:17.073858  1873 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:17.071559  7046 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.074746  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.075167  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.081846  1724 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:24:17.082682  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.082813  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3204) took 1099 ms. Trace:
I20260709 12:24:17.084432  1724 rpcz_store.cc:276] 0709 12:24:15.983734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.080070 (+1096336us) service_pool.cc:224] Handling call
0709 12:24:17.081443 (+  1373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.082480 (+  1037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.082717 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:17.075305  6875 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:17.084266  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57996.8us
I20260709 12:24:17.088958  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.095557  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.058211  7223 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:17.058669  7138 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:17.082023  7226 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.100s	user 0.013s	sys 0.002s
I20260709 12:24:17.098541  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:17.076398  6963 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.101279  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.061378  7201 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.042999  7220 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.090538  7228 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:17.112485  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 732) took 1766 ms (client timeout 1790 ms). Trace:
I20260709 12:24:17.083639  6945 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:17.112962  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:17.104913  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.127544  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:17.083914  7273 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:17.111598  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3187) took 1873 ms (client timeout 1811 ms). Trace:
W20260709 12:24:17.131479  1726 rpcz_store.cc:269] 0709 12:24:15.238334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.422279 (+183945us) service_pool.cc:224] Handling call
0709 12:24:17.111310 (+1689031us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b54002b7fd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:17.111512 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":659606784}
W20260709 12:24:17.128247  1879 rpcz_store.cc:269] 0709 12:24:15.346315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.346471 (+   156us) service_pool.cc:224] Handling call
0709 12:24:17.112266 (+1765795us) spinlock_profiling.cc:243] Waited 835 ms on lock 0x7b54002dfad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:17.112392 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1920834816}
W20260709 12:24:17.107436  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 60303.5us
I20260709 12:24:17.124791  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:17.136132  7278 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
W20260709 12:24:17.110447  7257 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.311s	user 0.000s	sys 0.002s
I20260709 12:24:17.114719  1877 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 10: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:17.107017   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:17.099993  7223 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.130048  1874 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:17.111263  6963 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:17.140045  1807 raft_consensus.cc:2368] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10.
I20260709 12:24:17.144131  2413 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:17.140334  1845 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.926s (SENT)
W20260709 12:24:17.155499  7280 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:24:17.155117  7203 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.091028  7046 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.159159  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2260) took 2102 ms. Trace:
I20260709 12:24:17.160602  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 734) took 1038 ms. Trace:
I20260709 12:24:17.159621  1729 rpcz_store.cc:276] 0709 12:24:15.056544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.605801 (+1549257us) service_pool.cc:224] Handling call
0709 12:24:16.634276 (+ 28475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:16.777261 (+142985us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:24:16.777297 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:16.795568 (+ 18271us) log.cc:844] Serialized 26 byte log entry
0709 12:24:16.919675 (+124107us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:16.919729 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.919827 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:17.158882 (+239055us) raft_consensus.cc:1685] finished
0709 12:24:17.158906 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.159063 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101331,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"spinlock_wait_cycles":9088,"thread_start_us":252413,"threads_started":2,"wal-append.queue_time_us":187662}
I20260709 12:24:17.140782  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:17.161005  1878 rpcz_store.cc:276] 0709 12:24:16.121953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.122139 (+   186us) service_pool.cc:224] Handling call
0709 12:24:16.122474 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:16.122713 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.122741 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:16.135380 (+ 12639us) log.cc:844] Serialized 26 byte log entry
0709 12:24:16.544413 (+409033us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:16.544475 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.544597 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:17.160362 (+615765us) raft_consensus.cc:1685] finished
0709 12:24:17.160386 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.160530 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":395678,"threads_started":1,"wal-append.queue_time_us":608967}
W20260709 12:24:17.159962  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.163790  1804 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:17.165078  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.166098  1804 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:17.166697  1873 raft_consensus.cc:2376] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15. Re-sending same reply.
I20260709 12:24:17.182826  2382 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:17.184046  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:17.158149  7093 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:17.138465  7276 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
I20260709 12:24:17.190537  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:17.188781  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.188898  1878 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:24:17.191742  1732 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:24:17.188190  2382 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:17.137414  7283 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:17.207270  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:17.193508  7193 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:17.141672  7216 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.119436  7226 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.145282  7223 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:17.251160  1872 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:17.252009  7093 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:17.252276  6766 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.579s	user 0.013s	sys 0.000s
I20260709 12:24:17.252892  6766 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.254982  1727 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:17.214032  1880 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:17.200608  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:17.256769  1874 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:17.234903  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.257762  1807 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:17.192059  6626 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:17.259011  6626 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.259449  6626 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:17.188791  7046 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:17.215773  7283 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.239280  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:17.264389  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 124096us
I20260709 12:24:17.264726  2412 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:17.265314  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.193701  7207 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:17.239974  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:17.274362  2415 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:17.241218  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.210871  7138 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.246167  7221 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:17.246922  7216 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.237915   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:17.249066  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.201231  7285 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:17.249548  7226 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:17.271521  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.272656  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.277200  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:17.280092  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 145727us
I20260709 12:24:17.281206  2411 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
W20260709 12:24:17.281364  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44734.1us
I20260709 12:24:17.289119  1802 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:17.142216  7228 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.283686  1804 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:17.240697  7217 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.558s	user 0.010s	sys 0.002s
I20260709 12:24:17.204622  7267 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:17.216445  7090 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:17.293565  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.239950  7220 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.295425  2410 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:24:17.293308  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:17.297470  2416 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:17.297650  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.300318  1807 raft_consensus.cc:2376] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16. Re-sending same reply.
I20260709 12:24:17.302801  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.302902  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.304064  1802 raft_consensus.cc:2376] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13. Re-sending same reply.
W20260709 12:24:17.291622  7287 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
W20260709 12:24:17.306401  7288 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:24:17.308166  7283 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:17.303220  7090 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:17.297287  7217 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.313131  7220 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:17.313448  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:17.333442  2411 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:17.334388  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.287904  7273 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.324518  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.336350  1802 raft_consensus.cc:2376] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13. Re-sending same reply.
I20260709 12:24:17.337553  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.338531  1802 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:24:17.314877  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.342902  2412 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:17.326058  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:17.330410  6856 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.725s	user 0.007s	sys 0.007s
I20260709 12:24:17.344842  1804 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:17.346261  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.347857  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.349287  1804 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:24:17.290957  7285 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:17.388078  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50337.7us
I20260709 12:24:17.425460  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:17.433586  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.435557  1804 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:17.444228  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.445114  1802 raft_consensus.cc:2376] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13. Re-sending same reply.
I20260709 12:24:17.401500  7111 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:17.416967  6856 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:24:17.447947  7289 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:24:17.448338  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.342947  7220 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.341281  7273 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.406558  6875 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.362171  6766 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.368698  7193 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:17.440088  7290 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:24:17.403019  7046 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 12:24:17.446707  7201 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:17.470877  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.397517  7196 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.475312  2412 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:17.505721  2411 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:17.507210  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.458424  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3048) took 3098 ms. Trace:
W20260709 12:24:17.393646  7230 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.736s	user 0.012s	sys 0.002s
I20260709 12:24:17.479737  1804 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:17.436542  7211 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:24:17.497563  7285 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.475919  7203 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.499100  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:59174
I20260709 12:24:17.511188  1729 raft_consensus.cc:606] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Received request to transfer leadership
I20260709 12:24:17.511561  1729 raft_consensus.cc:614] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:17.499563  7217 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.499466  7292 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:17.500530  7193 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.501628  7273 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.475095  7111 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:17.535581  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 246081us
I20260709 12:24:17.503228  6875 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:17.488474  7263 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.616s	user 0.004s	sys 0.002s
I20260709 12:24:17.432637  7221 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 12:24:17.483206  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.572031  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.508069  1875 rpcz_store.cc:276] 0709 12:24:14.360133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.655732 (+295599us) service_pool.cc:224] Handling call
0709 12:24:15.441008 (+785276us) spinlock_profiling.cc:243] Waited 8.54 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:24:15.441288 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:15.856287 (+414999us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:16.248223 (+391936us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.562081 (+313858us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.707484 (+145403us) spinlock_profiling.cc:243] Waited 5.42 ms on lock 0x7b18004ba060. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:17.036431 (+328947us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b18004ba060. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:24:17.036633 (+   202us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:17.036705 (+    72us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:17.036780 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.456779 (+419999us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x7b18004ba060. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419
0709 12:24:17.457805 (+  1026us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b5400117e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:17.457883 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.458251 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":536643,"prepare.queue_time_us":613090,"prepare.run_cpu_time_us":243,"prepare.run_wall_time_us":5183,"raft.queue_time_us":478450,"raft.run_cpu_time_us":328,"raft.run_wall_time_us":14447,"spinlock_wait_cycles":997563136,"thread_start_us":670877,"threads_started":2,"wal-append.queue_time_us":448368}
I20260709 12:24:17.444898  7090 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.575297  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:17.515297  1806 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:17.515921  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.517768  7223 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.503935  7296 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:17.521142  6890 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:17.522461  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3201) took 1539 ms. Trace:
W20260709 12:24:17.524649  7210 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.260s	user 0.008s	sys 0.005s
W20260709 12:24:17.525372  7269 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.620s	user 0.000s	sys 0.006s
I20260709 12:24:17.528877  6766 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:17.529091  1803 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:24:17.529477  7196 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:17.530567  6626 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:17.592830  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 146624us
I20260709 12:24:17.536410  7111 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:17.581813  7301 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
W20260709 12:24:17.551241  2418 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1163) took 1722 ms (client timeout 2092 ms). Trace:
I20260709 12:24:17.552313  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2264) took 1959 ms. Trace:
I20260709 12:24:17.579734  7058 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:17.615145  2418 rpcz_store.cc:269] 0709 12:24:15.828474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.831470 (+  2996us) service_pool.cc:224] Handling call
0709 12:24:17.551056 (+1719586us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54006b0154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:17.551174 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2451307264}
I20260709 12:24:17.566953  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:17.572867  1804 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:17.576769  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:17.583102  1730 rpcz_store.cc:276] 0709 12:24:15.982853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.955552 (+972699us) service_pool.cc:224] Handling call
0709 12:24:16.983813 (+ 28261us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:17.023678 (+ 39865us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:17.023717 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:17.025724 (+  2007us) log.cc:844] Serialized 26 byte log entry
0709 12:24:17.209092 (+183368us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:17.209151 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.209250 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:17.522242 (+312992us) raft_consensus.cc:1685] finished
0709 12:24:17.522264 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.522401 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36051,"raft.queue_time_us":39587,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":4094,"thread_start_us":152926,"threads_started":1,"wal-append.queue_time_us":264677}
I20260709 12:24:17.583475  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2160) took 1415 ms. Trace:
I20260709 12:24:17.668959  1800 rpcz_store.cc:276] 0709 12:24:16.167717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.211197 (+ 43480us) service_pool.cc:224] Handling call
0709 12:24:16.211602 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:16.220564 (+  8962us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.220596 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:16.221161 (+   565us) log.cc:844] Serialized 26 byte log entry
0709 12:24:16.561092 (+339931us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:16.561173 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.561284 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:17.462098 (+900814us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:17.577715 (+115617us) raft_consensus.cc:1685] finished
0709 12:24:17.577749 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.583380 (+  5631us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1288,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"spinlock_wait_cycles":24251776,"thread_start_us":339425,"threads_started":1,"wal-append.queue_time_us":562230}
W20260709 12:24:17.670902  7269 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:24:17.513669  7230 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.672070  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.673954  7230 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.563100  6862 raft_consensus.cc:2764] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20260709 12:24:17.675258  7308 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.064s	user 0.002s	sys 0.000s
I20260709 12:24:17.531879  7220 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.606678  1881 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:17.610178  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.679636  2410 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:17.545437  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:17.683215  2416 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:17.684393  7307 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:24:17.689553  1728 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:24:17.615854  1727 rpcz_store.cc:276] 0709 12:24:15.593115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.788356 (+1195241us) service_pool.cc:224] Handling call
0709 12:24:16.795159 (+  6803us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.255376 (+460217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.552075 (+296699us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b54002b75d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:17.552255 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1057033088}
I20260709 12:24:17.564062  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:17.616981  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.618726  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2263) took 2241 ms. Trace:
I20260709 12:24:17.642495  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.692044  2411 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:24:17.692521  2412 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:17.609259  7120 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.705s	user 0.014s	sys 0.000s
I20260709 12:24:17.586683  7211 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:17.602958  7217 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:17.656563  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62161.2us
I20260709 12:24:17.591825  7226 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:17.662209  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:17.664242  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.666497  2417 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:17.535581  7043 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:17.646396  7092 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.963s	user 0.011s	sys 0.004s
I20260709 12:24:17.692044  1732 rpcz_store.cc:276] 0709 12:24:15.377867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.775141 (+1397274us) service_pool.cc:224] Handling call
0709 12:24:16.783193 (+  8052us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:17.192239 (+409046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.618293 (+426054us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b54002b6bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:17.618626 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":938948992}
I20260709 12:24:17.693750  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3202) took 1710 ms. Trace:
W20260709 12:24:17.694311  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 157581us
I20260709 12:24:17.649039  7111 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:17.706476  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:17.716410  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:17.662083  7288 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
W20260709 12:24:17.710883  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 144057us
I20260709 12:24:17.643455  7228 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.713339  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.707828  7043 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:17.607164  6890 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.088s
I20260709 12:24:17.714524  6862 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:17.665122  6968 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:24:17.713161  7271 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.740934  1877 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:17.723826  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:17.722602  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:17.733068  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:17.725782  7217 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.607447  6766 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.715287  7092 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:24:17.617154  6963 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.723495  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:17.741926  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:17.747386  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 170685us
I20260709 12:24:17.764896  1880 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:17.765051  1804 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:17.747835  1875 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:24:17.725507  7295 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:24:17.752409  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:17.695801  7216 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.725864  7226 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.709964  7211 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:17.690380  7058 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.719275  6856 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:17.790134  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:17.791536  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.548s (SENT)
I20260709 12:24:17.774417  7303 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:17.820941  6933 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:17.821573  6933 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:17.798013  7290 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:17.824946  6626 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.826902  6766 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:17.830291  6963 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.809962  7312 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:24:17.796123  7293 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.485s	user 0.005s	sys 0.000s
I20260709 12:24:17.799688  7058 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:17.788079  7211 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:17.802119  7217 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:17.760735  6862 consensus_queue.cc:1243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:17.760819  7271 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.759703  7292 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:24:17.796164  7046 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:24:17.804798  7223 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:17.773005  7230 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.818885  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:17.834448  2418 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
W20260709 12:24:17.839668  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:17.857448  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 92364.6us
W20260709 12:24:17.860288  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68773.2us
I20260709 12:24:17.847673  6862 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.833958  7303 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:17.816486  7196 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:24:17.849817  7312 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:17.865028  6933 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.885440  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.861740  7217 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:24:17.889991  2413 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1152) took 2576 ms (client timeout 3245 ms). Trace:
I20260709 12:24:17.856738  7230 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.894265  2416 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:17.903308  2418 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:17.905929  2413 rpcz_store.cc:269] 0709 12:24:15.307345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.307543 (+   198us) service_pool.cc:224] Handling call
0709 12:24:17.883101 (+2575558us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b54007ce554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:17.883276 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4139952896}
I20260709 12:24:17.916221  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:17.918728  2418 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:24:17.887343  7198 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:24:17.920814  2419 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:24:17.917714  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 142563us
I20260709 12:24:17.924537  2413 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:17.892083  6862 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.926360  1845 leader_election.cc:400] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10.
I20260709 12:24:17.928902  7277 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:17.932318  1845 leader_election.cc:403] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260709 12:24:17.962159  7277 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:17.963186  7277 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.964345  7277 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:17.964804  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:17.965585  1725 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:17.938716  6933 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:17.945266  6875 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.967671  6875 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:17.969905  7316 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:24:17.951627  7292 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:17.917229  7231 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.971017  7292 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:17.971551  7303 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.972276  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:17.972079  7292 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:17.972863  7226 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:17.973378  7292 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:17.973783  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:17.973904  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:17.974443  1726 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:17.974628  1724 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:17.975232  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:17.955574  7312 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:17.972906  1729 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:17.958899  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.959388  1806 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:17.976859  1804 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:17.958006  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.959758  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:17.978075  1803 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:17.960671  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.979244  1799 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:17.960222  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:17.980551  1872 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:24:17.965651  7317 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:24:17.940395  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:17.957962  1805 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:17.970432  6875 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:17.982409  1800 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:17.999529  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.000406  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:18.003805  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:18.010246  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:18.276077  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:18.276796  1806 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:18.010217  1799 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:24:18.260355  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.142293  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.022616  1804 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:18.180791  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:18.178802  2418 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 15 index: 13. (term mismatch)
I20260709 12:24:18.283792  2418 pending_rounds.cc:85] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 12
W20260709 12:24:18.031664  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 142037us
I20260709 12:24:18.059943  1726 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
W20260709 12:24:18.285975  2383 leader_election.cc:336] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.883s (SENT)
I20260709 12:24:18.281373  1807 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:18.285354  2418 pending_rounds.cc:107] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 14.13
I20260709 12:24:18.287262  2418 raft_consensus.cc:2889] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:18.069496  1803 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:18.133031  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 733) took 4093 ms. Trace:
I20260709 12:24:18.289004  1731 rpcz_store.cc:276] 0709 12:24:14.039866 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.148307 (+2108441us) service_pool.cc:224] Handling call
0709 12:24:16.202960 (+ 54653us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.072096 (+869136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.132657 (+1060561us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54001905d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:18.132938 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1957564800}
I20260709 12:24:18.090931  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:18.152374  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.179487  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2161) took 1948 ms. Trace:
I20260709 12:24:18.286530  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:18.178969  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2261) took 3094 ms. Trace:
I20260709 12:24:18.112102  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.291903  1878 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 9.
I20260709 12:24:18.293422  2419 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:18.292652  1728 rpcz_store.cc:276] 0709 12:24:15.084781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.739592 (+1654811us) service_pool.cc:224] Handling call
0709 12:24:16.740104 (+   512us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:17.690413 (+950309us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.178548 (+488135us) spinlock_profiling.cc:243] Waited 949 ms on lock 0x7b5400414954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:18.178880 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2182400768}
I20260709 12:24:18.027211  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:18.291016  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:18.292119  1801 rpcz_store.cc:276] 0709 12:24:16.231435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.249103 (+ 17668us) service_pool.cc:224] Handling call
0709 12:24:16.269396 (+ 20293us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:16.269864 (+   468us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.269894 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:16.334011 (+ 64117us) log.cc:844] Serialized 52 byte log entry
0709 12:24:16.831324 (+497313us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:16.831358 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.831414 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.947409 (+115995us) raft_consensus.cc:1685] finished
0709 12:24:16.947435 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.179160 (+1231725us) spinlock_profiling.cc:243] Waited 5.29 ms on lock 0x7b54001d4f50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:18.179404 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":136,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":5701,"spinlock_wait_cycles":12169088,"thread_start_us":351639,"threads_started":1,"wal-append.queue_time_us":463804,"wal-append.run_cpu_time_us":3055,"wal-append.run_wall_time_us":1063482}
W20260709 12:24:18.291615  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 409748us
I20260709 12:24:18.145571  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:18.091590  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.059692  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:18.300884  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:18.061657  1725 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:24:18.257815  1880 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:18.295445  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:18.311123  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 220368us
I20260709 12:24:18.316141  1729 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
W20260709 12:24:18.253129  7309 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.656s	user 0.000s	sys 0.004s
I20260709 12:24:18.297580  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:18.321187  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.113830  6890 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:18.151458  7270 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.247s	user 0.000s	sys 0.007s
I20260709 12:24:18.323953  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:18.274766  7325 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.144s	user 0.000s	sys 0.000s
I20260709 12:24:17.977522  7218 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:18.327629  1730 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:18.334829  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48861.1us
I20260709 12:24:18.339342  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.176416  7201 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:24:18.344571  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.153016  7313 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:24:18.345836  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48374.3us
I20260709 12:24:18.347376  1877 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:24:18.327669  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.347422  7120 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:18.367089  1726 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:18.325393  7222 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:18.089268  7043 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:18.327028  7046 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.339622  7218 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:18.392143  1807 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:18.297184  7230 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:18.386401  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.349174  7111 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:18.382845  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66909.2us
I20260709 12:24:18.348956  7201 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:18.396061  1730 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:18.397238  1731 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:18.354297  7313 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:18.372047  6875 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:18.363274  7228 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.401400  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.328166  7320 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 12:24:18.372710  7203 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:18.397217  7196 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.410104  7203 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:18.410461  7196 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:18.412684  7196 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:18.415452  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.415777  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:18.416326  1726 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:18.416555  1730 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:18.417546  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:18.382733  7222 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.418452  1730 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:24:18.408114  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 484859us
I20260709 12:24:18.419464  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.409282  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:18.423640  1732 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:18.423983  1878 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:18.421836  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2162) took 2160 ms. Trace:
W20260709 12:24:18.425779  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 247500us
I20260709 12:24:18.351747  6890 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:18.428009  1880 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:18.431488  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:18.433826  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:18.424757  1798 rpcz_store.cc:276] 0709 12:24:16.261507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.561917 (+300410us) service_pool.cc:224] Handling call
0709 12:24:16.576083 (+ 14166us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:16.576600 (+   517us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.576639 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:16.577216 (+   577us) log.cc:844] Serialized 26 byte log entry
0709 12:24:16.723540 (+146324us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:16.723588 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.723667 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.150671 (+1427004us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:18.405694 (+255023us) raft_consensus.cc:1685] finished
0709 12:24:18.405746 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.421756 (+ 16010us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4167,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":21402,"spinlock_wait_cycles":2920704,"thread_start_us":145846,"threads_started":1,"wal-append.queue_time_us":229517}
I20260709 12:24:18.435300  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.435869  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.436858  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:18.434659  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.442590  1731 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
W20260709 12:24:18.425963  7211 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.574s	user 0.012s	sys 0.002s
I20260709 12:24:18.408455  7310 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10.
I20260709 12:24:18.447911  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.403853  7327 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:24:18.422448  6862 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:18.427515  7222 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:18.425369  6875 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:18.451985  7053 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20260709 12:24:18.464390  7313 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:18.465101  1730 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:18.466137  7313 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:18.466300  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:18.467254  1730 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:18.454902  1803 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:24:18.469974  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:18.448014  1770 connection.cc:256] RPC call timeout handler was delayed by -1.7852196039998489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3245s for a response
I20260709 12:24:18.399173  7046 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:18.470103  1806 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:18.471283  1731 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
W20260709 12:24:18.471486  1770 connection.cc:256] RPC call timeout handler was delayed by -1.7509759230001691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
W20260709 12:24:18.476001  1770 connection.cc:256] RPC call timeout handler was delayed by -1.6743454069996915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32010000000000005s for a response
I20260709 12:24:18.477859  1726 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:18.454525  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:18.405233  7111 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:18.474967  1732 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:18.480089  1730 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:18.481418  1730 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:18.450484  7290 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.626s	user 0.011s	sys 0.002s
I20260709 12:24:18.482535  1730 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:18.485276  7203 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:18.489868  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:18.490686  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:18.483007  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:18.491820  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:18.422709  7228 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:18.492278  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.484364  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:18.485351  7258 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:18.491912  1798 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:18.479177  2410 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:18.461869  7221 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:18.502089  1731 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:24:18.482916  1732 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:24:18.442824  7328 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:24:18.500799  7228 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:18.450699  7327 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:18.506935  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:18.507774  1728 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 10: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:18.509261  1725 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:18.505318  7221 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:18.511335  7221 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:18.512842  1770 connection.cc:256] RPC call timeout handler was delayed by -1.4940316890001668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20260709 12:24:18.515522  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:18.523382  1732 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:18.494112  7290 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:18.494516  7306 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:18.462100  7303 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:24:18.449661  7211 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:18.528762  2417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1070) took 1304 ms (client timeout 1606 ms). Trace:
I20260709 12:24:18.533526  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:18.534106  2417 rpcz_store.cc:269] 0709 12:24:17.224533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.226027 (+  1494us) service_pool.cc:224] Handling call
0709 12:24:18.528474 (+1302447us) spinlock_profiling.cc:243] Waited 373 ms on lock 0x7b54005f05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:18.528684 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":857822080}
I20260709 12:24:18.500682  7111 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:18.501938  6968 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:18.542398  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.518116  7067 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:18.548324  1879 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
W20260709 12:24:18.552631  2383 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.147s (SENT)
I20260709 12:24:18.553223  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:18.516736  7327 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:18.564913  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:18.565912  1725 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:18.482992  7201 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:18.538187  7306 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:18.545430  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53418.8us
I20260709 12:24:18.543630  7252 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:24:18.584785  2383 leader_election.cc:336] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.255s (SENT)
I20260709 12:24:18.603811  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.605049  1732 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:18.548524  6968 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:18.568634  7111 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.577383  7306 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:18.620826  1729 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:18.622720  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1017) took 1149 ms. Trace:
I20260709 12:24:18.623234  2418 rpcz_store.cc:276] 0709 12:24:17.473573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.924580 (+451007us) service_pool.cc:224] Handling call
0709 12:24:17.924933 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:18.287823 (+362890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.622453 (+334630us) spinlock_profiling.cc:243] Waited 253 ms on lock 0x7b5400488d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:18.622649 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":582714752}
I20260709 12:24:18.625068  1725 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:18.629473  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:18.518024  6875 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:18.581086  7220 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:18.585367  7252 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:18.605320  7201 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:18.641455  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:18.643903  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.647608  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.647907  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:18.663741  7216 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:24:18.664279  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 268949us
I20260709 12:24:18.664367  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:18.665431  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 315071us
I20260709 12:24:18.667536  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:18.669135  7312 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:18.670151  7312 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:18.674608  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.576839  7218 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:18.675386  1875 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:18.666141  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:18.692966  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:18.678761  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:18.692085  7005 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.057s	user 0.007s	sys 0.010s
I20260709 12:24:18.645747  7093 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:24:18.703490  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.642051  7043 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:18.705986  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 190538us
I20260709 12:24:18.700029  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:18.673830  7312 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:18.701994  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:18.696065  7231 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:18.755014  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:18.614298  7111 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:18.755908  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:18.756131  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.757124  1724 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:18.757196  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77571.9us
I20260709 12:24:18.688959  7320 consensus_queue.cc:1243] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:18.766273  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:18.766994  1732 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:18.763990  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64085.4us
I20260709 12:24:18.768219  1876 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:18.769201  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:18.770267  1729 raft_consensus.cc:2410] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 13 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:24:18.651213  7198 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 3, Time since last communication: 0.104s
I20260709 12:24:18.770434  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:18.772126  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:18.773056  1881 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:18.776664  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:18.777204  1875 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: [14.13-14.13]
I20260709 12:24:18.778486  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:18.781723  1729 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:24:18.708971  7258 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:24:18.698243  7334 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:18.768006  7332 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.266s	user 0.001s	sys 0.000s
W20260709 12:24:18.792312  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 127231us
W20260709 12:24:18.758176  7005 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.124s	user 0.007s	sys 0.013s
I20260709 12:24:18.695693  7067 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:18.691781  7090 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:24:18.793567  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.850039  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.790203  1729 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:18.794756  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:18.846088  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67650.9us
W20260709 12:24:18.846489  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69981.8us
I20260709 12:24:18.771236  7093 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:18.767318  6890 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:18.762672  7322 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.709712  7053 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:18.846547  1732 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:18.795550  7043 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.776455  7223 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:18.794122  7210 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:18.852952  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:18.870517  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.785435  7312 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:18.676254  7321 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.204s
I20260709 12:24:18.705754  7303 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.147s
I20260709 12:24:18.708218  7228 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.160s
I20260709 12:24:18.791158  7258 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:18.785653  7336 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:18.796818  7334 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.853440  7198 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:24:18.870972  7292 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.897s	user 0.005s	sys 0.012s
I20260709 12:24:18.887486  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:18.856674  7231 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:18.901011  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53507.9us
I20260709 12:24:18.927615  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:18.906028  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47625.5us
W20260709 12:24:18.907226  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54398.2us
I20260709 12:24:18.866354  7053 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.932089  1874 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:18.907761  1798 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:18.872409  6890 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:18.909101  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:18.785429  6968 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:18.922065  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.879637  7211 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:18.908185  7258 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:18.863068  7337 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
I20260709 12:24:18.923010  7320 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:18.887403  7043 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:18.921051  7334 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:18.951653  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:18.955243  7216 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:18.847633  7252 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:18.872023  7322 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:18.935156  7053 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:18.937898  6890 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:18.971943  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:18.996600  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.009194  1732 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
W20260709 12:24:19.009778  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80572.9us
W20260709 12:24:19.009769  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.937956  7198 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 12:24:18.924768  7336 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.020730  1877 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:19.010944  7346 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:24:19.020171  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:18.970093  7218 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.011731  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:19.006757  6890 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:19.014209  7211 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:19.010026  7320 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:19.007793  7347 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:19.019199  7252 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:19.035697  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:19.048341  7318 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.065s	user 0.000s	sys 0.004s
I20260709 12:24:19.012861  7223 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:19.062721  7216 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:18.953321  7342 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.092s
I20260709 12:24:19.037474  7347 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.033082  7211 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:19.049396  7344 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
I20260709 12:24:18.964305  7093 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:19.095078  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 139282us
I20260709 12:24:19.049826  7359 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:17.708925  1723 rpcz_store.cc:276] 0709 12:24:15.983134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.026147 (+1043013us) service_pool.cc:224] Handling call
0709 12:24:17.035905 (+  9758us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.036171 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:17.051881 (+ 15710us) log.cc:844] Serialized 14 byte log entry
0709 12:24:17.531161 (+479280us) spinlock_profiling.cc:243] Waited 9.73 ms on lock 0x7b1800431980. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:24:17.569146 (+ 37985us) log.cc:844] Serialized 14 byte log entry
0709 12:24:17.579368 (+ 10222us) log.cc:844] Serialized 14 byte log entry
0709 12:24:17.682202 (+102834us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b1800431980. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:24:17.682453 (+   251us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:17.682533 (+    80us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:17.682606 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.682729 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.693684 (+ 10955us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":594130,"prepare.queue_time_us":17722,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":3105,"raft.queue_time_us":170597,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":15335,"spinlock_wait_cycles":43640576,"thread_start_us":28957,"threads_started":1,"wal-append.queue_time_us":107547}
I20260709 12:24:19.082420  1881 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:19.038537  7111 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:19.060011  7120 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:19.034372  7321 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:19.100658  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:19.090131  7347 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:19.106011  7313 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.639s	user 0.008s	sys 0.002s
I20260709 12:24:19.059841  7353 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.056s
W20260709 12:24:19.110061  7340 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.073s	user 0.001s	sys 0.001s
I20260709 12:24:19.118428  7111 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:19.135803  7320 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:19.049896  7231 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.139315  7211 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.099339  7350 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:24:19.109148  7321 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:19.108757  7120 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:19.114617  7339 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
I20260709 12:24:19.165122  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:19.178308  1845 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.595s (SENT)
I20260709 12:24:19.178956  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:19.153474  7320 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:19.180152  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 321918us
I20260709 12:24:19.183367  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.101161  7223 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.186424  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:19.186647  1845 leader_election.cc:336] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.673s (SENT)
I20260709 12:24:19.199546  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.187170  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:19.161948  7111 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:19.164999  7120 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:19.117738  7358 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.088s
I20260709 12:24:19.204869  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:19.209617  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.212212  1803 raft_consensus.cc:2376] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15. Re-sending same reply.
I20260709 12:24:19.172369  7211 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:19.212527  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:19.189879  7361 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:19.216526  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.209529  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.216276  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.213486  2414 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:19.218240  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:19.222749  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.224983  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:19.198304  7360 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:19.279325  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:19.276384  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62284.1us
I20260709 12:24:19.272611  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:19.282341  2417 raft_consensus.cc:2410] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 17 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
I20260709 12:24:19.283237  1803 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:19.283568  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.284495  1807 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:24:19.222069  7355 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:24:19.208436  7090 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:19.221776  7303 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:17.307646   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:19.293550  1802 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:19.223443  7361 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:19.210673  7357 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:19.299849  1877 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
W20260709 12:24:19.303901  1845 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.131s (SENT)
W20260709 12:24:19.278263  7222 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.784s	user 0.009s	sys 0.006s
W20260709 12:24:19.289269  7277 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.324s	user 0.017s	sys 0.005s
I20260709 12:24:19.318511  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.327502  1804 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:19.319659  1878 raft_consensus.cc:1217] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Deduplicated request from leader. Original: 12.6->[13.7-15.8]   Dedup: 13.7->[15.8-15.8]
I20260709 12:24:19.281479  7336 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.318320  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.323520  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.281745  7322 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:19.275242  7210 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:19.275437  7043 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.201570  7321 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.328801  1878 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: [13.7-15.8]
I20260709 12:24:19.320791  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.215907  6968 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 7, Time since last communication: 0.090s
I20260709 12:24:19.288059  7313 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:19.324435  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.227242  7292 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.089s
I20260709 12:24:19.395905  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:19.316560  7221 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.805s	user 0.014s	sys 0.003s
I20260709 12:24:19.309986  7355 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:19.370674  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.410533  1801 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:19.371055  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:19.371335  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:19.416920  1881 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:19.311455  7361 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:19.420032  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.377524  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 152587us
I20260709 12:24:19.227213  7362 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.113s
W20260709 12:24:19.394384  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.325354  7348 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:19.422171  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 409996us
I20260709 12:24:19.392678  7313 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:19.378230  7357 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:19.424599  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.331739  7222 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.312099  7360 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.102s
I20260709 12:24:19.435353  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.366869  7364 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:19.436499  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.442116  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.433846  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:19.444649  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.444815  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.414043  7321 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:19.450883  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.453851  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.419838  7322 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:19.423743  6968 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:19.460731  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.377437  7320 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.474471  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:19.467825  1728 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
W20260709 12:24:19.471057  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.473413  2416 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
W20260709 12:24:19.483350  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51185.3us
I20260709 12:24:19.483958  2418 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:24:19.475293  2414 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:19.485713  1800 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [12.9-14.10]
I20260709 12:24:19.487892  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:19.488067  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:19.488699  2410 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:19.488814  2417 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:19.413136  7221 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.495285  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.498000  1802 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: [12.9-15.10]
I20260709 12:24:19.499310  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:19.500313  2415 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:19.500634  1802 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:19.503036  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.376019  7111 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.459246  7322 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:19.470010  6968 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:19.509528  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:19.549602  1875 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:19.583973  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 74558.4us
I20260709 12:24:19.584127  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:19.492620  7271 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:19.499141  7360 consensus_queue.cc:1243] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 15.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:24:19.581734  7365 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:19.495836  7342 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.095s
W20260709 12:24:19.604615  7258 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.653s	user 0.013s	sys 0.009s
I20260709 12:24:19.458441  7090 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.686084  1727 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:19.687021  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2265) took 1326 ms. Trace:
I20260709 12:24:19.687369  1727 rpcz_store.cc:276] 0709 12:24:18.360276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.685486 (+1325210us) service_pool.cc:224] Handling call
0709 12:24:19.685865 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.686605 (+   740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.686954 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:19.646349  7365 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:19.656903  7366 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.365s	user 0.001s	sys 0.000s
W20260709 12:24:19.695058  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 111008us
I20260709 12:24:19.701200  7310 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:19.715250  7223 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:19.715507  7310 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:19.707140  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2273) took 1331 ms. Trace:
I20260709 12:24:19.716619  7310 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.717941  7310 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:19.724617  7360 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:19.727468  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:19.729360  7005 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:19.438352  7348 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.639657  7319 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.111s
I20260709 12:24:19.588757  7377 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.105s
W20260709 12:24:19.697674  7367 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.369s	user 0.000s	sys 0.002s
I20260709 12:24:19.692662  7355 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:19.444487  7357 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.511976  7210 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.586925  7313 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.744107  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2284) took 1315 ms. Trace:
I20260709 12:24:19.780169  1730 rpcz_store.cc:276] 0709 12:24:18.428231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.493503 (+ 65272us) service_pool.cc:224] Handling call
0709 12:24:18.507490 (+ 13987us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:18.507800 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:18.640614 (+132814us) log.cc:844] Serialized 14 byte log entry
0709 12:24:19.743730 (+1103116us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:19.743791 (+    61us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:19.743841 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.743918 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:19.744041 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":941748,"prepare.queue_time_us":28470,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":2181,"thread_start_us":249687,"threads_started":2,"wal-append.queue_time_us":247988}
W20260709 12:24:19.742990  7369 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.336s	user 0.001s	sys 0.001s
W20260709 12:24:19.748103  7201 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.064s	user 0.008s	sys 0.006s
I20260709 12:24:19.742811  7337 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:19.794646  7313 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:19.803776  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:19.795650  7046 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:19.797101  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2293) took 1183 ms. Trace:
I20260709 12:24:19.762501  7374 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.806912  1724 rpcz_store.cc:276] 0709 12:24:18.613880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.663653 (+ 49773us) service_pool.cc:224] Handling call
0709 12:24:18.664019 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:18.762931 (+ 98912us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.796773 (+1033842us) spinlock_profiling.cc:243] Waited 92.9 ms on lock 0x7b54004362d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:19.797026 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":213644160}
I20260709 12:24:19.681318  7258 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.808176  7377 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:19.821803  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.804888  7222 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.833875  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.824411  7375 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:19.824218  7373 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.830911  7377 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.842947  7377 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:19.846395  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 1.14431e+06us
I20260709 12:24:19.847438  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:19.823359  7319 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
W20260709 12:24:19.855134  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.858748  7347 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.718s	user 0.014s	sys 0.000s
I20260709 12:24:19.859411  7347 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.860853  1800 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
W20260709 12:24:19.859922  6890 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.805s	user 0.012s	sys 0.009s
I20260709 12:24:19.862421  7221 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:19.826573  7374 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.862564  6890 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.864208  1807 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:19.865089  7374 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:19.837831  7375 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:19.865870  7347 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.866665  7375 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.868315  7375 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:19.869422  6890 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:19.807559  7337 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.873453  1872 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: [11.10-13.11]
W20260709 12:24:19.861989  7381 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
W20260709 12:24:19.884018  7053 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.861s	user 0.012s	sys 0.008s
I20260709 12:24:19.884517  7053 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:19.885102  7334 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.921s	user 0.008s	sys 0.002s
I20260709 12:24:19.885540  7334 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.876070  7319 consensus_queue.cc:1243] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 14.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:19.887130  1874 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:19.717391  1726 rpcz_store.cc:276] 0709 12:24:18.375597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.478494 (+102897us) service_pool.cc:224] Handling call
0709 12:24:18.478909 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:18.479126 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:18.503331 (+ 24205us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.694172 (+190841us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.698988 (+  4816us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.705138 (+  6150us) log.cc:844] Serialized 14 byte log entry
0709 12:24:19.706149 (+1001011us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b1800487120. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:24:19.706396 (+   247us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:19.706476 (+    80us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:19.706565 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.706829 (+   264us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:19.707044 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1012335,"prepare.queue_time_us":132010,"prepare.run_cpu_time_us":146,"prepare.run_wall_time_us":4870,"raft.queue_time_us":1090269,"raft.run_cpu_time_us":293,"raft.run_wall_time_us":63718,"spinlock_wait_cycles":131259136,"thread_start_us":187406,"threads_started":1,"wal-append.queue_time_us":287630}
I20260709 12:24:19.896073  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:19.905143  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.928325  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:19.906165  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 746) took 1137 ms. Trace:
I20260709 12:24:19.929817  1877 rpcz_store.cc:276] 0709 12:24:18.769225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.769480 (+   255us) service_pool.cc:224] Handling call
0709 12:24:19.905975 (+1136495us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54002df0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:19.906104 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":378959104}
I20260709 12:24:19.906409  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:19.906603  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3073) took 1492 ms. Trace:
I20260709 12:24:19.910390  2412 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:19.931715  1879 rpcz_store.cc:276] 0709 12:24:18.414356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.414569 (+   213us) service_pool.cc:224] Handling call
0709 12:24:18.414957 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:18.549174 (+134217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.906350 (+1357176us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b540021c5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:19.906537 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":306016000}
I20260709 12:24:19.911662  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.934047  1727 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:19.934178  2417 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:24:19.872794  7361 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.935082  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.935992  1727 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:19.935918  1805 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:19.905752  7092 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.937208  7361 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:19.937755  7092 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:19.939376  7211 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.666s	user 0.011s	sys 0.002s
I20260709 12:24:19.940207  7092 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.911223  7277 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:19.941401  7303 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.617s	user 0.006s	sys 0.005s
I20260709 12:24:19.941228  7277 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:19.941867  7303 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:19.896363  7337 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.943526  7277 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:19.944911  1724 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:19.946053  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.947281  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.948010  1724 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:19.948813  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.949723  7334 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.950018  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.912003  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:19.951675  1731 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:19.912384  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:19.912065  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.914855  7228 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.954406  1730 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:19.954715  7228 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.955911  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.913213  7203 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:19.957973  7203 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:19.961731  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.852648  7373 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:19.914191  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.980611  7203 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.981076  1725 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:19.982098  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:19.918246  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.983314  1726 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:19.983915  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 751) took 1019 ms. Trace:
I20260709 12:24:19.984256  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.984529  1874 rpcz_store.cc:276] 0709 12:24:18.964377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.964647 (+   270us) service_pool.cc:224] Handling call
0709 12:24:19.983624 (+1018977us) spinlock_profiling.cc:243] Waited 858 ms on lock 0x7b54001f7f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:19.983835 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1972473728}
I20260709 12:24:19.921644  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:19.985594  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:19.986584  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 65061.3us
W20260709 12:24:19.926388  7327 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.352s	user 0.006s	sys 0.010s
I20260709 12:24:19.987648  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:19.936926  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.989159  7322 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:19.989948  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:19.990347  7322 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:19.990855  1727 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:24:19.991786  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:19.991919  7196 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:19.992317  7196 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:19.992625  1727 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 12:24:19.993196  7312 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.094s	user 0.013s	sys 0.002s
I20260709 12:24:19.993747  7203 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:19.993875  7312 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:19.920917  7120 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.694s	user 0.008s	sys 0.003s
I20260709 12:24:19.919881  1878 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:19.957492  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:19.911667  7354 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:19.922356  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:20.007164  1881 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:20.007206  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:19.919231  7390 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:19.921304  7271 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:20.013254  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 106938us
I20260709 12:24:19.897476  7370 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:24:19.894716  7252 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.839s	user 0.005s	sys 0.005s
I20260709 12:24:19.901542  7365 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:19.908166  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:19.977614  1804 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [12.8-14.9]
I20260709 12:24:19.908458  6862 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:19.911435  7348 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:19.900768  7335 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:19.977722  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:19.944911  7306 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.269s	user 0.014s	sys 0.000s
I20260709 12:24:19.912717  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:19.979420  7228 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:20.034322  7005 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:20.034830  7005 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:19.919761  1880 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:24:19.913023  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:19.912245  6875 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.987468  7327 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:19.988729  7334 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:19.918990  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 71744.8us
I20260709 12:24:19.974218  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:19.899886  7067 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.046s	user 0.011s	sys 0.003s
I20260709 12:24:19.986673  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:20.042650  1726 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:20.044512  1726 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:20.009017  1724 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260709 12:24:20.011142  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.016719  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.048777  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.049348  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.017535  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:20.027601  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:20.029739  1731 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:20.030339  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:20.031788  1732 raft_consensus.cc:2368] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 14.
I20260709 12:24:20.006489  7354 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:20.017639  7370 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:20.038905  1730 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.10-14.10]
I20260709 12:24:20.050711  2412 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:20.022567  6862 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:20.054421  1723 raft_consensus.cc:2410] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:24:20.018244  7203 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:20.052950  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:20.022296  7365 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:20.051890  1726 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:24:20.026382  7211 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:20.013115  7120 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.030861  7374 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.039s
I20260709 12:24:20.062235  7391 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:20.076246  1731 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:24:20.077216  7111 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:20.081892  7231 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:19.993243  7196 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.084038  2413 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:20.084740  7222 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:20.085657  7218 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:20.086428  7092 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:20.086967  1804 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:24:20.087361  1732 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:20.082420  7368 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 12:24:20.029327  7335 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.067399  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:20.025403  7390 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:20.039276  7306 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.104198  6862 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.116068  6862 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:20.116015  7390 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.117007  7377 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.117498  7306 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:20.118111  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.118455  1873 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:20.058007  6875 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.110642  7335 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:20.119644  1725 raft_consensus.cc:2376] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17. Re-sending same reply.
I20260709 12:24:20.121071  6875 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:20.123088  7120 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:20.123509  1731 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:20.092722  2410 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:20.082412  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:20.099242  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46430.4us
I20260709 12:24:20.090896  7228 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.018s
I20260709 12:24:20.021232  7277 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.070s
I20260709 12:24:20.102048  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:20.101893  7230 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:20.103509  7111 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.144510  7210 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 6, Time since last communication: 0.007s
W20260709 12:24:20.153692  7376 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.513s	user 0.000s	sys 0.001s
I20260709 12:24:20.024937  7271 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.155509  2415 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:20.156705  7271 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:20.059651  7067 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:20.158046  7359 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:20.158654  7359 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:20.159582  7359 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.160359  7053 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:20.160666  7359 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:20.162626  7320 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:20.163998  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:20.164892  1806 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:20.165072  7221 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:20.165436  7221 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:20.028982  7303 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.065s
W20260709 12:24:20.169342  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.169838  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.173349  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 752) took 1155 ms. Trace:
I20260709 12:24:20.174196  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:20.176635  7230 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:20.177035  7222 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.177847  7222 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.179454  1800 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:20.180644  7327 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:20.181188  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:20.182102  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:20.182768  1723 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:20.148370  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:20.184414  1802 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:24:20.107561  2411 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:20.109560  1732 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:20.186156  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:20.095182  7203 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:20.119246  7377 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:20.062871  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 148163us
I20260709 12:24:20.100145  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:20.125907  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:20.127063  7370 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.133693  7058 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:24:20.208500  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.687s (SENT)
I20260709 12:24:20.215624  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:20.104424  7252 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
W20260709 12:24:20.220558  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94725.8us
I20260709 12:24:20.106060  7354 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.105988  7211 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:20.177743  7230 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.225272  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:20.226265  1730 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:20.225677  7211 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.227017  7092 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:20.227237  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:20.227602  7092 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:20.227851  1730 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:20.111757  7005 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.145160  7357 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:20.179464  1731 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:24:20.173830  1875 rpcz_store.cc:276] 0709 12:24:19.017754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.017898 (+   144us) service_pool.cc:224] Handling call
0709 12:24:19.031599 (+ 13701us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.550645 (+519046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.173073 (+622428us) spinlock_profiling.cc:243] Waited 518 ms on lock 0x7b540021bbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:20.173281 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1190433024}
W20260709 12:24:20.090967  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 139746us
W20260709 12:24:20.248633  1770 leader_election.cc:336] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.520s (SENT)
I20260709 12:24:20.249248  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:20.193487  1800 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:24:20.197837  1876 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:20.197940  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:20.250771  2414 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:20.197949  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:20.207286  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:20.212379  1724 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:20.114008  7196 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:20.145529  7228 consensus_queue.cc:1243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:20.212193  7368 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:20.220043  1799 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:24:20.215143  7058 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:20.149407  7277 consensus_queue.cc:1243] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:20.239838  1805 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:20.268474  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2286) took 1808 ms. Trace:
I20260709 12:24:20.268850  1728 rpcz_store.cc:276] 0709 12:24:18.460228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.523276 (+ 63048us) service_pool.cc:224] Handling call
0709 12:24:18.539204 (+ 15928us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:19.478191 (+938987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.268234 (+790043us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b5400435dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:20.268407 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2134949248}
I20260709 12:24:20.243165  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:20.166266  7221 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:20.254657  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.270818  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.209704  7203 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:20.145470  7120 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:20.224329  7354 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:20.158125  7365 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.244330  7090 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:20.234752  7111 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:20.211860  7303 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 3, Time since last communication: 0.057s
I20260709 12:24:20.236637  7349 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.218854  7310 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 12:24:20.211951  7337 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 12:24:20.266142  7368 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.305605  1801 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:20.245154  7321 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 6, Time since last communication: 0.056s
I20260709 12:24:20.301656  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:20.316493  1724 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:20.320134  1730 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:20.339136  1730 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:20.331236  1723 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:20.260095  7217 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:24:20.305749  7221 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:20.336931  7368 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:20.234711  7092 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.359769  1731 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:24:20.368691  1730 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:20.370371  1730 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:20.371974  1730 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:20.374323  1730 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:20.387835  1723 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:20.337965  7111 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:20.335430  7365 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:20.334973  7222 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:20.431327  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.437933  1723 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
W20260709 12:24:20.445254  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.343575  7310 consensus_queue.cc:1243] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 15.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:24:20.343858  7290 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.452821  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:20.454355  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:20.325541  7218 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.064s
W20260709 12:24:20.490336  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.512640  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 58310.6us
I20260709 12:24:20.515096  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2211) took 1034 ms. Trace:
I20260709 12:24:20.333624  7349 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:20.520655  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.528771  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 280176us
I20260709 12:24:20.414391  7111 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:20.510380  7252 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:20.554006  2410 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
W20260709 12:24:20.549463  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 328071us
I20260709 12:24:20.559257  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:20.520093  7303 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:24:20.542416  7404 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260709 12:24:20.565483  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 322569us
I20260709 12:24:20.453059  7210 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
W20260709 12:24:20.568360  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.569543  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:20.508705  7053 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.093s
I20260709 12:24:20.572441  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:20.573782  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:20.574869  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.510607  7198 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:24:20.546861  7111 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:20.582486  1727 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 15 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:24:20.549949  7328 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:20.593679  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:20.511030  7290 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.598991  7390 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:20.570604  7120 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:20.601528  7399 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.143s	user 0.000s	sys 0.000s
I20260709 12:24:20.602725  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:20.633826  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:20.635145  7390 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:20.635833  7390 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:20.605006  7408 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:20.639109  7222 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:20.605201  7217 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:20.639766  7222 consensus_queue.cc:1243] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 12.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:20.640450  1798 raft_consensus.cc:2410] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:24:20.640498  7290 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:20.565286  7231 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:20.579164  7230 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:20.641987  7230 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:20.641710  7120 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.643002  7230 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.606233  7306 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:20.644022  7306 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:20.607569  7198 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:20.580021  7405 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:24:20.644282  7334 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.585s	user 0.010s	sys 0.000s
I20260709 12:24:20.614200  7370 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:20.619902  7335 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.500s	user 0.007s	sys 0.003s
I20260709 12:24:20.645212  7414 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.645311  7335 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.645903  7414 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.647596  7335 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:20.620409  7258 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:20.648036  7414 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:20.620781  7349 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:20.548058  7368 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.648573  7414 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:20.649371  1729 raft_consensus.cc:2368] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11.
I20260709 12:24:20.650198  7368 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:20.573609  7252 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:20.610637  7303 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:20.651475  7252 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.644573  7198 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:20.655448  7198 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.656644  7198 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:20.660332  7303 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.663372  7306 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.664747  7306 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:20.635768  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:20.666903  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2212) took 1182 ms. Trace:
I20260709 12:24:20.667302  1806 rpcz_store.cc:276] 0709 12:24:19.484667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.165697 (+681030us) service_pool.cc:224] Handling call
0709 12:24:20.166072 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:20.166496 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:20.166526 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:20.207744 (+ 41218us) log.cc:844] Serialized 26 byte log entry
0709 12:24:20.301821 (+ 94077us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:24:20.301869 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.301942 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:20.666580 (+364638us) raft_consensus.cc:1685] finished
0709 12:24:20.666602 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.666838 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14928,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"thread_start_us":77766,"threads_started":1,"wal-append.queue_time_us":364518}
I20260709 12:24:20.678158  1806 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:20.676235  2414 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:20.625756  7090 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:20.671555  7407 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:20.708777  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:20.746593  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.747226  7322 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.746s	user 0.009s	sys 0.004s
W20260709 12:24:20.748672  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 57920.2us
I20260709 12:24:20.749646  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:20.750222  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 552300us
I20260709 12:24:20.751035  7398 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:20.751729  6875 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:20.752565  7231 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.753203  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:20.743782  7359 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.583s	user 0.014s	sys 0.002s
I20260709 12:24:20.755054  7358 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:20.757107  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:20.897405  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2223) took 1152 ms. Trace:
I20260709 12:24:20.897610  1729 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:20.897863  1805 rpcz_store.cc:276] 0709 12:24:19.745092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.268665 (+523573us) service_pool.cc:224] Handling call
0709 12:24:20.269039 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.269264 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:20.529504 (+260240us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.896956 (+367452us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:20.897038 (+    82us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:20.897091 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.897177 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.897337 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":219867,"prepare.queue_time_us":301858,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":5023,"thread_start_us":385495,"threads_started":2}
I20260709 12:24:20.757895  6890 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:20.761281  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2221) took 1018 ms. Trace:
W20260709 12:24:20.902940  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 149808us
I20260709 12:24:20.761083  7111 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.766041  7373 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:20.768079  1770 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.557s (SENT)
I20260709 12:24:20.905246  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:20.909291  7412 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:20.915572  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 147541us
W20260709 12:24:20.916539  1770 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.562s (SENT)
W20260709 12:24:20.917632  1770 leader_election.cc:336] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.606s (SENT)
I20260709 12:24:20.918094  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:20.902803  1800 rpcz_store.cc:276] 0709 12:24:19.742745 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.251020 (+508275us) service_pool.cc:224] Handling call
0709 12:24:20.251494 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.251786 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:20.413086 (+161300us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.760595 (+347509us) spinlock_profiling.cc:243] Waited 7 ms on lock 0x7b18003915a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:24:20.760841 (+   246us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:20.760905 (+    64us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:20.760959 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.761037 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.761207 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":285765,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":1457,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"spinlock_wait_cycles":16100096,"thread_start_us":272937,"threads_started":2,"wal-append.queue_time_us":370580}
I20260709 12:24:20.764257  7390 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:20.952214  7391 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:20.952894  7391 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:20.955925  6968 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.957598  6968 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:20.684839  7277 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.218s
I20260709 12:24:20.964056  1802 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:20.795270  7416 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:20.778311  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.975766  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.977594  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:20.778395  7328 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.979678  7328 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:20.980944  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:20.779914  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:20.780740  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:20.781234  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:20.782138  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:20.783371  2419 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:20.681927  7334 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:20.721321  7203 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:20.791049  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 750) took 1854 ms (client timeout 2272 ms). Trace:
I20260709 12:24:20.792979  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3072) took 2379 ms. Trace:
I20260709 12:24:20.685945  2414 pending_rounds.cc:85] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 11
W20260709 12:24:20.866335  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 73439.4us
W20260709 12:24:20.854417  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.768963  7218 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:24:20.944274  7337 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.772482  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:20.896214  7231 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:20.959311  7211 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.032s	sys 0.036s
I20260709 12:24:20.963824  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2219) took 1223 ms. Trace:
I20260709 12:24:20.776744  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2222) took 1032 ms. Trace:
I20260709 12:24:20.973101  2416 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:20.515643  1804 rpcz_store.cc:276] 0709 12:24:19.484352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.115158 (+630806us) service_pool.cc:224] Handling call
0709 12:24:20.122937 (+  7779us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.123199 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:20.127808 (+  4609us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.217677 (+ 89869us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.459151 (+241474us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b18003f3040. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:24:20.459694 (+   543us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.460099 (+   405us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.462212 (+  2113us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:20.462292 (+    80us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:20.462370 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.514840 (+ 52470us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.515022 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3115,"prepare.queue_time_us":3950,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":345833,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":7515,"spinlock_wait_cycles":42993920,"thread_start_us":89037,"threads_started":1,"wal-append.queue_time_us":208904}
I20260709 12:24:21.008874  7313 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:20.979019  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.902577  6890 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.983685  2410 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [14.13-16.14]
I20260709 12:24:20.982650  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:20.988992  7203 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:20.989395  1872 rpcz_store.cc:276] 0709 12:24:18.414015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.416440 (+  2425us) service_pool.cc:224] Handling call
0709 12:24:18.417169 (+   729us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:19.873927 (+1456758us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.792579 (+918652us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54002058d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:20.792898 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3348428160}
I20260709 12:24:20.992846  7337 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.013355  7111 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:20.986596  2415 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:20.899106  7358 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:21.007445  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89831.5us
W20260709 12:24:20.989192  1876 rpcz_store.cc:269] 0709 12:24:18.936337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.936487 (+   150us) service_pool.cc:224] Handling call
0709 12:24:20.790767 (+1854280us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002dfad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:20.790958 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2771837824}
I20260709 12:24:20.992568  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1082) took 1098 ms. Trace:
I20260709 12:24:20.989653  2414 pending_rounds.cc:107] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 12:24:20.993337  1726 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:21.020082  2414 raft_consensus.cc:2889] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:20.993916  1807 rpcz_store.cc:276] 0709 12:24:19.744338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.261430 (+517092us) service_pool.cc:224] Handling call
0709 12:24:20.281062 (+ 19632us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.281338 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:20.543256 (+261918us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.775942 (+232686us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.776168 (+   226us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:20.776232 (+    64us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:20.776307 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.776481 (+   174us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.776662 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":114719,"prepare.queue_time_us":297744,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":98606,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":192,"thread_start_us":346780,"threads_started":2}
I20260709 12:24:20.993841  1801 rpcz_store.cc:276] 0709 12:24:19.740478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.907891 (+167413us) service_pool.cc:224] Handling call
0709 12:24:20.963245 (+1055354us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b54002c0454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:20.963716 (+   471us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":501097216}
I20260709 12:24:21.004179  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:21.019634  2411 rpcz_store.cc:276] 0709 12:24:19.894478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.895787 (+  1309us) service_pool.cc:224] Handling call
0709 12:24:20.983072 (+1087285us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54005609d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:20.992479 (+  9407us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":416660224}
I20260709 12:24:20.945869  6862 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:20.913690  7375 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:20.953931  7391 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.14, Last appended by leader: 14, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:20.896926  7377 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
I20260709 12:24:21.018286  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:21.038797  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 709) took 1792 ms. Trace:
I20260709 12:24:21.049938  1798 rpcz_store.cc:276] 0709 12:24:19.246438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.279016 (+ 32578us) service_pool.cc:224] Handling call
0709 12:24:21.038419 (+1759403us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400067054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:21.038688 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3125756160}
I20260709 12:24:21.039870  1803 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
W20260709 12:24:21.045248  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.051191  1803 pending_rounds.cc:85] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 9
I20260709 12:24:21.051456  1803 pending_rounds.cc:107] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:24:21.051702  1803 raft_consensus.cc:2889] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:20.993669  7211 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.032s	sys 0.036s
I20260709 12:24:21.053625  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:21.056010  1876 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:21.057016  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.044796  7310 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:21.050848  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.072579  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.053566  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.054103  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.010018  6890 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:21.059576  1880 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 13. (term mismatch)
I20260709 12:24:21.081418  1880 pending_rounds.cc:85] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 12
I20260709 12:24:21.081745  1880 pending_rounds.cc:107] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 17.13
I20260709 12:24:21.082000  1880 raft_consensus.cc:2889] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:21.060024  7198 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.060711  7058 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.054976  7370 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.056209  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:21.063406  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 62426.1us
I20260709 12:24:21.108531  1726 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:21.064014  7303 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:21.065663  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.114143  1874 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:21.010022  7313 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.068246  1732 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:21.068881  7337 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:20.999414  7328 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:21.121273  1732 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:21.123878  1876 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:20.990885  7354 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:24:20.997133  7373 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.996999  7252 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:21.097060  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.104689  7391 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:21.103601  1881 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:21.112008  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:21.119442  1726 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:24:21.069191  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3078) took 1751 ms. Trace:
W20260709 12:24:21.124779  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 219560us
W20260709 12:24:21.138442  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.083338  6862 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:21.141101  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:21.126698  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 328873us
I20260709 12:24:21.035215  7336 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.075s
I20260709 12:24:21.127866  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 720) took 1741 ms. Trace:
I20260709 12:24:21.138491  1873 rpcz_store.cc:276] 0709 12:24:19.317325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.318046 (+   721us) service_pool.cc:224] Handling call
0709 12:24:19.318527 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.118925 (+800398us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.068877 (+949952us) spinlock_profiling.cc:243] Waited 800 ms on lock 0x7b54001f8454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:21.069113 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1839097600}
I20260709 12:24:21.144016  1799 rpcz_store.cc:276] 0709 12:24:19.386070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.387346 (+  1276us) service_pool.cc:224] Handling call
0709 12:24:19.387798 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.266353 (+878555us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.118242 (+851889us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54002f2e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:21.127786 (+  9544us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1899576064}
I20260709 12:24:21.139223  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:21.036348  7348 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.069s
W20260709 12:24:21.099491  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.089079  7310 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.052073  7375 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.015589  7416 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.109607  7390 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.109755  7258 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:21.142429  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:21.152896  7354 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:21.117123  7358 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.163965  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:21.164319  7358 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:21.143317  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:21.165372  7303 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:21.166117  7303 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:21.167831  7303 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:21.142231  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:21.154913  6862 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.173288  7416 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:21.173926  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:21.175396  6862 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:21.176220  7111 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:21.176851  7111 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:21.177433  7111 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:21.179332  7090 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:21.181136  7090 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:21.182251  7120 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:21.182649  7120 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:21.182921  7390 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.183624  7120 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.185115  1726 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:21.186113  7390 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:21.127527  7313 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.188905  7313 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:21.192785  7409 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:21.194336  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:21.195487  7359 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:21.195962  7359 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.196458  7359 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:21.196666  7375 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:21.122032  7364 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:24:21.197309  7313 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.192171  2415 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:21.198939  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:21.199560  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.200238  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.126256  7058 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:21.091413  7203 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:24:21.113078  7370 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.134991  7337 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:21.112592  7198 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.116627  7357 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 12:24:21.205454  7222 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:21.275921  7198 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:21.198422  7313 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.280248  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 769) took 1077 ms. Trace:
I20260709 12:24:21.280668  1723 rpcz_store.cc:276] 0709 12:24:20.203003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.440074 (+237071us) service_pool.cc:224] Handling call
0709 12:24:20.440560 (+   486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:20.457902 (+ 17342us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:20.667429 (+209527us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.946619 (+279190us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:20.947312 (+   693us) log.cc:844] Serialized 26 byte log entry
0709 12:24:20.947487 (+   175us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:20.947531 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.947623 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:21.279777 (+332154us) raft_consensus.cc:1685] finished
0709 12:24:21.280029 (+   252us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.280183 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":303988,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":15626,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":14820,"thread_start_us":421731,"threads_started":2,"wal-append.queue_time_us":600471}
I20260709 12:24:21.150111  7290 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:21.154717  7336 consensus_queue.cc:1243] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:21.279389  7313 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:21.206966  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 64954.8us
I20260709 12:24:21.198489  7328 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:21.283531  7328 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:21.283918  7120 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:21.284767  7328 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.286300  7328 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:21.289014  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.290266  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.318763  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.319451  1877 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:21.319636  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.320524  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.321166  1877 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:21.322208  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.140276  7252 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.323657  1878 raft_consensus.cc:2410] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:24:21.326048  7337 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:21.326313  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.327237  7337 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.329881  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:21.332293  1878 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:21.333461  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:21.334085  1878 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:21.321055  1726 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:24:21.340739  7210 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:21.341141  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:21.341454  7210 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.341840  1726 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:21.344167  7210 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:21.343830  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:21.267409  7420 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.153s	user 0.000s	sys 0.000s
I20260709 12:24:21.209153  7258 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.211411  7334 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:21.269435  7221 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.884s	user 0.009s	sys 0.004s
I20260709 12:24:21.209851  7416 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:21.289045  7409 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:21.400261  1877 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:24:21.290921  1873 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:21.291234  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:21.291945  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:21.405560  1872 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:21.407721  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.408886  1799 raft_consensus.cc:2410] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 13 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:24:21.409801  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.260344  7310 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:21.294009  7417 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.391s	user 0.002s	sys 0.000s
I20260709 12:24:21.295318  7005 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:21.424296  1873 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [13.12-16.14]
I20260709 12:24:21.425567  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.296990  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:21.426293  1801 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:21.426669  1876 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:21.299994  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:59190 (request call id 0) took 1231 ms. Trace:
I20260709 12:24:21.304224  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2218) took 1567 ms. Trace:
I20260709 12:24:21.266446  7111 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.273197  6890 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.307592  7231 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.134596  7373 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.317988  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:21.339735  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 719) took 1954 ms. Trace:
I20260709 12:24:21.348052  7363 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:21.349421  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:21.351799  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2295) took 1446 ms. Trace:
I20260709 12:24:21.351845  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:21.290547  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.339660  7271 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:24:21.461059  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 111682us
I20260709 12:24:21.400620  7198 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:21.415985  1771 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.540s (SENT) [suppressed 36 similar messages]
I20260709 12:24:21.420276  1874 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
W20260709 12:24:21.347582  7092 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.023s	sys 0.027s
I20260709 12:24:21.427136  1723 rpcz_store.cc:276] 0709 12:24:20.068384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.282711 (+1214327us) service_pool.cc:224] Handling call
0709 12:24:21.299880 (+ 17169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:21.427266  1802 rpcz_store.cc:276] 0709 12:24:19.736348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.212911 (+476563us) service_pool.cc:224] Handling call
0709 12:24:20.963573 (+750662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.970441 (+  6868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.303934 (+333493us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b54002c0450. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:21.304134 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1725313920}
I20260709 12:24:21.473896  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.475136  1801 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:24:21.440017  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.345469  1726 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
W20260709 12:24:21.418403  7365 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.965s	user 0.004s	sys 0.009s
I20260709 12:24:21.445633  1803 rpcz_store.cc:276] 0709 12:24:19.385607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.385799 (+   192us) service_pool.cc:224] Handling call
0709 12:24:21.039312 (+1653513us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.052418 (+ 13106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.339516 (+287098us) spinlock_profiling.cc:243] Waited 1.65 s on lock 0x7b5400067050. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:21.339666 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3786032896}
I20260709 12:24:21.450822  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.450881  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:21.452652  1727 rpcz_store.cc:276] 0709 12:24:19.911906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.993232 (+ 81326us) service_pool.cc:224] Handling call
0709 12:24:20.001524 (+  8292us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:20.583261 (+581737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.351528 (+768267us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54002b75d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:21.351736 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1332537216}
W20260709 12:24:21.483568  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 131808us
I20260709 12:24:21.459326  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:21.488541  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:21.488349  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.490271  1879 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:21.466094  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.491291  1799 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:21.467059  2416 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:21.425254  7353 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.477703  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:21.413960  7218 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
W20260709 12:24:21.489439  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.536007  1880 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:21.462710  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:21.415165  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:21.438071  7409 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:21.453083  7230 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.017s	sys 0.012s
I20260709 12:24:21.396241  7334 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.511189  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:21.525501  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:21.491484  7196 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.227s	user 0.008s	sys 0.006s
I20260709 12:24:21.567003  1723 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:21.468384  7252 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:21.483871  1798 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:21.537388  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 74773.6us
W20260709 12:24:21.473250  7092 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.023s	sys 0.028s
I20260709 12:24:21.541442  1805 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:21.538331  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:21.513878  7290 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.514163  7221 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:21.452347  7363 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:21.538600  7313 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:21.531323  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:21.585889  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2226) took 1768 ms. Trace:
I20260709 12:24:21.585769  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.586881  1805 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:21.586591  1806 rpcz_store.cc:276] 0709 12:24:19.817020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.682637 (+865617us) service_pool.cc:224] Handling call
0709 12:24:20.744220 (+ 61583us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:20.867373 (+123153us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:20.867427 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:20.911737 (+ 44310us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.147396 (+235659us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:21.147455 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.147551 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:21.585637 (+438086us) raft_consensus.cc:1685] finished
0709 12:24:21.585660 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.585822 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":137019,"raft.queue_time_us":152110,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":23695,"thread_start_us":235112,"threads_started":1,"wal-append.queue_time_us":537948}
I20260709 12:24:21.461685  7271 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:21.495878  1770 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:21.484078  7430 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:21.571931  1723 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:24:21.572655  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.525859  7361 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:21.609149  1723 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:24:21.614962  1678 auto_rebalancer.cc:893] Removing replace marker failed: Replica 075704438992440697732181e49ccd09 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:24:21.615866  1678 auto_rebalancer.cc:901] Could not move replica: Illegal state: not currently leader
W20260709 12:24:21.616261  1678 auto_rebalancer.cc:330] scheduled replica move failed to complete: Illegal state: not currently leader
I20260709 12:24:21.607096  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:21.620599  1803 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:21.621023  1799 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:24:21.525555  7328 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:21.572018  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:21.577215  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:21.543274  7412 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.633s	user 0.007s	sys 0.006s
W20260709 12:24:21.556845  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 538575us
I20260709 12:24:21.135043   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:21.580883  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:21.582048  1845 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.839s (SENT)
I20260709 12:24:21.674402  1807 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
W20260709 12:24:21.675259  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 93270.2us
I20260709 12:24:21.583673  7221 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:21.675921  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.518206  7111 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:21.674146  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:21.589937  7310 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:21.679519  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.681257  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.682394  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.607733  7430 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.686115  1805 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:21.515664  7370 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:21.540988  7218 consensus_queue.cc:1243] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:21.693431  7218 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:24:21.565876  7230 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.017s	sys 0.014s
I20260709 12:24:21.512470  7231 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:21.500219  7258 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:21.508896  7373 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:21.492022  7067 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.558591  6875 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:21.592505  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 96673.9us
I20260709 12:24:21.713973  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:21.713889  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.715303  1806 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:21.715451  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:21.488293  7198 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:21.605610  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.604331  7358 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:21.718180  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.603713  7277 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.656155  1728 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:21.721681  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:21.722553  2410 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:24:21.665145  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.668491  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:21.510618  7359 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.683256  1801 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:21.516507  6890 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:21.608343  7361 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:21.685202  1803 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:21.685935  1732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.008s	sys 0.022s
W20260709 12:24:21.741739  1732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.009s	sys 0.022s
W20260709 12:24:21.682461  1729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.011s	sys 0.019s
W20260709 12:24:21.743047  1729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.011s	sys 0.019s
I20260709 12:24:21.687332  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:21.647392  7398 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:21.590371  7391 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.092s
I20260709 12:24:21.690148  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:21.767095  1799 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:21.542959  7353 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.681130  7310 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.691674  7334 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:21.692657  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.684809  1878 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:21.590725  7290 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.681962  7412 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:21.677327  7221 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:21.708491  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.710985  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:21.689641  7313 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:21.719533  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:21.703025  6968 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.745s	user 0.009s	sys 0.005s
I20260709 12:24:21.689381  7430 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:21.685282  7409 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.755231  1802 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: [12.9-15.10]
I20260709 12:24:21.756852  7252 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:21.718927  6875 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.700914  7203 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 12:24:21.759181  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.811458  1798 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:21.812808  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.813619  1798 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:21.759081  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.815213  1803 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:21.744208  1805 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:24:21.760246  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.019s	sys 0.021s
W20260709 12:24:21.816396  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.019s	sys 0.021s
W20260709 12:24:21.817474  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 653556us
I20260709 12:24:21.722077  7211 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.707530  7271 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.681305  7363 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.768472  7361 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.730660  7373 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:24:21.731622  7370 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:21.771076  1629 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 171) took 58694 ms (client timeout 15000 ms). Trace:
I20260709 12:24:21.770830  7357 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:21.772090  7321 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:21.732883  7377 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:24:21.741204  7365 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:21.774226  7354 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:21.774545  1875 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:21.742190  7277 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.768517  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:21.782083  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.016s	sys 0.033s
W20260709 12:24:21.878252  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.016s	sys 0.034s
I20260709 12:24:21.853456  7353 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:21.888553  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:21.720184  7067 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.864933  7271 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:21.803815  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84338.2us
I20260709 12:24:21.778475  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:21.874547  7361 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:21.808207  1725 raft_consensus.cc:2410] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:24:21.824342  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1026) took 1142 ms. Trace:
I20260709 12:24:21.913434  2414 rpcz_store.cc:276] 0709 12:24:20.681853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.685221 (+  3368us) service_pool.cc:224] Handling call
0709 12:24:20.685657 (+   436us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:21.046226 (+360569us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.046277 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:21.047090 (+   813us) log.cc:844] Serialized 52 byte log entry
0709 12:24:21.429234 (+382144us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:21.429285 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.429353 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:21.824025 (+394672us) raft_consensus.cc:1685] finished
0709 12:24:21.824052 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.824272 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19159,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":21041,"thread_start_us":330582,"threads_started":1,"wal-append.queue_time_us":700260}
I20260709 12:24:21.820683  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.821808  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:21.819728  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:21.826363  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:21.780591  7231 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:21.823319  7434 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 12:24:21.819378  7290 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:21.837548  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.814664  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:21.806725  7058 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.594s	user 0.012s	sys 0.000s
W20260709 12:24:21.838948  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.928582  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:21.931519  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.933483  1803 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 14 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:24:21.867090  7370 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:21.842360  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:21.779752  7310 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:21.804080  7322 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.765462  7398 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.809217  7320 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:21.861403  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:21.816504  6875 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:21.836798  7335 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.003s	sys 0.028s
W20260709 12:24:21.862960  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.660s	user 0.013s	sys 0.027s
I20260709 12:24:21.806559  7313 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:21.870028  7357 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:21.840391  7203 consensus_queue.cc:1243] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 12.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:24:21.879191  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 680317us
I20260709 12:24:21.793421  1807 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:21.881060  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 465077us
I20260709 12:24:21.890887  2410 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 10.
W20260709 12:24:21.894337  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.896560  1805 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:21.756438  7258 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:24:21.770879  7198 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:21.910327  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:21.917088  1878 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:22.010051  1807 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:24:22.010406  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 771) took 1805 ms. Trace:
I20260709 12:24:22.010872  1731 rpcz_store.cc:276] 0709 12:24:20.204673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.002222 (+797549us) service_pool.cc:224] Handling call
0709 12:24:21.002776 (+   554us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.003642 (+   866us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.003686 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.004334 (+   648us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.355627 (+351293us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:21.355672 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.355746 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.010206 (+654460us) raft_consensus.cc:1685] finished
0709 12:24:22.010226 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.010344 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":140,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":106,"thread_start_us":350918,"threads_started":1,"wal-append.queue_time_us":589447}
I20260709 12:24:21.917243  1881 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:21.918889  1806 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:24:21.919339  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.920423  2416 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:24:21.921416  1770 leader_election.cc:400] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 14.
W20260709 12:24:22.013358  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 108096us
I20260709 12:24:22.014143  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:21.922657  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:22.014945  2414 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:21.855185  7211 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:21.898617  7363 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:21.928591  1879 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:21.947546  7398 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:21.949536  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:22.046135  1803 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:22.047290  1803 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:22.047631  1803 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:21.947198  1723 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:21.951237  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:21.952369  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:21.954985  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:24:21.975265  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.772s	user 0.013s	sys 0.027s
I20260709 12:24:21.986280  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:21.988391  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.058110  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.988886  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.986522  7359 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:22.066393  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 111511us
I20260709 12:24:22.067526  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:21.993656  1880 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:22.023676  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:22.011827  7258 consensus_queue.cc:1243] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:24:22.028402  2415 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:24:21.942734  1874 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
W20260709 12:24:21.975693  7335 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.003s	sys 0.028s
I20260709 12:24:21.925673  7196 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
W20260709 12:24:22.059418  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.078917  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:22.080092  1878 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:21.926420  7373 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.974649  7390 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:21.926149  7231 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.048395  7320 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:22.069357  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45762.6us
W20260709 12:24:22.026206  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.000022  7377 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.064272  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:22.048533  1803 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.097848  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:22.043320  7436 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 12:24:22.084395  1723 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:21.947747  7322 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.049074  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:22.049568  2411 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:22.145059  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:22.050908  1798 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:22.146068  1878 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:22.145176  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:22.149726  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.057459  1805 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
W20260709 12:24:21.860126  1629 rpcz_store.cc:269] 0709 12:23:23.107020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:23.108389 (+  1369us) service_pool.cc:224] Handling call
0709 12:24:21.757819 (+58649430us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:13.330863 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:13.331693 (+   830us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:13.331771 (+    78us) write_op.cc:435] Acquired schema lock
0709 12:24:13.331812 (+    41us) tablet.cc:662] Decoding operations
0709 12:24:13.332737 (+   925us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:13.332861 (+   124us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:13.332893 (+    32us) tablet.cc:685] Acquiring locks for 5 operations
0709 12:24:13.333741 (+   848us) tablet.cc:701] Row locks acquired
0709 12:24:13.333801 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:24:13.334003 (+   202us) write_op.cc:270] Start()
0709 12:24:13.334159 (+   156us) write_op.cc:276] Timestamp: P: 1783599853333931 usec, L: 0
0709 12:24:13.334193 (+    34us) op_driver.cc:348] REPLICATION: starting
0709 12:24:13.420193 (+ 86000us) log.cc:844] Serialized 2116 byte log entry
0709 12:24:14.672839 (+1252646us) op_driver.cc:464] REPLICATION: finished
0709 12:24:15.259573 (+586734us) write_op.cc:301] APPLY: starting
0709 12:24:15.259699 (+   126us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:15.356806 (+ 97107us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:15.356976 (+   170us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:15.515615 (+158639us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:15.515880 (+   265us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=10,key_file_lookups=10,delta_file_lookups=10,mrs_lookups=0
0709 12:24:15.515923 (+    43us) write_op.cc:312] APPLY: finished
0709 12:24:15.516253 (+   330us) log.cc:844] Serialized 57 byte log entry
0709 12:24:15.911385 (+395132us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:15.925753 (+ 14368us) write_op.cc:454] Released schema lock
0709 12:24:15.929632 (+  3879us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:16.011358 (+ 81726us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:16.015142 (+  3784us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3589668,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":562354,"apply.run_cpu_time_us":37858,"apply.run_wall_time_us":677188,"cfile_cache_hit":22,"cfile_cache_hit_bytes":25047,"num_ops":5,"prepare.queue_time_us":387107,"prepare.run_cpu_time_us":20777,"prepare.run_wall_time_us":323963,"raft.queue_time_us":675088,"raft.run_cpu_time_us":43856,"raft.run_wall_time_us":446981,"replication_time_us":1288102,"thread_start_us":1236996,"threads_started":5,"wal-append.queue_time_us":819624,"wal-append.run_cpu_time_us":60505,"wal-append.run_wall_time_us":2677548}]]}
I20260709 12:24:22.080922  7218 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:22.133016  1803 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:22.134927  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:22.161011  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:22.071616  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:22.025374  7357 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.138306  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:22.051334  1801 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:24:22.168339  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70599.7us
I20260709 12:24:22.169008  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:22.056756  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.071346  7365 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:22.152621  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:22.153792  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:22.175601  1798 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 10: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:22.155130  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:22.047927  7445 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.092418  7353 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.187776  1801 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:22.189195  1798 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:24:22.163843  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149404us
I20260709 12:24:22.164038  1873 raft_consensus.cc:1217] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Deduplicated request from leader. Original: 12.6->[13.7-15.8]   Dedup: 13.7->[15.8-15.8]
I20260709 12:24:22.190447  1873 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: [13.7-15.8]
I20260709 12:24:22.074051  7412 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:22.166364  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:22.150439  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.172499  2419 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:22.136862  7290 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.163256  7218 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:22.170149  1878 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
W20260709 12:24:22.171177  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 102204us
I20260709 12:24:22.203722  1878 pending_rounds.cc:85] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 8
I20260709 12:24:22.204093  1878 pending_rounds.cc:107] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:24:22.204360  1878 raft_consensus.cc:2889] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.204990  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:22.187083  2415 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:24:22.206391  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:22.190637  1798 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:22.044816  7444 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.155s
W20260709 12:24:22.137814  7337 consensus_queue.cc:1175] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 16 status { last_received { term: 14 index: 8 } last_committed_idx: 8 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:22.191066  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:22.194871  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:22.190655  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:22.139951  7390 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:22.173763  7328 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.502s	user 0.009s	sys 0.005s
I20260709 12:24:22.201267  1873 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 10. (term mismatch)
I20260709 12:24:22.224152  1873 pending_rounds.cc:85] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 9
I20260709 12:24:22.224556  1873 pending_rounds.cc:107] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 12:24:22.224903  1873 raft_consensus.cc:2889] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.159238  7358 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:22.096542  7373 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.160759  7334 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:22.211476  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:22.213649  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:22.144929  7377 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.169823  7433 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:22.217208  1798 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:24:22.176573  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:22.277411  1805 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:22.220772  1877 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:22.218438  2417 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
W20260709 12:24:22.279251  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 107643us
I20260709 12:24:22.279546  2412 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 15 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:24:22.208577  7218 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:22.201756  7359 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.205366  7290 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.210423  7230 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:24:22.259373  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.274507  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63107.3us
I20260709 12:24:22.291886  1879 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:22.197336  7353 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.232977  7328 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:22.324214  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:22.300186  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.288103  7452 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:24:22.253099  7443 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.327s	user 0.001s	sys 0.001s
I20260709 12:24:22.173523  7313 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:22.326236  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.269620  7435 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:24:22.252938  7334 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:22.220846  7277 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:22.231506  7111 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:22.193615  7445 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.290474  7198 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.305028  7370 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:22.319806  7306 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.276937  7412 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:22.234109  7067 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:22.323457  7221 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.521s	user 0.008s	sys 0.004s
W20260709 12:24:22.327272  7442 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.356s	user 0.001s	sys 0.000s
I20260709 12:24:22.340026  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:24:22.329661  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 162062us
W20260709 12:24:22.354744  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 216520us
I20260709 12:24:22.304812  7433 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:22.343161  1807 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:22.346626  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 126691us
W20260709 12:24:22.334017  7440 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.309s	user 0.001s	sys 0.001s
W20260709 12:24:22.336440  7446 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.278s	user 0.002s	sys 0.000s
I20260709 12:24:22.319401  7359 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.379235  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:22.389544  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:22.329758  7452 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:22.341763  7230 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:22.337059  7435 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:22.377341  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:22.391444  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:22.318277  1805 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:22.393373  1770 leader_election.cc:400] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11.
I20260709 12:24:22.399443  1770 leader_election.cc:403] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:24:22.345348  7277 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:24:22.350131  7306 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.383427  7463 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:22.344143  7198 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.376399  7444 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.030s
I20260709 12:24:22.421772  7435 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:22.423192  7457 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
I20260709 12:24:22.395288  7452 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:22.426128  7476 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:22.395373  7211 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:22.383054  7221 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.427112  7231 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:22.389698  7334 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.398629  7335 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:22.428674  7451 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:24:22.429395  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 778) took 1779 ms. Trace:
I20260709 12:24:22.429979  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:22.430210  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 773) took 1866 ms. Trace:
W20260709 12:24:22.479219  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49342.2us
I20260709 12:24:22.478649  1729 rpcz_store.cc:276] 0709 12:24:20.563925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.118001 (+554076us) service_pool.cc:224] Handling call
0709 12:24:21.118319 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.118565 (+   246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.118600 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.126598 (+  7998us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.743614 (+617016us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:21.743681 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.743784 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.429979 (+686195us) raft_consensus.cc:1685] finished
0709 12:24:22.430001 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.430147 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":587967,"threads_started":1,"wal-append.queue_time_us":763851}
I20260709 12:24:22.430771  7466 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11.
I20260709 12:24:22.431054  7390 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:22.431612  7373 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:22.432721  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 777) took 1782 ms. Trace:
I20260709 12:24:22.419590  7463 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:22.439414  7454 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
W20260709 12:24:22.444267  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66997.9us
I20260709 12:24:22.403119  7467 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:22.429385  7433 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.506954  2416 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:22.453560  7445 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:22.413002  7230 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:24:22.471909  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 80535.5us
I20260709 12:24:22.466421  7290 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:22.482653  7465 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:22.483212  7476 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.484663  1732 rpcz_store.cc:276] 0709 12:24:20.650075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.122261 (+472186us) service_pool.cc:224] Handling call
0709 12:24:21.122547 (+   286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.135301 (+ 12754us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.135338 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.143987 (+  8649us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.742516 (+598529us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:21.742593 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.742702 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.432522 (+689820us) raft_consensus.cc:1685] finished
0709 12:24:22.432543 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.432664 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5129,"raft.queue_time_us":9941,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":7152,"thread_start_us":571326,"threads_started":1,"wal-append.queue_time_us":731528}
I20260709 12:24:22.476830  1730 rpcz_store.cc:276] 0709 12:24:20.650409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.128641 (+478232us) service_pool.cc:224] Handling call
0709 12:24:21.141935 (+ 13294us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:21.142326 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.142355 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:21.143047 (+   692us) log.cc:844] Serialized 52 byte log entry
0709 12:24:21.677312 (+534265us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:21.703541 (+ 26229us) log.cc:844] Serialized 14 byte log entry
0709 12:24:21.704586 (+  1045us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.704698 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.429113 (+724415us) raft_consensus.cc:1685] finished
0709 12:24:22.429145 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.429327 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15031,"prepare.queue_time_us":35337,"prepare.run_wall_time_us":69,"raft.queue_time_us":4618,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":11811,"spinlock_wait_cycles":6450048,"thread_start_us":470556,"threads_started":1,"wal-append.queue_time_us":710239}
I20260709 12:24:22.502750  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:22.525874  1801 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:22.503154  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:22.465377  7320 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:22.512702  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:22.517781  7466 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:22.509871  7363 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:22.519869  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2379) took 1220 ms. Trace:
I20260709 12:24:22.535856  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:22.520852  7271 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.602s	user 0.009s	sys 0.002s
I20260709 12:24:22.454334  7444 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 12:24:22.528926  1805 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:22.506542  7334 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:22.570564  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.497285  7463 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:22.547122  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:22.536402  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.536198  1872 rpcz_store.cc:276] 0709 12:24:21.299901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.406460 (+106559us) service_pool.cc:224] Handling call
0709 12:24:21.406956 (+   496us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.433053 (+ 26097us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.433101 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.606523 (+173422us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b1800249040. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a
0709 12:24:21.760079 (+153556us) spinlock_profiling.cc:243] Waited 6.93 ms on lock 0x7b2c00009c38. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:24:21.769157 (+  9078us) log.cc:844] Serialized 26 byte log entry
0709 12:24:22.015174 (+246017us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:22.015230 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.015334 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.519622 (+504288us) raft_consensus.cc:1685] finished
0709 12:24:22.519646 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.519803 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31819,"raft.queue_time_us":5324,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":17654,"spinlock_wait_cycles":52797568,"thread_start_us":245524,"threads_started":1,"wal-append.queue_time_us":417271}
I20260709 12:24:22.610697  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:22.617569  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2382) took 1269 ms. Trace:
I20260709 12:24:22.618257  1876 rpcz_store.cc:276] 0709 12:24:21.348095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.427317 (+ 79222us) service_pool.cc:224] Handling call
0709 12:24:21.444267 (+ 16950us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.444850 (+   583us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.444881 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.459092 (+ 14211us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.932641 (+473549us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:21.932707 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.932842 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.617226 (+684384us) raft_consensus.cc:1685] finished
0709 12:24:22.617259 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.617477 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33818,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":30840,"thread_start_us":429183,"threads_started":1,"wal-append.queue_time_us":708473}
I20260709 12:24:22.510490  7377 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:22.544998  7230 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:24:22.623871  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76866.2us
I20260709 12:24:22.535817  7363 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:22.653697  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 173479us
W20260709 12:24:22.550617  7430 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.741s	user 0.015s	sys 0.000s
W20260709 12:24:22.660528  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 239376us
I20260709 12:24:22.530543  7111 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:22.524994  7465 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:22.522428  7092 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:22.544626  7277 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.111s
I20260709 12:24:22.510610  7467 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.746757  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:22.656230  7328 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:22.650434  7470 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:22.751828  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:22.762015  7328 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:22.655961  7473 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:22.545972  7335 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.183s
I20260709 12:24:22.762727  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:22.522629  7479 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.243s
I20260709 12:24:22.695061  7363 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:22.691896  7353 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:22.660190  7357 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:22.775161  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83241.7us
I20260709 12:24:22.762679  2416 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:22.748848  7466 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:22.760002  7361 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.840s	user 0.015s	sys 0.000s
I20260709 12:24:22.748379  7092 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:22.730568  7480 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:22.735242  7221 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:22.750521  7228 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:22.762315  7470 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:22.775998  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:22.607137  7390 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.140s
I20260709 12:24:22.741919  7465 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:22.751262  7306 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:22.756985  7359 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:22.785015  1804 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
W20260709 12:24:22.786190  7485 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:24:22.735589  7377 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
W20260709 12:24:22.856537  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.907948  7477 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.377s	user 0.000s	sys 0.001s
W20260709 12:24:22.912178  7490 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:24:22.900274  7328 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.915396  7467 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:22.787328  7472 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:24:22.808193  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2243) took 1095 ms. Trace:
W20260709 12:24:22.837500  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.535856  7476 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.775334  7473 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.724233  7430 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:22.922729  7092 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.936430  7430 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:22.936586  7092 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:22.937191  7430 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.844556  7470 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:22.939800  1726 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:22.941604  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2317) took 1632 ms. Trace:
I20260709 12:24:22.942134  7473 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:22.901619  7478 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.355s	user 0.001s	sys 0.000s
I20260709 12:24:22.944006  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3091) took 1116 ms. Trace:
I20260709 12:24:22.944434  7489 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.945209  7489 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:22.945855  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 758) took 1893 ms (client timeout 1916 ms). Trace:
W20260709 12:24:22.946460  1875 rpcz_store.cc:269] 0709 12:24:21.052854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.053715 (+   861us) service_pool.cc:224] Handling call
0709 12:24:22.945605 (+1891890us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:22.945777 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1609955072}
I20260709 12:24:22.948786  7476 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:22.939874  7398 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.871s	user 0.012s	sys 0.001s
I20260709 12:24:22.937767  7430 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.956103  1803 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:24:22.956611  1803 pending_rounds.cc:85] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 8
I20260709 12:24:22.956861  1803 pending_rounds.cc:107] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:24:22.957118  1803 raft_consensus.cc:2889] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.957937  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2241) took 1245 ms. Trace:
I20260709 12:24:22.908658  7221 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.958244  7430 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:22.951730  7398 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.900200  7335 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.942036  1726 rpcz_store.cc:276] 0709 12:24:21.308744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.477367 (+168623us) service_pool.cc:224] Handling call
0709 12:24:22.941351 (+1463984us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54002b66d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:22.941527 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3224537472}
I20260709 12:24:22.944675  1881 rpcz_store.cc:276] 0709 12:24:21.827709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.168407 (+340698us) service_pool.cc:224] Handling call
0709 12:24:22.168846 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.192712 (+ 23866us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.192771 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:22.192814 (+    43us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:22.192844 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.943648 (+750804us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b5400117954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:22.943717 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.943926 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54397312}
I20260709 12:24:22.972076  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2227) took 1287 ms. Trace:
I20260709 12:24:22.804435  7228 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:22.958310  1803 rpcz_store.cc:276] 0709 12:24:21.712339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.046875 (+334536us) service_pool.cc:224] Handling call
0709 12:24:22.047175 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.955744 (+908569us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b540011eca0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abddad87 00007f16abddb470 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:24:22.957606 (+  1862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.957862 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":68279680}
W20260709 12:24:22.849315  7468 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.386s	user 0.001s	sys 0.001s
I20260709 12:24:22.921499  1804 pending_rounds.cc:85] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 11
I20260709 12:24:23.009227  1804 pending_rounds.cc:107] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:24:22.975250  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3084) took 1156 ms. Trace:
I20260709 12:24:23.009804  1880 rpcz_store.cc:276] 0709 12:24:21.820276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.820408 (+   132us) service_pool.cc:224] Handling call
0709 12:24:22.975008 (+1154600us) spinlock_profiling.cc:243] Waited 77.3 ms on lock 0x7b54000f0854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:22.975168 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":177897856}
I20260709 12:24:23.002364  1802 rpcz_store.cc:276] 0709 12:24:21.684400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.688337 (+  3937us) service_pool.cc:224] Handling call
0709 12:24:21.754911 (+ 66574us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:21.804036 (+ 49125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.971736 (+1167700us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b54003b79d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:22.971971 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":152125312}
I20260709 12:24:23.012084  1804 raft_consensus.cc:2889] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:22.827689  7198 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.003221  1806 rpcz_store.cc:276] 0709 12:24:21.713080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.758158 (+ 45078us) service_pool.cc:224] Handling call
0709 12:24:22.807805 (+1049647us) spinlock_profiling.cc:243] Waited 40.6 ms on lock 0x7b54001d5454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:22.808078 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93430656}
W20260709 12:24:22.970360  6875 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.977s	user 0.010s	sys 0.002s
I20260709 12:24:23.004061  7335 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.959234  7221 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.999397  7258 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.005828  7398 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:22.985865  7489 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:22.971830  7310 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.999s	user 0.012s	sys 0.002s
W20260709 12:24:23.009132  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 232723us
W20260709 12:24:23.012768  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.829674  1876 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:24:22.986892  7470 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.985442  7465 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.784058  7466 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:22.985059  7363 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:22.845554  7111 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:23.008708  7231 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.022s	sys 0.026s
I20260709 12:24:23.026176  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2387) took 1081 ms. Trace:
I20260709 12:24:23.057150  1878 rpcz_store.cc:276] 0709 12:24:21.945265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.203100 (+257835us) service_pool.cc:224] Handling call
0709 12:24:22.203505 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.207964 (+  4459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.208018 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:22.208819 (+   801us) log.cc:844] Serialized 26 byte log entry
0709 12:24:22.398420 (+189601us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:22.398458 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.398535 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.025838 (+627303us) raft_consensus.cc:1685] finished
0709 12:24:23.025872 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.026092 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":189216,"threads_started":1,"wal-append.queue_time_us":315835}
I20260709 12:24:23.059537  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.063494  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:23.064581  1803 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:23.036412  7310 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.041651  7228 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.043857  7470 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.044467  7221 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.045013  7465 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.025511  7258 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.016929  6875 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.047426  7466 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.047739  7335 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.007866  7463 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.072013  1876 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:23.086334  1806 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:23.087591  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.073155  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.088265  1880 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:23.088703  1881 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:23.074126  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.060752  1802 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
W20260709 12:24:23.053429  7218 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.732s	user 0.009s	sys 0.005s
I20260709 12:24:22.958735  7452 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.052301  7398 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:24:23.050635  7377 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.080420  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.104512  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:23.105340  7310 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:23.105858  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:23.106880  1798 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:23.092585  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.094388  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.109815  1881 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:23.111776  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.112834  1878 raft_consensus.cc:2410] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
W20260709 12:24:23.069298  7231 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.022s	sys 0.026s
I20260709 12:24:23.096061  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.116570  1803 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:23.088907  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2335) took 1075 ms. Trace:
I20260709 12:24:23.089491  7228 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:23.121985  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:23.127220  7334 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.513s	user 0.014s	sys 0.000s
I20260709 12:24:23.129009  7363 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.130551  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2333) took 1256 ms. Trace:
I20260709 12:24:23.144318  1731 rpcz_store.cc:276] 0709 12:24:21.874499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.059651 (+185152us) service_pool.cc:224] Handling call
0709 12:24:22.088070 (+ 28419us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:22.210706 (+122636us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.210748 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:22.211392 (+   644us) log.cc:844] Serialized 52 byte log entry
0709 12:24:22.381586 (+170194us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:22.381653 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.130137 (+748484us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b540030aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:23.130189 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.130236 (+    47us) raft_consensus.cc:1685] finished
0709 12:24:23.130248 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.130495 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9251,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":129,"spinlock_wait_cycles":257717120,"thread_start_us":82657,"threads_started":1,"wal-append.queue_time_us":172950}
I20260709 12:24:23.101680  7218 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.117215  1723 rpcz_store.cc:276] 0709 12:24:22.013921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.143466 (+129545us) service_pool.cc:224] Handling call
0709 12:24:22.176171 (+ 32705us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.376553 (+200382us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:22.496029 (+119476us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.008073 (+512044us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:23.019490 (+ 11417us) log.cc:844] Serialized 26 byte log entry
0709 12:24:23.019703 (+   213us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:23.019738 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.019804 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.088525 (+ 68721us) raft_consensus.cc:1685] finished
0709 12:24:23.088560 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.088805 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11021,"prepare.queue_time_us":279437,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":62,"raft.queue_time_us":146926,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":11131,"thread_start_us":805254,"threads_started":4,"wal-append.queue_time_us":271612}
I20260709 12:24:23.147679  7486 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:23.148249  7486 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:23.148514  7218 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:23.151556  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.150692  1803 raft_consensus.cc:2410] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:24:23.151908  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.153111  1806 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:23.161461  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:23.096853  7480 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:23.100890  7230 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:23.166131  1880 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:23.176100  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:23.177830  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.185384  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:23.192613  1881 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:23.204320  1798 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:23.149376  7486 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.164129  7334 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:23.206177  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:23.111147  7377 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.173391  7335 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.213299  7463 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.214763  7258 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:23.209981  7482 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.182440  7474 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:23.204078  7479 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:23.178043  7222 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.173794  7489 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:24:23.173426  7473 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:24:23.229684  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:23.185253  7359 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:24:23.221607  7452 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.225760  7005 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:24:23.269958  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:23.221763  7483 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:23.232019  7313 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.229031  7479 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:23.266749  7231 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.003s
I20260709 12:24:23.219996  7334 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.224258  7335 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.277385  7479 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:23.276549  7474 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:23.279062  6875 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:23.280416  7437 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.470s	user 0.007s	sys 0.000s
I20260709 12:24:23.247386  7222 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.272166  7005 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:23.277251  7313 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.294456  7482 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:23.296985  7377 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.297675  7222 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.295153  7313 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.295468  7474 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:23.275794  7480 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.300367  7474 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:23.313241  7491 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.418s	user 0.003s	sys 0.000s
W20260709 12:24:23.313335  7450 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.182s	user 0.000s	sys 0.003s
W20260709 12:24:23.316543  7459 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.062s	user 0.001s	sys 0.002s
I20260709 12:24:23.318951  7479 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:23.360853  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 156988us
I20260709 12:24:23.364315  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2313) took 2948 ms. Trace:
I20260709 12:24:23.368851  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2262) took 1964 ms. Trace:
I20260709 12:24:23.369933  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:23.371948  1798 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:23.372226  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:23.364977  1724 rpcz_store.cc:276] 0709 12:24:20.415650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.006090 (+590440us) service_pool.cc:224] Handling call
0709 12:24:21.012255 (+  6165us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:21.012857 (+   602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.012899 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:21.013638 (+   739us) log.cc:844] Serialized 52 byte log entry
0709 12:24:21.416736 (+403098us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:21.416809 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.416926 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.279793 (+1862867us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:23.363309 (+ 83516us) raft_consensus.cc:1685] finished
0709 12:24:23.363393 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.364016 (+   623us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":14265,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":14013,"spinlock_wait_cycles":48292736,"thread_start_us":337826,"threads_started":1,"wal-append.queue_time_us":740702}
I20260709 12:24:23.355088  1626 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 9 to 8, leader changed from <none> to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 8 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:23.369568  1800 rpcz_store.cc:276] 0709 12:24:21.404442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.410957 (+  6515us) service_pool.cc:224] Handling call
0709 12:24:21.411426 (+   469us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:21.422839 (+ 11413us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.422904 (+    65us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:21.449039 (+ 26135us) log.cc:844] Serialized 52 byte log entry
0709 12:24:21.889812 (+440773us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:22.135327 (+245515us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.148834 (+ 13507us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.148951 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.143721 (+994770us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b1800449c80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:23.312582 (+168861us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:23.367798 (+ 55216us) raft_consensus.cc:1685] finished
0709 12:24:23.367867 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.368456 (+   589us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45539,"prepare.queue_time_us":452119,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":42373,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":21799,"spinlock_wait_cycles":39064704,"thread_start_us":619470,"threads_started":2,"wal-append.queue_time_us":536178}
I20260709 12:24:23.378086  1626 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 10, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 10 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:23.381227  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:23.383042  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:23.385546  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:23.389036  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:23.397547  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 766) took 1371 ms. Trace:
I20260709 12:24:23.399117  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:23.398348  1875 rpcz_store.cc:276] 0709 12:24:22.026235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.947784 (+921549us) service_pool.cc:224] Handling call
0709 12:24:22.948256 (+   472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.948736 (+   480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.948763 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:22.949352 (+   589us) log.cc:844] Serialized 26 byte log entry
0709 12:24:23.222337 (+272985us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:23.222397 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.222486 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.394209 (+171723us) raft_consensus.cc:1685] finished
0709 12:24:23.394271 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.397201 (+  2930us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":272539,"threads_started":1,"wal-append.queue_time_us":336354}
I20260709 12:24:23.388545  1626 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 9, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 9 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:23.425946  7222 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 12:24:23.428988  6875 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:23.429770  6875 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.430322  6875 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.432009  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.433076  7005 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.259330  7230 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.436918  7222 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:23.436465  7277 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:23.449515  7334 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.405613  1626 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 5 to 2, term changed from 4 to 10, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66), 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from VOTER to NON_VOTER, now has a pending config: VOTER 075704438992440697732181e49ccd09 (127.0.140.65), VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66), VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67), VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 10 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:23.455148  7391 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:23.456120  7391 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:23.457161  1626 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 8 to 11. New cstate: current_term: 11 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:23.454586  7409 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.314s	user 0.000s	sys 0.003s
W20260709 12:24:23.463971  7409 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.332s	user 0.001s	sys 0.003s
I20260709 12:24:23.464942  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 776) took 2844 ms. Trace:
I20260709 12:24:23.465523  1725 rpcz_store.cc:276] 0709 12:24:20.619901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.982306 (+362405us) service_pool.cc:224] Handling call
0709 12:24:23.464227 (+2481921us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b54000d2ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:23.464627 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1832176896}
I20260709 12:24:23.469870  7005 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:24:23.471608  7479 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.449617  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 765) took 1435 ms. Trace:
I20260709 12:24:23.458046  7391 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.451647  7277 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.446389  7452 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 12:24:23.486128  7480 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:23.489344  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 730) took 1810 ms. Trace:
I20260709 12:24:23.489619  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.489921  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.490299  7335 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.490648  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2231) took 1774 ms. Trace:
I20260709 12:24:23.491930  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.492429  7463 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:23.495913  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:23.476899  7222 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:23.513983  1626 catalog_manager.cc:5697] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:23.516848  1626 catalog_manager.cc:5697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 5 to 8, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 8 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:23.496343  7258 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.519222  7258 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:23.503182  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3085) took 1681 ms (client timeout 2155 ms). Trace:
W20260709 12:24:23.522037  1874 rpcz_store.cc:269] 0709 12:24:21.821423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.821558 (+   135us) service_pool.cc:224] Handling call
0709 12:24:23.496511 (+1674953us) spinlock_profiling.cc:243] Waited 26 ms on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:23.502760 (+  6249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59795840}
I20260709 12:24:23.497979  7277 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:23.524968  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:23.527138  7335 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:23.527500  7480 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:23.509950  7463 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.529842  7463 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.500519  7452 consensus_queue.cc:1243] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 17.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:24:23.507123  1807 rpcz_store.cc:276] 0709 12:24:21.702758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.013339 (+310581us) service_pool.cc:224] Handling call
0709 12:24:22.013631 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.378137 (+364506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.475996 (+1097859us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b54003b88d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:23.476571 (+   575us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":757354240}
I20260709 12:24:23.538712  7337 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:23.539621  7337 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.542204  7337 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.477519  1873 rpcz_store.cc:276] 0709 12:24:22.013433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.925654 (+912221us) service_pool.cc:224] Handling call
0709 12:24:22.926198 (+   544us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.963430 (+ 37232us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:23.160616 (+197186us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.395297 (+234681us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:23.396501 (+  1204us) log.cc:844] Serialized 26 byte log entry
0709 12:24:23.396872 (+   371us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:23.396931 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.397087 (+   156us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.449017 (+ 51930us) raft_consensus.cc:1685] finished
0709 12:24:23.449049 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.449299 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":296302,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":2034,"raft.queue_time_us":41464,"raft.run_cpu_time_us":517,"raft.run_wall_time_us":521,"thread_start_us":383383,"threads_started":2,"wal-append.queue_time_us":111409}
I20260709 12:24:23.548346  1626 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 10, leader changed from <none> to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 10 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:23.558569  1798 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
W20260709 12:24:23.559485  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.562685  7258 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:23.563086  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.563637  7258 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.564922  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2275) took 1806 ms. Trace:
I20260709 12:24:23.564775  7258 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.505824  1804 rpcz_store.cc:276] 0709 12:24:21.678650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.681816 (+  3166us) service_pool.cc:224] Handling call
0709 12:24:21.700688 (+ 18872us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:23.012774 (+1312086us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.488612 (+475838us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400066654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:23.489120 (+   508us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2492423040}
I20260709 12:24:23.565441  1799 rpcz_store.cc:276] 0709 12:24:21.758175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.051225 (+293050us) service_pool.cc:224] Handling call
0709 12:24:22.808065 (+756840us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.899025 (+ 90960us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.899060 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:22.904100 (+  5040us) log.cc:844] Serialized 26 byte log entry
0709 12:24:23.157211 (+253111us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:23.157260 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.157335 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.375033 (+217698us) raft_consensus.cc:1685] finished
0709 12:24:23.375086 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.564389 (+189303us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54001d5450. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:23.564716 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36084,"raft.queue_time_us":91021,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":1710780288,"thread_start_us":248871,"threads_started":1,"wal-append.queue_time_us":340468}
I20260709 12:24:23.566138  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:23.570381  1798 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:23.571789  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.572763  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.560325  1626 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 9, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 9 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:23.573645  7452 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.628897  7203 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.573712  7222 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:24:23.645812  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.646466  7474 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.645855  7407 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.647142  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.647462  7474 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.647866  7407 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.649113  7218 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:23.650039  7505 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.370s	user 0.000s	sys 0.001s
I20260709 12:24:23.650163  7218 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.655515  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.656525  1873 raft_consensus.cc:2376] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15. Re-sending same reply.
I20260709 12:24:23.663470  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.674886  7322 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:23.670857  7228 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:23.677639  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.674876  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 112781us
I20260709 12:24:23.633332  7452 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:23.684270  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 93200.9us
I20260709 12:24:23.684633  7203 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.686667  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:23.687788  2410 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:23.690294  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:23.691082  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:23.692060  2414 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:23.695923  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.696970  1724 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:23.697232  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.698555  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:23.698442  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.699699  1723 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:23.700886  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.701300  1626 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 9 to 11, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 11 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:23.703429  1725 raft_consensus.cc:2410] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:24:23.705423  7258 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.658301  7407 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.644863  7361 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.731382  7228 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.741427  7444 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.742255  7444 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.743168  7361 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:23.747345  7519 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:24:23.748826  7328 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:23.749565  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.752017  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:23.753623  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.756278  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:23.761234  2411 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:23.762059  1877 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:24:23.763331  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:23.763563  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 763) took 1750 ms. Trace:
I20260709 12:24:23.763815  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.764568  7452 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.764314  1731 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:23.766700  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.767633  1731 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:23.769109  7409 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.002s
I20260709 12:24:23.773424  7228 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.750659  1874 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.774993  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.775254  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:23.776623  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:23.761226  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2385) took 1816 ms. Trace:
I20260709 12:24:23.777079  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.777791  1724 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:23.777477  1879 rpcz_store.cc:276] 0709 12:24:21.944576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.155822 (+211246us) service_pool.cc:224] Handling call
0709 12:24:23.760755 (+1604933us) spinlock_profiling.cc:243] Waited 99.6 ms on lock 0x7b54002dfad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:23.760989 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228975872}
I20260709 12:24:23.769410  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.779635  1731 raft_consensus.cc:2410] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:24:23.779839  7322 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.780400  7484 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:23.780570  7322 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.783267  7218 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.784368  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:23.763913  1877 rpcz_store.cc:276] 0709 12:24:22.012901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.299848 (+286947us) service_pool.cc:224] Handling call
0709 12:24:23.761398 (+1461550us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:23.762580 (+  1182us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.763323 (+   743us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54002dfad0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:23.763467 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3360110464}
I20260709 12:24:23.785665  1731 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:24:23.784052  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:23.788270  7337 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:23.788126  7277 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.770174  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2247) took 1270 ms. Trace:
I20260709 12:24:23.790966  1806 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 12.
I20260709 12:24:23.784032  7322 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:23.784143  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.749595  7486 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:23.792738  1801 rpcz_store.cc:276] 0709 12:24:22.499636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.527107 (+ 27471us) service_pool.cc:224] Handling call
0709 12:24:22.527574 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.528035 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.528078 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:22.535506 (+  7428us) log.cc:844] Serialized 26 byte log entry
0709 12:24:23.136637 (+601131us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b18003e4520. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb557 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:23.473921 (+337284us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b18003e4520. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:23.627251 (+153330us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:23.627316 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.635218 (+  7902us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.769193 (+133975us) raft_consensus.cc:1685] finished
0709 12:24:23.769218 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.769508 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":92727,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":31994,"spinlock_wait_cycles":76362368,"thread_start_us":104970,"threads_started":1,"wal-append.queue_time_us":1050530}
I20260709 12:24:23.798478  7361 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.799571  7474 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.821815  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:23.825222  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.834833  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.842656  1731 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:23.832878  1806 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:23.842325  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:23.860765  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 193054us
I20260709 12:24:23.881560  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.822686  7258 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:23.830192  7322 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.835404  1626 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:23.833721  7277 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:23.895838  1801 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:23.893898  1875 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:23.887507  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.904881  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.876595  7203 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.891386  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:23.909502  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.906107  7222 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:23.913617  7222 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.914026  7444 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:23.914923  7322 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:23.924973  7277 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:23.926024  7277 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.928046  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.928958  1801 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:23.929416  7277 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:23.930997  1876 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:23.931916  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:23.932823  7524 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:23.933409  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.934230  1880 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:23.935093  1626 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:23.937433  1626 catalog_manager.cc:5697] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:23.938110  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.938997  1881 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:23.902065  7361 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.939453  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:23.941067  1872 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:23.943559  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.944417  1875 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:23.945537  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.947160  2410 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:23.948340  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:23.949144  2416 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:23.948968  7005 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:23.884850  7337 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.919615  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.953253  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.946355  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:23.974319  1875 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:23.985968  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:23.987331  1876 pending_rounds.cc:85] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 8
I20260709 12:24:23.989668  1798 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:23.990420  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:23.984241  7321 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:23.993070  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:23.967837  7277 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:23.996402  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:23.992089  7321 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.992012  7203 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:23.998420  7328 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.998739  7358 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:24.009907  6875 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.550s	user 0.016s	sys 0.002s
I20260709 12:24:24.034070  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:23.987814  1876 pending_rounds.cc:107] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 12:24:23.947202  7463 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:23.940209  7361 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:24.014840  1875 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:24:24.013522  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:23.955734  7258 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.012415  7321 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:23.981087  7337 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:24.036252  1876 raft_consensus.cc:2889] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:24.048354  1807 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 14 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:24:24.037165  2416 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:24.007975  1874 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
W20260709 12:24:24.060286  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 228411us
I20260709 12:24:24.063675  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.072153  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:24.079756  1880 pending_rounds.cc:85] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 10
I20260709 12:24:24.080332  1880 pending_rounds.cc:107] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260709 12:24:24.073532  2410 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:24.034705  6875 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.080655  1880 raft_consensus.cc:2889] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:24.079368  7252 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.024s
I20260709 12:24:24.088158  7322 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:24.088660  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.083453  1881 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:24:24.089517  1725 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:24.030967  7277 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:24.096154  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 141593us
I20260709 12:24:24.037276  7203 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.100288  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.101382  1872 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:24.052765  1626 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 9 to 11, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 11 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:24.110289  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 164093us
I20260709 12:24:24.109540  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:24.114802  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 220818us
I20260709 12:24:24.054144  7328 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.122875  7230 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:24.124522  7407 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.125341  7407 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:24.127857  7407 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:24.128465  7407 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.129132  7407 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.131810  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:24.133841  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:24.134078  7361 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:24.135108  7365 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:24.131817  7407 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.136603  6875 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.140436  7337 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:24.146864  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:24.148196  2419 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.157509  2410 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:24.164536  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.173848  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.175320  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:24.177145  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:24.134734  7361 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:24.196249  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:24.209129  7222 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:24.209698  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45788.7us
I20260709 12:24:24.210059  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:24.211616  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:24.212492  7277 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.213460  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:24.211613  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.215963  2410 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:24.218343  1724 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:24.237849  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.209110  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:24.247085  2411 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:24.221508  7005 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:24.251808  2416 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:24.256853  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.255652  1723 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:24.266510  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:24.275826  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 268272us
I20260709 12:24:24.172422  1731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:24.268078  7005 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:24.284535  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 71820.3us
I20260709 12:24:24.295192  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:24.297495  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:24.300045  7544 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.300262  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.301178  1725 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:24.302069  2416 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:24.302845  2412 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:24:24.300843  7544 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.285109  7005 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.282670  7463 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.317528  1730 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
W20260709 12:24:24.318158  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 184491us
I20260709 12:24:24.307116  1731 raft_consensus.cc:606] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:24.345224  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.346344  1726 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:24.345350  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.345253  1731 raft_consensus.cc:614] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:24.361986  1732 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:24.363001  1724 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:24.363551  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86282.3us
W20260709 12:24:24.359128  7486 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.539s	user 0.011s	sys 0.004s
I20260709 12:24:24.337747  7258 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.371658  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:24.373317  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:24.368970  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:24.366946  7484 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.586s	user 0.011s	sys 0.005s
I20260709 12:24:24.380800  1725 raft_consensus.cc:2410] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:24:24.381122  1626 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:24.383579  7328 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.383278  1802 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
W20260709 12:24:24.391906  7530 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:24:24.408223  7544 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:24.398029  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:24.394640  7534 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:24:24.411414  2415 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:24.412631  7407 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:24.413314  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:24.413599  7407 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:24.415307  7407 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:24.417009  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:24.417845  1724 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.422693  7222 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:24.425526  7222 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.425995  7222 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:24.398110  7484 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:24.427871  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:24.428993  7230 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:24.429561  7230 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.429970  7230 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.432436  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:24.433487  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.433895  1804 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:24.434226  1878 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
W20260709 12:24:24.432677  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 58559us
I20260709 12:24:24.433020  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:24.436297  1807 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:24.411072  7203 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.437863  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:24.439244  1800 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 17 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:24:24.447904  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:24.448611  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.451263  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:24.451313  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.452165  1804 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:24:24.396034  7537 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:24:24.453121  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 129784us
I20260709 12:24:24.452183  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.385681  7545 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:24.480334  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:24.453130  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.449620  1881 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:24.485760  1881 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:24.476076  1807 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:24.498921  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63284us
I20260709 12:24:24.502488  1724 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:24.523893  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:24.524780  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:24.525828  1872 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:24.523598  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:24.531718  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 154446us
W20260709 12:24:24.532240  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 183343us
I20260709 12:24:24.532094  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.532658  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:24.535774  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.536621  1725 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:24.543336  1800 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:24:24.441573  7321 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.539166  1873 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 10.
I20260709 12:24:24.546519  1626 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 4 to 6, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 6 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:24:24.531734  1845 connection.cc:256] RPC call timeout handler was delayed by -2.2047031819997756s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21589999999999998s for a response
I20260709 12:24:24.537680  7203 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.562624  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:24.543020  7444 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:24.566545  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:24.578264  2419 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:24.579972  2417 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:24.583385  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.583741  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.576190  7484 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:24.586513  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:24.587314  1724 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:24.588326  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:24.587798  1626 catalog_manager.cc:5697] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:24.589022  7535 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.589701  7535 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.591328  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.591594  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:24.592236  1732 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:24.593457  6875 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:24.593487  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.595877  1726 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:24.599032  1725 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:24.590291  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:24.600315  1724 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:24.602035  7452 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:24.602248  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.601684  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:24.603335  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.603565  7452 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:24.604100  7452 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.604173  1880 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:24.604664  7452 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.606065  1880 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:24.607280  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.608172  1881 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:24.609098  7463 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.609638  1874 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:24.609894  7463 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.610385  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:24.611404  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:24.612159  1879 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:24.612574  7463 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:24.615734  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68993.9us
I20260709 12:24:24.617368  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:24.548910  7358 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:24.620882  7358 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:24.582988  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:24.623973  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67420.5us
W20260709 12:24:24.625043  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89078.8us
I20260709 12:24:24.626253  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:24.627758  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:24.628739  7398 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:24.624091  7321 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.633744  1801 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:24.580580  7203 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:24.655978  7548 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.272s	user 0.002s	sys 0.000s
I20260709 12:24:24.656783  1724 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 12:24:24.658532  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.659624  1725 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 15 index: 13. (term mismatch)
I20260709 12:24:24.660171  1725 pending_rounds.cc:85] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 12
I20260709 12:24:24.660455  1725 pending_rounds.cc:107] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 14.13
I20260709 12:24:24.660730  1725 raft_consensus.cc:2889] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:24.661118  1725 pending_rounds.cc:107] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 12:24:24.661383  1725 raft_consensus.cc:2889] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:24.663292  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.664526  1732 raft_consensus.cc:2410] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:24:24.665704  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:24.670253  7543 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.294s	user 0.000s	sys 0.002s
I20260709 12:24:24.670832  7479 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:24.685483  7358 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:24.671106  1727 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:24.671383  7484 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:24.666575  7005 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:24.672351  2383 leader_election.cc:336] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.847s (SENT)
I20260709 12:24:24.675379  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.576395  7444 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.703998  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:24.705557  7484 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:24.706526  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:24.713268  7444 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.701121  1800 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:24.706362  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:24.704069  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:24.724546  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:24.724025  1872 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:24:24.719972  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 497979us
I20260709 12:24:24.734064  1727 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:24.736677  1875 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:24.761464  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61181.5us
W20260709 12:24:24.761988  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55745.8us
I20260709 12:24:24.701028  7545 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:24.764742  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:24.726567  7321 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:24.751533  7479 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:24.766146  1732 raft_consensus.cc:2376] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15. Re-sending same reply.
I20260709 12:24:24.780205  1728 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:24:24.762914  7203 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.763269  7463 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:24.722267  7409 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.045s
I20260709 12:24:24.768185  7358 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:24.717734  7230 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.726110  7544 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.061s
I20260709 12:24:24.759809  7222 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:24.778388  7321 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.776087  7552 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:24:24.815639  7321 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:24.818557  7474 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:24.819690  7535 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:24.809573  7545 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 12:24:24.821563  7545 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:24.821971  7545 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:24.823468  7545 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:24.825124  7545 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:24.826666  7222 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:24.798962  7358 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.788754  7452 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.838383  1724 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:24.845937  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 745) took 1347 ms (client timeout 1749 ms). Trace:
I20260709 12:24:24.844594  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:24:24.848312  1802 rpcz_store.cc:269] 0709 12:24:23.488764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.489008 (+   244us) service_pool.cc:224] Handling call
0709 12:24:24.836143 (+1347135us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b54001e3f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:24.836342 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1999727360}
I20260709 12:24:24.833807  7358 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:24.836826  6875 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:24.848857  7391 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:24.866643  7328 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:24.816864  7463 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.883823  7540 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:24.886994  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.896072  1730 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:24.851737  7484 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:24.929126  1807 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:24.881650  7328 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.927733  7005 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:24.947738  7452 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:24.950624  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 119556us
W20260709 12:24:24.940547  7361 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.720s	user 0.017s	sys 0.000s
W20260709 12:24:24.989176  1845 connection.cc:256] RPC call timeout handler was delayed by -1.6517771680000806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1824s for a response
W20260709 12:24:24.989957  1845 connection.cc:256] RPC call timeout handler was delayed by -1.5052938329995413s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
W20260709 12:24:24.990350  1845 connection.cc:256] RPC call timeout handler was delayed by -1.482577167999807s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20120000000000002s for a response
W20260709 12:24:24.990732  1845 connection.cc:256] RPC call timeout handler was delayed by -1.3639826449998509s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
W20260709 12:24:24.991122  1845 connection.cc:256] RPC call timeout handler was delayed by -1.3195036799997979s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1874s for a response
W20260709 12:24:24.991488  1845 connection.cc:256] RPC call timeout handler was delayed by -1.27737716799993s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22400000000000003s for a response
I20260709 12:24:24.943210  7328 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:25.015130  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 548730us
W20260709 12:24:24.959699  7536 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.012s	sys 0.009s
I20260709 12:24:24.952102  7486 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.048965  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:25.051616  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:25.024005  7547 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
W20260709 12:24:25.030623  7536 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.012s	sys 0.009s
I20260709 12:24:25.024446  7361 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:25.107578  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:25.108456  2413 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:25.078341  7444 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.109807  2417 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:25.110347  7444 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:25.111276  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:25.111491  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:25.111937  2412 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.113294  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:25.114003  2414 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:25.116503  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:25.117086  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.118285  1880 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:25.119307  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:25.119583  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:25.120043  1878 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:25.120577  1876 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:24:25.122258  2412 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:25.122720  7058 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:25.123263  7058 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.123638  7058 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.126947  7412 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:25.123397  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 398951us
W20260709 12:24:25.133502  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 371166us
I20260709 12:24:25.063755  7005 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:25.145304  2411 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:25.134733  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.141640  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:25.154546  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.126519  2414 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:25.160728  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:25.166826  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.169111  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:25.177657  2411 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:25.178975  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 153549us
I20260709 12:24:25.144776  7005 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:25.122880  7486 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.134225  7564 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:24:25.181640  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:25.196235  7222 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:25.196911  7444 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:25.183630  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:25.192859  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:25.198030  7222 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:25.194410  7564 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:25.197906  7444 consensus_queue.cc:1243] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 12.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:25.194078  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:25.196621  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:25.201026  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:25.181159  7407 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:25.164901  7361 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:25.210878  7550 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:25.176499  7544 raft_consensus.cc:2764] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:24:25.164207  7230 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.212239  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.189782  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:25.231063  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.127774  7412 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.232645  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:25.190577  7005 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.269645  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:25.273200  2410 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 17 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:24:25.274602  2416 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:25.032850  7484 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:24.884727  7540 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:25.279422  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:25.280320  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:25.283206  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52212.9us
I20260709 12:24:25.230401  7407 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.214663  7252 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:25.224200  7222 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:25.267323  7398 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:25.232844  7561 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:25.223256  7564 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:25.310876  7322 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:24:25.300846  7407 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:25.319132  7545 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.319293  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:25.320115  2418 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:25.319963  7545 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.320567  7564 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.322216  7545 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:25.323050  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 122172us
W20260709 12:24:25.324111  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 127552us
W20260709 12:24:25.324561  7321 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.505s	user 0.011s	sys 0.000s
I20260709 12:24:25.325073  7321 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:25.325759  1698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.000s	sys 0.027s
W20260709 12:24:25.326067  1698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.000s	sys 0.027s
I20260709 12:24:25.307494  7544 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:25.327527  7544 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.328095  7544 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:25.271584  7550 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:25.331838  1872 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
W20260709 12:24:25.327654  7570 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:24:25.322463  7564 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:25.198802  7558 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.114s
I20260709 12:24:25.303218  7222 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:25.303831  7398 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.287287  7361 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.362361  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:25.367782  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:25.359045  7545 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:25.363584  7398 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:25.395859  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:25.396901  1874 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:25.371829  7463 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.380781  7444 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:25.362262  7358 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.359290  7561 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.381505  7321 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.426474  7058 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.431870  7412 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:25.438340  7573 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:25.440253  7550 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.441558  7576 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:25.443723  7484 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.444643  7486 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.384684  7558 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
W20260709 12:24:25.486624  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.387934  7328 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.447520  7361 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:25.447667  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.399240  7545 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.450129  7444 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:25.450361  7358 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:25.451781  7588 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:25.493894  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:25.495908  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:25.497331  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49793.8us
I20260709 12:24:25.497682  1730 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:25.471943  7463 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20260709 12:24:25.481122  7412 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:25.491117  7230 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:25.508654  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 210688us
I20260709 12:24:25.492431  7484 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.505844  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:25.508253  1727 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 12 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:25.487995  7576 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:25.477149  7005 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.512056  7361 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.514572  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:25.526258  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:25.526654  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:25.527029  1725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:25.527318  1730 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:25.528434  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:25.529119  1802 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:25.529371  1800 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:24:25.503360  7561 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.516952  7588 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.530828  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:25.531431  1806 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:25.532567  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 236582us
I20260709 12:24:25.534044  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.533933  7561 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:25.494151  7590 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.536598  7590 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.539206  7588 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:25.518378  7444 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.540277  1806 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:25.541435  7252 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:25.544281  7590 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:25.498874  7573 raft_consensus.cc:2764] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:24:25.566479  1806 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
W20260709 12:24:25.563998  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62404.5us
W20260709 12:24:25.567785  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.502693  7486 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:25.571457  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:25.575490  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:25.577366  1805 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:25.576611  1728 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:25.580566  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:25.582924  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 198827us
I20260709 12:24:25.585326  1727 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:25.586364  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.592888  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2312) took 1093 ms. Trace:
W20260709 12:24:25.612016  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 102247us
I20260709 12:24:25.623406  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:25.582834  7590 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.573520  7358 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:25.592262  7321 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:25.566422  7484 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.631045  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:25.653324  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70153.5us
I20260709 12:24:25.606527  7561 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:25.679343  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 169401us
I20260709 12:24:25.575552  7576 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.584910  7412 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:25.697311  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 66388.6us
I20260709 12:24:25.606889  7575 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:25.542552  7252 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.699347  7604 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:25.703274  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 79932.2us
I20260709 12:24:25.602036  7361 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.705108  1803 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 15 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:24:25.705293  1725 raft_consensus.cc:606] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:24:25.665319  7561 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.606712  7486 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:25.758845  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44057.9us
I20260709 12:24:25.730907  7604 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:25.660578  7590 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.762467  1725 raft_consensus.cc:614] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:25.824167  1800 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 9. Current term is 10. Ops: []
I20260709 12:24:25.829952  7412 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.831441  7599 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:25.727437  7463 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.692878  7358 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:25.839860  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 77776.2us
I20260709 12:24:25.842605  1806 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: [13.10-15.11]
I20260709 12:24:25.762357  7058 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.775449  7561 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:25.839938  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 117724us
I20260709 12:24:25.845337  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:25.851029  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.844182  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:25.854866  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:25.859819  1800 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:24:25.813755  7604 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:25.854509  7407 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.539s	user 0.008s	sys 0.005s
I20260709 12:24:25.854866  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:25.838753  7328 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:25.868875  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54632us
I20260709 12:24:25.832808  7398 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.872133  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 789) took 1114 ms. Trace:
I20260709 12:24:25.875504  7321 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:25.875972  7576 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:25.865907  7463 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:25.876950  7587 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:24:25.866200  7005 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:25.845558  7599 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:25.879705  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:25.857576  7545 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:25.886605  7585 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
W20260709 12:24:25.887647  7596 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
I20260709 12:24:25.889539  6875 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:25.890398  7590 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.957931  7391 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:25.959151  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51784.3us
I20260709 12:24:25.960125  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:25.962071  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:25.962934  7486 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.963639  7612 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:25.966320  7557 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:25.966802  7557 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:25.967736  7557 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.969211  7557 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:25.892994  1727 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:25.895709  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:25.879655  7544 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.003s
I20260709 12:24:25.984820  7544 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:25.986477  7581 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:25.814388  7575 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.987175  7581 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.987646  7581 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:25.989367  7575 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:25.991042  7058 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:25.991370  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:25.991863  7544 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:25.991084  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 110494us
I20260709 12:24:25.994185  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:25.995888  7550 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:24:25.997164  7550 consensus_queue.cc:1243] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:24:25.958611  7391 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:25.998871  7550 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:25.999087  7578 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:25.999658  7550 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:26.000324  7550 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:25.883833  7321 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:25.923756  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69029.4us
I20260709 12:24:25.925086  7599 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.003047  7321 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:25.907068  7564 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.548s	user 0.011s	sys 0.001s
I20260709 12:24:25.925612  7484 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:25.926985  7591 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:25.931639  7561 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.992384  7258 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:26.007427  7258 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:26.008870  7561 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:26.008694  7258 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:26.009586  7222 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.637s	user 0.013s	sys 0.000s
I20260709 12:24:25.880517  1729 rpcz_store.cc:276] 0709 12:24:24.588699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.650594 (+ 61895us) service_pool.cc:224] Handling call
0709 12:24:24.651280 (+   686us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:24.723144 (+ 71864us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:24.723194 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:24.776297 (+ 53103us) log.cc:844] Serialized 52 byte log entry
0709 12:24:25.276276 (+499979us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:25.276321 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.276416 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:25.703219 (+426803us) raft_consensus.cc:1685] finished
0709 12:24:25.703245 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.703439 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27512,"raft.queue_time_us":68163,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":38194,"thread_start_us":479880,"threads_started":1,"wal-append.queue_time_us":724652}
I20260709 12:24:25.996419  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.010130  7222 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.012648  7222 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.012969  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:25.982721  7005 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 12:24:26.014175  7558 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:25.908025  1806 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:26.014758  7558 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.015161  7558 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:26.015332  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2421) took 1415 ms. Trace:
I20260709 12:24:26.015733  7005 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:26.016006  7561 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:26.016851  7561 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:25.879983  7407 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.017362  7561 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:26.017349  7606 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:26.018155  7606 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:26.019227  7606 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.992502  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.021086  7573 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:26.021533  7573 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.021868  7573 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:25.977262  7595 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.247s	user 0.000s	sys 0.002s
I20260709 12:24:26.015805  1875 rpcz_store.cc:276] 0709 12:24:24.600296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.610109 (+  9813us) service_pool.cc:224] Handling call
0709 12:24:26.015114 (+1405005us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b5400222c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:26.015268 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":257010048}
I20260709 12:24:25.936470  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.005774  7230 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:26.023195  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:26.037091  1800 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 9.
I20260709 12:24:26.051065  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.020056  7407 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.047425  7058 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:26.035880  7230 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:26.063642  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 56902.1us
I20260709 12:24:26.064515  1802 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:24:26.065541  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.066955  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.068318  1731 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:26.069722  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.064388  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.023243  6875 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.039s
W20260709 12:24:26.088850  2383 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.505s (SENT)
I20260709 12:24:26.064949  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.091253  1729 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:26.091897  1726 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:26.093271  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.036073  7564 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:24:26.073467  7606 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:26.106115  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.115933  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.106215  7620 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.087s
I20260709 12:24:26.139434  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.140180  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.177605  1805 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:26.143054  1726 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:26.178470  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.143240  1802 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:26.143782  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49608.6us
I20260709 12:24:26.135730  7591 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:26.072284  7603 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.175700  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:26.183022  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.185833  1874 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:26.187548  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.188283  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.189145  1798 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:26.189327  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.101346  7578 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.190577  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.190222  1876 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:26.208158  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.195322  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.179955  7252 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.248375  7252 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.184381  7590 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.069s
W20260709 12:24:26.252818  7618 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20260709 12:24:26.253453  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 164633us
W20260709 12:24:26.254604  2383 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.510s (SENT)
I20260709 12:24:26.257305  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.258291  1798 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:26.259351  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.260353  1798 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:26.261510  1798 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:24:26.262291  1875 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 10.
I20260709 12:24:26.209391  7321 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.265128  1877 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:26.265439  7321 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:26.267482  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.210906  7603 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.268381  1697 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.269254  6875 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.270113  7218 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.270041  6875 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.270747  7218 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.174185  7484 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.272277  6875 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.272033  7484 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.272819  6875 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:26.272943  7218 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.273358  6875 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.184343  7591 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.274164  7591 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:26.274255  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.251987  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:26.275220  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.275301  1805 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:26.276305  7600 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:26.094558  7486 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.098s
I20260709 12:24:26.277791  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:26.251739  7621 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:24:26.274255  7484 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.188175  7564 consensus_queue.cc:1243] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 12.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:24:26.279429  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3114) took 1163 ms. Trace:
W20260709 12:24:26.279716  7589 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
I20260709 12:24:26.280821  1806 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:26.281008  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.281345  6875 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.273628  7218 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:26.284716  1872 rpcz_store.cc:276] 0709 12:24:25.116093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.116227 (+   134us) service_pool.cc:224] Handling call
0709 12:24:26.279210 (+1162983us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b540001ead4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:26.279353 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":493089024}
I20260709 12:24:26.208184  7588 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:26.181380  7545 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:26.296872  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:26.143612  7544 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:24:26.307396  1878 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:26.313545  6875 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:26.311112  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:26.316094  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.320529  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.191395  7328 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.121s
I20260709 12:24:26.270238  7603 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.324256  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.325188  1724 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:26.324923  7545 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.326103  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59174 (request call id 1) took 1489 ms. Trace:
I20260709 12:24:26.326807  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.326057  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.327253  1732 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:26.328037  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.328147  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.328699  1732 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:26.329107  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.329499  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.330111  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.330823  1724 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:26.333850  7561 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.324115  7252 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.343465  7604 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.343704  7391 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.346151  7391 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.273635  7540 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:26.349426  7558 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:25.597623  1804 rpcz_store.cc:276] 0709 12:24:24.499184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.499427 (+   243us) service_pool.cc:224] Handling call
0709 12:24:24.545371 (+ 45944us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:24.579590 (+ 34219us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:25.064090 (+484500us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.531230 (+467140us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.592389 (+ 61159us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:25.592451 (+    62us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:25.592560 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.592650 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.592814 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70678,"prepare.queue_time_us":727264,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":88,"raft.queue_time_us":171436,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":2454,"thread_start_us":762081,"threads_started":3,"wal-append.queue_time_us":511790}
W20260709 12:24:26.351034  7601 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.132s	user 0.005s	sys 0.000s
I20260709 12:24:26.316042  7486 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:26.352470  7361 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.352573  7486 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.353075  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.354297  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.354959  1801 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:26.359488  7545 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.359558  7573 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.360863  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.361840  1875 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:26.362429  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.308422  7588 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.364270  7588 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:26.363153  1724 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:26.365943  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.366525  1724 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:26.364904  7604 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.367947  7604 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.368382  7604 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.314651  7620 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.057s
I20260709 12:24:26.371214  7604 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.367318  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.372352  1724 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:26.372840  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.373270  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.373594  1801 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.374439  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.375430  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.375597  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.376140  1798 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:26.376758  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.376870  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:26.377079  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.377588  1724 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:26.377902  7602 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:26.378247  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.378345  7602 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:26.378877  7602 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:26.380774  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.382349  7603 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:26.383330  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.384248  1727 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:26.386137  7463 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.387253  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.388022  1799 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:26.388836  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.389320  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:26.389545  1799 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:26.390718  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:26.390960  7590 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:26.391475  7590 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.391886  7590 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:26.394245  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:26.363224  7486 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.395568  7361 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:26.396035  7361 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.396355  7361 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:26.397612  7557 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:26.398176  7557 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.399384  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:26.394827  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.402050  7550 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.312106  7484 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.377449  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.403648  7005 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:26.403883  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:26.404031  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:26.404263  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:26.405700  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:26.406421  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:26.407366  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:26.403306  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:26.401193  7557 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.407761  1806 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:24:26.403416  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.477275  1874 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:26.412220  7558 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:26.413591  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:26.414825  7604 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:26.326601  1723 rpcz_store.cc:276] 0709 12:24:24.851005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.859674 (+  8669us) service_pool.cc:224] Handling call
0709 12:24:26.325832 (+1466158us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54002d0ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdde735 00007f16abdd69fa 00007f16b0d9e4c9 00007f16aa31ac54 00007f16aa31abe2 00007f16aa31ab11 00007f16aa31aa8c 00007f16aa319d42 00007f16aa11348c 00007f16aa1128c6
0709 12:24:26.326007 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":916682112}
W20260709 12:24:26.421111  2382 connection.cc:256] RPC call timeout handler was delayed by -1.0576267690003078s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
I20260709 12:24:26.421597  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:26.423308  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:26.404263  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.506278  1804 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:26.425702  7590 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.427208  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:26.429296  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:26.390651  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.506944  1878 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.507655  1799 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:26.330541  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.433553  1801 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:26.418129  7591 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:26.444639  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 138621us
I20260709 12:24:26.509959  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:26.510618  2411 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:24:26.445663  1802 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:26.403198  7328 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:26.477016  1872 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:26.476415  1726 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:26.478156  2419 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.477759  2417 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:26.514039  2383 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.916s (SENT)
I20260709 12:24:26.477111  2415 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:26.476895  2413 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:26.478418  2416 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:26.408800  7322 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:26.476970  2410 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:26.485749  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.419188  7486 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:26.453752  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.517627  1877 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:26.481931  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.479857  7218 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:26.480540  1806 pending_rounds.cc:85] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 7
I20260709 12:24:26.508878  1732 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:26.438529  7573 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:26.503681  1873 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:26.515921  1807 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:26.516430  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.518384  1727 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:26.541637  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.528394  7590 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.521308  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.517369  7486 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.547226  7486 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.547329  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:26.547662  1723 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:26.480010  7005 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:26.518260  7627 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.530285  7322 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.523615  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.556847  1873 raft_consensus.cc:2368] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15.
I20260709 12:24:26.522408  1806 pending_rounds.cc:107] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:24:26.513949  7328 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:26.528728  7391 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.532073  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.563335  1799 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:26.532673  1614 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8bb813e4fab14a61b810a24b56899586 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 075704438992440697732181e49ccd09 (127.0.140.65:35067) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 075704438992440697732181e49ccd09 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 13 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:26.533082  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.529577  7620 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:26.523914  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.567466  1804 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:26.529763  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 102159us
I20260709 12:24:26.522225  7218 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.541834  7603 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:26.571411  7463 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.571762  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.572048  7603 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:26.572427  1873 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:26.572748  7603 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.573309  7603 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:26.573693  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:26.538064  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:26.573889  7391 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.574828  2414 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:26.575076  7590 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:26.536705  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:26.575636  7590 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:26.575714  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:26.576074  7391 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.576525  2414 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.561591  7627 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:26.577021  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72549.6us
I20260709 12:24:26.542174  2411 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:26.557029  7536 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:26.579892  2411 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:26.580561  7328 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.581789  7222 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:26.582271  7222 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.582887  7544 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.547662  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:26.583621  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47011.1us
I20260709 12:24:26.584928  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.397336  7581 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.585767  7599 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:26.586068  1872 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:24:26.586267  7599 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:26.586822  7599 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:26.303498  7578 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.555929  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.589676  7591 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.590370  7591 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:26.590714  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.591092  7614 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:24:26.591625  1876 pending_rounds.cc:85] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 11
I20260709 12:24:26.592017  1876 pending_rounds.cc:107] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 12.12
I20260709 12:24:26.592300  1876 raft_consensus.cc:2889] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:26.592881  1874 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:26.592670  7005 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.593976  7005 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:26.594889  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.596032  1874 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:26.542847  1805 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:26.598032  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.598289  1880 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:24:26.557673  1806 raft_consensus.cc:2889] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:26.598670  1874 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:26.599000  1729 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:26.557684  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.599485  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:26.559392  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.600625  1807 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:26.600858  7536 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:26.583067  7222 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:26.558382  7557 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:24:26.601956  7557 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:26.576265  7590 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.585013  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:26.556840  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:26.572021  7463 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:26.615521  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.020s	sys 0.039s
W20260709 12:24:26.615888  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.020s	sys 0.039s
I20260709 12:24:26.583670  7544 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.604153  2415 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 10.
I20260709 12:24:26.627882  1771 leader_election.cc:400] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15.
I20260709 12:24:26.628222  1771 leader_election.cc:403] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:24:26.561061  1801 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [14.8-16.9]
I20260709 12:24:26.628357  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:26.597018  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:26.600507  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.502599  7484 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.573474  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.531587  1878 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:26.630216  2414 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:26.582191  7328 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:26.610270  7600 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:26.609889  7602 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.610570  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:26.637001  1872 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:26.638938  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:26.446933  7540 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:26.602381  7557 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:26.616575  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.649943  1802 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:24:26.629190  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.609653  7218 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.632444  1873 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:26.635159  6875 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:26.632877  7591 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:26.627588  7536 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:26.635975  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.634773  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:26.609812  7627 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.625937  1799 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:26.652740  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:26.652863  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52437.3us
I20260709 12:24:26.662714  1807 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:26.663524  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:26.634585  7600 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.635514  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.573405  7361 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.643388  7444 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:26.575646  7230 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.644666  7540 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:26.645850  7620 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:26.683617  1872 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:26.662151  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.666460  1878 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:26.672883  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.673125  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.668107  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:26.679512  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.689162  1799 raft_consensus.cc:2410] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 18 because replica has last-logged OpId of term: 15 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 13.
W20260709 12:24:26.689482  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.683617  2419 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:26.669893  2417 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:26.709076  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.710772  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.709487  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.715088  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.714460  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.716118  1731 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:26.718858  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.717559  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.719499  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.720656  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.722060  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:26.723181  1878 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:26.724781  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.666426  7606 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:26.726735  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.727110  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.727841  1723 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:26.728816  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:26.671277  7600 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:26.726251  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.649259  7222 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.736310  1724 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:26.736315  1806 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:26.732065  1872 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:26.739658  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.745287  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.742700  1731 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:26.720332  7252 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:26.739720  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:26.751652  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:26.759325  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.759922  1807 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:26.760408  1799 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:26.761582  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.716105  7452 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:26.763196  1807 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:26.764830  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.731801  7614 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:26.726148  7540 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.780406  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.740763  7391 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:26.772678  7602 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:26.773445  7550 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:26.752013  7444 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:26.665344  7536 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.775190  7484 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.752215  7557 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.734205  7606 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.777962  7588 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:26.779455  1872 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:26.779837  1878 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:26.801870  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40200 (request call id 43) took 1272 ms. Trace:
I20260709 12:24:26.805271  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.825222  1725 rpcz_store.cc:276] 0709 12:24:25.493081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.526889 (+ 33808us) service_pool.cc:224] Handling call
0709 12:24:26.765542 (+1238653us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54000d20d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd2853 00007f16b0da060a 00007f16aa3267d4 00007f16aa326762 00007f16aa326691 00007f16aa32660c 00007f16aa3258c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:26.765638 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":408733568}
I20260709 12:24:26.826027  1799 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
W20260709 12:24:26.792302  1845 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.599s (SENT) [suppressed 16 similar messages]
I20260709 12:24:26.810499  1805 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:26.812877  1807 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:24:26.811825  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:26.801894  7452 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:26.816926  1804 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:26.824862  1874 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:24:26.788946  7544 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.812527  7252 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:26.840575  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.841636  1799 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:26.820267  7540 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:26.823807  7484 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:26.848462  1872 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:26.850803  1798 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:26.816273  7614 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:26.823776  7606 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:26.832450  7602 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:26.947192  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:26.949955  1725 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:26.834704  7544 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.094s
I20260709 12:24:26.955219  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:26.958022  6875 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:26.958418  6875 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:26.958850  6875 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:26.917896  7557 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:26.965153  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 541992us
I20260709 12:24:26.918056  7484 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.920849  7550 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.963701  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:26.970510  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.960788  1800 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.005s	sys 0.022s
W20260709 12:24:26.976450  1800 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.005s	sys 0.022s
I20260709 12:24:26.978295  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:26.971604  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:27.046456  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74914.8us
I20260709 12:24:27.048513  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:27.049580  1725 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:27.051813  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 528332us
I20260709 12:24:27.061581  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.917718  7486 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.070257  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:26.928622  7614 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:26.970507  7484 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:27.083856  7588 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:26.984328  7545 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:26.969547  7407 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:27.077881  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:27.088759  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.985366  7602 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:27.096963  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 285221us
I20260709 12:24:27.074218  7463 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.067286  7361 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.108918  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:27.105227  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:27.111454  6875 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:27.095728  7630 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260709 12:24:27.139009  1725 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:27.047063  7550 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:27.141853  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.961840  7252 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:27.110508  7632 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 12:24:27.151822  1798 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:24:27.159344  6875 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.096014  7554 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.081s
W20260709 12:24:27.163511  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 499983us
I20260709 12:24:27.095516  7602 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:27.168031  1725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:27.169304  1725 raft_consensus.cc:606] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.169847  1725 raft_consensus.cc:614] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:27.141876  7463 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:27.114223  7588 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:27.183825  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 466327us
I20260709 12:24:27.185003  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.073990  7452 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:27.188328  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 396060us
W20260709 12:24:27.174540  7637 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.331s	user 0.001s	sys 0.000s
I20260709 12:24:27.173656  7544 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:27.189087  7328 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.554s	user 0.009s	sys 0.001s
I20260709 12:24:27.181685  7463 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.178036  7588 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.172655  7602 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:27.225754  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.228083  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.229377  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 782) took 1053 ms. Trace:
W20260709 12:24:27.232947  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 269346us
W20260709 12:24:27.239575  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.235455  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:27.229787  1880 rpcz_store.cc:276] 0709 12:24:26.176158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.176292 (+   134us) service_pool.cc:224] Handling call
0709 12:24:26.188862 (+ 12570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:26.598636 (+409774us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.229033 (+630397us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b54002dfad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:27.229297 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":940990336}
I20260709 12:24:27.255904  1800 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:24:27.257462  6875 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.258224  6875 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.260633  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 756) took 1734 ms. Trace:
W20260709 12:24:27.265000  1723 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.529s	user 0.011s	sys 0.000s
I20260709 12:24:27.195037  7640 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:27.226315  7544 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.261229  1803 rpcz_store.cc:276] 0709 12:24:25.526179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.526322 (+   143us) service_pool.cc:224] Handling call
0709 12:24:27.252345 (+1726023us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b5400132254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:27.260414 (+  8069us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":407471488}
I20260709 12:24:27.178967  7444 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:27.271687  1723 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
W20260709 12:24:27.266919  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.282305  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 803) took 1758 ms. Trace:
W20260709 12:24:27.284389  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 179509us
I20260709 12:24:27.273998  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:27.222659  7536 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.226694  7361 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:27.289121  6875 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:27.239988  7463 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:27.237346  7588 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:27.253275  7620 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:27.282958  1730 rpcz_store.cc:276] 0709 12:24:25.523699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.550716 (+ 27017us) service_pool.cc:224] Handling call
0709 12:24:25.583375 (+ 32659us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:26.097651 (+514276us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.097688 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:26.098375 (+   687us) log.cc:844] Serialized 52 byte log entry
0709 12:24:26.552310 (+453935us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:26.552358 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.552438 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.278565 (+726127us) raft_consensus.cc:1685] finished
0709 12:24:27.278599 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.282211 (+  3612us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4368,"raft.queue_time_us":184607,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":48021,"thread_start_us":826037,"threads_started":2,"wal-append.queue_time_us":639587}
I20260709 12:24:27.307544  6875 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:27.272182  7627 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:27.313583  6875 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:27.290498  7637 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:24:27.256304  7649 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:27.318543  6875 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:27.292145  7544 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:27.277395  7640 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:27.345695  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 162383us
I20260709 12:24:27.230599  7641 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.084s
W20260709 12:24:27.350581  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46040us
W20260709 12:24:27.346724  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 293821us
W20260709 12:24:27.317296  7005 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.723s	user 0.012s	sys 0.003s
I20260709 12:24:27.364714  1723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:27.318222  7627 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:27.354024  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 245194us
W20260709 12:24:27.352604  7603 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.774s	user 0.014s	sys 0.000s
I20260709 12:24:27.367450  1881 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:24:27.290875  7391 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 12:24:27.367558  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:27.316812  7407 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.049s
W20260709 12:24:27.379091  7648 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:24:27.344411  7358 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:27.289533  7645 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.090s
W20260709 12:24:27.391433  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 163418us
I20260709 12:24:27.393536  6875 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.395416  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.351490  7640 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:27.405088  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:27.408012  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.411345  7667 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:27.364571  7486 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:27.374361  7627 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:27.380861  7614 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:27.354264  7361 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.409062  7666 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.011s
I20260709 12:24:27.405246  7328 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:27.404508  7444 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.393092  7663 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:27.383608  7603 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.448655  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:27.466800  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2436) took 1154 ms. Trace:
I20260709 12:24:27.460320  7646 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.461624  7664 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:27.463303  7486 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:27.481119  7599 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.894s	user 0.009s	sys 0.003s
W20260709 12:24:27.439992  7653 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.118s	user 0.001s	sys 0.001s
I20260709 12:24:27.511194  7361 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:27.413718  7358 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:27.476341  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2434) took 1197 ms. Trace:
W20260709 12:24:27.453169  7639 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
W20260709 12:24:27.454656  7647 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:24:27.515661  7603 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:27.460657  7218 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.794s	user 0.011s	sys 0.002s
I20260709 12:24:27.495647  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.495795  1878 rpcz_store.cc:276] 0709 12:24:26.312795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.824819 (+512024us) service_pool.cc:224] Handling call
0709 12:24:26.825202 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:26.834981 (+  9779us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.835013 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:26.835531 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 12:24:27.250998 (+415467us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:27.251058 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.251158 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.466563 (+215405us) raft_consensus.cc:1685] finished
0709 12:24:27.466583 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.466719 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2954,"raft.queue_time_us":7683,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":111,"thread_start_us":290404,"threads_started":1,"wal-append.queue_time_us":547114}
I20260709 12:24:27.440001  7391 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:27.453444  7660 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 12:24:27.510035  7651 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:27.571977  7651 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:27.509725  7599 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:27.485097  7540 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.568s	user 0.001s	sys 0.011s
W20260709 12:24:27.535104  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 143739us
I20260709 12:24:27.532262  1873 rpcz_store.cc:276] 0709 12:24:26.278629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.763790 (+485161us) service_pool.cc:224] Handling call
0709 12:24:26.764153 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:27.193564 (+429411us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:27.194000 (+   436us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:27.224675 (+ 30675us) log.cc:844] Serialized 26 byte log entry
0709 12:24:27.383898 (+159223us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:27.383937 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.384002 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.476091 (+ 92089us) raft_consensus.cc:1685] finished
0709 12:24:27.476121 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.476284 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17289,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":74024,"thread_start_us":446948,"threads_started":2,"wal-append.queue_time_us":174776}
I20260709 12:24:27.492602  6875 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:27.574337  7599 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.487486  7658 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:27.575742  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.575459  7658 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.577687  7658 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.586326  1878 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [14.10-16.11]
I20260709 12:24:27.587535  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3132) took 1081 ms. Trace:
I20260709 12:24:27.587932  1878 rpcz_store.cc:276] 0709 12:24:26.506552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.585385 (+1078833us) service_pool.cc:224] Handling call
0709 12:24:27.585950 (+   565us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:27.587151 (+  1201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.587462 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:27.529255  7641 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:27.595988  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 212394us
W20260709 12:24:27.599903  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 225931us
W20260709 12:24:27.597373  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 189427us
I20260709 12:24:27.583853  7218 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:27.603250  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 107716us
I20260709 12:24:27.605270  7218 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:27.608413  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.610404  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:27.612433  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:27.582494  7391 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:27.634845  1732 raft_consensus.cc:2368] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16.
I20260709 12:24:27.613286  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:27.650012  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:27.613214  7590 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.010s	user 0.009s	sys 0.006s
I20260709 12:24:27.651429  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:27.664937  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.650702  2411 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:27.654013  2417 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:27.665745  1798 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:27.530721  7646 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.666373  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:27.670667  2411 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:24:27.667500  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.569484  7361 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 12:24:27.677516  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:27.677765  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:27.678759  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:27.613154  7675 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:27.674443  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 66791.2us
W20260709 12:24:27.633985  7647 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:27.684298  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:27.674237  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:27.676479  1804 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:27.674249  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46506.5us
I20260709 12:24:27.679764  2417 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:27.680995  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2430) took 1403 ms. Trace:
I20260709 12:24:27.633986  7649 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:27.689448  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.569698  7602 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:27.695093  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94280.1us
I20260709 12:24:27.691514  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:27.688117  1875 rpcz_store.cc:276] 0709 12:24:26.277642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.369210 (+ 91568us) service_pool.cc:224] Handling call
0709 12:24:26.369644 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:26.370339 (+   695us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.370377 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:26.421075 (+ 50698us) log.cc:844] Serialized 26 byte log entry
0709 12:24:26.816091 (+395016us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:26.816141 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.816226 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.680593 (+864367us) raft_consensus.cc:1685] finished
0709 12:24:27.680626 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.680926 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":109,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":91,"thread_start_us":394528,"threads_started":1,"wal-append.queue_time_us":668667}
W20260709 12:24:27.678957  7591 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.018s	user 0.012s	sys 0.002s
I20260709 12:24:27.696451  1803 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
W20260709 12:24:27.700037  7668 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.128s	user 0.000s	sys 0.001s
I20260709 12:24:27.710330  7670 raft_consensus.cc:2764] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election decision vote started in defunct term 15: won
I20260709 12:24:27.711447  7679 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.714087  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:27.714826  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.712081  7679 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.695186  7663 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:27.707746  7484 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.600s	user 0.009s	sys 0.003s
I20260709 12:24:27.719693  2419 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:27.694516  7641 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:24:27.710624  7685 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:24:27.727120  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.727824  1803 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:27.728855  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:27.713097  7591 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15.
I20260709 12:24:27.677208  7590 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.737733  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.738423  1804 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:27.728094  7606 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.779s	user 0.013s	sys 0.000s
I20260709 12:24:27.743640  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:27.695144  7675 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.747287  7675 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:27.750419  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:27.751142  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2338) took 1076 ms. Trace:
I20260709 12:24:27.752334  7679 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.751633  1805 rpcz_store.cc:276] 0709 12:24:26.674459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.836041 (+161582us) service_pool.cc:224] Handling call
0709 12:24:26.917400 (+ 81359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:26.928357 (+ 10957us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:27.160058 (+231701us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.625858 (+465800us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:27.626481 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 12:24:27.639374 (+ 12893us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:27.639518 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.639699 (+   181us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.750740 (+111041us) raft_consensus.cc:1685] finished
0709 12:24:27.750762 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.751072 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12572,"prepare.queue_time_us":343399,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":3096,"raft.queue_time_us":354156,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":10836,"thread_start_us":633103,"threads_started":3,"wal-append.queue_time_us":322251}
I20260709 12:24:27.753083  7452 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:24:27.753521  7557 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.783s	user 0.011s	sys 0.000s
I20260709 12:24:27.753705  7452 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:27.754084  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:27.755133  7452 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 14.14, Last appended by leader: 14, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.750026  7649 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.758337  2414 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:27.725117  7641 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:27.754127  7218 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:24:27.678169  7651 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.763494  7641 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:27.759826  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63751.4us
I20260709 12:24:27.760352  7606 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.772563  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:27.776203  1798 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:27.777304  2411 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:27.777416  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.785244  1804 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:27.786423  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:27.787096  1804 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
W20260709 12:24:27.788546  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.789674  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.734091  7659 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:27.727626  7484 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.797600  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:27.801663  1805 raft_consensus.cc:2368] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17.
I20260709 12:24:27.761723  7604 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:27.764541  7218 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:24:27.813081  7687 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:24:27.815757  7646 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.816494  7604 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.734186  7391 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.820960  7544 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.822402  2412 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:27.823232  7391 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:27.823619  7463 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.825867  7463 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.808813  7218 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:27.826486  7561 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:27.827430  1723 raft_consensus.cc:606] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:24:27.827316  7218 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.828176  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2441) took 1452 ms. Trace:
I20260709 12:24:27.828616  1881 rpcz_store.cc:276] 0709 12:24:26.376081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.112137 (+736056us) service_pool.cc:224] Handling call
0709 12:24:27.112459 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:27.373219 (+260760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.827876 (+454657us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:27.828100 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":583704832}
I20260709 12:24:27.829151  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:27.830410  7600 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.078s	user 0.014s	sys 0.000s
I20260709 12:24:27.830952  7600 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.767043  7673 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:27.827889  1723 raft_consensus.cc:614] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:27.834035  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:27.854943  1798 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:27.842443  7640 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.844794  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3130) took 1371 ms. Trace:
I20260709 12:24:27.845628  1803 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:27.844211  7606 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:27.828781  7218 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:27.847162  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2437) took 1472 ms. Trace:
I20260709 12:24:27.795631  7659 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:27.847637  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49162.4us
I20260709 12:24:27.725804  7663 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.831660  7604 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.809623  7679 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:27.844271  7544 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.830525  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:27.881022  1730 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:27.881675  1724 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:27.864856  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:27.874706  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:27.884123  1798 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:27.884038  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.871445  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:27.865806  1880 rpcz_store.cc:276] 0709 12:24:26.473730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.271383 (+797653us) service_pool.cc:224] Handling call
0709 12:24:27.300326 (+ 28943us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:27.595672 (+295346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:27.595706 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:27.596271 (+   565us) log.cc:844] Serialized 26 byte log entry
0709 12:24:27.732240 (+135969us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:27.732284 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.732367 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:27.844533 (+112166us) raft_consensus.cc:1685] finished
0709 12:24:27.844555 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.844713 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":150751,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":54,"thread_start_us":430380,"threads_started":2,"wal-append.queue_time_us":207676}
I20260709 12:24:27.871445  1872 rpcz_store.cc:276] 0709 12:24:26.375009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.928693 (+553684us) service_pool.cc:224] Handling call
0709 12:24:27.092263 (+163570us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:27.092531 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:27.111126 (+ 18595us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.287694 (+176568us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.757040 (+469346us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x7b18003e55a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:24:27.846234 (+ 89194us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:27.846308 (+    74us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:27.846381 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.846462 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.847098 (+   636us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7046,"prepare.queue_time_us":21932,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":125,"raft.queue_time_us":203110,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":23657,"spinlock_wait_cycles":9787648,"thread_start_us":255395,"threads_started":2,"wal-append.queue_time_us":339847}
I20260709 12:24:27.890130  7579 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:27.891419  7597 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:27.891674  7579 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:27.892328  7579 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:27.892831  7689 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:24:27.894687  7600 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.900926  7663 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:27.905236  7597 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:27.906805  7597 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.886725  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:27.832229  7658 consensus_queue.cc:1175] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 17 status { last_received { term: 17 index: 12 } last_committed_idx: 11 last_received_current_leader { term: 17 index: 12 } }
I20260709 12:24:27.914573  6875 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:27.868727  7463 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:27.918910  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:27.922950  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2429) took 1645 ms. Trace:
I20260709 12:24:27.916075  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:27.919041  6875 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:27.919968  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:27.923383  1877 rpcz_store.cc:276] 0709 12:24:26.277346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.323321 (+ 45975us) service_pool.cc:224] Handling call
0709 12:24:26.323585 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.531359 (+207774us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.922622 (+1391263us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54001f8954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:27.922870 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":443271936}
I20260709 12:24:27.924322  1798 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:27.893683  7579 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.920149  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:27.853157  7694 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.065s
I20260709 12:24:27.929121  7675 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:27.929606  7675 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.881153  7679 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.929967  7675 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:27.930220  7679 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:27.921787  7557 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.011s
I20260709 12:24:27.933880  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:27.934940  1801 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:27.935992  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.936761  1801 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:27.937587  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.938094  1801 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:27.938367  7581 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:27.926730  7463 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:27.922282  7645 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:27.944186  7463 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:27.945214  7588 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.599s	user 0.012s	sys 0.000s
I20260709 12:24:27.945335  7398 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.945914  7588 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:27.944062  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.947151  1806 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:27.947264  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:27.949031  7673 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:27.950768  7670 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:27.951315  7670 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:27.951689  7696 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:24:27.943823  1731 raft_consensus.cc:2410] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:24:27.942072  7627 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.876369  7659 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:27.958150  1806 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:27.970661  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.982893  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.983628  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:27.983640  1806 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:27.982458  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:27.991369  2415 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:24:27.991798  7600 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.992554  7600 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.994872  7600 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:27.995744  1798 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:27.997658  1798 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:24:27.985446  6875 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:27.999900  1803 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:28.029309  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47450.7us
I20260709 12:24:28.031291  1727 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:27.981143  7620 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.635s	user 0.012s	sys 0.000s
W20260709 12:24:28.049453  2383 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.749s (SENT)
I20260709 12:24:27.998989  7611 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:28.050302  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:27.976467  7579 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:28.053753  1806 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:28.057701  1801 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:28.033830  7452 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:28.059716  7452 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:28.060844  7452 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.062124  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3278) took 1078 ms. Trace:
W20260709 12:24:28.063025  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.062566  1732 rpcz_store.cc:276] 0709 12:24:26.983643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.984200 (+   557us) service_pool.cc:224] Handling call
0709 12:24:28.056167 (+1071967us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b5400327dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:28.062025 (+  5858us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1229900544}
I20260709 12:24:28.064101  7611 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:28.065274  7611 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.066501  7691 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:27.983897  7645 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:28.069958  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:28.069747  1801 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:28.071192  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.073318  1803 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:28.075821  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:28.083425  1801 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:28.087213  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:28.055474  7620 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:28.092793  1806 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 15: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:28.097690  7222 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.084395  7391 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:27.968230  7670 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.084652  7398 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:28.118685  7627 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:28.139406  7645 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.141923  1723 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:28.133669  7651 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:28.148316  7678 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:28.130283  7641 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.110854  7391 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:28.156716  1806 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:28.157424  7391 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.159181  7641 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:28.145412  7550 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.160332  1723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb48ef23a7434834929994987a36f360"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:28.161569  7675 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:28.130033  7644 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.823s	user 0.003s	sys 0.000s
I20260709 12:24:28.168041  7398 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:28.166466  7675 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:28.199525  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.209281  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:28.210909  7398 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.212891  1697 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:28.196617  7659 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.217445  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:28.221274  1697 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:28.226750  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:28.228552  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:28.194267  7670 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.231710  1801 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:28.192140  7578 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:28.241926  7704 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:24:28.242564  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.230759  7545 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:28.230729  7673 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:28.230997  7588 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.232316  7627 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:28.231142  7218 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:28.248271  7663 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:28.254330  7663 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:28.248401  7218 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:28.247105  7675 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.255766  7218 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:28.261107  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:28.261718  7627 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:28.261941  1730 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.255353  7663 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.283680  7230 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.284842  7679 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.256433  7675 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.288935  7675 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:28.275117  1807 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:28.289695  7674 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.687s	user 0.002s	sys 0.000s
I20260709 12:24:28.295125  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2435) took 2016 ms. Trace:
I20260709 12:24:28.295634  1874 rpcz_store.cc:276] 0709 12:24:26.285993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.772438 (+486445us) service_pool.cc:224] Handling call
0709 12:24:26.824556 (+ 52118us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.831884 (+  7328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.286759 (+1454875us) spinlock_profiling.cc:243] Waited 51.7 ms on lock 0x7b5400118350. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:28.295056 (+  8297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":119015296}
W20260709 12:24:28.285925  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2346) took 1340 ms (client timeout 1734 ms). Trace:
W20260709 12:24:28.298530  1800 rpcz_store.cc:269] 0709 12:24:26.945710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.977875 (+ 32165us) service_pool.cc:224] Handling call
0709 12:24:28.285702 (+1307827us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b540013a6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:28.285862 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":635242368}
I20260709 12:24:28.262960  7627 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.248268  7486 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.302772  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.306263  1730 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:28.312475  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:28.313161  1800 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.313915  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:28.315344  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:28.317414  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.315968  2415 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:28.318528  2411 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:28.266402  7641 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:28.320785  1770 leader_election.cc:400] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16.
I20260709 12:24:28.324093  1770 leader_election.cc:403] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:24:28.326267  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.327260  2410 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:28.328862  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.329741  2419 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:28.331820  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:28.342083  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:28.343075  1725 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:28.309880  7486 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.346983  7358 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.340715  7588 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16.
I20260709 12:24:28.317684  7463 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:28.378229  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 112768us
I20260709 12:24:28.378962  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2269) took 1958 ms. Trace:
I20260709 12:24:28.379562  1802 rpcz_store.cc:276] 0709 12:24:26.420345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.518358 (+ 98013us) service_pool.cc:224] Handling call
0709 12:24:26.534471 (+ 16113us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:26.650384 (+115913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.378511 (+1728127us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54003dbdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:28.378870 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":264968960}
I20260709 12:24:28.381067  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.385010  2419 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:28.386389  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 790) took 1966 ms. Trace:
W20260709 12:24:28.391491  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 942823us
I20260709 12:24:28.498811  7398 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:28.510751  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.511267  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.511613  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.395358  7671 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:24:28.397848  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.400084  7391 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.402768  7679 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:28.405375  7564 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.523283  7564 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.526147  7564 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:28.405884  7550 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:28.405825  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:28.527078  7550 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.527897  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:28.407333  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:28.528877  1875 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:28.529654  7550 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:28.529829  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:28.530565  1875 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:28.531680  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:28.408071  7581 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.532656  7581 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.533252  7590 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.534390  7602 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.409499  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:28.410404  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:28.411458  7705 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
I20260709 12:24:28.390658  7218 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.407333  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:28.416291  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:28.420105  1800 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
W20260709 12:24:28.421090  7634 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.324s	user 0.002s	sys 0.003s
I20260709 12:24:28.420775  2410 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
W20260709 12:24:28.423792  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.428845  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:28.433228  7643 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.210s	user 0.000s	sys 0.003s
I20260709 12:24:28.434026  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:28.437136  7322 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.442442  7230 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.444407  1805 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:24:28.416324  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:28.579958  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 163776us
I20260709 12:24:28.464146  1732 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:28.383388  7659 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:28.580826  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:28.512930  7573 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:28.592585  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:28.483987  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87051.9us
I20260709 12:24:28.495291  1772 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:28.501387  1802 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:28.502605  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:28.508023  7658 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:28.507042  1879 rpcz_store.cc:276] 0709 12:24:26.419888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.271118 (+851230us) service_pool.cc:224] Handling call
0709 12:24:27.295241 (+ 24123us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:27.295496 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:27.295898 (+   402us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.493905 (+198007us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.830573 (+336668us) spinlock_profiling.cc:243] Waited 6.24 ms on lock 0x7b180037ee80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:28.289337 (+458764us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b240023a9f8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d2a09d 00007f16ab95c10e 00007f16ab97388c 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221
0709 12:24:28.383181 (+ 93844us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b180037ee80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:24:28.385987 (+  2806us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.386045 (+    58us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:28.386106 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.386177 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.386322 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":892601,"prepare.queue_time_us":4410,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":3897,"raft.queue_time_us":903062,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":6346,"spinlock_wait_cycles":29450240,"thread_start_us":197278,"threads_started":1,"wal-append.queue_time_us":306756}
I20260709 12:24:28.519969  7391 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.516881  7005 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:28.408422  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:28.535394  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.536928  7673 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:28.540469  7484 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.543392  7463 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:28.545349  7688 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.735s	user 0.002s	sys 0.000s
I20260709 12:24:28.546401  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3280) took 1485 ms. Trace:
I20260709 12:24:28.547839  7640 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:28.549419  7486 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:28.536921  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2343) took 1708 ms. Trace:
I20260709 12:24:28.555959  1725 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:28.526049  1803 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:28.579128  7671 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:28.565726  1801 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.646929  1799 rpcz_store.cc:276] 0709 12:24:26.828834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.842426 (+ 13592us) service_pool.cc:224] Handling call
0709 12:24:28.378799 (+1536373us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:28.387173 (+  8374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.387204 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:28.388050 (+   846us) log.cc:844] Serialized 52 byte log entry
0709 12:24:28.476665 (+ 88615us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:28.476714 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.476790 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:28.535832 (+ 59042us) raft_consensus.cc:1685] finished
0709 12:24:28.535853 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.536641 (+   788us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54003dbdd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:28.536855 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4194,"raft.queue_time_us":15150,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":159,"spinlock_wait_cycles":3350003584,"thread_start_us":88194,"threads_started":1,"wal-append.queue_time_us":112915}
I20260709 12:24:28.577648  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:28.651216  1727 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.588709  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:28.376679  7358 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.589823  1723 raft_consensus.cc:606] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:28.653627  1723 raft_consensus.cc:614] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:28.591240  1877 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.591815  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:28.595439  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 100422us
I20260709 12:24:28.556912  7631 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.655881  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:28.598130  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 95592.5us
I20260709 12:24:28.590179  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:28.564409  7564 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.602458  1872 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:28.592108  7322 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:28.643637  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:28.517776  7398 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.668883  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:28.669857  1879 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:28.599495  7550 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:28.556629  1880 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:28.673501  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.648059  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:28.674957  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.674995  1725 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:28.643328  1731 rpcz_store.cc:276] 0709 12:24:27.060910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.061137 (+   227us) service_pool.cc:224] Handling call
0709 12:24:27.940167 (+879030us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b5400436128. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bb55f4 00007f16b0d9e227 00007f16b0d9dab8 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:28.546203 (+606036us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b54004362d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:28.546344 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":30484096}
I20260709 12:24:28.652086  1875 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:24:28.662384  7695 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:24:28.657788  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:28.681394  7620 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:28.682492  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:28.685312  2418 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:28.607080  7561 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:28.660233  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 70145.2us
I20260709 12:24:28.687242  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:28.690232  7697 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:28.690838  7697 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:28.694352  7218 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.663606  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:28.608018  7252 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:28.600696  7573 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.754477  7463 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:28.657222  7579 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.585s	user 0.012s	sys 0.000s
I20260709 12:24:28.757711  1881 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:28.600701  7452 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:28.673528  1801 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:28.676991  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.678018  1805 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 13: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:28.685454  1803 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:28.591243  7230 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.600696  7602 raft_consensus.cc:2764] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:24:28.712915  7620 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:28.694581  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:28.710937  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.709101  1798 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:28.713672  7407 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:24:28.720027  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:28.785696  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.786926  1806 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:28.787352  7657 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:28.787894  7657 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:28.788090  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:28.686126  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1123) took 1107 ms. Trace:
I20260709 12:24:28.717034  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.713171  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:28.672492  7606 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:24:28.696012  7391 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:28.695248  7561 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:28.697065  1725 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:28.703790  1877 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:28.647756  7673 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.703065  1875 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:28.672325  7564 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.733134  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:28.801371  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:28.735384  1845 connection.cc:256] RPC call timeout handler was delayed by -1.0518701359997067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
I20260709 12:24:28.801719  7407 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:28.803328  7407 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.803732  7407 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:28.811408  7727 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:24:28.812355  7230 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.812500  7561 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:28.777573  7670 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:28.737469  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.739318  7358 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.793145  7606 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:28.739950  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:28.740532  7718 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:24:28.686081  7671 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.691701  7697 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.665853  7631 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.743947  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.826009  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.745780  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:28.603812  7658 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.827718  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:28.828347  1879 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:28.829221  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:28.829360  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 786) took 2652 ms. Trace:
I20260709 12:24:28.829768  1879 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:28.829815  7658 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.608676  7005 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.832149  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.832434  7005 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.702457  7398 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.833000  7564 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.833560  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.833839  7631 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:28.821954  7561 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.014s
I20260709 12:24:28.834827  1874 raft_consensus.cc:2410] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:24:28.835048  7561 consensus_queue.cc:1243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:28.835335  7657 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.836453  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.839282  2419 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:28.749895  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:28.752627  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.698705  7599 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:28.836526  7657 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:28.671569  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.755488  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:28.756636  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:28.731633  7550 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:28.762517  2412 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:28.764179  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:28.847427  1807 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:28.765861  2413 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:28.754704  7252 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.761525  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 810) took 1020 ms. Trace:
I20260709 12:24:28.768070  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.829792  1876 rpcz_store.cc:276] 0709 12:24:26.177136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.246035 (+ 68899us) service_pool.cc:224] Handling call
0709 12:24:26.256821 (+ 10786us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:26.593156 (+336335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.593195 (+    39us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:26.593985 (+   790us) log.cc:844] Serialized 78 byte log entry
0709 12:24:26.957936 (+363951us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:27.239339 (+281403us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.239961 (+   622us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.270549 (+1030588us) spinlock_profiling.cc:243] Waited 8.29 ms on lock 0x7b180049cf60. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db
0709 12:24:28.432343 (+161794us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:28.434930 (+  2587us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b5400223654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:28.435013 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:28.827327 (+392314us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x7b240023a968. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:24:28.829116 (+  1789us) raft_consensus.cc:1685] finished
0709 12:24:28.829135 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.829282 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":128542,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":12956,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":15509,"spinlock_wait_cycles":809109760,"thread_start_us":607739,"threads_started":2,"wal-append.queue_time_us":553610}
I20260709 12:24:28.772615  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.775820  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.857091  1805 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:28.775063  1800 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:28.858170  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:28.777951  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83441.6us
I20260709 12:24:28.779278  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.791285  1799 raft_consensus.cc:2368] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16.
I20260709 12:24:28.759447  7452 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.757098  7463 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.861155  1805 raft_consensus.cc:2376] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15. Re-sending same reply.
I20260709 12:24:28.864845  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.791706  2418 rpcz_store.cc:276] 0709 12:24:27.578685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.578891 (+   206us) service_pool.cc:224] Handling call
0709 12:24:28.684831 (+1105940us) spinlock_profiling.cc:243] Waited 925 us on lock 0x7b5800003f98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3eee 000000000036eb39 00007f16b0d271e0 00007f16b0df00c4 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:24:28.684981 (+   150us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.685837 (+   856us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.686050 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2127616}
I20260709 12:24:28.820932  7670 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.866317  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:28.867060  2418 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:28.867947  7670 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:28.797916  1727 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
W20260709 12:24:28.798352  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 85256.6us
I20260709 12:24:28.869625  1801 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:28.869973  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:28.794916  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:28.797735  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:28.734833  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.870939  1803 raft_consensus.cc:2376] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16. Re-sending same reply.
W20260709 12:24:28.813694  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80645.9us
I20260709 12:24:28.793751  7640 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:28.802691  1806 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:28.817097  2414 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:28.795429  7321 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:28.818539  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:28.873960  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.819072  1804 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:28.818991  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:28.876917  1725 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:28.843688  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 93912.5us
I20260709 12:24:28.755209  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:28.878094  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 59258.6us
I20260709 12:24:28.878376  1799 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:28.818037  7602 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:28.849541  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:28.836700   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:28.885562  7573 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.821698  7322 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:24:28.886518  7322 consensus_queue.cc:1243] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:24:28.887136  7322 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:28.887488  7322 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:28.821902  7606 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:28.888023  7631 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:28.888649  7631 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:28.888429  7322 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.889776  7322 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.889601  7631 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.855386  1724 rpcz_store.cc:276] 0709 12:24:27.741163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.927016 (+185853us) service_pool.cc:224] Handling call
0709 12:24:27.939790 (+ 12774us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:27.940033 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:28.573918 (+633885us) log.cc:844] Serialized 14 byte log entry
0709 12:24:28.661702 (+ 87784us) log.cc:844] Serialized 14 byte log entry
0709 12:24:28.689390 (+ 27688us) spinlock_profiling.cc:243] Waited 9.96 ms on lock 0x7b1800485500. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1
0709 12:24:28.733994 (+ 44604us) spinlock_profiling.cc:243] Waited 6.88 ms on lock 0x7b1800485500. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:28.760431 (+ 26437us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.760498 (+    67us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:28.760571 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.761330 (+   759us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.761463 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":471629,"prepare.queue_time_us":479145,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":88,"raft.queue_time_us":102289,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":10881,"spinlock_wait_cycles":38737536,"thread_start_us":337379,"threads_started":2,"wal-append.queue_time_us":172688}
I20260709 12:24:28.891072  7321 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:28.891497  7550 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.893200  7678 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:28.893615  7321 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:28.876699  7640 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.895674  7640 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:28.896641  7602 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:28.897756  7602 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:28.883347  7690 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.059s	user 0.000s	sys 0.009s
I20260709 12:24:28.884490  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:28.870265  7452 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.899982  1729 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:28.900487  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:28.901584  1729 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:28.901942  7678 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:28.902848  7678 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:28.902992  7679 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:28.872463  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:28.904340  7463 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:28.902715  7444 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:24:28.903082  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:28.905982  7444 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:28.873085  7561 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:28.736696  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2365) took 2682 ms. Trace:
I20260709 12:24:28.873853  1697 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:28.876402  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.907550  7604 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:28.909832  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:28.871217  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73612.3us
I20260709 12:24:28.913187  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:28.915275  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.915990  1724 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:28.878846  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:28.916936  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.906474  7444 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:28.847270  7579 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:28.903450  7398 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:28.903903  7678 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:28.905237  7536 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:28.910039  7676 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:28.913421  1878 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:28.949946  1807 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:28.914367  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.907181  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:28.952129  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.887568   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:28.879568  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:28.917972  1800 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:28.907430  7561 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:28.872702  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.821110  1881 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:28.925078  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.927405  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.914664  1803 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
W20260709 12:24:28.933460  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 84015.2us
I20260709 12:24:28.931488  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:28.931488  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:28.933723  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:28.934999  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.937099  2414 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:28.929013  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:28.929029  7579 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:28.976045  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:28.946568  1725 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:28.908423  1728 rpcz_store.cc:276] 0709 12:24:26.053860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.053988 (+   128us) service_pool.cc:224] Handling call
0709 12:24:26.063540 (+  9552us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:27.771081 (+1707541us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:27.771119 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:27.771853 (+   734us) log.cc:844] Serialized 52 byte log entry
0709 12:24:27.875387 (+103534us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:28.582581 (+707194us) log.cc:844] Serialized 14 byte log entry
0709 12:24:28.589711 (+  7130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.736138 (+146427us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54002b7ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:28.736196 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:28.736268 (+    72us) raft_consensus.cc:1685] finished
0709 12:24:28.736401 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.736615 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":437941,"prepare.queue_time_us":539630,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":9243,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":8927,"spinlock_wait_cycles":3912554752,"thread_start_us":375498,"threads_started":2,"wal-append.queue_time_us":179041}
I20260709 12:24:28.952245  1799 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:24:28.758210  7258 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.015s
I20260709 12:24:28.882175  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:28.959884  1878 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:28.960892  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:28.999279   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:28.970743  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:28.936754  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:28.971560  1723 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:29.002102  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.004245  1876 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:28.971066  1804 raft_consensus.cc:2410] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:24:28.973711  1724 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 11: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:29.005596  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74172.8us
I20260709 12:24:28.974617  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:29.007647  1731 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:28.947679  7486 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:28.943827  7670 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:28.933835  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:29.012576  1806 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:28.976817  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.945802  7321 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:24:28.982121  1725 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:29.025105  1725 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 12 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:28.983510  1728 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:28.946619  7536 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:28.949146  7676 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.985591  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:28.986102  2419 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:24:29.031068  7676 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:28.928591  7230 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:24:28.971508  7679 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:28.999501  1807 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:28.938202  7544 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.033842  7544 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.037863  2410 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:28.947553  7484 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:24:29.039072  7536 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.039687  7484 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:29.040449  7484 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:29.040807  7536 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:29.041488  7484 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:29.002367  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 65676.5us
W20260709 12:24:29.043303  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.043695  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.044926  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:29.046257  7670 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:29.046854  7670 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:29.047843  7670 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.036463  7544 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.054518  7484 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:28.980158  7579 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:29.063161  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 181095us
I20260709 12:24:29.072944  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:29.065797  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59235.9us
W20260709 12:24:29.067070  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.995668  7258 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:29.084080  1806 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:29.068286  7321 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.071573  7663 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:29.081616  7544 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:29.115227  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:29.116114  1806 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:24:29.117880  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.087854  7558 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.089905  7258 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:29.152287  2414 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:29.142295  7730 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:29.154388  7730 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.154876  7730 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:29.158484  1806 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:29.111831  7663 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.092552  7564 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.058s
I20260709 12:24:29.174387  7606 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.176687  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.186733  1876 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:29.096206  7573 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
W20260709 12:24:29.187579  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 96678.1us
I20260709 12:24:29.111397  7681 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.074s
I20260709 12:24:29.069710  7444 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.160006  7544 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:29.170495  7663 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:29.210638  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 239950us
I20260709 12:24:29.111482  7561 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.198127  7673 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.196784  7659 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:29.223320  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2371) took 2533 ms (client timeout 2555 ms). Trace:
I20260709 12:24:29.226035  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:29.226236  1726 rpcz_store.cc:269] 0709 12:24:26.689453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.714169 (+ 24716us) service_pool.cc:224] Handling call
0709 12:24:29.223077 (+2508908us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54002d05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:29.223240 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1987824384}
I20260709 12:24:29.191279  7645 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:24:29.209816  7252 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.234616  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.242123  1873 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:29.243661  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 161679us
I20260709 12:24:29.245081  1879 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:29.247947  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.248603  1879 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.236356  7444 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.264050  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 813) took 1436 ms. Trace:
I20260709 12:24:29.237349  7694 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:29.244289  7710 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:24:29.273181  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:29.264786  1732 rpcz_store.cc:276] 0709 12:24:27.827208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.064368 (+237160us) service_pool.cc:224] Handling call
0709 12:24:28.064699 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.589228 (+524529us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.263648 (+674420us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b5400436cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:29.263962 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":918048640}
I20260709 12:24:29.249866  7730 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.246469  7740 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:29.291517  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:29.292377  2414 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
W20260709 12:24:29.283994  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 81664.7us
I20260709 12:24:29.236845  7659 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:29.276937  7734 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:24:29.295279  1845 leader_election.cc:400] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16.
I20260709 12:24:29.284762  7740 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:29.297601  1845 leader_election.cc:403] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:29.249971  7252 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
I20260709 12:24:29.305186  7730 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.305569  1879 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:29.306388  7606 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.308696  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.309449  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:29.281035  7676 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.313056  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:29.314950  2414 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:29.315927  1846 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:29.316020  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:29.279983  7694 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.303889  7005 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:29.320302  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 12 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:29.333812  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:29.296998  7452 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.327562  7252 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:24:29.312062  7736 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:24:29.342446  2414 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:29.332585  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:29.341607  1724 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:29.340031  1873 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:29.333752  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.347873  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:29.325892  7579 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.338267  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:29.347846  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.339572  7252 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:29.351358  7737 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:24:29.350960  7252 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.352588  7252 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:29.339196  7742 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.355337  7743 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:29.355347  7742 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.358924  7676 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.358611  7740 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.340802  7631 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:29.360589  7631 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:29.360139  7740 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:29.360991  7631 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:29.370834  2412 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1126) took 1424 ms (client timeout 1662 ms). Trace:
I20260709 12:24:29.371733  1846 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:29.371721  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.372864  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:29.373236  1873 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:29.373651  7681 raft_consensus.cc:2764] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:24:29.373857  7581 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.374337  7681 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:29.371376  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:29.371361  2412 rpcz_store.cc:269] 0709 12:24:27.946463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.946897 (+   434us) service_pool.cc:224] Handling call
0709 12:24:29.370562 (+1423665us) spinlock_profiling.cc:243] Waited 814 ms on lock 0x7b54005f05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:29.370714 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1872425344}
I20260709 12:24:29.333912  7730 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.342662  7606 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:29.385509  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:29.389503  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:29.290580  7544 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.402377  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:29.402582  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:29.374766  7581 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.408977  1873 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:29.410440  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.404182  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.411808  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.388677  7602 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:29.418324  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55345.1us
I20260709 12:24:29.420631  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.385807  7573 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:29.403080  7484 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.407037  7740 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:24:29.407302  7681 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:29.403257  7640 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.436106  7484 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.439157  1805 raft_consensus.cc:2368] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19.
I20260709 12:24:29.438519  7640 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.385684  7564 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:24:29.452517  7452 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.455128  7681 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:29.455751  7484 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.455889  7681 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:29.384413  7730 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.462277  1876 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:29.463973  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.464680  1876 raft_consensus.cc:2376] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 12. Re-sending same reply.
I20260709 12:24:29.468954  7667 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:29.469758  7667 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.470324  7631 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.473568  7631 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.473555  7667 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.480028  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:29.483723  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.490680  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.464623  7407 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.660s	user 0.015s	sys 0.000s
I20260709 12:24:29.490232  7730 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.496330  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:29.497385  7564 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:29.497843  7564 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.498263  7564 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:29.499657  7322 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.609s	user 0.017s	sys 0.000s
I20260709 12:24:29.501302  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.422214  7602 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.502816  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.504242  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.504774  7602 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.505311  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:29.506649  7657 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.660s	user 0.014s	sys 0.000s
I20260709 12:24:29.507412  7407 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.508213  7564 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.509394  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.509770  7407 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.510056  1876 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.510660  1730 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:29.510536  7564 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.514441  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:29.462107  7740 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:29.516711  7740 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:29.519747  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.521879  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.523427  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.524068  1875 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:29.525112  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.527120  7659 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.527313  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 779) took 1221 ms. Trace:
I20260709 12:24:29.529276  1802 rpcz_store.cc:276] 0709 12:24:28.305536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.380684 (+ 75148us) service_pool.cc:224] Handling call
0709 12:24:29.527100 (+1146416us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54001eb9d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:29.527248 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":274583424}
I20260709 12:24:29.526446  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.530617  1876 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:29.529932  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:29.532248  1875 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:29.532207  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.484669  7640 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.534373  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.534637  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.544806  1802 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:29.548602  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 689004us
I20260709 12:24:29.549973  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:29.554332  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.555613  1872 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 15: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:29.560182  1803 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: [18.9-18.9]
I20260709 12:24:29.563993  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.564599  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:29.565713  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:29.534230  7667 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:29.568253  7667 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:29.568939  7667 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:29.533313  7739 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:29.565713  7738 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:29.533793  7602 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.567605  1805 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:29.572124  7602 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.578339  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:29.583657  7659 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:29.583855  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.584867  1803 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:29.555524  7579 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.586076  7611 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:29.587039  7611 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:29.587482  7611 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:29.588382  7611 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:29.588876  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 638938us
I20260709 12:24:29.589551  7611 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:29.590828  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:29.591459  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:29.592288  7599 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:24:29.595484  7729 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:24:29.595533  7579 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.596066  7729 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:29.596505  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2452) took 1077 ms. Trace:
I20260709 12:24:29.596841  7579 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.597007  7729 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.600348  7579 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:29.601226  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.603577  1732 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:29.604436  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.605144  1807 raft_consensus.cc:2376] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15. Re-sending same reply.
I20260709 12:24:29.586133  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.596905  1878 rpcz_store.cc:276] 0709 12:24:28.519206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.591462 (+ 72256us) service_pool.cc:224] Handling call
0709 12:24:29.596334 (+1004872us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b5400223b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:29.596450 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":595424512}
I20260709 12:24:29.608709  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.609637  1878 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:29.611714  1878 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:24:29.612563  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2449) took 1096 ms. Trace:
I20260709 12:24:29.613021  1878 rpcz_store.cc:276] 0709 12:24:28.515736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.610802 (+1095066us) service_pool.cc:224] Handling call
0709 12:24:29.611349 (+   547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.612198 (+   849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.612487 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.605038  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:29.611560  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:29.598672  7729 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:29.625890  7733 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:29.626354  7733 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.626695  7733 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:29.627671  7573 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.623313  7398 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:29.594703  7599 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:29.628780  7398 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.629544  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:29.630494  1723 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 11.
I20260709 12:24:29.620548  7358 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:29.635160  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:29.635538  7398 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.587459  7743 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:29.636602  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 777) took 1461 ms (client timeout 1871 ms). Trace:
I20260709 12:24:29.637152  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.637909  1802 raft_consensus.cc:2376] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16. Re-sending same reply.
I20260709 12:24:29.638769  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.638950  1806 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 12:24:29.639298  1806 pending_rounds.cc:85] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 10
I20260709 12:24:29.639524  1806 pending_rounds.cc:107] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 16.11
W20260709 12:24:29.636982  1798 rpcz_store.cc:269] 0709 12:24:28.175243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.266256 (+ 91013us) service_pool.cc:224] Handling call
0709 12:24:29.636404 (+1370148us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b540004f6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:29.636544 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":906371328}
I20260709 12:24:29.639668  7452 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.636089  1803 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:24:29.635221  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:29.536715  7005 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16.
I20260709 12:24:29.658298  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:29.659505  1876 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:29.660329  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:29.660694  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.661074  2416 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:29.662222  1875 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:29.634495  7742 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.621966  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:29.639398  1802 raft_consensus.cc:2368] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17.
I20260709 12:24:29.644410  1806 raft_consensus.cc:2889] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:29.617904  7670 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.023s	sys 0.028s
I20260709 12:24:29.644932  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:29.649350  1805 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:29.566285  7663 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.650171  1732 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: [13.12-15.13]
I20260709 12:24:29.651762  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.677652  1807 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:29.653671  2413 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:29.654587  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:29.650133  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:29.681406  1727 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:29.642100  1878 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:29.651644  7738 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.685025  6875 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:29.667055  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:29.636057  7627 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:29.619721  7659 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:29.669100  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47263.6us
I20260709 12:24:29.690141  1727 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:29.672747  1723 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:24:29.651243  7452 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.672963  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.693217  1805 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:29.649802  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:29.673352  7358 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:29.673173  7676 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:29.640477  7673 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:29.674495  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.685921  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2451) took 1169 ms. Trace:
I20260709 12:24:29.693495  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:29.693259  1798 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260709 12:24:29.675060  7670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.023s	sys 0.028s
I20260709 12:24:29.713011  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:29.721065  1726 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
W20260709 12:24:29.714435  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64740.5us
I20260709 12:24:29.724646  1807 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:29.704183  1802 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:29.717658  1878 rpcz_store.cc:276] 0709 12:24:28.516173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.614254 (+1098081us) service_pool.cc:224] Handling call
0709 12:24:29.614763 (+   509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.685483 (+ 70720us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.685793 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.644824  7739 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
I20260709 12:24:29.730611  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.636152  7579 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:24:29.728855  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2457) took 1122 ms. Trace:
I20260709 12:24:29.735091  1878 rpcz_store.cc:276] 0709 12:24:28.606709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.727694 (+1120985us) service_pool.cc:224] Handling call
0709 12:24:29.728140 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.728392 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.728434 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:29.728498 (+    64us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:29.728560 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.728636 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.728779 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.744246  1878 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:24:29.737345  7730 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:29.711349  7738 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.747725  1875 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: [16.11-16.11]
W20260709 12:24:29.847555  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 256787us
I20260709 12:24:29.756543  1879 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:29.858134  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 793) took 1253 ms. Trace:
I20260709 12:24:29.759001  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.858609  1875 rpcz_store.cc:276] 0709 12:24:28.604860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.692897 (+1088037us) service_pool.cc:224] Handling call
0709 12:24:29.737782 (+ 44885us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.857740 (+119958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.858061 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.859313  1877 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:29.859817  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2459) took 1196 ms. Trace:
I20260709 12:24:29.860090  1877 rpcz_store.cc:276] 0709 12:24:28.663517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.758682 (+1095165us) service_pool.cc:224] Handling call
0709 12:24:29.859758 (+101076us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.715267  7358 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.791311  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 797) took 1164 ms. Trace:
I20260709 12:24:29.711676  7640 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:29.749876  1805 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:29.951804  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:29.953202  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:29.733115  7444 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:29.694145  7597 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:24:29.949527  1807 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:29.753645  7673 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.942612  6875 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:29.862752  1878 rpcz_store.cc:276] 0709 12:24:28.627401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.743379 (+1115978us) service_pool.cc:224] Handling call
0709 12:24:29.743911 (+   532us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.790835 (+ 46924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.791226 (+   391us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.711470  7659 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.953203  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:29.955358  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 350220us
I20260709 12:24:29.957413  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:29.962261  1877 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:29.988493  1875 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:29.988909  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 799) took 1321 ms. Trace:
I20260709 12:24:29.989444  1877 rpcz_store.cc:276] 0709 12:24:28.668856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.942937 (+1274081us) service_pool.cc:224] Handling call
0709 12:24:29.961794 (+ 18857us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.988514 (+ 26720us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.988836 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.844448  7730 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.717726  7627 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:29.964039  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3140) took 1559 ms (client timeout 1507 ms). Trace:
I20260709 12:24:29.979791  6875 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:29.861459  7691 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:29.981630  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:30.000677  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.001591  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:29.941898  7358 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:30.016942  7452 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.017133  7258 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.018342  7258 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:30.019757  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:30.020188  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.985512  2416 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:24:29.992388  1874 rpcz_store.cc:269] 0709 12:24:28.405023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.406943 (+  1920us) service_pool.cc:224] Handling call
0709 12:24:29.963714 (+1556771us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:29.963933 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":622200192}
W20260709 12:24:29.959481  7681 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.503s	user 0.017s	sys 0.004s
I20260709 12:24:29.992535  7730 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:30.005815  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:30.031204  1877 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:30.009872  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:30.034035  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:30.011437  1807 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:30.035987  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:30.026954  7735 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:30.011710  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:30.038111  2418 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:30.015739  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.023499  7738 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:30.044068  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:30.044901  1875 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:30.050688  7663 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.958773  7602 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.980512  7640 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:29.956771  7398 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.056553  7398 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.057372  1800 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [11.7-13.8]
I20260709 12:24:30.057730  7640 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
W20260709 12:24:30.059149  7752 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:24:30.060897  1803 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:29.992331  7673 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.063361  1803 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:30.063470  7673 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:29.956343  7444 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.064306  7444 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:30.026711  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.067260  7659 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:29.999500  7691 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:30.029129  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.067924  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.069343  7691 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:29.962603  7739 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.071075  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:30.071164  7739 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.072225  1874 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:30.070662  7691 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:30.074771  7739 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:30.092253  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3293) took 1409 ms (client timeout 1524 ms). Trace:
W20260709 12:24:30.092909  1730 rpcz_store.cc:269] 0709 12:24:28.729376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.749995 (+ 20619us) service_pool.cc:224] Handling call
0709 12:24:30.091973 (+1341978us) spinlock_profiling.cc:243] Waited 667 ms on lock 0x7b54000d16d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:30.092170 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1533460736}
I20260709 12:24:30.095002  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:30.080017  6875 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.080503  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.097779  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:30.099491  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:30.100291  1803 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:30.100579  1730 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:30.102164  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:30.103127  1803 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:30.104594  1727 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:24:30.106770  1730 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:30.116945  2411 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:30.080152  7681 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.121809  7681 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:30.041981  7733 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.107613  1800 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:30.127272  6875 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:30.134091  6875 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:30.127934  1877 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:30.129420  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:30.140254  1728 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.014s	sys 0.041s
W20260709 12:24:30.141193  1728 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.014s	sys 0.041s
I20260709 12:24:30.143937  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2386) took 1580 ms. Trace:
I20260709 12:24:30.144556  1724 rpcz_store.cc:276] 0709 12:24:28.563232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.006539 (+443307us) service_pool.cc:224] Handling call
0709 12:24:29.006965 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.347027 (+340062us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.143660 (+796633us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b54000d2ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:30.143866 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":767438080}
I20260709 12:24:30.153682  1803 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:24:30.091445  7391 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:30.142633  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 821) took 1216 ms. Trace:
I20260709 12:24:30.155457  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.145699  6875 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.156569  1877 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:24:30.136498  7735 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.022s
I20260709 12:24:30.158607  7735 consensus_queue.cc:1243] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 16.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:30.144832  7005 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:24:30.079536  7398 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.153998  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:30.158694  7358 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.134905  7739 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:30.155500  1728 rpcz_store.cc:276] 0709 12:24:28.926602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.028816 (+102214us) service_pool.cc:224] Handling call
0709 12:24:29.029163 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.142286 (+1113123us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.142324 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.142368 (+    44us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:30.142404 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.142471 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.142578 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":573440,"raft.queue_time_us":594057,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":539222,"threads_started":1}
I20260709 12:24:30.135385  7627 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:30.136308  7321 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 12:24:30.135202  7659 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:30.175675  1803 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:30.179628  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:30.167665  7667 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.598s	user 0.012s	sys 0.002s
I20260709 12:24:30.198204  2411 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:30.198837  2411 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:30.160797  7391 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:30.210659  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.229449  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2387) took 1662 ms. Trace:
I20260709 12:24:30.229736  7667 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.196091  7739 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:30.160157  7558 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:24:30.145217  7679 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:24:30.275867  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:30.278800  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 326820us
I20260709 12:24:30.281116  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:30.199894  2411 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.199869  1875 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:30.297575  2411 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:30.297870  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.298365  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:30.229126  1805 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:30.203145  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:30.202692  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.300403  1877 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:30.245491  1874 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:30.240602  1731 rpcz_store.cc:276] 0709 12:24:28.566763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.008334 (+441571us) service_pool.cc:224] Handling call
0709 12:24:29.008727 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.009021 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:29.009544 (+   523us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.199017 (+1189473us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.199988 (+   971us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.229071 (+ 29083us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.229130 (+    59us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:30.229176 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.229249 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.229384 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":607903,"prepare.queue_time_us":2223,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":601766,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":7253,"thread_start_us":609354,"threads_started":3,"wal-append.queue_time_us":83120}
I20260709 12:24:30.300853  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:30.193478  7602 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:30.306355  1803 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:30.315866  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2466) took 1634 ms. Trace:
I20260709 12:24:30.316423  1874 rpcz_store.cc:276] 0709 12:24:28.681910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.314866 (+1632956us) service_pool.cc:224] Handling call
0709 12:24:30.315241 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.315448 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.315486 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.315541 (+    55us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:30.315586 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.315656 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.315792 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.318357  1874 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:30.319208  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2470) took 1636 ms. Trace:
I20260709 12:24:30.313244  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:30.320042  1874 rpcz_store.cc:276] 0709 12:24:28.682999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.317699 (+1634700us) service_pool.cc:224] Handling call
0709 12:24:30.318095 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.318877 (+   782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.319137 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.322029  1874 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
I20260709 12:24:30.273485  7673 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.324164  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.201066  7573 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 12:24:30.299571  7627 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:30.330893  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:30.331836  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3149) took 1625 ms. Trace:
I20260709 12:24:30.326488  7611 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:24:30.333377  7670 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.342082  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:30.342519  7670 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.343834  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:30.345103  1875 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:30.346020  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3153) took 1638 ms. Trace:
I20260709 12:24:30.335280  7754 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:30.337524  1874 rpcz_store.cc:276] 0709 12:24:28.706004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.321372 (+1615368us) service_pool.cc:224] Handling call
0709 12:24:30.321768 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.322647 (+   879us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.331752 (+  9105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.201792  7640 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.130s
I20260709 12:24:30.346390  1875 rpcz_store.cc:276] 0709 12:24:28.707149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.325695 (+1618546us) service_pool.cc:224] Handling call
0709 12:24:30.344913 (+ 19218us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.345706 (+   793us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.345956 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.341903  7599 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:30.322667  7602 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.201043  7659 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.353801  1874 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:24:30.355793  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2474) took 1460 ms. Trace:
I20260709 12:24:30.343750  7597 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:30.363415  1875 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 16 index: 15. (index mismatch)
I20260709 12:24:30.364306  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2476) took 1443 ms. Trace:
I20260709 12:24:30.364693  1875 rpcz_store.cc:276] 0709 12:24:28.920614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.347329 (+1426715us) service_pool.cc:224] Handling call
0709 12:24:30.362683 (+ 15354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.363968 (+  1285us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.364240 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.356781  1874 rpcz_store.cc:276] 0709 12:24:28.895749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.353132 (+1457383us) service_pool.cc:224] Handling call
0709 12:24:30.353520 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.355387 (+  1867us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.355699 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:30.345214  7670 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.352744  7452 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:30.337044  7673 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.380126  2411 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:24:30.374198  7463 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.003s	sys 0.011s
I20260709 12:24:30.369786  7754 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:30.375656  7667 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.346918  7391 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.358314  7602 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:30.399713  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.404179  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.400323  7663 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.406121  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.386914  7733 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:30.409394  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 425652us
I20260709 12:24:30.410590  1875 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
W20260709 12:24:30.410684  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 254858us
W20260709 12:24:30.419456  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.386353  7463 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.746s	user 0.004s	sys 0.011s
I20260709 12:24:30.408666  7759 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.421368  7597 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.409909  7391 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:30.390440  7670 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.022s
I20260709 12:24:30.427276   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:30.428340  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:30.438200  7757 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:30.439033  7757 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:30.439503  7757 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:30.439682  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:30.436139  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.443935  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.445449  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.446959  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.450984  7660 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:30.451174  7729 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:30.452018  7729 consensus_queue.cc:1243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 10.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:30.453645  7398 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:30.451905  7660 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:30.456344  1873 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:30.382045  7739 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:30.463850  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 43506us
I20260709 12:24:30.413170  7663 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.455111  7670 consensus_queue.cc:1243] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 15.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:30.413028  7730 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.424996  7754 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:30.471341  7751 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
I20260709 12:24:30.451030  7611 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20260709 12:24:30.434074  7597 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.485285  7673 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.463344  7676 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:24:30.433851  7759 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.487856  7730 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.490327  2418 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [14.14-16.15]
I20260709 12:24:30.576905  7660 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.466035  7358 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.495334  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:30.465324  7599 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:24:30.468091  7321 consensus_queue.cc:1243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:24:30.487493  7659 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:30.476828  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:30.587409  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 110647us
W20260709 12:24:30.534528  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.572705  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.590474  2415 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:30.595763  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2460) took 1928 ms. Trace:
I20260709 12:24:30.481925  7640 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.100s
I20260709 12:24:30.588342  7597 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.611881  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2462) took 1931 ms. Trace:
I20260709 12:24:30.613802  1876 rpcz_store.cc:276] 0709 12:24:28.680788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.533952 (+853164us) service_pool.cc:224] Handling call
0709 12:24:30.611643 (+1077691us) spinlock_profiling.cc:243] Waited 98.6 ms on lock 0x7b54001f8e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:30.611803 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":226678528}
I20260709 12:24:30.615249  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:30.616083  2415 pending_rounds.cc:85] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 12
I20260709 12:24:30.616154  1876 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:30.616737  2415 pending_rounds.cc:107] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 12:24:30.617146  2415 raft_consensus.cc:2889] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:30.620612  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.597654  7659 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:30.621671  1876 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:30.625336  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.638064  7550 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:30.638661  7550 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:30.639670  7550 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.640733  7550 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:30.629791  7444 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.565s	user 0.012s	sys 0.002s
I20260709 12:24:30.646708  7444 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.649214  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 785) took 1173 ms. Trace:
I20260709 12:24:30.649693  1802 rpcz_store.cc:276] 0709 12:24:29.475812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.727079 (+251267us) service_pool.cc:224] Handling call
0709 12:24:29.757985 (+ 30906us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:29.904994 (+147009us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.960314 (+ 55320us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:29.999085 (+ 38771us) log.cc:844] Serialized 52 byte log entry
0709 12:24:30.477461 (+478376us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:30.477512 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.478834 (+  1322us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.648961 (+170127us) raft_consensus.cc:1685] finished
0709 12:24:30.648996 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.649144 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":102583,"raft.run_cpu_time_us":9656,"raft.run_wall_time_us":87143,"thread_start_us":472720,"threads_started":1,"wal-append.queue_time_us":642626}
I20260709 12:24:30.651930  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:30.652827  1802 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:30.629549  7321 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:30.656862  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:30.657181  7321 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:30.657796  1802 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:24:30.658191  7321 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.659196  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:30.659271  7759 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.660166  1802 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:30.659368  7321 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:30.624414  7659 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.623718  7670 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.671000  7757 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.672941  7670 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.674015  7757 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:30.667830  7659 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:30.675737  7670 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:30.677953  7663 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.678887  1874 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:30.679308  1874 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:30.680222  1874 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.661862  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:30.682224  7759 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.687688  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:30.690816  1805 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:30.689544  1874 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:30.681510  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:30.689997  1802 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 11.
I20260709 12:24:30.712252  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 806) took 1196 ms. Trace:
I20260709 12:24:30.666698  7444 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:30.736279  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 282801us
W20260709 12:24:30.736471  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.724917  1874 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:30.749001  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2477) took 1827 ms. Trace:
I20260709 12:24:30.749543  1874 rpcz_store.cc:276] 0709 12:24:28.921309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.367213 (+1445904us) service_pool.cc:224] Handling call
0709 12:24:30.367566 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.678751 (+311185us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b5400235120. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:24:30.747075 (+ 68324us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.748661 (+  1586us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":495006976}
I20260709 12:24:30.753968  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.760293  1805 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:30.761457  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:30.762148  1805 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:30.762059  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:30.755553  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.776964  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:30.778846  1874 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:24:30.780750  7742 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:24:30.782150  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3152) took 2075 ms (client timeout 1885 ms). Trace:
W20260709 12:24:30.782642  1879 rpcz_store.cc:269] 0709 12:24:28.706913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.348705 (+641792us) service_pool.cc:224] Handling call
0709 12:24:30.781914 (+1433209us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b5400117454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:30.782045 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":692346240}
I20260709 12:24:30.783780  1877 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:30.764158  7681 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 14 current term: 14 is higher.
I20260709 12:24:30.786480  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:30.791332  1879 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:30.793179  1879 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 12:24:30.790221  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2286) took 2067 ms (client timeout 1717 ms). Trace:
W20260709 12:24:30.794991  1801 rpcz_store.cc:269] 0709 12:24:28.710648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.763037 (+ 52389us) service_pool.cc:224] Handling call
0709 12:24:30.778037 (+2015000us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:30.778251 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27366144}
I20260709 12:24:30.794688  1879 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:30.797647  1802 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:24:30.788235  7742 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:30.799707  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 792) took 2181 ms (client timeout 1672 ms). Trace:
W20260709 12:24:30.800177  1880 rpcz_store.cc:269] 0709 12:24:28.604520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.719599 (+115079us) service_pool.cc:224] Handling call
0709 12:24:30.785158 (+2065559us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b540021c5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:30.785350 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":758336256}
I20260709 12:24:30.800690  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:30.801898  1874 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:24:30.707511  7759 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.805708  1874 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:30.809397  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:30.773916  7444 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.817153  1880 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:24:30.817878  1874 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:30.819032  1874 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:30.820344  1874 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:30.789698  7667 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:30.826969  1801 pending_rounds.cc:85] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 6
I20260709 12:24:30.829494  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:30.830009  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 241590us
I20260709 12:24:30.830336  1879 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:30.802208  7391 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:30.832809  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2444) took 2318 ms (client timeout 1864 ms). Trace:
W20260709 12:24:30.836076  1881 rpcz_store.cc:269] 0709 12:24:28.514585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.591485 (+ 76900us) service_pool.cc:224] Handling call
0709 12:24:30.832562 (+2241077us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b5400025b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:30.832707 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379408000}
W20260709 12:24:30.837455  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.837496  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.829834  1801 pending_rounds.cc:107] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:24:30.838392  1801 raft_consensus.cc:2889] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:30.840075  1874 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:30.776317  7398 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:24:30.846863  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:30.847859  1804 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:30.808143  7663 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:30.862383  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:30.820295  7597 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:30.799396  7676 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.047s
I20260709 12:24:30.864841  1880 raft_consensus.cc:2376] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 19. Re-sending same reply.
I20260709 12:24:30.807734  7218 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.043s
W20260709 12:24:30.849823  7536 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260709 12:24:30.813048  7660 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:24:30.891153  1874 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:30.851085  7667 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:30.891603  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:30.828631  7444 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.865319  7663 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:30.895352  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:30.899369  1804 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:30.869803  7597 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.873347  7733 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:30.877782  7636 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.550s	user 0.029s	sys 0.052s
I20260709 12:24:30.956672  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2479) took 1096 ms. Trace:
I20260709 12:24:31.239913  7597 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:31.242100  7636 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.914s	user 0.030s	sys 0.053s
I20260709 12:24:31.242173  1876 rpcz_store.cc:276] 0709 12:24:29.860031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.626597 (+766566us) service_pool.cc:224] Handling call
0709 12:24:30.627031 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.627816 (+   785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.627858 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.628642 (+   784us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.825711 (+197069us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:30.825778 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.829317 (+  3539us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.956445 (+127128us) raft_consensus.cc:1685] finished
0709 12:24:30.956468 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.956609 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":137,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"thread_start_us":196677,"threads_started":1,"wal-append.queue_time_us":273729}
I20260709 12:24:30.940465  7663 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:30.963160  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:31.245095  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 282021us
I20260709 12:24:31.246568  7733 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.247349  7660 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:31.247844  7676 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.248595  7676 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.249003  7733 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:30.944631  7788 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
W20260709 12:24:30.986433  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.886986  7739 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:31.143064  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.943917  7785 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
W20260709 12:24:30.946565  7795 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.135s	user 0.001s	sys 0.000s
I20260709 12:24:30.942819  7444 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:30.889590  7742 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:30.965200  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 790) took 1487 ms. Trace:
W20260709 12:24:31.039563  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.263201  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.004863  1880 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:24:30.967327  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.945210  7777 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
W20260709 12:24:30.945914  7782 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
W20260709 12:24:30.941071  7602 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.533s	user 0.008s	sys 0.004s
W20260709 12:24:30.986245  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.269891  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.270380  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.956776  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:31.271386  1803 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:31.188481  7759 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:31.189325  7667 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:31.203861  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.272508  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.272830  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.272990  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.034142  7659 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.205191  7761 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:30.958134  7743 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:24:30.962258  7675 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:24:31.278926  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.713059  1875 rpcz_store.cc:276] 0709 12:24:29.515545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.447989 (+932444us) service_pool.cc:224] Handling call
0709 12:24:30.448367 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.448655 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:30.449161 (+   506us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.711633 (+262472us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.711866 (+   233us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.711912 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:30.711968 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.712037 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.712187 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":400,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":184,"spinlock_wait_cycles":2304,"thread_start_us":261763,"threads_started":1}
I20260709 12:24:31.261668  1807 rpcz_store.cc:276] 0709 12:24:29.477467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.037115 (+559648us) service_pool.cc:224] Handling call
0709 12:24:30.037521 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.808262 (+770741us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.808298 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.808347 (+    49us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:30.808404 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.964867 (+156463us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b5400067054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:30.964935 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.965116 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1771956480}
W20260709 12:24:31.267103  1845 connection.cc:256] RPC call timeout handler was delayed by -1.0500531589996172s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17170000000000002s for a response
W20260709 12:24:31.261933  7321 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.601s	user 0.014s	sys 0.000s
I20260709 12:24:31.284044  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:31.286108  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:31.286484  1872 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:31.286857  1874 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:31.267560  7678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.027s	sys 0.021s
I20260709 12:24:31.267535  7740 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.291919  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:31.292966  1880 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:31.279115  7742 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:31.297622  1875 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:31.283093  7671 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.280944  7761 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.262938  7597 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.302496  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:31.304296  1874 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:31.281481  7675 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:31.289139  7678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.028s	sys 0.022s
I20260709 12:24:31.283285  7733 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 12:24:31.306625  1876 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:31.307053  1879 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:31.317678  1879 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:31.307588  1881 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:31.285033  7676 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:31.309178  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:31.319437  1872 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.309538  1845 leader_election.cc:400] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19.
I20260709 12:24:31.305079  7671 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.320556  1845 leader_election.cc:403] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:31.321563  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:31.310272  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:31.278821  7602 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.312769  7761 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:31.310305  7800 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:24:31.314240  7659 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:31.318583  1879 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.315444  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:31.280326  7444 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.021s
I20260709 12:24:31.330173  7444 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:31.329869  7675 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.330709  7321 raft_consensus.cc:2764] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
W20260709 12:24:31.305887  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59192.7us
I20260709 12:24:31.261116  7739 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.328755  1879 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:31.329205  1880 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:31.346238  7597 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:31.330727  7444 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:31.328413  7606 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19.
I20260709 12:24:31.347975  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.348708  1879 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 10. (term mismatch)
I20260709 12:24:31.349138  1879 pending_rounds.cc:85] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 9
I20260709 12:24:31.305835  7740 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.340581  7671 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:31.349409  1879 pending_rounds.cc:107] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:24:31.349695  1879 raft_consensus.cc:2889] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:31.351192  7739 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:31.332587  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.322433  1874 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:31.354025  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:31.358949  7444 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.360447  7444 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:31.363678  7740 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:31.366557  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.355432  7321 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:31.371163  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:31.374687  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.355676  7675 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.377539  1872 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:31.375967  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.353286  7597 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 12:24:31.379806  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.382004  1845 leader_election.cc:400] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17.
I20260709 12:24:31.382411  1845 leader_election.cc:403] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:31.386572  1880 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:31.375720  7676 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:31.388144  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.378762  7660 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:31.391147  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.389381  7740 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17.
I20260709 12:24:31.397974  7218 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 12:24:31.401329  7218 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:31.401867  7218 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:31.403730  7755 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:31.403079  7218 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.404435  7755 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:31.405539  7218 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:31.405438  7755 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.398977  7611 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.009s
I20260709 12:24:31.390738  7606 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:31.411633  7681 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:31.397397  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.419768  7733 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.420380  7573 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:31.420387  7733 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.422497  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.423072  7729 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:31.423430  7733 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:31.423702  7729 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:31.423475  7573 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:31.424286  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:31.425201  7573 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:31.425648  7573 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:31.426018  7573 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:31.399385  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:31.427327  7611 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.424726  7729 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.388664  7321 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:24:31.430259  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 629625us
I20260709 12:24:31.431955  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:31.433969  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:31.435555  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.389916  7676 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:31.441063  2415 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:31.447273  7742 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.448120  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:31.448958  2416 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:31.448877  7681 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:31.450150  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:31.450590  7729 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:31.450693  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:31.451611  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:31.448282  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:31.452950  2416 raft_consensus.cc:2368] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15.
I20260709 12:24:31.454897  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:31.516650  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.523018  2412 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:31.521394  7681 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:31.527490  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:31.549201  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 789) took 2072 ms. Trace:
I20260709 12:24:31.525878  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.452201  7627 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:31.452648  7676 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:31.528934  7799 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:30.596405  1878 rpcz_store.cc:276] 0709 12:24:28.667119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.131129 (+1464010us) service_pool.cc:224] Handling call
0709 12:24:30.142892 (+ 11763us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.143127 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:30.410890 (+267763us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.595314 (+184424us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.595418 (+   104us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:30.595471 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.595544 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.595691 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4950,"prepare.queue_time_us":225121,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":5349,"raft.queue_time_us":5184,"thread_start_us":420442,"threads_started":2}
I20260709 12:24:31.543823  7793 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:31.544361  7358 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:31.544994  7484 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:31.559067  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 104707us
I20260709 12:24:31.528622  7678 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:31.550387  1798 rpcz_store.cc:276] 0709 12:24:29.477140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.987891 (+510751us) service_pool.cc:224] Handling call
0709 12:24:30.031725 (+ 43834us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.032078 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:30.436212 (+404134us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.799635 (+363423us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.812055 (+ 12420us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.548635 (+736580us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:31.548724 (+    89us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:31.548824 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.548903 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.549107 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":751082,"prepare.queue_time_us":510786,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":3645,"raft.queue_time_us":65348,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":24998,"spinlock_wait_cycles":8869248,"thread_start_us":705211,"threads_started":3,"wal-append.queue_time_us":174916}
I20260709 12:24:31.542951  7640 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.523434  7606 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:31.553292  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 105073us
I20260709 12:24:31.572459  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:31.554903  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.574489  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.576030  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.524322  7761 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:24:31.459760  7742 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.521982  7757 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:24:31.553787  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:31.564608  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:31.566339  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:31.604880  1875 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:31.604979  1872 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
W20260709 12:24:31.605170  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52398.1us
I20260709 12:24:31.566362  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2385) took 1587 ms. Trace:
I20260709 12:24:31.577564  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:31.606091  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.558602  7739 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:31.606962  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.579501  7640 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:31.609349  7796 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:24:31.612095  7660 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:31.613679  7742 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:31.620182  1846 leader_election.cc:400] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15.
I20260709 12:24:31.620587  1846 leader_election.cc:403] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:24:31.606832  1800 rpcz_store.cc:276] 0709 12:24:29.979091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.149163 (+170072us) service_pool.cc:224] Handling call
0709 12:24:30.149572 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.149838 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:30.600007 (+450169us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.555353 (+955346us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.556940 (+  1587us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:31.556991 (+    51us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:31.557054 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.557118 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.566293 (+  9175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":620028,"prepare.queue_time_us":338803,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":724673,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":5627,"thread_start_us":743127,"threads_started":3,"wal-append.queue_time_us":289905}
I20260709 12:24:31.567144  7678 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:31.608353  7645 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:31.605759  7667 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:31.646504  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.614300  7321 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.617350  7681 raft_consensus.cc:2764] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
W20260709 12:24:31.671803  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 65398.4us
I20260709 12:24:31.649914  7667 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.652598  7663 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:31.655330  7780 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.011s	user 0.004s	sys 0.012s
I20260709 12:24:31.608098  7733 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 12:24:31.628336  7599 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:24:31.640463  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.570708  7484 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.669469  7694 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:31.670207  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 707077us
I20260709 12:24:31.628017  7641 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:31.559207  7793 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.636615  7776 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15.
I20260709 12:24:31.604135  7627 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.043s
I20260709 12:24:31.629061  7740 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:24:31.636093  7678 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:31.676920  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 783) took 2889 ms. Trace:
I20260709 12:24:31.629033  7742 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:31.607252  7606 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:31.684887  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.682343  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.723801  1806 rpcz_store.cc:276] 0709 12:24:28.778200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.161405 (+383205us) service_pool.cc:224] Handling call
0709 12:24:29.638603 (+477198us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.674763 (+ 36160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.666904 (+1992141us) spinlock_profiling.cc:243] Waited 477 ms on lock 0x7b540004f6d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:31.667144 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1096026368}
I20260709 12:24:31.730383  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1090) took 1617 ms. Trace:
W20260709 12:24:31.749677  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.739432  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.741248  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2461) took 3063 ms. Trace:
I20260709 12:24:31.751892  1877 rpcz_store.cc:276] 0709 12:24:28.677647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.318390 (+1640743us) service_pool.cc:224] Handling call
0709 12:24:30.783522 (+465132us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.785164 (+  1642us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.740936 (+955772us) spinlock_profiling.cc:243] Waited 455 ms on lock 0x7b5400117450. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:31.741131 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1045533568}
I20260709 12:24:31.737002  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2367) took 2989 ms. Trace:
I20260709 12:24:31.759147  1799 rpcz_store.cc:276] 0709 12:24:28.747268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.091999 (+344731us) service_pool.cc:224] Handling call
0709 12:24:30.778383 (+1686384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:30.778634 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.778660 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:30.778728 (+    68us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:30.778804 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.778884 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.736704 (+957820us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54002f2950. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:31.736885 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3877467392}
I20260709 12:24:31.673002  7321 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.695279  7645 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:31.707500  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.747121  2413 rpcz_store.cc:276] 0709 12:24:30.113240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.113456 (+   216us) service_pool.cc:224] Handling call
0709 12:24:30.113873 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.135003 (+ 21130us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:30.361793 (+226790us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.502829 (+141036us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.505014 (+  2185us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.505142 (+   128us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:30.505173 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.505237 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.654667 (+1149430us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:31.730068 (+ 75401us) raft_consensus.cc:1685] finished
0709 12:24:31.730167 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.730314 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10311,"prepare.queue_time_us":353333,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":17911,"raft.run_cpu_time_us":506,"raft.run_wall_time_us":1881,"spinlock_wait_cycles":3824384,"thread_start_us":320814,"threads_started":2,"wal-append.queue_time_us":228040}
W20260709 12:24:31.708051  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94131.6us
W20260709 12:24:31.711220  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 106715us
I20260709 12:24:31.670441  7812 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:31.678048  7667 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:31.705422  7739 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:31.686292  7799 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:31.812762  7602 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:31.766929  7641 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:31.767735  7659 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.767932  1806 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:31.768705  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 86444.7us
I20260709 12:24:31.815065  7814 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:31.769496  2415 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:31.855242  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:31.859127  1806 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:31.770383  1800 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:31.771257  1799 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:31.861132  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:31.771733  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:31.861711  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:31.862808  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.772338  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:31.773320  1724 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:24:31.774173  1807 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:24:31.774533  1729 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:31.861773  1799 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:31.775036  1731 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: []
I20260709 12:24:31.817775  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:31.865765  1805 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:31.818663  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:31.867365  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54327.3us
I20260709 12:24:31.812541  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:31.720239  7218 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.093s
W20260709 12:24:31.777931  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.696121  7811 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.083s
I20260709 12:24:31.731472  7640 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:31.817567  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:31.845568  7799 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.821868  1730 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:24:31.822696  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:31.757855  7694 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:31.574955  7597 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.107s
I20260709 12:24:31.777369  7825 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:31.816243  7810 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.760017  7742 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.768793  1802 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:31.854629  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.908880  1799 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:31.859639  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:31.862433  1732 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:31.868244  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.726198  7817 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.913923  7573 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:31.914212  7817 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.915233  7810 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.917829  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:31.918465  2416 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.919021  7742 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:31.868873  1723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:31.890535  7681 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:31.777535  7757 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:24:31.971804  7321 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:31.872575  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:31.972815  7663 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:31.973013  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 100530us
I20260709 12:24:31.902495  7816 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:31.876402  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.975464  7816 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.893422  7627 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:31.981771  7755 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:31.982366  7755 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:31.982824  7755 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.987022  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:31.987671  7742 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:31.988137  1731 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:31.837713  1727 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:31.849840  7673 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:31.987085  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 118884us
I20260709 12:24:31.989617  7673 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.991118  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:31.991742  1727 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:24:31.899084  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2450) took 3383 ms (client timeout 2481 ms). Trace:
I20260709 12:24:31.992549  7733 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:31.994148  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.009s (SENT) [suppressed 38 similar messages]
I20260709 12:24:31.926179  1805 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:32.001928  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:32.002589  1805 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
W20260709 12:24:31.869825  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 129773us
W20260709 12:24:31.926749  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72171.9us
I20260709 12:24:31.966599  7573 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:31.967176  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2393) took 1151 ms. Trace:
I20260709 12:24:31.969117  7810 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:31.877348  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:32.004114  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:32.004609  1806 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:32.003928  1803 rpcz_store.cc:276] 0709 12:24:30.815817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.278393 (+462576us) service_pool.cc:224] Handling call
0709 12:24:31.278826 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.548430 (+269604us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.548464 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.548962 (+   498us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.778180 (+229218us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:31.778259 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.784482 (+  6223us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.966869 (+182387us) raft_consensus.cc:1685] finished
0709 12:24:31.966895 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.967090 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":304881,"raft.queue_time_us":335119,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":21958,"thread_start_us":223571,"threads_started":1,"wal-append.queue_time_us":342048}
I20260709 12:24:31.988251  7742 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:31.970659  7444 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.610s	user 0.005s	sys 0.005s
W20260709 12:24:31.994720  1873 rpcz_store.cc:269] 0709 12:24:28.515962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.420370 (+904408us) service_pool.cc:224] Handling call
0709 12:24:31.898757 (+2478387us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b5400052154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:31.898968 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2379327360}
I20260709 12:24:31.979521  7321 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:31.891001  7578 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.087s
I20260709 12:24:31.900219  7799 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:31.881631  7740 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.082s
I20260709 12:24:32.009865  2416 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:32.011210  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.115728  2415 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:31.882510  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:31.902431  7820 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:31.890116  7776 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.096s
I20260709 12:24:31.912204  1732 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:32.016314  1731 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:31.873083  7659 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:31.904600  7694 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.923202  7358 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.131507  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:31.955123  7681 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.080560  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.139321  1877 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:32.080628  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.082170  2414 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:24:31.845568  7812 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.093786  1805 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:32.112512  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:32.080874  7640 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:32.151582  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:31.993242  7733 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:31.996157  7673 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.022629  7742 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:32.128760  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:32.128374  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.153959  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.161154  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.161353  1879 raft_consensus.cc:2410] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:24:32.159147  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.161839  1877 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:32.132905  1723 raft_consensus.cc:606] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:24:32.133919  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.162757  1723 raft_consensus.cc:614] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:32.163115  1876 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:32.133961  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.163789  1873 raft_consensus.cc:2376] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17. Re-sending same reply.
I20260709 12:24:32.135324  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.164690  1881 raft_consensus.cc:2376] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16. Re-sending same reply.
I20260709 12:24:32.135331  2416 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:32.137290  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.081676  7321 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.170090  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.147608  7358 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.141011  1727 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:32.141281  7681 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:32.016081  7671 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.126s
I20260709 12:24:32.089479  7536 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:31.905653  7739 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.160800  1729 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:32.166081  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.229501  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.230027  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.231177  1879 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:32.132197  1874 raft_consensus.cc:2376] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15. Re-sending same reply.
I20260709 12:24:32.232210  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.232956  1874 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:32.175669  7606 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:32.125586  7811 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:32.168639  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.250440  1731 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:31.978446  7819 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:24:32.205314  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 433679us
I20260709 12:24:32.206738  1878 raft_consensus.cc:2376] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12. Re-sending same reply.
I20260709 12:24:32.171613  1798 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:24:32.207140  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 394757us
I20260709 12:24:32.139417  7820 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:32.207968  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.208092  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.209151  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.270195  1881 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
W20260709 12:24:32.209620  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.209941  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 215802us
I20260709 12:24:32.016670  7627 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.206050  7599 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.140479  7793 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:32.119761  7663 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:24:32.216652  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.222563  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:32.227274  2416 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:32.015110  7645 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.166967  1732 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:32.205456  7321 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:32.249259  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.340939  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.341493  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.244616  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:32.342639  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 98132.1us
I20260709 12:24:32.249782  7484 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:32.139165  7641 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:32.278573  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.260726  7819 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:32.337437  1873 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:32.345772  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:32.220409  7740 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:24:32.379601  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 115804us
I20260709 12:24:32.348007  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:32.360199  1770 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.524s (SENT)
W20260709 12:24:32.363242  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.366298  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.258701  1727 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:32.367851  1724 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
W20260709 12:24:32.371702  1728 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.509s	user 0.010s	sys 0.001s
W20260709 12:24:32.378862  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 156419us
I20260709 12:24:32.383702  1728 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:32.384541  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:32.285567  7825 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:32.311671  7218 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.386983  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.387727  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:32.390301  1877 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
W20260709 12:24:32.392871  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.388614  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:32.307101  7831 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.306615  7602 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:32.363337  7444 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:32.416877  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:32.372602  7606 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.267330  7811 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:32.412007  7830 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.513s	user 0.000s	sys 0.002s
I20260709 12:24:32.379946  7740 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:32.380159  7819 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:32.338920  7599 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.400621  7793 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:32.399977  7573 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:32.411209  7484 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:32.422572  7761 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:32.429674  7810 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:32.423343  7444 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:32.474758  7776 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:32.485901  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 101097us
I20260709 12:24:32.434870  7812 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.439952  7759 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:32.490479  1873 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:32.494540  1732 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:32.487283  7776 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.418139  7218 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:32.486277  7675 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.101s	user 0.013s	sys 0.001s
I20260709 12:24:32.418661  7826 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 12:24:32.475318  7793 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:32.435302  7602 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.491493  7444 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:32.521073  2413 raft_consensus.cc:2410] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:24:32.442224  7606 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.516614  7759 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:32.479421  7820 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.420226  7831 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.484808  7573 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.522388  1873 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:32.631500  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1138) took 1184 ms. Trace:
I20260709 12:24:32.608428  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.624027  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:32.646013  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:32.627625  7837 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.651793  7671 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:32.652387  7671 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:32.652136  7837 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.652493  7811 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:32.653463  7671 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.653959  7816 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.585290  7484 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:32.644932  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:32.649989  2412 rpcz_store.cc:276] 0709 12:24:31.447383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.452028 (+  4645us) service_pool.cc:224] Handling call
0709 12:24:31.452320 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:31.523484 (+ 71164us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.631208 (+1107724us) spinlock_profiling.cc:243] Waited 70.5 ms on lock 0x7b54000e6d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:32.631422 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162146688}
I20260709 12:24:32.648810  1873 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:32.577419  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1233) took 2154 ms. Trace:
I20260709 12:24:32.530126  1729 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
W20260709 12:24:32.626554  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 280793us
W20260709 12:24:32.663014  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 246237us
I20260709 12:24:32.663897  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:32.665637  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:32.650861  1877 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 11: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:32.608944  7834 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.277s	user 0.002s	sys 0.000s
I20260709 12:24:32.629390  7599 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.602871  7733 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:32.511348  7810 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:32.677109  7810 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.668035  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:32.655958  7484 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.659682  1873 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.690357  1873 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:32.671554  7675 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 12:24:32.673585  7826 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:32.582906  7676 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.023s	user 0.008s	sys 0.006s
I20260709 12:24:32.660329  2411 rpcz_store.cc:276] 0709 12:24:30.415612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.415727 (+   115us) service_pool.cc:224] Handling call
0709 12:24:30.416064 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.421458 (+  5394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.421487 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.421994 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.546401 (+1124407us) spinlock_profiling.cc:243] Waited 929 us on lock 0x7b180031ca80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:31.767076 (+220675us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:31.767125 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.767206 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.554988 (+787782us) raft_consensus.cc:1685] finished
0709 12:24:32.555023 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.570343 (+ 15320us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7377,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":8919,"spinlock_wait_cycles":2136832,"thread_start_us":220421,"threads_started":1,"wal-append.queue_time_us":1404604}
W20260709 12:24:32.686290  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 279260us
W20260709 12:24:32.608768  7840 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260709 12:24:32.608433  7843 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:32.531498  7755 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.548s	user 0.011s	sys 0.000s
I20260709 12:24:32.688694  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:32.754494  1799 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:32.755743  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:32.640470  7817 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.608139  7799 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.628649  7218 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:32.627730  7627 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.620952  7602 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.750546  7843 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:32.775075  7843 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.775940  7573 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.671340  7837 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:32.631461  7667 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.778s	user 0.012s	sys 0.000s
W20260709 12:24:32.527818  7729 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.077s	user 0.013s	sys 0.000s
I20260709 12:24:32.700878  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:32.521929  7793 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:32.672662  7611 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.245s	user 0.011s	sys 0.002s
I20260709 12:24:32.570499  7694 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:32.609292  7776 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:32.598791  7659 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:32.696460  7826 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:32.677966  7816 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:32.578212  7833 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:32.734444  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68876.9us
I20260709 12:24:32.524394  7842 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:32.748041  7807 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.132s	user 0.006s	sys 0.000s
W20260709 12:24:32.748420  7822 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.630s	user 0.002s	sys 0.000s
W20260709 12:24:32.749126  1697 connection.cc:256] RPC call timeout handler was delayed by -1.6842489730001944s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:24:32.811957  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:32.526080  7761 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.198s
I20260709 12:24:32.757323  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:32.815768  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58546.2us
I20260709 12:24:32.758855  1799 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:32.817116  7816 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:24:32.817693  7816 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:32.818626  7816 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.820468  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:32.821638  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.763553  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1093) took 1024 ms. Trace:
I20260709 12:24:32.823103  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:32.823503  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.824028  1729 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:32.822697  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.825011  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.825488  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.826264  1731 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
W20260709 12:24:32.528251  7832 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.313s	user 0.001s	sys 0.000s
I20260709 12:24:32.773170  7484 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.741364  7811 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:32.526283  7740 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.786968  1873 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:24:32.793174  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 92435.6us
I20260709 12:24:32.705650  7676 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:32.777673  7742 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.569s	user 0.011s	sys 0.002s
W20260709 12:24:32.785735  7678 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.021s	user 0.013s	sys 0.001s
I20260709 12:24:32.799690  1805 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:32.527273  7825 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 13.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.773444  7819 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.809196  7826 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:32.798851  7611 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:32.762099  7755 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.825067  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.867323  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.827122  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.823069  2413 rpcz_store.cc:276] 0709 12:24:31.739095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.816908 (+ 77813us) service_pool.cc:224] Handling call
0709 12:24:32.754187 (+937279us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b5400489754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:32.763453 (+  9266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1435499264}
I20260709 12:24:32.879516  1877 raft_consensus.cc:2376] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 15. Re-sending same reply.
I20260709 12:24:32.827208  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.858296  1878 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: [18.9-18.9]
I20260709 12:24:32.828786  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:32.800304  7859 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:32.826380  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:32.883095  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 804) took 1040 ms. Trace:
I20260709 12:24:32.793299  7667 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.879638  1731 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:32.866950  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1148) took 1416 ms. Trace:
I20260709 12:24:32.848845  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:32.831290  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.882479  1724 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:32.890626  1799 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:32.818615  7486 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:24:32.847424  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.884163  1805 rpcz_store.cc:276] 0709 12:24:31.842940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.796721 (+953781us) service_pool.cc:224] Handling call
0709 12:24:32.799262 (+  2541us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.882758 (+ 83496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.883028 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:32.831809  7863 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.850255  1876 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [14.14-16.15]
W20260709 12:24:32.887276  7867 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:24:32.887387  7740 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:32.863767  7818 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.131s	user 0.000s	sys 0.004s
I20260709 12:24:32.891070  2415 rpcz_store.cc:276] 0709 12:24:31.450328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.962638 (+512310us) service_pool.cc:224] Handling call
0709 12:24:32.115502 (+152864us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.116225 (+   723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.866688 (+750463us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54005f28d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:32.866886 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":350677248}
I20260709 12:24:32.882165  7859 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:32.884558  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55904.9us
W20260709 12:24:32.865302  7815 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.132s	user 0.003s	sys 0.000s
I20260709 12:24:32.837747  7729 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.911844  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:32.896607  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.899451  1730 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:32.914409  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.867362  7536 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:32.927297  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:32.928138  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2322) took 1162 ms. Trace:
I20260709 12:24:32.928603  1798 rpcz_store.cc:276] 0709 12:24:31.765587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.769187 (+  3600us) service_pool.cc:224] Handling call
0709 12:24:31.769513 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.262630 (+493117us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.927884 (+665254us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b54001671d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:32.928074 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":907107584}
I20260709 12:24:32.894150  7694 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:32.898823  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 391563us
I20260709 12:24:32.831555  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:32.836488  7739 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:32.843436  7776 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.947152  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:32.947425  2414 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1143) took 1498 ms (client timeout 1670 ms). Trace:
I20260709 12:24:32.948196  2413 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:24:32.948370  1724 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
W20260709 12:24:32.950196  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.861209  7847 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.386s	user 0.000s	sys 0.003s
I20260709 12:24:32.808679  7444 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.795912  7759 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.892491  7849 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.912073  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:32.858625  7814 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:32.923183  7831 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:32.923579  7838 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:24:32.813551  7842 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.924063  7819 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:32.924626  7729 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.925036  7755 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:32.925515  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2391) took 2110 ms. Trace:
I20260709 12:24:32.934156  7799 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.942997  7739 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:32.862099  7839 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260709 12:24:32.869534  7811 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:32.931535  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:32.812352  7833 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.932253  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2395) took 1100 ms. Trace:
I20260709 12:24:32.952617  7671 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.937933  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:32.983917  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1141) took 1535 ms. Trace:
I20260709 12:24:32.942075  7667 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:32.893226  7486 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:32.987058  1873 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
W20260709 12:24:32.945852  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.988123  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.988577  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.989017  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.989593  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.844717  7852 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.953140  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2396) took 1121 ms. Trace:
I20260709 12:24:32.982936  7844 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.926692  7536 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:32.948546  7640 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.956343  7814 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:32.889052  7742 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.911396  7859 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:32.914301  7606 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:32.960829  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.903225  7863 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.962862  1804 rpcz_store.cc:276] 0709 12:24:30.815180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.240176 (+424996us) service_pool.cc:224] Handling call
0709 12:24:31.250735 (+ 10559us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.668614 (+417879us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.668650 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.669279 (+   629us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.841482 (+172203us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:31.841528 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.841626 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.747750 (+906124us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:32.911681 (+163931us) raft_consensus.cc:1685] finished
0709 12:24:32.911701 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.925430 (+ 13729us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":305665,"raft.queue_time_us":319951,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":32922,"spinlock_wait_cycles":30771968,"thread_start_us":297174,"threads_started":2,"wal-append.queue_time_us":236798}
I20260709 12:24:33.049360  1724 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:33.049975  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:33.050493  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.965263  2417 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:33.052755  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1147) took 1602 ms. Trace:
I20260709 12:24:33.053125  2417 rpcz_store.cc:276] 0709 12:24:31.450033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.568336 (+118303us) service_pool.cc:224] Handling call
0709 12:24:31.614884 (+ 46548us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.965102 (+1350218us) spinlock_profiling.cc:243] Waited 71.5 ms on lock 0x7b54005922a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:24:33.052448 (+ 87346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.052667 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":164441856}
I20260709 12:24:33.051782  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:32.949085  7484 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:33.055287  1724 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:33.055377  7484 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:33.056370  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.056494  7484 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.927330  7862 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:24:32.928442  1805 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:32.955101  7776 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.063705  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:32.892753  7678 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.064944  7776 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:32.972210  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:32.977406  1800 rpcz_store.cc:276] 0709 12:24:31.831670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.866047 (+ 34377us) service_pool.cc:224] Handling call
0709 12:24:31.866401 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.866862 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.866899 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.884508 (+ 17609us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.264290 (+379782us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:32.264332 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.265103 (+   771us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.932086 (+666983us) raft_consensus.cc:1685] finished
0709 12:24:32.932100 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.932209 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2951,"raft.queue_time_us":6231,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":17621,"thread_start_us":268081,"threads_started":1,"wal-append.queue_time_us":905720}
W20260709 12:24:33.067068  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94971.2us
I20260709 12:24:32.935016  7761 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.032s
W20260709 12:24:32.967113  7865 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:24:32.993032  1725 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
W20260709 12:24:32.947911  2414 rpcz_store.cc:269] 0709 12:24:31.448907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.450398 (+  1491us) service_pool.cc:224] Handling call
0709 12:24:32.947219 (+1496821us) spinlock_profiling.cc:243] Waited 630 ms on lock 0x7b540046a354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:32.947358 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1449578880}
I20260709 12:24:32.984359  2419 rpcz_store.cc:276] 0709 12:24:31.448309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.452518 (+  4209us) service_pool.cc:224] Handling call
0709 12:24:31.452848 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.453283 (+   435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.453313 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.459885 (+  6572us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.684647 (+224762us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:31.772436 (+ 87789us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.773964 (+  1528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.774066 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.864598 (+1090532us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:32.969982 (+105384us) raft_consensus.cc:1685] finished
0709 12:24:32.983644 (+ 13662us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.983833 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31803,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":21812,"raft.queue_time_us":48680,"raft.run_cpu_time_us":456,"raft.run_wall_time_us":33036,"spinlock_wait_cycles":250282752,"thread_start_us":213472,"threads_started":2,"wal-append.queue_time_us":206270}
I20260709 12:24:32.991266  1802 rpcz_store.cc:276] 0709 12:24:31.832085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.961759 (+129674us) service_pool.cc:224] Handling call
0709 12:24:31.962248 (+   489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.980105 (+ 17857us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:32.090857 (+110752us) log.cc:844] Serialized 14 byte log entry
0709 12:24:32.637068 (+546211us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b180048cb80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:24:32.768103 (+131035us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.783096 (+ 14993us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.783606 (+   510us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:32.783731 (+   125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.783908 (+   177us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.942913 (+159005us) raft_consensus.cc:1685] finished
0709 12:24:32.952882 (+  9969us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.953069 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33374,"prepare.queue_time_us":74403,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":9821,"raft.queue_time_us":22665,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":28823,"spinlock_wait_cycles":32578816,"thread_start_us":126261,"threads_started":1,"wal-append.queue_time_us":680513}
I20260709 12:24:33.070824  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:33.070514  7811 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.994632  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:33.070016  7761 consensus_queue.cc:1243] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 12:24:33.072616  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 78184.7us
I20260709 12:24:32.996949  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:32.997172  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.071408  1802 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:32.970185  7850 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:32.976631  7739 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:33.074049  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:33.008495  7872 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:33.009054  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40200 (request call id 47) took 1140 ms. Trace:
I20260709 12:24:33.098186  1723 rpcz_store.cc:276] 0709 12:24:31.868527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.868741 (+   214us) service_pool.cc:224] Handling call
0709 12:24:33.008852 (+1140111us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b54003cdcd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd2853 00007f16b0da060a 00007f16aa3267d4 00007f16aa326762 00007f16aa326691 00007f16aa32660c 00007f16aa3258c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:33.008984 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":488633600}
I20260709 12:24:33.099731  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.010125  7660 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:33.011337  7640 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.013235  7486 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.018292  7358 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:33.019482  7667 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:33.021275  7536 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:32.998286  7743 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:24:33.024060  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:33.024287  7681 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.790s	user 0.014s	sys 0.000s
I20260709 12:24:33.023911  7814 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:32.995321  7671 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:33.027231  7321 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.648s	user 0.012s	sys 0.002s
I20260709 12:24:33.028817  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.037525  7578 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:33.039219  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:33.039943  7837 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.045008  7833 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:33.046254  7868 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:24:33.140487  7536 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:33.050638  1804 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:33.120544  7486 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:32.926082  1875 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:33.144866  7814 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:33.145179  1875 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:33.108040  7858 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:33.067449  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.147122  1800 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:33.147357  7842 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:33.118738  7667 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.148208  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.148847  7676 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:33.149454  1800 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 18 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
W20260709 12:24:33.149606  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 110458us
I20260709 12:24:33.150261  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:32.936631  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 106086us
I20260709 12:24:33.151010  7729 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:33.068217  7823 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:33.124078  7681 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.152678  7823 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.153167  7823 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:33.005509  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2326) took 1238 ms. Trace:
I20260709 12:24:33.154260  7681 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:32.986269  7839 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.076910  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:33.155073  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2406) took 1383 ms. Trace:
I20260709 12:24:33.154889  7839 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:33.086220  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 177995us
I20260709 12:24:33.155866  2415 raft_consensus.cc:2410] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:24:33.157007  7839 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:33.087692  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:33.088157  1727 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:33.158150  7810 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:33.158829  7810 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.154021  1807 rpcz_store.cc:276] 0709 12:24:31.771804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.864367 (+ 92563us) service_pool.cc:224] Handling call
0709 12:24:31.864682 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.894722 (+ 30040us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.894757 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.902303 (+  7546us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.306516 (+404213us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:32.306557 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.306626 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.005222 (+698596us) raft_consensus.cc:1685] finished
0709 12:24:33.005254 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.005431 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17042,"raft.queue_time_us":16736,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":7516,"thread_start_us":326732,"threads_started":1,"wal-append.queue_time_us":849227}
I20260709 12:24:33.161175  7810 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:33.087466  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:33.095311  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:33.161674  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.162410  1807 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:33.104471  1723 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:24:33.105216  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:33.170090  2412 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:33.161211  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.105777  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:33.050933  7863 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:33.115885  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:33.096764  7776 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:33.119019  1873 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:32.964294  7849 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.102535  7660 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:33.203370  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:33.131211  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.000535  7852 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.134900  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:33.139729  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.133232  7837 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:33.140571  7870 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:24:33.219954  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.078238  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:33.159152  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:33.162504  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 67324.3us
I20260709 12:24:33.155507  1726 rpcz_store.cc:276] 0709 12:24:31.771660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.771794 (+   134us) service_pool.cc:224] Handling call
0709 12:24:31.819316 (+ 47522us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:32.246498 (+427182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.246551 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:32.477670 (+231119us) log.cc:844] Serialized 52 byte log entry
0709 12:24:32.830068 (+352398us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:32.830119 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.830197 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.154672 (+324475us) raft_consensus.cc:1685] finished
0709 12:24:33.154695 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.155004 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":25784,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":7488,"thread_start_us":576595,"threads_started":2,"wal-append.queue_time_us":432010}
I20260709 12:24:33.164191  7627 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:33.064988  1805 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:33.097322  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:33.148069  7842 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:33.101742  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:33.171555  2413 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:33.171715  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.195013  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.173975  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.194658  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:33.224797  1800 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:33.187903  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.191243  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.198845  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:33.003757  7844 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.225143  1723 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:33.240281  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2390) took 2425 ms. Trace:
I20260709 12:24:33.201207  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:33.175192  1804 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:33.244462  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.245846  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.155043  7799 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:33.129302  7578 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:33.147971  7640 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:33.241657  1801 rpcz_store.cc:276] 0709 12:24:30.814811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.826163 (+ 11352us) service_pool.cc:224] Handling call
0709 12:24:30.826652 (+   489us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:31.743030 (+916378us) spinlock_profiling.cc:243] Waited 36.6 ms on lock 0x7b1800048be0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abdd0bfb 00007f16abddb64b
0709 12:24:31.743234 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.743273 (+    39us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:31.744133 (+   860us) log.cc:844] Serialized 78 byte log entry
0709 12:24:32.111054 (+366921us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:32.138507 (+ 27453us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.217186 (+1078679us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.217475 (+   289us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.217560 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.240021 (+ 22461us) raft_consensus.cc:1685] finished
0709 12:24:33.240053 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.240215 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1651972,"prepare.queue_time_us":1062494,"raft.queue_time_us":714010,"raft.run_cpu_time_us":299,"raft.run_wall_time_us":43421,"spinlock_wait_cycles":84114688,"thread_start_us":469190,"threads_started":2,"wal-append.queue_time_us":481700}
I20260709 12:24:33.229560  1772 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:33.239353  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:33.240192  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.247066  1804 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:33.194526  7858 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:24:33.283006  1724 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 18 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:33.270478  1801 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:33.270629  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.282569  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:33.284503  1799 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:33.285244  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:33.287269  1728 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:33.289233  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 831) took 1385 ms. Trace:
I20260709 12:24:33.292762  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.195803  7606 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.289160  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2398) took 1456 ms. Trace:
I20260709 12:24:33.297706  2410 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:33.100589  7850 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.293273  1730 rpcz_store.cc:276] 0709 12:24:31.905487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.905618 (+   131us) service_pool.cc:224] Handling call
0709 12:24:31.907867 (+  2249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.914844 (+1006977us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.288891 (+374047us) spinlock_profiling.cc:243] Waited 991 ms on lock 0x7b540035d754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:33.289150 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2279955200}
I20260709 12:24:33.294577  1801 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:33.306646  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:33.304076  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53135.7us
I20260709 12:24:33.326303  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.220808  7812 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:33.135448  7321 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.249187  7599 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.353107  7599 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.239596  7597 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:33.355365  7597 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.355849  7597 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:33.324775  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:33.239421  7627 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:33.351204  1799 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 9. (term mismatch)
I20260709 12:24:33.359104  1799 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 8
I20260709 12:24:33.359388  1799 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 12:24:33.359627  1799 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:33.249118  7641 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:33.301389  7659 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:33.240082  7863 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.110s
I20260709 12:24:33.392611  1802 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 12.
I20260709 12:24:33.393530  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.348613  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:33.396275  1802 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:24:33.398841  1802 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:24:33.400245  1881 raft_consensus.cc:2368] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19.
I20260709 12:24:33.400353  1802 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:24:33.269048  7831 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.111s
I20260709 12:24:33.318567  7799 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:33.379671  7840 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:24:33.250941  7842 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.403647  1799 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:33.406211  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:33.383351  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 58672us
I20260709 12:24:33.360826  7819 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:33.394686  7847 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:24:33.416554  1873 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:33.420008  1879 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:33.435905  1800 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:33.446044  1807 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:33.446189  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:33.446748  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:33.447304  1879 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:33.414194  1802 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:33.330482  7776 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.078s
I20260709 12:24:33.318910  7606 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.466769  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:33.392017  7659 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:33.473240  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:33.406071  7833 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.477298  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:33.478294  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:33.378639  7812 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.481148  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:33.482264  1873 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
W20260709 12:24:33.467674  7874 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.337s	user 0.000s	sys 0.001s
I20260709 12:24:33.410750  7743 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.385354  7641 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.419965  7819 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:33.509671  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:33.603065  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 318040us
I20260709 12:24:33.603498  1731 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:33.487847  7833 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.515262  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.610081  1730 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:33.479570  7659 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:33.469233  7573 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:24:33.486660  7627 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.541955  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:33.542040  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:33.622009  1873 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:33.520097  7799 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.576010  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:33.445705  7681 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.084s
I20260709 12:24:33.630245  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:33.376677  7873 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.132s
W20260709 12:24:33.602862  7218 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.050s	sys 0.082s
I20260709 12:24:33.407114  7578 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.649780  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:33.602577  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.650640  1879 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
W20260709 12:24:33.576560  7740 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.671s	user 0.013s	sys 0.005s
I20260709 12:24:33.586861  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:33.615237  1802 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:33.576020  7885 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.642006  1874 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
W20260709 12:24:33.648818  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 206493us
W20260709 12:24:33.604032  1876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.019s	sys 0.033s
I20260709 12:24:33.616007  7831 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:33.665793  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:33.655467  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 846) took 1431 ms. Trace:
I20260709 12:24:33.667306  1877 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:24:33.667197  1732 rpcz_store.cc:276] 0709 12:24:32.224193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.372875 (+148682us) service_pool.cc:224] Handling call
0709 12:24:32.373361 (+   486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.495057 (+121696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.655055 (+1159998us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54002aa654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:33.655372 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":277527808}
I20260709 12:24:33.660105  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:33.661250  1876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.019s	sys 0.033s
I20260709 12:24:33.668913  1873 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:33.609227  7819 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:33.652107  7218 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.051s	sys 0.083s
I20260709 12:24:33.672277  1879 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:33.680565  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3187) took 1294 ms. Trace:
I20260709 12:24:33.778295  1881 rpcz_store.cc:276] 0709 12:24:32.386485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.386656 (+   171us) service_pool.cc:224] Handling call
0709 12:24:33.680273 (+1293617us) spinlock_profiling.cc:243] Waited 995 ms on lock 0x7b54001f8954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:33.680477 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2289378560}
I20260709 12:24:33.600061  7743 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.648885  7667 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:33.827458  1879 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:24:33.671666  7627 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:33.844110  1879 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 10. Current term is 11. Ops: []
W20260709 12:24:33.862169  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 415473us
I20260709 12:24:33.673008  7882 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:24:33.869678  1802 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:33.869988  1873 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
I20260709 12:24:33.886222  1879 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:33.834432  7852 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:33.835424  7671 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:33.829766  7831 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:33.844137  7776 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.834749  7891 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:33.845283  7892 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:33.891108  7755 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.036s	sys 0.046s
W20260709 12:24:33.892477  7876 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.554s	user 0.000s	sys 0.002s
I20260709 12:24:33.775833  7885 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.894951  7321 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:33.895293  7825 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:33.932052  7671 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:33.936832  7755 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.043s	sys 0.048s
I20260709 12:24:33.915169  7611 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:33.907728  7850 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:33.940157  7892 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:34.025482  7793 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.224s	user 0.009s	sys 0.005s
W20260709 12:24:34.026299  7886 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.380s	user 0.001s	sys 0.000s
I20260709 12:24:33.941272  7776 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:33.977079  7823 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.823s	user 0.019s	sys 0.000s
W20260709 12:24:34.032289  7826 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.118s	user 0.004s	sys 0.006s
I20260709 12:24:34.032749  7823 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.033144  7826 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.972057  7849 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:34.005823  7859 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.911s	user 0.013s	sys 0.006s
I20260709 12:24:34.044224  7859 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:33.940827  7891 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:34.047637  7879 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.469s	user 0.002s	sys 0.000s
I20260709 12:24:34.050127  7891 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:33.977662  7833 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:34.007457  7858 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.661s	user 0.012s	sys 0.000s
I20260709 12:24:34.051472  7858 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:24:33.989635  7842 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.052534  7844 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.054055  7837 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:33.981087  7877 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.402s	user 0.000s	sys 0.001s
W20260709 12:24:33.992524  7880 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.418s	user 0.000s	sys 0.001s
I20260709 12:24:34.019665  7740 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:24:33.976598  7883 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.323s	user 0.000s	sys 0.000s
I20260709 12:24:34.057536  7740 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:34.058740  7740 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:33.980330  7884 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.402s	user 0.000s	sys 0.001s
I20260709 12:24:33.952371  7812 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.060276  7740 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:34.022563  7831 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:33.982376  7881 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:33.971374  7905 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.067581  7905 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.026181  7793 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.026576  7892 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:33.969977  7825 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.036106  7823 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.032294  7776 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:34.042270  7887 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.458s	user 0.001s	sys 0.000s
I20260709 12:24:34.001004  7743 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.121279  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.210191  1802 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:34.127022  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 812) took 1309 ms. Trace:
I20260709 12:24:34.212177  1804 rpcz_store.cc:276] 0709 12:24:32.817083 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.285434 (+468351us) service_pool.cc:224] Handling call
0709 12:24:33.285927 (+   493us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.328767 (+ 42840us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.328804 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:33.357472 (+ 28668us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.578676 (+221204us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:33.578740 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.578903 (+   163us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.126172 (+547269us) raft_consensus.cc:1685] finished
0709 12:24:34.126269 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.126681 (+   412us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16447,"raft.queue_time_us":31937,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":29513,"thread_start_us":220702,"threads_started":1,"wal-append.queue_time_us":503489}
W20260709 12:24:34.129616  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 542694us
I20260709 12:24:34.215031  7819 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.217698  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:34.132469  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:34.219702  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 87587.3us
I20260709 12:24:34.221167  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:34.137519  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2508) took 1286 ms. Trace:
I20260709 12:24:34.138883  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2512) took 1279 ms. Trace:
I20260709 12:24:34.140494  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2507) took 1290 ms. Trace:
I20260709 12:24:34.140559  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2509) took 1289 ms. Trace:
I20260709 12:24:34.143738  2414 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:34.145082  2419 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:33.989229  7606 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.017247  7641 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.014386  7660 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.15, Last appended by leader: 15, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:33.972086  7671 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.286967  7911 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:34.288707  7611 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:34.036322  7678 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.050369  7742 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:34.018060  7835 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:24:34.016551  7885 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.077499  7905 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.194921  7812 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.299615  7812 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:34.211545  7739 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.106s	user 0.011s	sys 0.011s
I20260709 12:24:34.202085  7826 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.226699  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2403) took 1371 ms. Trace:
I20260709 12:24:34.303792  7660 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:34.303966  1807 rpcz_store.cc:276] 0709 12:24:32.855575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.286568 (+430993us) service_pool.cc:224] Handling call
0709 12:24:33.445713 (+159145us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.471762 (+ 26049us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.225984 (+754222us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400133b50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:34.226415 (+   431us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":341348096}
I20260709 12:24:34.226333  7627 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:34.206431  7825 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.198518  7793 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.308410  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3317) took 1485 ms. Trace:
I20260709 12:24:34.277359  7819 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.312464  7826 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:34.313827  7826 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.322543  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.326071  7826 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:34.286021  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 68725.1us
I20260709 12:24:34.326581  2411 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:34.329193  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.224833  1878 rpcz_store.cc:276] 0709 12:24:32.850896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.093662 (+242766us) service_pool.cc:224] Handling call
0709 12:24:33.097096 (+  3434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.105832 (+  8736us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.105866 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:33.106329 (+   463us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.375453 (+269124us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:33.375504 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.375589 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.041334 (+665745us) spinlock_profiling.cc:243] Waited 26.7 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:34.139652 (+ 98318us) raft_consensus.cc:1685] finished
0709 12:24:34.139718 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.140302 (+   584us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":145,"raft.queue_time_us":32644,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":28808,"spinlock_wait_cycles":61306752,"thread_start_us":249425,"threads_started":1,"wal-append.queue_time_us":405508}
I20260709 12:24:34.332870  7444 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.224412  1876 rpcz_store.cc:276] 0709 12:24:32.850377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.898463 (+ 48086us) service_pool.cc:224] Handling call
0709 12:24:32.898990 (+   527us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:32.903010 (+  4020us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.903050 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:32.903883 (+   833us) log.cc:844] Serialized 52 byte log entry
0709 12:24:33.033965 (+130082us) spinlock_profiling.cc:243] Waited 969 us on lock 0x7b18001cfc60. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:33.668969 (+635004us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:33.669007 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.669078 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.139573 (+470495us) raft_consensus.cc:1685] finished
0709 12:24:34.139625 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.140247 (+   622us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3465,"raft.queue_time_us":4706,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":8498,"spinlock_wait_cycles":2229504,"thread_start_us":634779,"threads_started":1,"wal-append.queue_time_us":896915}
I20260709 12:24:34.055699  7859 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:33.297616  1806 rpcz_store.cc:276] 0709 12:24:31.832676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.005292 (+172616us) service_pool.cc:224] Handling call
0709 12:24:32.005690 (+   398us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:32.021569 (+ 15879us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.021614 (+    45us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:32.022430 (+   816us) log.cc:844] Serialized 78 byte log entry
0709 12:24:32.231258 (+208828us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:32.860386 (+629128us) spinlock_profiling.cc:243] Waited 4.72 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:33.249688 (+389302us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.249978 (+   290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.250074 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.250120 (+    46us) raft_consensus.cc:1685] finished
0709 12:24:33.288872 (+ 38752us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.289082 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":977567,"raft.queue_time_us":19441,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":47735,"spinlock_wait_cycles":10856320,"thread_start_us":92293,"threads_started":1,"wal-append.queue_time_us":272272}
I20260709 12:24:34.286721  7823 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.303488  7739 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.357102  7739 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.222983  7892 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:24:34.358449  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.224239  1875 rpcz_store.cc:276] 0709 12:24:32.858990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.145929 (+286939us) service_pool.cc:224] Handling call
0709 12:24:33.146227 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.146588 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.146621 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:33.170967 (+ 24346us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.663501 (+492534us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:33.663563 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.663667 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.138249 (+474582us) raft_consensus.cc:1685] finished
0709 12:24:34.138296 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.138637 (+   341us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5444,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"thread_start_us":380654,"threads_started":1,"wal-append.queue_time_us":816286}
I20260709 12:24:34.360292  7823 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.362324  7812 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.363497  7823 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.284329  7811 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:34.364176  7811 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:34.364611  7811 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:34.326443  7599 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.224174  1872 rpcz_store.cc:276] 0709 12:24:32.850629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.026930 (+176301us) service_pool.cc:224] Handling call
0709 12:24:33.027439 (+   509us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.027981 (+   542us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.028017 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:33.028639 (+   622us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.180336 (+151697us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:33.180403 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.180520 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.046816 (+866296us) spinlock_profiling.cc:243] Waited 4.92 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:34.136416 (+ 89600us) raft_consensus.cc:1685] finished
0709 12:24:34.136491 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.137170 (+   679us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4321,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":115,"spinlock_wait_cycles":11308416,"thread_start_us":151155,"threads_started":1,"wal-append.queue_time_us":382384}
I20260709 12:24:34.366598  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.351245  2417 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [14.14-16.15]
I20260709 12:24:34.309108  1729 rpcz_store.cc:276] 0709 12:24:32.822654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.883185 (+ 60531us) service_pool.cc:224] Handling call
0709 12:24:32.883526 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.005622 (+122096us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.005655 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:33.088987 (+ 83332us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.398012 (+309025us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:33.398054 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.025553 (+627499us) spinlock_profiling.cc:243] Waited 7.31 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:34.307503 (+281950us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54003cd2d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:34.307635 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.307714 (+    79us) raft_consensus.cc:1685] finished
0709 12:24:34.307732 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.308063 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":296933504,"thread_start_us":277417,"threads_started":1,"wal-append.queue_time_us":420016}
I20260709 12:24:34.349102  1805 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:34.372449  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2337) took 1175 ms. Trace:
I20260709 12:24:34.373268  1805 rpcz_store.cc:276] 0709 12:24:33.196661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.239908 (+ 43247us) service_pool.cc:224] Handling call
0709 12:24:34.340745 (+1100837us) spinlock_profiling.cc:243] Waited 1.19 ms on lock 0x7b54001ea928. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bb55f4 00007f16b0d9e227 00007f16b0d9dab8 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:34.372306 (+ 31561us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2742656}
I20260709 12:24:34.359567  7671 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.319902  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 822) took 2755 ms. Trace:
I20260709 12:24:34.376096  1880 rpcz_store.cc:276] 0709 12:24:31.564525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.564718 (+   193us) service_pool.cc:224] Handling call
0709 12:24:31.565186 (+   468us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:31.569667 (+  4481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.569716 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:31.581077 (+ 11361us) log.cc:844] Serialized 52 byte log entry
0709 12:24:32.089478 (+508401us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:32.306153 (+216675us) log.cc:844] Serialized 14 byte log entry
0709 12:24:32.338987 (+ 32834us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.339094 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.828937 (+1489843us) spinlock_profiling.cc:243] Waited 4.84 ms on lock 0x7b1800384880. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:34.318955 (+490018us) raft_consensus.cc:1685] finished
0709 12:24:34.319179 (+   224us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.319591 (+   412us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":347083,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":49748,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":51994,"spinlock_wait_cycles":11134336,"thread_start_us":581787,"threads_started":2,"wal-append.queue_time_us":712383}
I20260709 12:24:34.318005  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:34.381220  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63441.2us
W20260709 12:24:34.382474  1770 leader_election.cc:336] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.832s (SENT)
I20260709 12:24:34.383249  7812 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:34.383499  7816 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.397500  7816 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:34.399108  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.401464  7816 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.382644  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.385423  7599 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:34.403427  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.031s	user 0.008s	sys 0.015s
I20260709 12:24:34.403079  7599 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:34.403894  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.163s	user 0.008s	sys 0.015s
W20260709 12:24:34.404352  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 1.16344e+06us
W20260709 12:24:34.404536  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2516) took 1543 ms (client timeout 1669 ms). Trace:
W20260709 12:24:34.405272  1877 rpcz_store.cc:269] 0709 12:24:32.860655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.896253 (+ 35598us) service_pool.cc:224] Handling call
0709 12:24:34.403986 (+1507733us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b5400026054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:34.404273 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1638538624}
I20260709 12:24:34.406149  7599 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:34.407490  1724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3327) took 1377 ms (client timeout 1752 ms). Trace:
I20260709 12:24:34.407917  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3334) took 1189 ms. Trace:
W20260709 12:24:34.408025  1724 rpcz_store.cc:269] 0709 12:24:33.029833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.056060 (+ 26227us) service_pool.cc:224] Handling call
0709 12:24:34.406904 (+1350844us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b54002a9254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:34.407237 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":496014720}
I20260709 12:24:34.408456  1728 rpcz_store.cc:276] 0709 12:24:33.218394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.219666 (+  1272us) service_pool.cc:224] Handling call
0709 12:24:34.407464 (+1187798us) spinlock_profiling.cc:243] Waited 66.5 ms on lock 0x7b5400436cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:34.407695 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":152855808}
W20260709 12:24:34.382623  7597 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.026s	user 0.012s	sys 0.020s
W20260709 12:24:34.392237  2382 connection.cc:256] RPC call timeout handler was delayed by -1.1868033300002025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19010000000000002s for a response
I20260709 12:24:34.413654  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:34.420712  7922 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:34.407996  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.424299  7885 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:34.425344  7885 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.422344  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:34.429998  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.430598  7922 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:34.431080  7743 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.431738  7922 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.432368  7606 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:34.432063  7743 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.433313  7606 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.435133  7743 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.436015  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1243) took 1356 ms. Trace:
I20260709 12:24:34.436362  7885 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.436551  2414 rpcz_store.cc:276] 0709 12:24:33.079117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.079255 (+   138us) service_pool.cc:224] Handling call
0709 12:24:34.435572 (+1356317us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b5400840154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:34.435909 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2113332608}
I20260709 12:24:34.437037  7923 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.447229  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 833) took 1203 ms. Trace:
I20260709 12:24:34.447782  1874 rpcz_store.cc:276] 0709 12:24:33.243303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.249867 (+  6564us) service_pool.cc:224] Handling call
0709 12:24:34.446754 (+1196887us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b54003525d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:34.447002 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":543704960}
I20260709 12:24:34.450212  7606 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.440886  7599 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:34.443638  7922 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.482479  7578 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:34.460104  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77911.1us
I20260709 12:24:34.450939  7923 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.455932  1771 leader_election.cc:400] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19.
I20260709 12:24:34.452258  7885 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:34.450714  7659 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.802s	user 0.012s	sys 0.013s
I20260709 12:24:34.452265  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.480073  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:34.453374  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1152) took 1366 ms. Trace:
I20260709 12:24:34.491081  7599 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.538746  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.540072  7922 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:34.541011  7599 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.508234  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:34.540933  7922 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.544689  7922 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.493795  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.545456  7922 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:34.494297  2419 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1242) took 1416 ms (client timeout 1674 ms). Trace:
W20260709 12:24:34.457120  1803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.004s	sys 0.019s
W20260709 12:24:34.547361  1803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.007s	sys 0.021s
I20260709 12:24:34.546336  1724 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
W20260709 12:24:34.547096  2419 rpcz_store.cc:269] 0709 12:24:33.077818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.077974 (+   156us) service_pool.cc:224] Handling call
0709 12:24:34.493940 (+1415966us) spinlock_profiling.cc:243] Waited 923 ms on lock 0x7b54007ce554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:34.494182 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2122036096}
I20260709 12:24:34.493312  1771 leader_election.cc:403] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:34.510272  7885 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:34.551589  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.553013  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:34.510938  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58903.7us
I20260709 12:24:34.552822  7885 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:34.554402  7776 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.511686  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.498013  7694 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:34.555346  7694 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.555764  7694 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:34.555955  7891 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:24:34.557013  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.477826  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:34.497457  2410 rpcz_store.cc:276] 0709 12:24:33.087215 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.087369 (+   154us) service_pool.cc:224] Handling call
0709 12:24:34.452955 (+1365586us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400488d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:34.453170 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":320336640}
I20260709 12:24:34.509303  7606 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.559206  7844 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.559477  7606 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.560352  7844 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.562312  7606 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.513710  7573 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:34.562964  7573 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.507400  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.515038  2411 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:34.515153  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.516276  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.564911  1877 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:34.565150  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.566143  1732 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:34.517226  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.518344  1880 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 19 index: 9. (index mismatch)
I20260709 12:24:34.459993  7358 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:34.462673  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2514) took 1603 ms. Trace:
I20260709 12:24:34.519354  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.520606  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.520764  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:34.521739  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.525158  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.525550  1872 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:24:34.527081  1876 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:34.498113  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 109894us
I20260709 12:24:34.494630  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.494746  7923 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.511482  7729 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:34.495776  7659 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.534876  2413 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:34.577013  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.577299  7659 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.508163  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3188) took 1100 ms. Trace:
I20260709 12:24:34.569650  7573 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:34.578510  1881 rpcz_store.cc:276] 0709 12:24:33.408045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.831137 (+423092us) service_pool.cc:224] Handling call
0709 12:24:33.831701 (+   564us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.832222 (+   521us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.832267 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:33.833025 (+   758us) log.cc:844] Serialized 26 byte log entry
0709 12:24:34.443848 (+610823us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:34.446346 (+  2498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.507644 (+ 61298us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.507725 (+    81us) raft_consensus.cc:1685] finished
0709 12:24:34.507751 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.507962 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8994,"raft.queue_time_us":90434,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":1717,"thread_start_us":510245,"threads_started":1,"wal-append.queue_time_us":463702}
W20260709 12:24:34.580519  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.580906  7678 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:34.582661  7739 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:34.582953  7678 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:34.583353  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:34.584079  7484 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.087s	user 0.012s	sys 0.016s
I20260709 12:24:34.536702  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:34.587790  1725 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:34.496876  7811 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:34.511113  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 103299us
I20260709 12:24:34.589475  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.524776  1624 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 12, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 12 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:24:34.536471  1626 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 252) took 52101 ms (client timeout 15000 ms). Trace:
W20260709 12:24:34.552314  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 96664.3us
I20260709 12:24:34.558828  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:34.492458  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.513790  7816 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:34.569914  1873 rpcz_store.cc:276] 0709 12:24:32.859491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.669717 (+810226us) service_pool.cc:224] Handling call
0709 12:24:33.670157 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.880238 (+210081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.462090 (+581852us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b5400222754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:34.462466 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":458143104}
I20260709 12:24:34.575099  7729 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:34.584426  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:34.586251  7826 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:34.587272  7678 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:34.586956  7606 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:34.563203  7776 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.602715  2417 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:34.593818  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.616961  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:34.592576  7484 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.117s	user 0.013s	sys 0.016s
I20260709 12:24:34.606087  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.586150  7844 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.626938  7826 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.627056  1728 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:34.628417  7923 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:34.629472  7729 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.630275  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.601001  7816 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.647394  7816 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:34.647673  7218 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:34.648257  7218 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:34.648712  7218 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:34.623529  7892 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:34.651736  7892 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.651476  7627 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:34.652165  7892 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:34.652418  7627 consensus_queue.cc:1243] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 16.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:24:34.653049  7599 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:34.653577  7599 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:34.620625  7811 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.634018  1732 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:34.654837  7859 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:34.645745  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.633597  7694 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.628002  7826 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.666020  7826 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.605506  7678 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:34.637171  7358 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:34.666898  7844 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:34.638712  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.668795  7923 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:34.669044  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:34.639878  1624 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:34.670244  7923 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.672487  1624 catalog_manager.cc:5697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 6 to 16, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 16 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:34.640769  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.675402  1726 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:34.675526  7678 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.639981  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.677552  7678 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:34.645000  7776 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.660915  2413 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:24:34.645471  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:34.418761  7597 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.637096  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.663553  7606 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:34.667263  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44950.5us
I20260709 12:24:34.670394  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.678700  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:34.657845  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47838.7us
I20260709 12:24:34.676558  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:34.680389  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 61516.2us
I20260709 12:24:34.688362  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:34.689333  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52333.3us
I20260709 12:24:34.695565  2411 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:34.699036  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:34.706364  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.707114  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.714339  2412 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:34.696587  7776 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:34.680840  7922 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.655937  7599 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.723511  7905 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:34.753842  7599 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:34.754029  7905 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.754755  7905 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:34.731904  1729 raft_consensus.cc:2376] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17. Re-sending same reply.
I20260709 12:24:34.755759  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:34.739889  1697 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:34.756484  2413 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:34.759140  7920 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:34.763494  1874 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:34.763978  7611 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:24:34.765141  7757 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:34.765570  7757 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:34.765043  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:34.766637  1879 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:24:34.767354  1874 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:34.766697  7757 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.741323  7913 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:34.768240  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.743304  7606 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.742806  7694 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.768885  7757 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:34.744411  1873 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:34.769698  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:34.770327  1875 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:34.771071  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:34.771998  1881 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:34.745193  1727 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 19 index: 9. (term mismatch)
I20260709 12:24:34.772835  1727 pending_rounds.cc:85] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 8
I20260709 12:24:34.772802  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.773312  1727 pending_rounds.cc:107] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 18.9
I20260709 12:24:34.773653  1727 raft_consensus.cc:2889] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:34.773814  1881 raft_consensus.cc:2410] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:24:34.745743  1723 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:34.752439  7811 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:34.767637  1878 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:34.768337  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:34.743678  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:34.770237  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:34.781631  1873 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:34.706566  7358 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:34.739634  7484 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:34.782243  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.786960  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3342) took 1201 ms. Trace:
I20260709 12:24:34.787509  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.789187  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.789718  7606 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.790007  1876 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:34.791092  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.785159  7358 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:34.792785  7358 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:34.793808  7694 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:34.794481  7694 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:34.797019  1879 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:34.766625  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:34.800195  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.796376  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:34.787513  1723 rpcz_store.cc:276] 0709 12:24:33.585820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.587479 (+  1659us) service_pool.cc:224] Handling call
0709 12:24:33.587903 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.775138 (+1187235us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.785046 (+  9908us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540030be54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:34.786885 (+  1839us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2565704064}
I20260709 12:24:34.799472  1874 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:34.809310  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:34.590664  1626 rpcz_store.cc:269] 0709 12:23:42.435263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.435470 (+   207us) service_pool.cc:224] Handling call
0709 12:23:42.502646 (+ 67176us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5400002490. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb40b2 00007f16b1c2d759 00007f16b1bc5a76 00007f16b1b96d58 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc
0709 12:23:43.464878 (+962232us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b5400002490. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3e9c 00007f16a8cb40f6 00007f16b1c2d759 00007f16b1bc5a76 00007f16b1b96d58 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc
0709 12:23:44.023958 (+559080us) spinlock_profiling.cc:243] Waited 444 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd42090 00007f16abde0f3a 00007f16b1b96e9c 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:24:34.536069 (+50512111us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:25.451250 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:25.452131 (+   881us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:25.452191 (+    60us) write_op.cc:435] Acquired schema lock
0709 12:24:25.452225 (+    34us) tablet.cc:662] Decoding operations
0709 12:24:25.512172 (+ 59947us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:25.512279 (+   107us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:25.512305 (+    26us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:24:25.513784 (+  1479us) tablet.cc:701] Row locks acquired
0709 12:24:25.513830 (+    46us) write_op.cc:260] PREPARE: finished
0709 12:24:25.514083 (+   253us) write_op.cc:270] Start()
0709 12:24:25.514226 (+   143us) write_op.cc:276] Timestamp: P: 1783599865513948 usec, L: 0
0709 12:24:25.514261 (+    35us) op_driver.cc:348] REPLICATION: starting
0709 12:24:25.516526 (+  2265us) log.cc:844] Serialized 7087 byte log entry
0709 12:24:26.267141 (+750615us) op_driver.cc:464] REPLICATION: finished
0709 12:24:27.082605 (+815464us) write_op.cc:301] APPLY: starting
0709 12:24:27.082735 (+   130us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:27.941820 (+859085us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:27.942009 (+   189us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:28.260657 (+318648us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b30004a5e0c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16af742161 00007f16af749864 00007f16af991b69 00007f16af876775 00007f16af7228da 00007f16af7253cb 00007f16af724fd3 00007f16af823e34 00007f16af80f97c 00007f16af813d61 00007f16af813d19
0709 12:24:29.276731 (+1016074us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:29.327101 (+ 50370us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=34,mrs_lookups=0
0709 12:24:29.327158 (+    57us) write_op.cc:312] APPLY: finished
0709 12:24:29.327694 (+   536us) log.cc:844] Serialized 156 byte log entry
0709 12:24:30.686047 (+1358353us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16a8cce0ac 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b
0709 12:24:31.242721 (+556674us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:31.242959 (+   238us) write_op.cc:454] Released schema lock
0709 12:24:31.243058 (+    99us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:31.243597 (+   539us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:31.243630 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":49,"spinlock_wait_cycles":1051479424,"child_traces":[["op",{"apply.queue_time_us":803802,"apply.run_cpu_time_us":128603,"apply.run_wall_time_us":4161037,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75255,"num_ops":17,"prepare.queue_time_us":343335,"prepare.run_cpu_time_us":29606,"prepare.run_wall_time_us":246110,"raft.queue_time_us":304974,"raft.run_cpu_time_us":40260,"raft.run_wall_time_us":518278,"replication_time_us":752646,"spinlock_wait_cycles":47742080,"thread_start_us":2953789,"threads_started":5,"wal-append.queue_time_us":2264563,"wal-append.run_cpu_time_us":48032,"wal-append.run_wall_time_us":2252940}]]}
I20260709 12:24:34.810482  1877 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:34.801527  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:34.812901  1727 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:34.816943  7913 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.817812  7913 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.818209  7826 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:34.819075  7444 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:34.819837  7826 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:34.820078  7913 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.821103  7776 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.823247  7611 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:34.824292  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:34.824374  7776 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:34.818629  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.833329  1727 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:34.837316  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.838217  1880 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:34.845436  7859 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:34.841356  1881 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:34.824740  7611 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 12:24:34.852610  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.854413  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:34.856794  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:34.847735  7660 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:34.857815  1881 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
I20260709 12:24:34.701464  1624 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 16, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 16 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:34.846905  7826 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:34.863739  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49962.7us
W20260709 12:24:34.855938  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 404661us
I20260709 12:24:34.846102  7859 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:34.833375  7816 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:34.866324  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 99841.3us
I20260709 12:24:34.865254  7831 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 12:24:34.866991  1727 raft_consensus.cc:2376] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16. Re-sending same reply.
I20260709 12:24:34.800906  7694 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.876613  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.856830  7891 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19.
I20260709 12:24:34.877545  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.857123  7484 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:34.877449  7913 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.866613  7911 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:34.868161  7826 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.887755  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.889863  7811 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:34.876291  7859 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:34.892482  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.893815  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:34.866147  1624 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 reported cstate change: term changed from 4 to 11, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 11 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:34.890679  7811 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.895032  7811 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:34.895413  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.895812  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.896251  1807 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:34.896653  1804 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:34.898628  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.898944  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.897729  1624 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 16. New cstate: current_term: 16 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } }
I20260709 12:24:34.901966  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.901517  1624 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 11, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 11 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:34.902765  1802 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:34.903791  7922 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:34.899209  7826 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:34.905117  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 192067us
I20260709 12:24:34.906916  7911 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:34.912271  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:34.912590  7911 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.913458  7812 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:34.913808  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.914803  1798 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:34.915092  7444 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:34.915786  7444 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:34.917333  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.918419  1798 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:34.920528  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.921334  1800 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:34.884900  7913 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:34.922747  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.925060  7913 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.926178  7599 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:24:34.905313  2382 connection.cc:256] RPC call timeout handler was delayed by -1.0182389169999624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1527s for a response
I20260709 12:24:34.930984  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:34.932968  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:34.938725  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:34.923223  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.938532  7816 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:34.914230  7911 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:34.941851  1800 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:34.942883  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:34.943043  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.923732  1805 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:34.944211  1807 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:34.923550  7444 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:34.945734  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.903282  1624 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 reported cstate change: config changed from index 2 to 5, term changed from 10 to 15, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to <none>, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER 075704438992440697732181e49ccd09 (127.0.140.65), VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66), VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67), VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 15 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:34.943461  1800 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:34.946743  1807 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:34.966171  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.966813  1875 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:34.968703  7812 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:34.971355  7913 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:24:34.971834  7913 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.972244  7913 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:34.976830  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.977813  1799 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:34.982342  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.984496  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.973899  1624 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 12. New cstate: current_term: 12 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:34.985917  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:34.991137  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:34.993628  1806 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
W20260709 12:24:34.953735  7743 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.008s	sys 0.042s
I20260709 12:24:35.013985  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.015038  1727 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:35.016409  1732 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:34.952802  7597 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:35.022455  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:35.023221  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46306.1us
I20260709 12:24:35.018472  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:35.014954  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.032001  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:34.935803  7892 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:35.024223  1723 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:35.036223  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.033437  1729 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
W20260709 12:24:35.017725  7743 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.008s	sys 0.043s
I20260709 12:24:35.047956  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.050798  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:34.922775  7573 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:35.054703  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:35.052858  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:35.057252  1805 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:35.061488  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.071485  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.073393  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:35.056120  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:35.075423  1725 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:35.081060  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.083150  1732 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:35.035158  7812 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:24:35.093779  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:35.094471  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:35.096673  7892 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:35.097800  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.100147  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:35.100981  1732 raft_consensus.cc:2376] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15. Re-sending same reply.
I20260709 12:24:35.064116  7218 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:35.102047  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:35.103856  7218 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:35.104653  7573 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:35.073128  7358 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:35.107746  7358 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:35.094611  1876 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:35.114316  1732 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:35.116510  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.122633  1879 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:35.131600  1877 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:34.987882  1624 catalog_manager.cc:5697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:35.133975  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:35.148703  1806 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:35.135299  1873 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:35.136824  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:35.138185  7920 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:35.149326  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:35.149816  1807 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:35.139688  1723 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:35.140643  7905 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:35.142881  7859 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:35.196156  7859 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:35.144215  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.199416  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.200201  1729 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:35.200271  7913 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:35.201218  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:35.202349  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:35.202996  1727 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:35.204312  7811 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:35.204792  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.205444  1727 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:35.206295  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.206645  1799 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:35.144953  1726 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:35.208175  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:35.209187  1726 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:35.209324  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:35.145881  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:35.211710  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:35.212250  1876 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:35.146962  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.214917  1805 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:35.216564  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:35.218091  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 167199us
I20260709 12:24:35.181882  7358 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:35.182909  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.219761  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:35.187022  1728 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:35.187839  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:35.196238  7905 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:35.221994  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72818.4us
I20260709 12:24:35.183698  1624 catalog_manager.cc:5697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 5 to 13, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 13 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:35.180318  7826 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:35.210484  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:35.275318  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:35.276397  1873 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:35.278201  1799 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:35.246871  1806 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:35.218084  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:35.264838  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:35.265725  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:35.267082  1798 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:35.281747  1877 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:35.267231  1800 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:35.282320  1726 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:24:35.283035  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 237289us
W20260709 12:24:35.271494  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.268563  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:35.284576  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:35.284982  1805 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:35.284090  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:35.288488  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:35.289322  1800 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:35.271543  7892 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.277748  7358 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:35.275192  7218 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:35.305485  1807 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:35.293776  7891 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.005s
I20260709 12:24:35.299188  7913 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:35.299394  1624 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:35.275528  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:35.330546  1727 raft_consensus.cc:606] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 12 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:35.277691  7573 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.282671  7812 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:35.344544  1803 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.035s	sys 0.058s
I20260709 12:24:35.286944  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:35.321421  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 101742us
I20260709 12:24:35.306414  1879 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:35.324529  1876 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:35.325624  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:35.331700  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:35.346984  1877 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:35.288220  7911 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:35.337152  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:35.338747  1732 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:35.288901  7811 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:35.348151  1803 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.035s	sys 0.058s
I20260709 12:24:35.422761  1806 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:24:35.462283  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 184006us
W20260709 12:24:35.463685  2410 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.874s	user 0.010s	sys 0.018s
W20260709 12:24:35.371150  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.017s	sys 0.033s
W20260709 12:24:35.467500  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.889s	user 0.018s	sys 0.033s
W20260709 12:24:35.464525  2410 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.010s	sys 0.018s
W20260709 12:24:35.456446  1731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.863s	user 0.014s	sys 0.016s
W20260709 12:24:35.473798  1731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.015s	sys 0.016s
I20260709 12:24:35.476258  1873 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:35.322482  7859 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:35.337546  7358 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:35.491258  1800 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
W20260709 12:24:35.464080  7885 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.855s	user 0.011s	sys 0.057s
I20260709 12:24:35.315611  7892 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:35.504137  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 79749us
I20260709 12:24:35.506489  1806 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:35.466398  7891 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.512436  1798 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:35.507493  1772 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:35.512259  1880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.004s	sys 0.013s
W20260709 12:24:35.514714  1880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.004s	sys 0.013s
I20260709 12:24:35.510720  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2399) took 3670 ms. Trace:
W20260709 12:24:35.502612  7885 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.926s	user 0.011s	sys 0.058s
W20260709 12:24:35.491979  7627 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.009s	sys 0.005s
I20260709 12:24:35.493099  7484 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:35.470225  7905 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:35.542259  7905 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:35.545127  7905 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:35.547061  1877 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:35.426175  7218 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:35.525596  1879 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:35.546576  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:24:35.515790  7444 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.001s	sys 0.016s
I20260709 12:24:35.541061  1805 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:35.523628  1803 rpcz_store.cc:276] 0709 12:24:31.833026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.081282 (+248256us) service_pool.cc:224] Handling call
0709 12:24:32.121590 (+ 40308us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.133226 (+ 11636us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:33.168571 (+1035345us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b1800195be0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf 00007f16abdf2038
0709 12:24:33.573986 (+405415us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.548191 (+974205us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:34.549174 (+   983us) log.cc:844] Serialized 26 byte log entry
0709 12:24:34.549408 (+   234us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:35.425843 (+876435us) log.cc:844] Serialized 14 byte log entry
0709 12:24:35.461618 (+ 35775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.475397 (+ 13779us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:35.500029 (+ 24632us) raft_consensus.cc:1685] finished
0709 12:24:35.500065 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:35.503668 (+  3603us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51864,"prepare.queue_time_us":1164412,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":17861,"raft.queue_time_us":62650,"raft.run_cpu_time_us":493,"raft.run_wall_time_us":29802,"spinlock_wait_cycles":3207936,"thread_start_us":2248039,"threads_started":3,"wal-append.queue_time_us":765174}
I20260709 12:24:35.557559  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:35.558223  1877 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:24:35.559126  1874 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.000s	sys 0.012s
W20260709 12:24:35.550334  2418 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.904s	user 0.010s	sys 0.011s
W20260709 12:24:35.560685  2418 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.915s	user 0.010s	sys 0.011s
W20260709 12:24:35.561319  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.006s	sys 0.007s
W20260709 12:24:35.561760  1874 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.000s	sys 0.012s
W20260709 12:24:35.557686  1875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.005s	sys 0.012s
W20260709 12:24:35.563140  1875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.005s	sys 0.012s
I20260709 12:24:35.491755  7573 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:35.547863  7820 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.018s	sys 0.059s
W20260709 12:24:35.562183  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.006s	sys 0.007s
W20260709 12:24:35.568281  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66652.7us
I20260709 12:24:35.569859  1873 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:35.573523  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:35.557633  7444 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.001s	sys 0.016s
W20260709 12:24:35.534368  7627 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.010s	sys 0.005s
I20260709 12:24:35.565958  1805 pending_rounds.cc:85] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 9
I20260709 12:24:35.585222  1805 pending_rounds.cc:107] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 12:24:35.585649  1805 raft_consensus.cc:2889] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:35.591233  1877 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
W20260709 12:24:35.583535  1872 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.005s	sys 0.007s
W20260709 12:24:35.571111  7820 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.081s	user 0.018s	sys 0.060s
I20260709 12:24:35.592785  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "908089442ca44109875257bfbc9d06c2"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
W20260709 12:24:35.592823  1872 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.005s	sys 0.008s
W20260709 12:24:35.564554  1802 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.005s	sys 0.008s
I20260709 12:24:35.594756  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:35.596920  1802 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.005s	sys 0.008s
I20260709 12:24:35.597572  1879 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 12:24:35.587579  7694 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.020s	sys 0.009s
W20260709 12:24:35.601763  7740 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.012s	sys 0.002s
W20260709 12:24:35.597400  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.008s	sys 0.006s
I20260709 12:24:35.532929  7891 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:35.586091  7911 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:24:35.610096  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.009s	sys 0.006s
W20260709 12:24:35.595407  7923 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.909s	user 0.028s	sys 0.028s
I20260709 12:24:35.611436  7885 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:35.611660  7826 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:35.613054  7740 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.013s	sys 0.003s
I20260709 12:24:35.612493  1624 catalog_manager.cc:5697] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 6 to 15. New cstate: current_term: 15 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:24:35.624905  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 58315.5us
W20260709 12:24:35.613040  7930 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:24:35.609158  7694 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.021s	sys 0.009s
W20260709 12:24:35.614907  7923 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.940s	user 0.032s	sys 0.028s
I20260709 12:24:35.637702  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:35.645853  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:35.634356  7816 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:35.599241  7358 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:35.654284  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:35.661695  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56216.5us
I20260709 12:24:35.664182  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:35.667573  7931 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.065s	user 0.001s	sys 0.001s
I20260709 12:24:35.668382  7812 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:35.668951  7812 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:35.669975  1624 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 15, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 15 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:35.673616  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2431) took 1150 ms. Trace:
I20260709 12:24:35.681113  1799 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:35.674063  1730 rpcz_store.cc:276] 0709 12:24:34.523567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.575880 (+ 52313us) service_pool.cc:224] Handling call
0709 12:24:34.576409 (+   529us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:34.577175 (+   766us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.577221 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:34.578325 (+  1104us) log.cc:844] Serialized 52 byte log entry
0709 12:24:35.468589 (+890264us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:35.468678 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.468858 (+   180us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:35.673360 (+204502us) raft_consensus.cc:1685] finished
0709 12:24:35.673383 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:35.673543 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13196,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":93,"thread_start_us":889460,"threads_started":1,"wal-append.queue_time_us":979860}
W20260709 12:24:35.676699  1804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.003s	sys 0.022s
W20260709 12:24:35.690441  1804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.003s	sys 0.023s
W20260709 12:24:35.686713  1881 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.000s	sys 0.021s
I20260709 12:24:35.670104  7812 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:35.691426  1878 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.818s	user 0.005s	sys 0.017s
W20260709 12:24:35.700824  1878 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.005s	sys 0.017s
W20260709 12:24:35.674831  7659 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.028s	sys 0.028s
I20260709 12:24:35.698616  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:35.703212  2413 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:35.706182  2412 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:35.643344  7885 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:35.703454  1844 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.010s	sys 0.002s
I20260709 12:24:35.704797  1799 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: [16.12-16.12]
W20260709 12:24:35.711779  1844 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.011s	sys 0.003s
I20260709 12:24:35.655530  7816 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:35.696894  7729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.784s	user 0.016s	sys 0.008s
W20260709 12:24:35.694334  1881 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.000s	sys 0.022s
W20260709 12:24:35.716135  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.692539  7484 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:35.701764  7611 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.010s	sys 0.013s
W20260709 12:24:35.712217  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 928885us
I20260709 12:24:35.710765  7678 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:35.708207  7659 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.965s	user 0.029s	sys 0.028s
I20260709 12:24:35.700620  7892 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:35.727954  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2432) took 1153 ms. Trace:
I20260709 12:24:35.724651  1799 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:35.728673  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:35.735994  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43946.4us
I20260709 12:24:35.736827  1732 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:35.718128  7729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.016s	sys 0.008s
W20260709 12:24:35.737321  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73290.3us
I20260709 12:24:35.739755  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:35.741572  1723 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:35.740480  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.742822  1732 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 16 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:24:35.735678  1731 rpcz_store.cc:276] 0709 12:24:34.574640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.579464 (+  4824us) service_pool.cc:224] Handling call
0709 12:24:34.580088 (+   624us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:34.580816 (+   728us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.580849 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:34.582001 (+  1152us) log.cc:844] Serialized 52 byte log entry
0709 12:24:35.508365 (+926364us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:24:35.508439 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.508526 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:35.727618 (+219092us) raft_consensus.cc:1685] finished
0709 12:24:35.727648 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:35.727867 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2409,"raft.queue_time_us":9672,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":12567,"thread_start_us":892111,"threads_started":1,"wal-append.queue_time_us":1116690}
I20260709 12:24:35.754236  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:35.734760  7611 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.866s	user 0.010s	sys 0.014s
I20260709 12:24:35.754807  1732 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260709 12:24:35.760875  1732 pending_rounds.cc:85] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 10
I20260709 12:24:35.755578  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:35.757112  1728 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:35.763094  1728 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:35.772212  7891 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:35.773511  1801 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.016s	sys 0.007s
W20260709 12:24:35.774075  1801 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.017s	sys 0.007s
I20260709 12:24:35.776875  7660 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:35.777344  2382 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:35.779167  7885 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:35.779891  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:35.780530  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2426) took 1258 ms. Trace:
I20260709 12:24:35.780952  1724 rpcz_store.cc:276] 0709 12:24:34.521777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.552745 (+ 30968us) service_pool.cc:224] Handling call
0709 12:24:35.780338 (+1227593us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002a9754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:35.780470 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2779532160}
I20260709 12:24:35.781368  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:35.782290  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:35.783061  7816 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:35.784577  7816 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:35.788223  7694 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:35.789426  7694 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:35.795853  1723 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:35.735133  7218 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:35.790382  7611 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:24:35.817139  7859 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:35.777436  7660 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:35.819437  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:35.812011  7678 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:35.761252  1732 pending_rounds.cc:107] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:24:35.887212  1732 raft_consensus.cc:2889] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:35.797055  7945 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:35.820747  7885 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:35.838552  7743 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:35.842092  7923 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:35.842343  7573 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:35.947988  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 165772us
W20260709 12:24:35.960497  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78522.3us
I20260709 12:24:35.883491  7678 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:35.983634  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 164254us
I20260709 12:24:35.985301  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:35.990057  7952 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.177s	user 0.000s	sys 0.000s
W20260709 12:24:36.002812  7951 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:35.847754  7729 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.114s
I20260709 12:24:36.008948  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3212) took 1227 ms. Trace:
I20260709 12:24:36.009485  1874 rpcz_store.cc:276] 0709 12:24:34.781936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.824201 (+ 42265us) service_pool.cc:224] Handling call
0709 12:24:34.825466 (+  1265us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:34.825776 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.825813 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:34.834351 (+  8538us) log.cc:844] Serialized 26 byte log entry
0709 12:24:35.564471 (+730120us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:35.564524 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.564616 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.008605 (+443989us) raft_consensus.cc:1685] finished
0709 12:24:36.008654 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.008860 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":729582,"threads_started":1,"wal-append.queue_time_us":1103942}
W20260709 12:24:36.012645  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49480.7us
I20260709 12:24:36.013998  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:35.926092  7885 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:35.938591  7611 consensus_queue.cc:1243] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 20.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:35.955140  7905 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.060966  7935 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:24:36.063731  7945 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.064687  7949 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:36.070211  7218 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:36.055627  1728 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:36.032357  7885 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:36.031190  7920 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:24:36.098032  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86310.3us
I20260709 12:24:36.049490  7743 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:24:36.112497  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 98609.5us
I20260709 12:24:36.115419  1728 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:36.099248  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:36.077996  7218 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:36.122315  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2346) took 1209 ms. Trace:
W20260709 12:24:36.127815  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.126926  1804 rpcz_store.cc:276] 0709 12:24:34.900856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.903617 (+  2761us) service_pool.cc:224] Handling call
0709 12:24:34.904043 (+   426us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:34.904846 (+   803us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.904879 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:35.015278 (+110399us) log.cc:844] Serialized 52 byte log entry
0709 12:24:35.691205 (+675927us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:35.691252 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.691330 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.109704 (+418374us) raft_consensus.cc:1685] finished
0709 12:24:36.109740 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.110029 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":132,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":595225,"threads_started":1,"wal-append.queue_time_us":1041976}
I20260709 12:24:36.039386  7859 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.075s
I20260709 12:24:36.111444  7891 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.143252  1807 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.011s	sys 0.027s
W20260709 12:24:36.147497  1807 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.011s	sys 0.028s
I20260709 12:24:36.158600  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3213) took 1374 ms. Trace:
I20260709 12:24:36.076905  7905 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:36.159130  1880 rpcz_store.cc:276] 0709 12:24:34.784115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.869105 (+ 84990us) service_pool.cc:224] Handling call
0709 12:24:34.886317 (+ 17212us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:34.974884 (+ 88567us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:34.974953 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:34.975717 (+   764us) log.cc:844] Serialized 26 byte log entry
0709 12:24:35.523669 (+547952us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:35.523709 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.523788 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.158324 (+634536us) raft_consensus.cc:1685] finished
0709 12:24:36.158349 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.158525 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6326,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":91,"thread_start_us":547400,"threads_started":1,"wal-append.queue_time_us":744092}
I20260709 12:24:36.120260  7920 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:36.159699  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.004s	sys 0.008s
W20260709 12:24:36.165436  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.004s	sys 0.008s
I20260709 12:24:36.165779  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:36.166203  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 592724us
I20260709 12:24:36.176151  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:36.137226  7694 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:36.032768  7660 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:36.183336  2410 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.017s	sys 0.083s
W20260709 12:24:36.183797  2410 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.018s	sys 0.083s
I20260709 12:24:36.190197  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2428) took 1077 ms. Trace:
I20260709 12:24:36.191051  1804 rpcz_store.cc:276] 0709 12:24:35.113013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.140122 (+1027109us) service_pool.cc:224] Handling call
0709 12:24:36.163985 (+ 23863us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.178211 (+ 14226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.178250 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.189729 (+ 11479us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:36.189809 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.189911 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.190111 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.158490  7892 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:36.108623  7945 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:36.186038  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:36.213855  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:36.216810  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:36.213203  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1105) took 1619 ms. Trace:
I20260709 12:24:36.198669  7611 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:36.223716  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47696.1us
I20260709 12:24:36.180975  7694 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:36.227008  7218 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:36.220463  1803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.006s	sys 0.011s
W20260709 12:24:36.224969  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.035s	sys 0.069s
I20260709 12:24:36.219547  2412 raft_consensus.cc:2368] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15.
I20260709 12:24:36.201505  7739 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:36.164242  7891 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:36.217296  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:36.228792  1879 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.002s	sys 0.042s
I20260709 12:24:36.225965  2414 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 11.
W20260709 12:24:36.216904  7627 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.012s	sys 0.031s
I20260709 12:24:36.143936  7678 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:36.231253  7911 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.010s	sys 0.007s
W20260709 12:24:36.235925  1875 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.637s	user 0.003s	sys 0.015s
I20260709 12:24:36.198454  7729 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
W20260709 12:24:36.234658  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.007s	sys 0.036s
W20260709 12:24:36.262702  1802 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.009s	sys 0.011s
I20260709 12:24:36.218830  7949 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:36.229167  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:36.232211  7820 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.010s	sys 0.032s
I20260709 12:24:36.232244  7743 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:36.269727  1877 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.012s	sys 0.014s
I20260709 12:24:36.216435  7892 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:36.279803  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 75975.6us
W20260709 12:24:36.281657  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 82646us
W20260709 12:24:36.299115  1730 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.000s	sys 0.029s
W20260709 12:24:36.335189  1730 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.000s	sys 0.029s
I20260709 12:24:36.335215  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:36.303860  1878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.012s	sys 0.017s
I20260709 12:24:36.336194  7826 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:36.336570  1878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.012s	sys 0.017s
I20260709 12:24:36.353936  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 847) took 1587 ms. Trace:
I20260709 12:24:36.354456  1878 rpcz_store.cc:276] 0709 12:24:34.766112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.801063 (+ 34951us) service_pool.cc:224] Handling call
0709 12:24:34.813249 (+ 12186us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.836479 (+ 23230us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:35.701555 (+865076us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.353421 (+651866us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.353503 (+    82us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:36.353576 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.353668 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.353861 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":781307,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"thread_start_us":1433996,"threads_started":2}
I20260709 12:24:36.311156  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:24:36.311498  1879 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.003s	sys 0.042s
W20260709 12:24:36.318251  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.007s	sys 0.036s
W20260709 12:24:36.356841  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 722728us
I20260709 12:24:36.313856  2411 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:36.306748  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.035s	sys 0.069s
W20260709 12:24:36.318526  1802 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.010s	sys 0.012s
W20260709 12:24:36.293650  2418 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.688s	user 0.012s	sys 0.014s
W20260709 12:24:36.315538  1875 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.003s	sys 0.015s
W20260709 12:24:36.298432  1805 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.000s	sys 0.017s
W20260709 12:24:36.359450  1805 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.000s	sys 0.021s
W20260709 12:24:36.317122  1801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.007s	sys 0.004s
W20260709 12:24:36.360278  1801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.008s	sys 0.004s
W20260709 12:24:36.332947  2413 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.619s	user 0.009s	sys 0.005s
W20260709 12:24:36.298813  1806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.013s	sys 0.004s
W20260709 12:24:36.361076  1806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.013s	sys 0.004s
I20260709 12:24:36.362404  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3215) took 1540 ms. Trace:
I20260709 12:24:36.362808  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:36.331775  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:36.306547  1803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.007s	sys 0.011s
W20260709 12:24:36.317066  1881 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.001s	sys 0.011s
W20260709 12:24:36.366560  1881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.001s	sys 0.011s
W20260709 12:24:36.323279  1877 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.013s	sys 0.014s
W20260709 12:24:36.282910  1873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.012s	sys 0.013s
W20260709 12:24:36.358992  2418 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.012s	sys 0.018s
W20260709 12:24:36.367998  1873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.013s	sys 0.013s
I20260709 12:24:36.368109  2413 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
W20260709 12:24:36.332365  1800 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.010s	sys 0.020s
W20260709 12:24:36.371773  1800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.010s	sys 0.020s
I20260709 12:24:36.373520  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:36.370918  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:36.321821  7820 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.010s	sys 0.033s
W20260709 12:24:36.331130  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 105399us
W20260709 12:24:36.315358  7911 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.010s	sys 0.007s
I20260709 12:24:36.398002  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2350) took 1483 ms. Trace:
I20260709 12:24:36.362846  1875 rpcz_store.cc:276] 0709 12:24:34.821556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.970066 (+148510us) service_pool.cc:224] Handling call
0709 12:24:34.970525 (+   459us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:34.971038 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.971068 (+    30us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:34.972024 (+   956us) log.cc:844] Serialized 78 byte log entry
0709 12:24:35.563803 (+591779us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:36.359139 (+795336us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.362008 (+  2869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.362104 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.362155 (+    51us) raft_consensus.cc:1685] finished
0709 12:24:36.362170 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.362342 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2569,"prepare.queue_time_us":95185,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":58,"raft.queue_time_us":285,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":88,"thread_start_us":1351652,"threads_started":2,"wal-append.queue_time_us":867332}
I20260709 12:24:36.311339  7694 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:36.298486  7740 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.009s	sys 0.019s
W20260709 12:24:36.312520  7627 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.678s	user 0.013s	sys 0.032s
W20260709 12:24:36.293589  7659 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.016s	sys 0.010s
I20260709 12:24:36.381891  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:36.296430  2410 rpcz_store.cc:276] 0709 12:24:34.566302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.566908 (+   606us) service_pool.cc:224] Handling call
0709 12:24:34.589333 (+ 22425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.589799 (+   466us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:35.472687 (+882888us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.184811 (+712124us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.184892 (+    81us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:36.184968 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.185069 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.185238 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":970490,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":15228,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":14886,"thread_start_us":1543689,"threads_started":2,"wal-append.queue_time_us":817586}
I20260709 12:24:36.399928  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:36.404531  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:36.351301  7966 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:36.323930  7743 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:36.437212  7743 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.443271  7573 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:36.444994  7820 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:36.445523  7820 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:36.446496  7820 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:36.448323  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 835) took 1728 ms. Trace:
W20260709 12:24:36.449457  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 795236us
I20260709 12:24:36.448756  1872 rpcz_store.cc:276] 0709 12:24:34.721332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.725089 (+  3757us) service_pool.cc:224] Handling call
0709 12:24:34.725703 (+   614us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:34.808992 (+ 83289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.809031 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:34.839118 (+ 30087us) log.cc:844] Serialized 26 byte log entry
0709 12:24:35.596347 (+757229us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:35.596392 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.708596 (+112204us) spinlock_profiling.cc:243] Waited 71.3 ms on lock 0x7b54002df5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:35.708666 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.429961 (+721295us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b240023a968. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:24:36.448071 (+ 18110us) raft_consensus.cc:1685] finished
0709 12:24:36.448092 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.448249 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1123,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":29505,"spinlock_wait_cycles":173839104,"thread_start_us":741916,"threads_started":1,"wal-append.queue_time_us":971817}
I20260709 12:24:36.450299  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:36.407946  7740 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.010s	sys 0.023s
W20260709 12:24:36.412701  7659 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s	user 0.019s	sys 0.013s
I20260709 12:24:36.430301  7218 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:36.430539  7955 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.187s	user 0.001s	sys 0.001s
I20260709 12:24:36.431588  7597 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:36.431763  7891 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:36.432310  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:36.432695  7729 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:36.560446  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2522) took 1593 ms. Trace:
I20260709 12:24:36.560910  1874 rpcz_store.cc:276] 0709 12:24:34.928743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.057390 (+1128647us) service_pool.cc:224] Handling call
0709 12:24:36.057952 (+   562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.058320 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:36.073613 (+ 15293us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.522069 (+448456us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.522133 (+    64us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:36.522203 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.522274 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.522426 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":421,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":51,"thread_start_us":400291,"threads_started":1}
I20260709 12:24:36.436115  7945 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:36.380615  7678 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.414374  7966 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.398453  1802 rpcz_store.cc:276] 0709 12:24:34.915080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.915224 (+   144us) service_pool.cc:224] Handling call
0709 12:24:34.915611 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.915923 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:35.602441 (+686518us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.397606 (+795165us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.397666 (+    60us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:36.397726 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.397797 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.397939 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22167,"prepare.queue_time_us":900517,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":64,"thread_start_us":1424591,"threads_started":2,"wal-append.queue_time_us":683554}
I20260709 12:24:36.451599  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2356) took 1519 ms. Trace:
I20260709 12:24:36.451459  1872 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: [16.12-16.12]
W20260709 12:24:36.513706  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:36.514269  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.451176  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:36.573679  1806 rpcz_store.cc:276] 0709 12:24:34.932201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.517649 (+585448us) service_pool.cc:224] Handling call
0709 12:24:35.517982 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.562923 (+ 44941us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:24:35.562970 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.595452 (+ 32482us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.361618 (+766166us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:36.364752 (+  3134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.364840 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.451348 (+ 86508us) raft_consensus.cc:1685] finished
0709 12:24:36.451376 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.451528 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22428,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":9910,"thread_start_us":743016,"threads_started":1,"wal-append.queue_time_us":767858}
W20260709 12:24:36.517542  1798 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.852s	user 0.000s	sys 0.028s
W20260709 12:24:36.575284  1798 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.910s	user 0.001s	sys 0.029s
I20260709 12:24:36.406143  7911 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:36.433419  7892 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:36.565194  1878 raft_consensus.cc:2410] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:36.298664  7920 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:36.565160  1727 raft_consensus.cc:606] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:24:36.579197  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 852) took 1642 ms. Trace:
I20260709 12:24:36.464365  7905 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:36.590391  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.579643  1872 rpcz_store.cc:276] 0709 12:24:34.937237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.450328 (+1513091us) service_pool.cc:224] Handling call
0709 12:24:36.450694 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.578818 (+128124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.579129 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:36.583453  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 132325us
I20260709 12:24:36.522331  7923 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:36.597916  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:36.520923  7859 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:36.616411  1874 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:36.572790  7816 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.788s	user 0.012s	sys 0.006s
I20260709 12:24:36.621637  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2527) took 1549 ms. Trace:
I20260709 12:24:36.622157  1874 rpcz_store.cc:276] 0709 12:24:35.072185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.609482 (+1537297us) service_pool.cc:224] Handling call
0709 12:24:36.616206 (+  6724us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.621297 (+  5091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.621562 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.623816  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3214) took 1838 ms. Trace:
I20260709 12:24:36.624404  1881 rpcz_store.cc:276] 0709 12:24:34.785681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.874284 (+ 88603us) service_pool.cc:224] Handling call
0709 12:24:34.874755 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.883490 (+  8735us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:35.719471 (+835981us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.623369 (+903898us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.623443 (+    74us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:36.623506 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.623597 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.623737 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":755105,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":35899,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":35253,"thread_start_us":1646301,"threads_started":3}
W20260709 12:24:36.630291  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 267580us
I20260709 12:24:36.629683  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:36.578469  7627 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:36.573211  7729 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:36.580401  7966 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:36.631140  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 248977us
W20260709 12:24:36.643127  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 269674us
I20260709 12:24:36.644784  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 853) took 1707 ms. Trace:
I20260709 12:24:36.621210  7859 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:36.646528  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:36.650345  1872 rpcz_store.cc:276] 0709 12:24:34.937500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.619673 (+1682173us) service_pool.cc:224] Handling call
0709 12:24:36.638116 (+ 18443us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.644425 (+  6309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.644455 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.644501 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:36.644547 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.644612 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.644722 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.625892  7573 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:36.588887  7597 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:36.658180  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 347058us
I20260709 12:24:36.634095  7905 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:36.666412  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:36.669817  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2530) took 1532 ms. Trace:
I20260709 12:24:36.670347  1878 rpcz_store.cc:276] 0709 12:24:35.137101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.629128 (+1492027us) service_pool.cc:224] Handling call
0709 12:24:36.642973 (+ 13845us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.660147 (+ 17174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.660179 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.660227 (+    48us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:36.669540 (+  9313us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.669613 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.669750 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.646341  7740 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:36.646175  7627 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:36.650820  7823 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:36.657388  7859 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.683375  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 848) took 1591 ms. Trace:
I20260709 12:24:36.654168  7892 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:36.626679  7923 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:36.665135  7905 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.688694  1872 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:36.696614  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:36.692413  1725 rpcz_store.cc:276] 0709 12:24:35.091656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.094929 (+  3273us) service_pool.cc:224] Handling call
0709 12:24:35.095420 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.095958 (+   538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.095990 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.096685 (+   695us) log.cc:844] Serialized 26 byte log entry
0709 12:24:35.597124 (+500439us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:35.597180 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.597280 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.683036 (+1085756us) raft_consensus.cc:1685] finished
0709 12:24:36.683068 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.683271 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":134,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"thread_start_us":474784,"threads_started":1,"wal-append.queue_time_us":859607}
I20260709 12:24:36.707405  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2531) took 1570 ms. Trace:
I20260709 12:24:36.713230  1872 rpcz_store.cc:276] 0709 12:24:35.137407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.656297 (+1518890us) service_pool.cc:224] Handling call
0709 12:24:36.683198 (+ 26901us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.707029 (+ 23831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.707325 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:36.716238  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 316363us
W20260709 12:24:36.704674  7983 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.383s	user 0.000s	sys 0.002s
I20260709 12:24:36.678134  7740 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:36.720805  7740 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:36.705327  7993 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:36.723071  1881 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:24:36.723376  7993 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:36.723996  7993 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:36.725561  7979 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:24:36.726594  7911 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:36.728492  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1109) took 2084 ms. Trace:
I20260709 12:24:36.728863  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40200 (request call id 49) took 1136 ms. Trace:
I20260709 12:24:36.729772  7627 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:36.730484  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 851) took 1848 ms. Trace:
I20260709 12:24:36.731333  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2535) took 1592 ms. Trace:
I20260709 12:24:36.731422  1875 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [14.14-16.15]
I20260709 12:24:36.732013  1881 rpcz_store.cc:276] 0709 12:24:35.147152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.722408 (+1575256us) service_pool.cc:224] Handling call
0709 12:24:36.722811 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.723644 (+   833us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.731272 (+  7628us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.731009  1876 rpcz_store.cc:276] 0709 12:24:34.881629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.657195 (+775566us) service_pool.cc:224] Handling call
0709 12:24:35.690558 (+ 33363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.703048 (+ 12490us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.703080 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.739750 (+ 36670us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.312139 (+572389us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:36.312204 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.312296 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.730210 (+417914us) raft_consensus.cc:1685] finished
0709 12:24:36.730237 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.730407 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6426,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":13518,"thread_start_us":499056,"threads_started":1,"wal-append.queue_time_us":915346}
I20260709 12:24:36.729492  1727 rpcz_store.cc:276] 0709 12:24:35.592327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.592594 (+   267us) service_pool.cc:224] Handling call
0709 12:24:36.728659 (+1136065us) spinlock_profiling.cc:243] Waited 969 ms on lock 0x7b54002d05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd2853 00007f16b0da060a 00007f16aa3267d4 00007f16aa326762 00007f16aa326691 00007f16aa32660c 00007f16aa3258c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:36.728772 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2228268928}
I20260709 12:24:36.746279  7968 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:36.747263  7739 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:36.747787  7739 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:36.748637  7923 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:36.749505  7923 raft_consensus.cc:2764] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:24:36.750298  7923 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:36.751405  7960 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:24:36.751643  7989 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:24:36.751854  7885 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.632s	user 0.010s	sys 0.002s
I20260709 12:24:36.752067  7960 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:36.752341  7885 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:36.753201  7960 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:36.726531  7964 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.404s	user 0.000s	sys 0.002s
W20260709 12:24:36.756520  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 306263us
I20260709 12:24:36.757428  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:36.673678  7573 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:36.758980  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 835) took 1820 ms. Trace:
W20260709 12:24:36.726464  7967 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:36.758890  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 820) took 1825 ms. Trace:
I20260709 12:24:36.760794  1805 rpcz_store.cc:276] 0709 12:24:34.933806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.560108 (+626302us) service_pool.cc:224] Handling call
0709 12:24:35.560477 (+   369us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:36.380312 (+819835us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.380354 (+    42us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:36.392168 (+ 11814us) log.cc:844] Serialized 78 byte log entry
0709 12:24:36.651406 (+259238us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:36.651540 (+   134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.651764 (+   224us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.758586 (+106822us) raft_consensus.cc:1685] finished
0709 12:24:36.758612 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.758820 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37063,"raft.run_cpu_time_us":59,"raft.run_wall_time_us":56,"thread_start_us":897752,"threads_started":2,"wal-append.queue_time_us":331108}
I20260709 12:24:36.759621  1799 rpcz_store.cc:276] 0709 12:24:34.938309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.847573 (+909264us) service_pool.cc:224] Handling call
0709 12:24:35.847908 (+   335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.902343 (+ 54435us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:35.947292 (+ 44949us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.758488 (+811196us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.758545 (+    57us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:36.758609 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.758682 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.758911 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":301990,"prepare.queue_time_us":51991,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"thread_start_us":425331,"threads_started":1}
W20260709 12:24:36.764377  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94245.8us
I20260709 12:24:36.762223  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2419) took 1834 ms. Trace:
I20260709 12:24:36.712406  7905 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:36.728945  2418 rpcz_store.cc:276] 0709 12:24:34.644560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.644719 (+   159us) service_pool.cc:224] Handling call
0709 12:24:34.645156 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:34.645758 (+   602us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:35.562873 (+917115us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.634337 (+1071464us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:36.649200 (+ 14863us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.649414 (+   214us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:36.649460 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.649543 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.728190 (+ 78647us) raft_consensus.cc:1685] finished
0709 12:24:36.728214 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.728398 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":937719,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":2515,"raft.queue_time_us":77234,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":286,"thread_start_us":1931441,"threads_started":3,"wal-append.queue_time_us":822108}
I20260709 12:24:36.776868  7812 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:36.767371  1807 rpcz_store.cc:276] 0709 12:24:34.928255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.344653 (+416398us) service_pool.cc:224] Handling call
0709 12:24:35.486608 (+141955us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.486881 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:36.148127 (+661246us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.761907 (+613780us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.761957 (+    50us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:36.762002 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.762057 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.762164 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":244225,"prepare.queue_time_us":319806,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"thread_start_us":940942,"threads_started":2,"wal-append.queue_time_us":463440}
W20260709 12:24:36.783737  7958 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.374s	user 0.003s	sys 0.000s
I20260709 12:24:36.783756  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:36.750696  7923 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:36.874945  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 91311.2us
W20260709 12:24:36.875607  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 222552us
I20260709 12:24:36.771728  7885 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:36.821627  7949 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.500s	user 0.012s	sys 0.000s
I20260709 12:24:36.711630  7892 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:36.893414  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 133211us
I20260709 12:24:36.907341  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:36.890339  7967 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
W20260709 12:24:36.931403  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 253254us
I20260709 12:24:36.925616  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:36.918429  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 146778us
I20260709 12:24:36.916090  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2417) took 1988 ms. Trace:
I20260709 12:24:36.932613  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:36.886708  7816 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:24:36.887828  7968 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:36.886155  7923 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:36.954311  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 234215us
I20260709 12:24:36.883674  7913 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:36.922053  7949 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:36.854703  7945 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.088s
W20260709 12:24:36.979584  1803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.015s	sys 0.033s
W20260709 12:24:36.980190  1803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s	user 0.015s	sys 0.033s
W20260709 12:24:36.984958  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77730.7us
I20260709 12:24:36.986085  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:36.939073  1801 rpcz_store.cc:276] 0709 12:24:34.927399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.927656 (+   257us) service_pool.cc:224] Handling call
0709 12:24:34.928094 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.928440 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:35.774540 (+846100us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.391924 (+617384us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.715466 (+323542us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b180023b900. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:24:36.739009 (+ 23543us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.876643 (+137634us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.876706 (+    63us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:36.876768 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.876861 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.915992 (+ 39131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":491964,"prepare.queue_time_us":900529,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":3272,"raft.queue_time_us":536378,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":8616,"spinlock_wait_cycles":109717120,"thread_start_us":1431264,"threads_started":2,"wal-append.queue_time_us":1011175}
I20260709 12:24:37.001906  1806 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:24:37.012475  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:37.012898  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.013943  1805 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:37.015194  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 125290us
I20260709 12:24:37.018846  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:37.019026  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50337.3us
I20260709 12:24:37.021492  1804 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:37.023030  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.023067  1799 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:37.024137  1806 raft_consensus.cc:2376] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15. Re-sending same reply.
I20260709 12:24:37.029668  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:36.970644  7885 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:37.033798  1772 leader_election.cc:400] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15.
I20260709 12:24:36.989555  7811 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:37.030637  1802 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:37.038188  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.063751  1806 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:37.040541  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.042398  1807 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:37.043223  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:37.067328  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:37.068643  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.068804  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:36.966708  7994 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 12:24:37.067963  1799 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:37.080699  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:37.035109  1772 leader_election.cc:403] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:24:37.085356  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:37.075047  1846 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 2.085s (SENT) [suppressed 36 similar messages]
I20260709 12:24:37.088762  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.059751  7729 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:37.091262  7892 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:37.092757  7998 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:24:37.095212  7980 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:24:37.096089  8012 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.107s
W20260709 12:24:37.101294  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 100240us
W20260709 12:24:37.135277  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 149271us
W20260709 12:24:37.107041  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 61816.2us
I20260709 12:24:37.110855  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:37.103220  1804 raft_consensus.cc:2376] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15. Re-sending same reply.
I20260709 12:24:37.117471  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:37.130930  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:37.129422  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 100128us
I20260709 12:24:37.086083  1802 raft_consensus.cc:2376] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14. Re-sending same reply.
W20260709 12:24:37.105576  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.076676  7597 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:37.106549  7694 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.642s	user 0.007s	sys 0.006s
I20260709 12:24:37.136667  2382 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:37.087225  7994 consensus_queue.cc:1243] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:24:37.151543  1805 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:37.158211  1806 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
W20260709 12:24:37.169960  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 164048us
I20260709 12:24:37.129662  7659 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15.
W20260709 12:24:37.172435  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 404275us
I20260709 12:24:37.134734  7891 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:37.180260  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69553.6us
W20260709 12:24:37.177939  7743 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.740s	user 0.012s	sys 0.004s
W20260709 12:24:37.181406  7611 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.869s	user 0.015s	sys 0.001s
I20260709 12:24:37.177811  7970 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:24:37.224373  8002 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:24:37.162443  7694 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.227525  7994 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:37.235296  7994 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:37.236583  7994 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.238353  7994 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:37.246672  7981 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.306s	user 0.002s	sys 0.000s
I20260709 12:24:37.247483  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 829) took 2310 ms. Trace:
W20260709 12:24:37.248349  7986 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:24:37.248941  7660 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:37.248083  1798 rpcz_store.cc:276] 0709 12:24:34.936533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.664458 (+727925us) service_pool.cc:224] Handling call
0709 12:24:35.664933 (+   475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.575712 (+910779us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.575750 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:36.621896 (+ 46146us) log.cc:844] Serialized 26 byte log entry
0709 12:24:37.129035 (+507139us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:37.129252 (+   217us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.129386 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.247211 (+117825us) raft_consensus.cc:1685] finished
0709 12:24:37.247235 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.247409 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":105399,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":129,"thread_start_us":1365404,"threads_started":2,"wal-append.queue_time_us":601650}
W20260709 12:24:37.251574  7990 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:24:37.222138  7743 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.253724  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2524) took 2322 ms. Trace:
I20260709 12:24:37.221937  7611 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:37.255908  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3216) took 2433 ms. Trace:
I20260709 12:24:37.254285  1880 rpcz_store.cc:276] 0709 12:24:34.931595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.198701 (+1267106us) service_pool.cc:224] Handling call
0709 12:24:36.199157 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.213786 (+ 14629us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:36.232036 (+ 18250us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.583278 (+351242us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:36.645087 (+ 61809us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.645311 (+   224us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:36.645345 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.645420 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.253436 (+608016us) raft_consensus.cc:1685] finished
0709 12:24:37.253466 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.253646 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":668,"prepare.queue_time_us":14750,"prepare.run_cpu_time_us":131,"prepare.run_wall_time_us":1791,"raft.queue_time_us":59538,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":14241,"thread_start_us":283179,"threads_started":1,"wal-append.queue_time_us":804415}
I20260709 12:24:37.256359  1873 rpcz_store.cc:276] 0709 12:24:34.822191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.594445 (+772254us) service_pool.cc:224] Handling call
0709 12:24:35.606953 (+ 12508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.619289 (+ 12336us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.619317 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.653402 (+ 34085us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.368516 (+715114us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:36.368562 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.368695 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.255661 (+886966us) raft_consensus.cc:1685] finished
0709 12:24:37.255702 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.255846 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3709,"raft.queue_time_us":18800,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":5840,"thread_start_us":653963,"threads_started":1,"wal-append.queue_time_us":1526262}
I20260709 12:24:37.257434  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:37.258327  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:37.258888  7985 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.252s	user 0.001s	sys 0.001s
I20260709 12:24:37.259222  1873 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 12: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:37.260478  1873 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:37.261449  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2539) took 2049 ms. Trace:
I20260709 12:24:37.261794  1873 rpcz_store.cc:276] 0709 12:24:35.212317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.259948 (+2047631us) service_pool.cc:224] Handling call
0709 12:24:37.260301 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.261211 (+   910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.261395 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:37.264884  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2357) took 2332 ms. Trace:
I20260709 12:24:37.265336  1800 rpcz_store.cc:276] 0709 12:24:34.932505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.559867 (+627362us) service_pool.cc:224] Handling call
0709 12:24:35.580640 (+ 20773us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.619197 (+ 38557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.619229 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.649323 (+ 30094us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.372260 (+722937us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:36.372305 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.372519 (+   214us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.264642 (+892123us) raft_consensus.cc:1685] finished
0709 12:24:37.264668 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.264820 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":58976,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":49,"thread_start_us":665716,"threads_started":1,"wal-append.queue_time_us":1501758}
I20260709 12:24:37.118688  7968 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.258347  1880 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
W20260709 12:24:37.298282  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.312851  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 850) took 2432 ms. Trace:
I20260709 12:24:37.318635  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:37.313498  1879 rpcz_store.cc:276] 0709 12:24:34.880795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.604236 (+723441us) service_pool.cc:224] Handling call
0709 12:24:35.614203 (+  9967us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.369584 (+755381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.369630 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:36.410888 (+ 41258us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.668983 (+258095us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:36.669020 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.669094 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.312519 (+643425us) raft_consensus.cc:1685] finished
0709 12:24:37.312548 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.312760 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4646,"raft.queue_time_us":17731,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"thread_start_us":835645,"threads_started":2,"wal-append.queue_time_us":538630}
I20260709 12:24:37.324482  1874 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:37.309134  7885 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:37.234664  8010 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:37.299629  7970 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:37.309923  7659 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:37.334364  7218 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.802s	user 0.016s	sys 0.000s
I20260709 12:24:37.353544  1879 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
W20260709 12:24:37.367377  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.007s	sys 0.036s
W20260709 12:24:37.367897  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.007s	sys 0.037s
W20260709 12:24:37.368281  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 658043us
W20260709 12:24:37.370430  1878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.020s	sys 0.021s
W20260709 12:24:37.372165  1878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.021s	sys 0.021s
I20260709 12:24:37.347862  8010 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:37.367245  7966 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.703s	user 0.012s	sys 0.006s
I20260709 12:24:37.377238  8010 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:37.377908  1879 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
I20260709 12:24:37.378319  1879 pending_rounds.cc:85] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 12
I20260709 12:24:37.378582  1879 pending_rounds.cc:107] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 16.13
I20260709 12:24:37.378827  1879 raft_consensus.cc:2889] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:37.380304  1879 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:37.381709  7997 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:24:37.382767  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71136.8us
I20260709 12:24:37.358383  7911 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:37.387547  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69025.5us
I20260709 12:24:37.387995  7911 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.389549  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:37.367687  7970 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.370512  7659 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:37.408777  7740 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.687s	user 0.014s	sys 0.001s
I20260709 12:24:37.409139  7947 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:37.409615  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:37.411194  8005 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:37.427758  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 256378us
W20260709 12:24:37.424362  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 285152us
W20260709 12:24:37.412384  7678 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.834s	user 0.010s	sys 0.005s
I20260709 12:24:37.431924  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:37.387075  7966 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:37.438519  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:37.424800  7947 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:37.427464  8005 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:37.453588  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64113.8us
I20260709 12:24:37.458412  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2525) took 2526 ms. Trace:
I20260709 12:24:37.459000  1875 rpcz_store.cc:276] 0709 12:24:34.931858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.455376 (+1523518us) service_pool.cc:224] Handling call
0709 12:24:36.731103 (+275727us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:36.732209 (+  1106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.458138 (+725929us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002df0d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:37.458335 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":404529024}
I20260709 12:24:37.462841  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 855) took 2135 ms. Trace:
I20260709 12:24:37.463518  1873 rpcz_store.cc:276] 0709 12:24:35.327183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.262524 (+1935341us) service_pool.cc:224] Handling call
0709 12:24:37.262903 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.462098 (+199195us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:37.462150 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:37.462279 (+   129us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:37.462357 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.462475 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.462694 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":188471,"thread_start_us":10024,"threads_started":1}
I20260709 12:24:37.456940  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:37.449538  7358 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:37.444248  8020 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:37.426826  7740 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:37.414433  7996 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.225s
I20260709 12:24:37.426915  8010 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.469935  7911 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:37.479852  7974 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:24:37.497704  7743 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:37.492350  8020 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:37.499990  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.500276  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.500957  8009 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:37.501262  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.501762  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.502183  2416 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:37.480989  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:37.481261  7966 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:37.529808  2412 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
W20260709 12:24:37.473623  7905 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.594s	user 0.012s	sys 0.007s
I20260709 12:24:37.479863  8036 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:24:37.483382  8019 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:24:37.490483  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.452885  8022 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:37.471393  7859 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.754s	user 0.012s	sys 0.004s
I20260709 12:24:37.457006  8005 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:37.465773  7960 raft_consensus.cc:2764] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:24:37.493443  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2537) took 2281 ms. Trace:
I20260709 12:24:37.500401  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.502611  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.421634  7885 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:37.480562  7959 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:37.499141  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 849) took 2619 ms. Trace:
I20260709 12:24:37.547641  1877 rpcz_store.cc:276] 0709 12:24:34.879906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.600197 (+720291us) service_pool.cc:224] Handling call
0709 12:24:35.600644 (+   447us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:35.623000 (+ 22356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.623048 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:35.634254 (+ 11206us) log.cc:844] Serialized 52 byte log entry
0709 12:24:36.367656 (+733402us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:36.444220 (+ 76564us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.444428 (+   208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.444504 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.498894 (+1054390us) raft_consensus.cc:1685] finished
0709 12:24:37.498915 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.499079 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16746,"prepare.queue_time_us":34935,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":44,"raft.queue_time_us":29044,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":2835,"thread_start_us":693165,"threads_started":1,"wal-append.queue_time_us":1674432}
I20260709 12:24:37.498571  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:37.525764  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.534157  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2534) took 2395 ms. Trace:
I20260709 12:24:37.532614  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2536) took 2321 ms. Trace:
I20260709 12:24:37.538424  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.536197  2410 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
W20260709 12:24:37.543999  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87214us
I20260709 12:24:37.544636  1876 rpcz_store.cc:276] 0709 12:24:35.211715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.737736 (+1526021us) service_pool.cc:224] Handling call
0709 12:24:36.768628 (+ 30892us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.778295 (+  9667us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:36.925675 (+147380us) log.cc:844] Serialized 14 byte log entry
0709 12:24:37.492999 (+567324us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:37.493076 (+    77us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:37.493143 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.493223 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.493374 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":364077,"prepare.queue_time_us":125146,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":393132,"thread_start_us":208870,"threads_started":2,"wal-append.queue_time_us":358363}
I20260709 12:24:37.542218  2415 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:37.554489  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:37.550740  1872 rpcz_store.cc:276] 0709 12:24:35.138512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.714288 (+1575776us) service_pool.cc:224] Handling call
0709 12:24:36.764776 (+ 50488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.785563 (+ 20787us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:24:36.887512 (+101949us) log.cc:844] Serialized 14 byte log entry
0709 12:24:37.379428 (+491916us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b180000fbe0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:24:37.533698 (+154270us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:37.533774 (+    76us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:37.533850 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.533939 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.534086 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37955,"prepare.queue_time_us":145310,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":69,"spinlock_wait_cycles":3382528,"thread_start_us":115899,"threads_started":2}
W20260709 12:24:37.555930  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 127094us
I20260709 12:24:37.551244  1881 rpcz_store.cc:276] 0709 12:24:35.211362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.732927 (+1521565us) service_pool.cc:224] Handling call
0709 12:24:36.733255 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.733527 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:36.780965 (+ 47438us) log.cc:844] Serialized 14 byte log entry
0709 12:24:37.067637 (+286672us) log.cc:844] Serialized 14 byte log entry
0709 12:24:37.402750 (+335113us) spinlock_profiling.cc:243] Waited 1.67 ms on lock 0x7b18004988e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:24:37.532176 (+129426us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:37.532244 (+    68us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:37.532312 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.532411 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.532554 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":416619,"prepare.queue_time_us":158770,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":95,"raft.queue_time_us":586214,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":23584,"spinlock_wait_cycles":15945856,"thread_start_us":336328,"threads_started":3,"wal-append.queue_time_us":309415}
I20260709 12:24:37.533447  7911 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:37.559227  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78330us
I20260709 12:24:37.562659  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.563829  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2532) took 2426 ms. Trace:
I20260709 12:24:37.566710  2417 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:37.565896  1878 rpcz_store.cc:276] 0709 12:24:35.137753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.671326 (+1533573us) service_pool.cc:224] Handling call
0709 12:24:36.697156 (+ 25830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.711058 (+ 13902us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:36.762026 (+ 50968us) log.cc:844] Serialized 14 byte log entry
0709 12:24:37.563400 (+801374us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:37.563473 (+    73us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:37.563539 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.563618 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.563754 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96875,"prepare.queue_time_us":65602,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":185738,"thread_start_us":647452,"threads_started":2,"wal-append.queue_time_us":699273}
I20260709 12:24:37.564229  2415 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 11.
I20260709 12:24:37.541421  7743 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:37.568002  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:37.544898  8036 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:37.538589  7905 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:37.581693  7997 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:37.592406  7739 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.844s	user 0.010s	sys 0.006s
I20260709 12:24:37.592645  7997 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.593112  7739 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.500036  8020 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.601377  8036 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.610980  7911 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:37.593531  7905 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:37.590039  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.589083  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:37.545682  7859 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:37.630100  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59703.8us
I20260709 12:24:37.611709  7911 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:37.612481  8010 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:37.644570  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:37.646900  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57900.2us
I20260709 12:24:37.639432  1800 pending_rounds.cc:85] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 10
I20260709 12:24:37.648772  1800 pending_rounds.cc:107] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:24:37.649031  1800 raft_consensus.cc:2889] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:37.649838  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.650986  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.569983  8022 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.656014  7659 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:37.650468  2419 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:37.695171  2410 raft_consensus.cc:2410] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
W20260709 12:24:37.697168  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 219625us
I20260709 12:24:37.698940  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:37.700431  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69340.3us
I20260709 12:24:37.583173  7743 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.706647  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.701313  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:37.711974  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.712750  7573 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:37.713510  8030 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:24:37.713706  8042 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.143s
W20260709 12:24:37.714879  8035 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.117s	user 0.002s	sys 0.000s
I20260709 12:24:37.715642  8045 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:24:37.716101  8025 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:24:37.716256  8045 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.716833  8045 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:37.719128  8043 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.157s
W20260709 12:24:37.720691  8028 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.224s	user 0.004s	sys 0.000s
I20260709 12:24:37.721081  8039 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:37.721854  8039 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:37.722993  8039 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:37.725507  8007 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.205s	user 0.000s	sys 0.002s
I20260709 12:24:37.727892  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 828) took 2791 ms. Trace:
I20260709 12:24:37.728292  1803 rpcz_store.cc:276] 0709 12:24:34.936265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.571037 (+634772us) service_pool.cc:224] Handling call
0709 12:24:35.592213 (+ 21176us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.365874 (+773661us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.365918 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:36.399464 (+ 33546us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.980987 (+581523us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:36.981032 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.981104 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.727668 (+746564us) raft_consensus.cc:1685] finished
0709 12:24:37.727690 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.727832 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":75100,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":100,"thread_start_us":1302601,"threads_started":2,"wal-append.queue_time_us":819563}
I20260709 12:24:37.725767  8051 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:37.730422  8051 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:37.730829  8051 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:37.699745  8014 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:37.726033  7694 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:37.734994  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.739351  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 854) took 2658 ms. Trace:
I20260709 12:24:37.739817  1874 rpcz_store.cc:276] 0709 12:24:35.080718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.622932 (+1542214us) service_pool.cc:224] Handling call
0709 12:24:36.640289 (+ 17357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.325108 (+684819us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.739089 (+413981us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540001e5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:37.739286 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1569404160}
I20260709 12:24:37.740916  8022 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:37.744318  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:37.744740  8051 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:37.747776  8047 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:37.748384  8047 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:37.748800  8047 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:37.749737  8051 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:37.723327  7911 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:37.747640  8009 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:37.750681  8009 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:37.751950  7743 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:37.753509  7739 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:37.706960  8023 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.256s
I20260709 12:24:37.706313  7970 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:37.748836  7859 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:37.757822  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56547.1us
W20260709 12:24:37.758816  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57550.4us
I20260709 12:24:37.760164  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:37.766294  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:37.771088  1879 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:37.777554  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:37.788964  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:37.709666  7905 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:37.791460  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.792683  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.793741  1806 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:37.799816  1874 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:37.803277  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.812552  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.749809  7740 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:37.776515  7993 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.052s	user 0.013s	sys 0.004s
I20260709 12:24:37.816263  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.816852  2413 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:37.818977  1872 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:24:37.821172  2419 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:37.817026  1876 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:24:37.785287  7811 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:37.824878  7811 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:37.827020  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:37.828480  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.785143  8051 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:37.825443  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.783707  8023 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:37.837394  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.841459  2416 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:37.804603  7905 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:37.843458  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:37.845947  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54002.4us
I20260709 12:24:37.846935  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:37.845088  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:37.820385  7816 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.853312  1806 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:37.874056  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.879796  1807 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:37.877358  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.833251  7960 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:37.833279  8014 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:24:37.812294  7998 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.065s
I20260709 12:24:37.884541  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.898753  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.898417  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:37.932849  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.928232  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 857) took 2253 ms. Trace:
I20260709 12:24:37.830276  7993 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:37.932673  1805 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:37.850090  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.896402  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:37.934774  1880 rpcz_store.cc:276] 0709 12:24:35.675336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.297418 (+1622082us) service_pool.cc:224] Handling call
0709 12:24:37.297851 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.927628 (+629777us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:37.927667 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:37.927718 (+    51us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:37.927882 (+   164us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.927983 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.928149 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":153745,"raft.queue_time_us":243590,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":11313,"thread_start_us":472480,"threads_started":1}
W20260709 12:24:37.930544  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 153109us
I20260709 12:24:37.943035  1806 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:37.944774  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:37.899255  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:37.943665  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:37.848228  7905 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:37.839131  8022 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.975885  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:24:37.930325  8007 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:24:37.987756  1806 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:38.001904  1877 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:38.003048  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:37.941231  7573 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.941535  7885 raft_consensus.cc:2764] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:24:38.020756  2410 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:24:37.883353  8010 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.118s
W20260709 12:24:38.011126  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 99871.8us
I20260709 12:24:37.914036  7891 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.089s
I20260709 12:24:37.984097  7811 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:38.024421  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:38.039592  1803 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:38.008333  8012 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.000s	sys 0.009s
I20260709 12:24:38.034989  8020 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.909098  7959 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.103s
I20260709 12:24:37.877228  7816 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.041183  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:38.026291  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:38.076829  1805 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
W20260709 12:24:38.066053  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 120806us
I20260709 12:24:37.848629  8023 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:37.974761  7812 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 12:24:37.898394  8009 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.096829  8009 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:38.097950  8054 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
I20260709 12:24:38.100317  7993 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:38.045364  8012 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.000s	sys 0.010s
W20260709 12:24:38.126185  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 99261.7us
I20260709 12:24:37.985400  8045 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:38.040544  8043 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:38.180065  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 138947us
W20260709 12:24:38.106581  7923 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.094s	user 0.014s	sys 0.007s
I20260709 12:24:38.189324  1880 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:38.001936  8022 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:38.195221  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2450) took 2456 ms. Trace:
I20260709 12:24:38.023564  7573 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:38.191954  1876 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:24:38.203655  1877 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:24:38.205715  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 263481us
I20260709 12:24:38.212424  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:38.213367  1875 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 13: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:38.104682  8037 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.087s
I20260709 12:24:38.195690  1730 rpcz_store.cc:276] 0709 12:24:35.724655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.727077 (+  2422us) service_pool.cc:224] Handling call
0709 12:24:35.727472 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.727845 (+   373us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:36.335703 (+607858us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.869250 (+533547us) log.cc:844] Serialized 14 byte log entry
0709 12:24:37.596734 (+727484us) spinlock_profiling.cc:243] Waited 7.81 ms on lock 0x7b180038f440. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:38.180796 (+584062us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b180038f440. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:24:38.180958 (+   162us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:38.181008 (+    50us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:38.181059 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.181134 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:38.181310 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1313832,"prepare.queue_time_us":49097,"prepare.run_cpu_time_us":148,"prepare.run_wall_time_us":7003,"raft.queue_time_us":304565,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":22117,"spinlock_wait_cycles":54497920,"thread_start_us":1110037,"threads_started":3,"wal-append.queue_time_us":909508}
I20260709 12:24:38.215337  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:38.129410  8058 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:38.092192  8020 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.207340  8052 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:38.260823  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45603.1us
W20260709 12:24:38.257551  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 175555us
I20260709 12:24:38.265334  2418 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:24:38.266105  2418 pending_rounds.cc:85] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 11
I20260709 12:24:38.266456  2418 pending_rounds.cc:107] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:24:38.266829  2418 raft_consensus.cc:2889] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:38.216293  7960 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:38.196962  7996 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:24:38.192695  7659 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.059s
I20260709 12:24:38.201288  7923 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:38.245740  7949 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.208541  8003 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:24:38.216693  7743 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:24:38.313784  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:38.314582  7573 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:38.270589  8043 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:38.310026  7959 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:24:38.279513  7960 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:38.251233  7972 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 12:24:38.282995  7678 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:38.301395  7627 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:38.357929  8060 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
W20260709 12:24:38.358523  8056 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
W20260709 12:24:38.363080  8061 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
W20260709 12:24:38.365408  8071 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:24:38.372666  1807 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:38.374703  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:38.281490  8068 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260709 12:24:38.345580  7678 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:38.424250  1877 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:38.374393  8052 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:38.375020  7358 raft_consensus.cc:2764] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 12:24:38.317158  7949 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:38.421435  8062 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:24:38.472621  8027 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 1.095s	user 0.002s	sys 0.012s
W20260709 12:24:38.473922  8047 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.724s	user 0.008s	sys 0.006s
W20260709 12:24:38.477921  8066 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.227s	user 0.001s	sys 0.000s
I20260709 12:24:38.297694  8023 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:38.497165  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 737410us
I20260709 12:24:38.418169  8043 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 7, Time since last communication: 0.047s
W20260709 12:24:38.510668  8064 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.178s	user 0.000s	sys 0.000s
I20260709 12:24:38.512387  8004 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:38.513090  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:38.513988  1876 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:38.425501  8036 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.073s
I20260709 12:24:38.520751  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.521696  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.521325  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:38.567613  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.423900  7627 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.571282  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.519752  1876 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:38.573292  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:38.497000  8020 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:38.515910  8069 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
W20260709 12:24:38.516527  8005 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.945s	user 0.014s	sys 0.004s
I20260709 12:24:38.474406  8047 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:38.498363  7945 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 6, Time since last communication: 0.088s
I20260709 12:24:38.593160  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:38.704643  1732 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:38.588958  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.726541  1729 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:38.675786  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.592724  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.727443  1724 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.596206  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.679458  1727 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:38.704483  1728 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:38.585132  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:38.708165  1803 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.644s	user 0.011s	sys 0.000s
W20260709 12:24:38.708527  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:38.613967  8063 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:24:38.615609  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 110554us
W20260709 12:24:38.627555  8057 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.400s	user 0.002s	sys 0.000s
I20260709 12:24:38.737691  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:38.600208  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 1.2307e+06us
I20260709 12:24:38.516963  7823 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:38.739189  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.739238  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:38.740170  1798 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.013s	sys 0.024s
W20260709 12:24:38.740532  1798 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.013s	sys 0.024s
I20260709 12:24:38.740435  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.589334  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:38.573081  7597 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.568071  7987 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:38.595733  8051 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.763s	user 0.011s	sys 0.003s
W20260709 12:24:38.592360  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 94025.2us
I20260709 12:24:38.735728  1803 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:38.524027  8004 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:38.651036  7923 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:38.607131  7991 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:38.748409  7991 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:38.573381  7611 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.749718  7991 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:38.741227  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 845) took 1128 ms. Trace:
I20260709 12:24:38.736922  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.614070  8070 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.759425  1732 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:38.627235  8052 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:38.745890  1731 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:38.745429  1723 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:38.746656  1878 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:38.675483  8076 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.738696  1726 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:24:38.718194  8003 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.760564  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:38.750746  7611 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.734357  7729 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:38.779209  8069 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:24:38.761456  7597 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.818212  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.825312  8052 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:38.830291  1728 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:38.760804  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.725502  8005 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.839639  7729 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:38.609356  7995 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.213s
I20260709 12:24:38.854295  8005 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:38.760910  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.855253  1727 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:38.821249  8004 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.855526  1879 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:38.771940  7987 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.736866  7920 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.850272  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:38.741693  7823 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.863911  8012 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:38.864589  7987 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:38.804678  8014 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.822s	user 0.010s	sys 0.002s
I20260709 12:24:38.832314  1724 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:38.866981  8014 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:24:38.607563  7678 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.844633  2417 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:38.868870  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.869756  8047 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:38.845280  2385 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.003s	sys 0.029s
I20260709 12:24:38.870560  1724 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:38.871196  2385 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.003s	sys 0.030s
I20260709 12:24:38.837921  1732 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:38.771544  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:38.862221  7920 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.828606  7949 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:38.865037  8012 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.765866  8051 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:38.854065  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.855477  1729 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
W20260709 12:24:38.872913  1696 connection.cc:256] RPC call timeout handler was delayed by -2.5408249080001042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1513s for a response
I20260709 12:24:38.855712  1876 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260709 12:24:38.850196  8085 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
W20260709 12:24:38.873549  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.868932  2419 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:24:38.910919  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.905635  7996 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.916103  7996 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.882802  7987 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.918051  7987 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:38.921285  1731 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.878527  1727 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:38.804723  7218 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:38.759164  1798 rpcz_store.cc:276] 0709 12:24:37.612532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.613239 (+   707us) service_pool.cc:224] Handling call
0709 12:24:37.616527 (+  3288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.616825 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:37.845948 (+229123us) log.cc:844] Serialized 14 byte log entry
0709 12:24:38.044584 (+198636us) log.cc:844] Serialized 14 byte log entry
0709 12:24:38.063771 (+ 19187us) log.cc:844] Serialized 14 byte log entry
0709 12:24:38.740879 (+677108us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:38.740939 (+    60us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:38.740995 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.741064 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:38.741177 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":197927,"prepare.queue_time_us":57099,"prepare.run_cpu_time_us":1270,"prepare.run_wall_time_us":8116,"raft.queue_time_us":404877,"raft.run_cpu_time_us":384,"raft.run_wall_time_us":27616,"spinlock_wait_cycles":48849664,"thread_start_us":911968,"threads_started":3,"wal-append.queue_time_us":415147}
W20260709 12:24:38.940516  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 169103us
I20260709 12:24:38.892979  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:38.942565  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:38.943184  1732 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:38.921307  7949 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.841768  7991 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:38.846880  7627 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:38.957151  7816 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:38.959470  7218 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:38.960834  7218 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:38.899485  8012 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:38.926959  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:38.944693  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:38.943516  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:38.948169  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:38.977066  1732 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:38.865089  8045 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:38.880178  8039 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:38.944298  7885 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:38.952085  7949 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.923614  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:38.900570  7995 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:38.986464  1727 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:38.829207  8070 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:38.993997  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:38.995000  2412 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:38.991032  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 131304us
I20260709 12:24:38.987891  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:38.998857  1798 raft_consensus.cc:2410] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 12:24:38.902032  1696 connection.cc:256] RPC call timeout handler was delayed by -2.48862490800002s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
W20260709 12:24:39.000337  1696 connection.cc:256] RPC call timeout handler was delayed by -2.3887249079998583s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16820000000000002s for a response
W20260709 12:24:39.006985  1696 connection.cc:256] RPC call timeout handler was delayed by -1.9405249080000431s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21800000000000003s for a response
W20260709 12:24:39.007975  1696 connection.cc:256] RPC call timeout handler was delayed by -1.9054249079999863s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2219s for a response
W20260709 12:24:39.008829  1696 connection.cc:256] RPC call timeout handler was delayed by -1.5229249079998226s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2644s for a response
I20260709 12:24:39.005898  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:39.023687  1732 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:39.025424  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.027015  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:39.027762  1732 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.837378  8076 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.028596  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.029177  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:39.029619  1727 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:39.029863  1729 raft_consensus.cc:2368] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15.
I20260709 12:24:39.032466  1724 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:39.035322  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.039508  1724 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:39.010053  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:38.842664  8003 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.103613  1798 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:39.104171  1877 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:24:39.112079  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:39.114055  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:39.159441  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:39.114998  2418 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:38.992978  7995 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:39.174983  8040 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.212s
W20260709 12:24:39.176708  7905 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.157s	user 0.011s	sys 0.008s
I20260709 12:24:38.995334  7987 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.182512  1798 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.9-19.9]
I20260709 12:24:39.044545  7811 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:39.195508  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.219672  1801 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:39.226156  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.227660  1732 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:38.994755  8039 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.247227  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:39.247610  1801 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:39.102222  7627 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.170s
I20260709 12:24:39.251991  7920 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.252238  8024 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:24:39.253235  8006 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:39.253684  8018 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:39.253574  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:39.290846  1805 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.255381  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:39.256783  1804 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:39.252082  8095 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.106s	user 0.001s	sys 0.015s
W20260709 12:24:39.262279  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:39.262213  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 256399us
I20260709 12:24:39.211645  7923 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.221154  7987 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:39.278918  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:39.288439  1731 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:39.295019  1770 leader_election.cc:400] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15.
I20260709 12:24:39.315323  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:39.197376  7905 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.297513  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:39.317498  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:39.321357  1805 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:39.296700  7811 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:39.323132  7923 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.290575  8018 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:39.296365  8023 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:39.353662  1730 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.511s	user 0.006s	sys 0.007s
I20260709 12:24:39.328567  7920 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:39.360558  1874 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:39.359625  1730 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:39.329917  7996 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:39.364688  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49510.9us
W20260709 12:24:39.378314  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:39.380362  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:39.380411  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 370460us
I20260709 12:24:39.348693  7816 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:39.384720  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.344499  8067 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:39.347780  7949 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.339321  7611 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.387684  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:39.351194  7597 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.360803  8018 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:39.380338  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:39.401407  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.401830  7816 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:39.354820  8088 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:39.403275  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:39.362195  7920 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.341035  7811 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.407873  7811 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.363811  7911 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:39.411586  7911 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:39.411640  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.412303  7911 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:39.413533  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.415334  7920 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:39.416332  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:39.415531  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.417590  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.402565  7816 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:39.419199  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.419672  7816 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:39.403375  8088 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.423523  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.425863  1873 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:24:39.426442  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.418649  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:39.431957  1872 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.433351  1873 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.9-19.9]
I20260709 12:24:39.438654  1873 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:39.442488  8076 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.443321  8076 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.443897  8076 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.445739  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 861) took 1672 ms. Trace:
I20260709 12:24:39.447216  1873 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:39.448607  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:39.446257  1879 rpcz_store.cc:276] 0709 12:24:37.773088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.781085 (+  7997us) service_pool.cc:224] Handling call
0709 12:24:37.781546 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:38.860345 (+1078799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.445360 (+585015us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54002df5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:39.445647 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2469421440}
I20260709 12:24:39.450446  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:39.451515  1725 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:39.451622  1879 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:39.293884  8006 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.458518  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.460637  1725 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:39.291857  7995 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.395551  8024 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:39.455493  8103 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 12:24:39.464421  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 278392us
I20260709 12:24:39.386291  8023 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.468667  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.484606  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:39.486343  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:39.489689  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 102127us
I20260709 12:24:39.491644  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:39.483516  8040 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.494078  8040 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.496717  8040 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.500093  7923 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:39.500523  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.500638  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 844) took 1888 ms. Trace:
I20260709 12:24:39.501375  1725 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:39.469136  8024 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.501188  1800 rpcz_store.cc:276] 0709 12:24:37.612177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.612323 (+   146us) service_pool.cc:224] Handling call
0709 12:24:37.639203 (+ 26880us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:37.649998 (+ 10795us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:37.650030 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:37.650552 (+   522us) log.cc:844] Serialized 52 byte log entry
0709 12:24:39.177932 (+1527380us) spinlock_profiling.cc:243] Waited 4.65 ms on lock 0x7b1800499000. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:39.361557 (+183625us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:24:39.361614 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.361699 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:39.500215 (+138516us) raft_consensus.cc:1685] finished
0709 12:24:39.500237 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:39.500562 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46236,"raft.queue_time_us":14080,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":5620,"spinlock_wait_cycles":10696960,"thread_start_us":183391,"threads_started":1,"wal-append.queue_time_us":1833452}
I20260709 12:24:39.474141  7997 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.502434  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.473838  8039 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:39.531883  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:39.505643  8024 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.508044  7959 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.507735  8108 raft_consensus.cc:2764] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:39.516779  8052 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.394452  8067 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.523382  7923 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:39.533349  7892 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:39.506235  7997 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.533937  7913 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:39.582963  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55597.3us
I20260709 12:24:39.584133  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:39.585773  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3241) took 1269 ms. Trace:
I20260709 12:24:39.552392  8088 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.587664  1881 rpcz_store.cc:276] 0709 12:24:38.316505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.332918 (+ 16413us) service_pool.cc:224] Handling call
0709 12:24:38.350505 (+ 17587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:38.402714 (+ 52209us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:38.880965 (+478251us) log.cc:844] Serialized 14 byte log entry
0709 12:24:39.266932 (+385967us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.267628 (+   696us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.267809 (+   181us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:39.267849 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.267939 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:39.585442 (+317503us) raft_consensus.cc:1685] finished
0709 12:24:39.585475 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:39.585683 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120526,"prepare.queue_time_us":192044,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":96719,"raft.run_cpu_time_us":383,"raft.run_wall_time_us":16997,"thread_start_us":705846,"threads_started":2,"wal-append.queue_time_us":351376}
W20260709 12:24:39.603250  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 198517us
I20260709 12:24:39.606107  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.607167  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.608790  1881 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:39.551622  7959 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.588882  8076 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.611363  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.591840  7905 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.614446  1879 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:39.578294  8052 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:39.609637  7991 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.655s	user 0.018s	sys 0.000s
I20260709 12:24:39.582144  7923 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.614667  7996 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.588313  7913 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:39.620522  1872 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:39.636617  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.630012  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.657210  1879 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:39.643445  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.608258  7892 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:39.657467  1873 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 12.
I20260709 12:24:39.668522  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:39.635545  7823 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.772s	user 0.014s	sys 0.000s
I20260709 12:24:39.671983  7823 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.673349  2410 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:39.674062  8110 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:39.629338  7959 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.666736  8006 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:39.640709  7911 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.676767  8052 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:39.655447  7920 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.679441  7920 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.679735  7997 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.679926  7947 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.680472  7959 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:39.681068  7947 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.674386  7823 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.676884  7911 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.620050  8076 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.670843  8112 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:39.699160  7678 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.699609  7913 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:24:39.699180  8076 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.702608  8076 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.693491  7892 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:39.707144  8111 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:24:39.708990  8074 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.359s	user 0.004s	sys 0.000s
I20260709 12:24:39.709343  7892 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.695262  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2470) took 1167 ms. Trace:
I20260709 12:24:39.699427  8112 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.670061  8068 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.016s
I20260709 12:24:39.724402  8112 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:39.712680  8012 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.727s	user 0.013s	sys 0.000s
I20260709 12:24:39.725705  8012 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.656473  7991 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:24:39.615967  8088 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.634057  8018 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.694989  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:39.743570  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48659.5us
I20260709 12:24:39.744799  7991 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:39.744975  7358 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.745913  7358 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.631737  8067 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.637518  7923 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:39.714668  1725 rpcz_store.cc:276] 0709 12:24:38.528155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.523822 (+995667us) service_pool.cc:224] Handling call
0709 12:24:39.545982 (+ 22160us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:39.553310 (+  7328us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.553343 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.560982 (+  7639us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.669106 (+108124us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:39.669168 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.669250 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:39.694998 (+ 25748us) raft_consensus.cc:1685] finished
0709 12:24:39.695023 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:39.695191 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":129,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":22973,"thread_start_us":59424,"threads_started":1,"wal-append.queue_time_us":99952}
I20260709 12:24:39.638006  7996 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.751197  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.712190  8023 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:39.760176  1725 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:39.758535  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:39.768767  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:39.768944  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:39.769739  1806 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:39.770263  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:39.771603  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:39.771927  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:39.772719  1806 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:39.773669  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:39.774343  1723 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:39.775275  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.777618  1806 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:39.778208  1877 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:39.780896  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:39.781845  1803 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:39.783867  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.788280  1723 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:39.790656  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:39.791388  1805 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:39.784920  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:39.793488  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:39.794217  1805 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 18 because replica has last-logged OpId of term: 17 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:24:39.795640  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:39.797487  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:39.798525  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.810314  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.724761  7816 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.814196  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:39.816529  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.817559  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:39.755827  7923 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:39.821640  1800 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:39.818344  1873 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:39.758039  7996 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.818434  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:39.825997  1804 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:39.780892  7947 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:39.837791  1802 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:39.833431  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:39.755069  7597 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:24:39.780486  8029 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:39.840478  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.790499  8067 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:39.848066  1879 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:24:39.756891  8108 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:24:39.859649  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:39.813769  7991 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.757659  7218 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:24:39.758006  7611 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.058s
I20260709 12:24:39.795953  8023 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:24:39.847348  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 162553us
I20260709 12:24:39.808708  8006 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:39.835968  8045 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:39.822324  8051 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:39.751072  8088 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.845557  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3240) took 1535 ms. Trace:
I20260709 12:24:39.861411  7947 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:39.869738  7816 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.821120  8047 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:39.824692  8050 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:39.835234  7358 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.857524  8029 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:39.859414  7987 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:39.884859  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:39.888852  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70533.3us
I20260709 12:24:39.849464  7923 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.891732  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:39.950331  1879 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:24:39.891516  1878 rpcz_store.cc:276] 0709 12:24:38.310189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.326111 (+ 15922us) service_pool.cc:224] Handling call
0709 12:24:38.333090 (+  6979us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:38.837151 (+504061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.845134 (+1007983us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:39.845450 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":945630592}
I20260709 12:24:39.885905  8045 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:39.879312  7991 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:39.863824  8067 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:39.955803  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71040.4us
I20260709 12:24:39.887204  8012 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:39.961884  1803 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:39.956945  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:39.987411  7947 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:39.947657  8029 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:40.001531  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44686us
I20260709 12:24:39.961001  7816 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:39.969321  8045 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:39.973896  7611 consensus_queue.cc:1243] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 11.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:39.999770  7218 consensus_queue.cc:1243] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:24:39.955756  8050 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.044852  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.045579  1879 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:40.019081  8006 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:40.050530  8006 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.014008  7911 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:24:40.022745  7816 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.055774  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.030357  8129 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:40.059377  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.064837  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:40.054909  7816 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:40.030853  7597 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.031s
I20260709 12:24:40.059429  7218 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:40.085695  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.086531  1879 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:40.089774  8023 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:40.090492  8051 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:40.091897  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.093420  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.072734  8003 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:40.095189  7597 consensus_queue.cc:1243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:24:40.097009  1879 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:40.100608  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.058997  8128 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 12:24:40.104837  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.060884  8126 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:24:40.105870  1879 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:40.105362  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:40.107465  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:40.097623  8108 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:40.109395  1878 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:40.101224  8003 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:40.098626  7923 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:40.096920  7611 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:24:40.111518  8067 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:40.112241  8100 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.848s	user 0.000s	sys 0.003s
I20260709 12:24:40.111979  8023 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.112771  8050 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:40.116484  8051 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:40.106254  8124 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:24:40.118953  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:40.107883  8131 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:40.122973  7218 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:40.127209  7991 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:40.125074  1879 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:24:40.135419  7611 consensus_queue.cc:1243] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:24:40.136875  7970 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:40.137445  7970 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:40.139772  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.141148  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:40.141644  1724 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:40.122496  8003 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:40.147790  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1270) took 2309 ms. Trace:
I20260709 12:24:40.147886  8003 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.149874  1878 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:40.151033  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:40.152060  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.153021  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.153595  1878 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:40.160392  7920 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:40.160928  7920 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:40.161962  7920 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.171181  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.171319  2416 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:40.172219  7920 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:40.172395  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.173163  1727 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:40.174057  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.167164  1875 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:40.165717  8022 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:40.166148  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.176244  2418 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:40.167132  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.174634  1727 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:40.167850  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:40.168628  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.170377  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:40.180716  8029 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.181494  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:40.182415  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:40.183229  8076 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:40.183727  8076 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.184105  8076 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:40.184252  8067 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:40.184800  8067 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.185268  8067 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:40.184732  7960 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.186136  7960 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:40.185956  7816 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.165246  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.148242  2419 rpcz_store.cc:276] 0709 12:24:37.838410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.838604 (+   194us) service_pool.cc:224] Handling call
0709 12:24:37.850735 (+ 12131us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:38.910087 (+1059352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.147499 (+1237412us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003fc854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:40.147695 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2340748032}
I20260709 12:24:40.187197  8022 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:40.136937  8088 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:40.189118  8116 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.205464  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:40.189738  1772 connection.cc:256] RPC call timeout handler was delayed by -1.7560364620003384s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15380000000000002s for a response
I20260709 12:24:40.190974  7911 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.194468  8045 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.198177  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1122) took 1355 ms. Trace:
I20260709 12:24:40.200340  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.214056  2411 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:40.213330  2412 rpcz_store.cc:276] 0709 12:24:38.842839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.842996 (+   157us) service_pool.cc:224] Handling call
0709 12:24:38.843279 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.000577 (+157298us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.197912 (+1197335us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400525154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:40.198106 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":341868416}
I20260709 12:24:40.201043  2417 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:40.202736  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:40.203039  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 866) took 1672 ms (client timeout 2220 ms). Trace:
I20260709 12:24:40.218166  1727 raft_consensus.cc:2376] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18. Re-sending same reply.
W20260709 12:24:40.218103  1731 rpcz_store.cc:269] 0709 12:24:38.530933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.852024 (+321091us) service_pool.cc:224] Handling call
0709 12:24:40.202889 (+1350865us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b54000d1bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:40.202990 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":48640896}
I20260709 12:24:40.219240  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:40.209139  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 90260.2us
I20260709 12:24:40.219908  1727 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:40.209580  1724 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:40.219343  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:40.207413  1772 connection.cc:256] RPC call timeout handler was delayed by -1.1402426990002823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1547s for a response
I20260709 12:24:40.223381  1805 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:40.224916  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:40.226894  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:40.138432  7970 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.236619  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:40.237574  1727 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:40.237845  1806 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:40.237895  7970 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:40.239288  1806 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:40.237018  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:40.240473  1806 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:40.241364  1805 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:40.241587  1806 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:40.242704  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.201403  8067 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.244092  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.211510  8137 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:40.245591  8067 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:40.234367  8022 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:40.246447  7988 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:24:40.247292  7988 consensus_queue.cc:1243] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:24:40.247113  8022 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.249713  7660 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:40.250115  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 863) took 2117 ms. Trace:
I20260709 12:24:40.250288  1727 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:40.250733  1874 rpcz_store.cc:276] 0709 12:24:38.133268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.133507 (+   239us) service_pool.cc:224] Handling call
0709 12:24:38.133934 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.363350 (+1229416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.249819 (+886469us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400025b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:40.250033 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2820286336}
I20260709 12:24:40.251434  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.251915  8088 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.254302  8088 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.255039  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:40.255566  8096 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.154s	user 0.002s	sys 0.024s
I20260709 12:24:40.256440  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:40.258443  8088 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:40.259378  7729 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:40.259361  1727 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:40.260569  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:40.262488  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:40.267544  8112 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:40.268661  8022 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:40.268528  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:40.267568  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:40.269523  1799 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:40.233982  8111 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:24:40.271193  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.264562  8018 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:40.252018  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:40.242583  1806 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:40.344509  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.351058  1799 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:40.487172  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:40.489095  1874 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:24:40.489238  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.281841  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.489670  1798 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:40.490311  1724 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
W20260709 12:24:40.460918  2414 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1163) took 2942 ms (client timeout 3152 ms). Trace:
I20260709 12:24:40.488524  1731 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
W20260709 12:24:40.491328  2414 rpcz_store.cc:269] 0709 12:24:37.499708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.501515 (+  1807us) service_pool.cc:224] Handling call
0709 12:24:40.441999 (+2940484us) spinlock_profiling.cc:243] Waited 709 ms on lock 0x7b54007ce554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:40.442259 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1629888256}
I20260709 12:24:40.360383  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.489748  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.250497  7660 consensus_queue.cc:1243] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:40.499123  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 862) took 2368 ms. Trace:
I20260709 12:24:40.500455  1880 rpcz_store.cc:276] 0709 12:24:38.130847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.131104 (+   257us) service_pool.cc:224] Handling call
0709 12:24:38.189107 (+ 58003us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:38.194845 (+  5738us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.498735 (+2303890us) spinlock_profiling.cc:243] Waited 57.5 ms on lock 0x7b540021c5d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:40.499039 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":132316416}
I20260709 12:24:40.501665  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:40.502396  1799 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:40.504806  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 144578us
I20260709 12:24:40.506160  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.502000  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:40.508358  1880 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:40.509992  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:40.514568  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:40.514947  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.515565  1731 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:40.515962  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.517563  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.518385  1724 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:40.519286  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.315620  8137 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.343675  7611 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.206326  8116 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.342362  8042 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:40.523376  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:40.345723  7891 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:40.339702  8029 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:40.520505  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:40.412279  7959 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:40.269073  8022 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:40.575156  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.575919  1731 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:40.507632  8112 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:40.531481  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2471) took 1996 ms. Trace:
W20260709 12:24:40.525009  8047 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.011s	sys 0.013s
I20260709 12:24:40.508756  7947 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:40.539930  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:40.543576  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:40.604987  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:40.605021  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 65154.8us
I20260709 12:24:40.570623  1805 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:40.527480  7660 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:40.519366  8067 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.080s
I20260709 12:24:40.531726  8018 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:40.215178  7911 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.595062  1724 rpcz_store.cc:276] 0709 12:24:38.528491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.524406 (+1995915us) service_pool.cc:224] Handling call
0709 12:24:40.524710 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.524892 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.524914 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.524954 (+    40us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:40.524997 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.525047 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.525154 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.537792  7891 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:40.594390  7988 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:40.615435  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:40.533492  8042 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:40.589738  1799 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:40.531416  7970 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:40.598475  8047 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.012s	sys 0.014s
I20260709 12:24:40.633812  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:40.634994  1799 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:40.531976  7611 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.639098  1800 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: [14.8-17.10]
I20260709 12:24:40.639254  1806 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
W20260709 12:24:40.646874  2382 connection.cc:256] RPC call timeout handler was delayed by -2.0954270629999883s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1507s for a response
I20260709 12:24:40.652416  1874 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:40.638872  8047 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:40.667059  7891 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.670149  1874 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:40.669751  7891 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:40.670545  1799 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
W20260709 12:24:40.671617  8140 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:24:40.671986  1801 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:40.669456  8047 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:40.672112  8104 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.408s	user 0.002s	sys 0.000s
W20260709 12:24:40.655558  2382 connection.cc:256] RPC call timeout handler was delayed by -1.8965270630001214s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
W20260709 12:24:40.676302  2382 connection.cc:256] RPC call timeout handler was delayed by -1.7147270629998275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.193s for a response
W20260709 12:24:40.676817  2382 connection.cc:256] RPC call timeout handler was delayed by -1.4537270629998602s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
W20260709 12:24:40.677270  2382 connection.cc:256] RPC call timeout handler was delayed by -1.382736991999991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1503s for a response
W20260709 12:24:40.677843  2382 connection.cc:256] RPC call timeout handler was delayed by -1.1954270629998973s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25070000000000003s for a response
W20260709 12:24:40.678298  2382 connection.cc:256] RPC call timeout handler was delayed by -1.0047369919998346s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19230000000000003s for a response
W20260709 12:24:40.678534  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.680480  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.680928  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.683559  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:40.684836  1726 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 15. Preceding OpId from leader: term: 17 index: 14. (term mismatch)
I20260709 12:24:40.685292  1726 pending_rounds.cc:85] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
I20260709 12:24:40.685551  1726 pending_rounds.cc:107] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 14.14
I20260709 12:24:40.685808  1726 raft_consensus.cc:2889] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:40.686131  1726 pending_rounds.cc:107] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 16.15
I20260709 12:24:40.686368  1726 raft_consensus.cc:2889] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:40.689531  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:40.690239  1731 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:40.619021  8131 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:40.699980  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.700886  2418 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:40.681282  2412 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:40.703320  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.704041  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.704456  1874 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:40.594805  7959 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.716686  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.624315  7988 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.709590  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:40.712512  1880 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:24:40.712954  1727 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:40.716780  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:40.718606  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.704136  2414 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:40.636559  7970 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.731354  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:40.731840  7970 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:40.732878  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.717903  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 868) took 2186 ms. Trace:
I20260709 12:24:40.733913  1726 rpcz_store.cc:276] 0709 12:24:38.531643 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.652999 (+2121356us) service_pool.cc:224] Handling call
0709 12:24:40.653368 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.717555 (+ 64187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.717813 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.723347  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.717696  2416 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 12.
I20260709 12:24:40.735821  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:40.736548  8076 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.552s	user 0.009s	sys 0.002s
I20260709 12:24:40.719522  8131 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:40.737135  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.622478  8018 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.739032  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:40.739290  7988 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:40.740207  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.717788  8022 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.718403  8045 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:40.753616  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.751241  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:40.649999  7994 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.655249  7816 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:40.761005  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.755137  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.762444  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:40.765939  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 76081.9us
I20260709 12:24:40.766769  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:40.729501  1880 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:40.781641  2415 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:40.787632  1880 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:40.790422  1880 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
W20260709 12:24:40.790961  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 293486us
I20260709 12:24:40.800707  1880 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:40.804956  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:40.754073  7920 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.581s	user 0.008s	sys 0.003s
I20260709 12:24:40.751960  8018 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:40.760499  7959 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:40.824927  1731 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:40.838879  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.837816  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 871) took 1985 ms. Trace:
W20260709 12:24:40.840132  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47709.6us
I20260709 12:24:40.840618  1876 rpcz_store.cc:276] 0709 12:24:38.852468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.922443 (+ 69975us) service_pool.cc:224] Handling call
0709 12:24:38.922852 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:38.945207 (+ 22355us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:38.945252 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:38.945841 (+   589us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.162361 (+216520us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:39.162434 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.162530 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.671437 (+1508907us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:40.821822 (+150385us) raft_consensus.cc:1685] finished
0709 12:24:40.821855 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.837705 (+ 15850us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21796,"raft.queue_time_us":22779,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":3550,"spinlock_wait_cycles":36787968,"thread_start_us":198613,"threads_started":1,"wal-append.queue_time_us":300197}
I20260709 12:24:40.841976  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:40.810822  8014 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:40.844211  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.737061  8076 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.784986  7994 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.856205  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:40.857228  1731 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:40.853978  1798 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:40.842844  7959 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.016s
W20260709 12:24:40.860275  8086 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 2.093s	user 0.004s	sys 0.007s
W20260709 12:24:40.860972  8143 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
W20260709 12:24:40.862141  8099 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.598s	user 0.002s	sys 0.000s
W20260709 12:24:40.862716  8144 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
I20260709 12:24:40.863102  7611 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:40.863755  8146 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 12:24:40.868897  7994 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:40.872301  7960 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.686s	user 0.011s	sys 0.000s
I20260709 12:24:40.873152  8120 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.873097  7960 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:40.859967  7959 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:40.875370  2418 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:40.873864  8120 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:40.737813  8042 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.885272  7959 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:40.885591  7597 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:40.738022  8131 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.527494  8137 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.881610  8112 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:40.891412  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.892190  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.915222  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.892437  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:40.893119  7988 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.895741  7911 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:40.933534  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:40.934460  2412 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:40.896106  7995 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:40.878494  8014 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:40.899652  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.915920  1732 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:40.937067  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.882759  7960 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:40.945214  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45684us
I20260709 12:24:40.946590  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.947885  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.949355  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:40.951063  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.907466  7597 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:40.955552  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 13 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.910864  8042 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:40.959280  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.971958  8076 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:40.972918  8116 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:40.981395  7627 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:40.980769  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.978645  1697 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:40.995663  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 228942us
I20260709 12:24:40.944782  7960 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.983794  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:40.996559  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:40.953953  7911 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:40.986761  2415 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:40.960798  8112 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:40.974365  1729 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:40.984808  8161 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:40.999488  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 870) took 2462 ms. Trace:
I20260709 12:24:40.957147  7959 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:41.000072  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2473) took 2471 ms. Trace:
I20260709 12:24:40.999855  1729 rpcz_store.cc:276] 0709 12:24:38.537544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.938517 (+2400973us) service_pool.cc:224] Handling call
0709 12:24:40.974077 (+ 35560us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.999185 (+ 25108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.999420 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.997051  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:40.990406  8101 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.749s	user 0.005s	sys 0.001s
I20260709 12:24:41.003389  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:41.000499  1724 rpcz_store.cc:276] 0709 12:24:38.529160 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.619669 (+2090509us) service_pool.cc:224] Handling call
0709 12:24:40.620037 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.620296 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:40.768992 (+148696us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.995565 (+226573us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.996504 (+   939us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.999760 (+  3256us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:40.999830 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.999902 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.000014 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":717,"prepare.queue_time_us":226946,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":2084,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":1053,"thread_start_us":294733,"threads_started":2}
I20260709 12:24:41.004613  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:40.997869  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:41.001945  7729 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:41.006973  7947 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:41.022158  8137 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:41.024505  7988 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:40.973312  7597 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:41.026634  7597 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:41.031213  8014 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:40.995702  7627 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:41.036870  7627 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:41.003433  8112 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.046897  8116 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:41.105865  8112 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:41.105944  8116 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:41.117609  8116 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:41.152055  8116 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:41.153903  8116 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:24:41.155066  7970 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:41.157056  8112 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:41.157518  7970 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:42.066774  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08002cd960 after lost signal to thread 561
W20260709 12:24:42.067487  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08001f29c0 after lost signal to thread 564
I20260709 12:24:42.235952  1724 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:42.237075  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 875) took 3614 ms. Trace:
I20260709 12:24:42.237485  1724 rpcz_store.cc:276] 0709 12:24:38.623721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.005110 (+2381389us) service_pool.cc:224] Handling call
0709 12:24:42.235618 (+1230508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.236672 (+  1054us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.236985 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.257901  1724 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:42.258656  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 876) took 3607 ms. Trace:
I20260709 12:24:42.258811  8112 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:42.259110  1724 rpcz_store.cc:276] 0709 12:24:38.651592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.238622 (+3587030us) service_pool.cc:224] Handling call
0709 12:24:42.257697 (+ 19075us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.258359 (+   662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.258569 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.260782  1724 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:24:42.261351  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 877) took 3600 ms. Trace:
I20260709 12:24:42.261656  1724 rpcz_store.cc:276] 0709 12:24:38.660588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.260080 (+3599492us) service_pool.cc:224] Handling call
0709 12:24:42.260434 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.261116 (+   682us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.261297 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.262907  1724 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 12. Current term is 13. Ops: [12.11-12.11]
I20260709 12:24:42.263445  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 878) took 3539 ms. Trace:
W20260709 12:24:42.263616  7959 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.258s	user 0.015s	sys 0.007s
I20260709 12:24:42.263743  1724 rpcz_store.cc:276] 0709 12:24:38.723930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.262350 (+3538420us) service_pool.cc:224] Handling call
0709 12:24:42.262668 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.263253 (+   585us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.263408 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.264389  7959 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:42.273411  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2476) took 3451 ms. Trace:
I20260709 12:24:42.273837  1724 rpcz_store.cc:276] 0709 12:24:38.822305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.264824 (+3442519us) service_pool.cc:224] Handling call
0709 12:24:42.265110 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.265305 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:42.271469 (+  6164us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.272549 (+  1080us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.272946 (+   397us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:42.272997 (+    51us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:42.273073 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.273167 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.273324 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85,"prepare.queue_time_us":6256,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":100,"thread_start_us":6354,"threads_started":2,"wal-append.queue_time_us":1012}
I20260709 12:24:42.282251  8123 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:42.282963  8123 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:42.283883  8123 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:42.288156  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2478) took 3465 ms. Trace:
I20260709 12:24:42.288681  1724 rpcz_store.cc:276] 0709 12:24:38.823278 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.275363 (+3452085us) service_pool.cc:224] Handling call
0709 12:24:42.275705 (+   342us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:42.276087 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:42.276437 (+   350us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.283668 (+  7231us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.284205 (+   537us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.285816 (+  1611us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:42.286424 (+   608us) log.cc:844] Serialized 52 byte log entry
0709 12:24:42.286601 (+   177us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:42.286637 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.286705 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.287875 (+  1170us) raft_consensus.cc:1685] finished
0709 12:24:42.287899 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.288058 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":153,"prepare.queue_time_us":6764,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":114,"raft.queue_time_us":18453,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":160,"thread_start_us":7596,"threads_started":2,"wal-append.queue_time_us":6632}
I20260709 12:24:42.296326  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2479) took 3472 ms. Trace:
I20260709 12:24:42.296754  1724 rpcz_store.cc:276] 0709 12:24:38.823692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.290634 (+3466942us) service_pool.cc:224] Handling call
0709 12:24:42.291090 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.291377 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.291418 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:42.291976 (+   558us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.293374 (+  1398us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:42.293412 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.293485 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.296062 (+  2577us) raft_consensus.cc:1685] finished
0709 12:24:42.296084 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.296263 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1008,"threads_started":1,"wal-append.queue_time_us":1409}
I20260709 12:24:42.298652  1724 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:42.299810  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3360) took 3386 ms. Trace:
I20260709 12:24:42.300153  1724 rpcz_store.cc:276] 0709 12:24:38.913467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.298116 (+3384649us) service_pool.cc:224] Handling call
0709 12:24:42.298460 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.299085 (+   625us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.299750 (+   665us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.301853  1724 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [13.12-16.14]
I20260709 12:24:42.302520  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3362) took 3388 ms. Trace:
I20260709 12:24:42.302861  1724 rpcz_store.cc:276] 0709 12:24:38.914023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.301105 (+3387082us) service_pool.cc:224] Handling call
0709 12:24:42.301539 (+   434us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:42.302226 (+   687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.302467 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.304126  1724 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:42.304720  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3365) took 3364 ms. Trace:
I20260709 12:24:42.305356  1724 rpcz_store.cc:276] 0709 12:24:38.946331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.303622 (+3357291us) service_pool.cc:224] Handling call
0709 12:24:42.303930 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.304436 (+   506us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.304663 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:42.310062  8067 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.695s	user 0.011s	sys 0.016s
W20260709 12:24:42.310549  8067 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.696s	user 0.011s	sys 0.016s
W20260709 12:24:42.315927  1696 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.628s	user 0.059s	sys 0.128s
W20260709 12:24:42.316645  1696 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.629s	user 0.060s	sys 0.128s
W20260709 12:24:42.317003  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 1.70791e+06us
W20260709 12:24:42.321336  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1608283900004608s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1544s for a response
I20260709 12:24:42.323257  8174 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.323849  8174 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.325263  8175 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.325469  7987 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:42.325954  8174 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.325932  8175 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.327373  8174 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.327840  8174 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:42.329229  7920 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:42.329696  8174 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:42.329710  8175 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.330139  7920 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.330559  8175 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.330525  8174 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.331390  8175 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.331423  8174 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:42.333201  7920 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.333608  8174 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:42.333611  8175 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.334491  8175 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:42.327154  7987 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:42.337071  2419 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.366s	user 0.014s	sys 0.037s
W20260709 12:24:42.337574  2419 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.366s	user 0.015s	sys 0.037s
I20260709 12:24:42.338184  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1131) took 1424 ms. Trace:
I20260709 12:24:42.338477  2419 rpcz_store.cc:276] 0709 12:24:40.913861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.917116 (+  3255us) service_pool.cc:224] Handling call
0709 12:24:40.917445 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:40.917815 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.917843 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:40.955158 (+ 37315us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.337936 (+1382778us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:42.337965 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.338012 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.338037 (+    25us) raft_consensus.cc:1685] finished
0709 12:24:42.338044 (+     7us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.338140 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1639,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":22359,"thread_start_us":1366679,"threads_started":1,"wal-append.queue_time_us":179457,"wal-append.run_cpu_time_us":2543,"wal-append.run_wall_time_us":1002551}
I20260709 12:24:42.339476  7920 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:42.340729  7920 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:42.341099  7920 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:42.336707  7987 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:42.342024  7920 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:42.343216  7987 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:42.343941  8175 consensus_queue.cc:1175] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 18 status { last_received { term: 18 index: 10 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: [17.10-17.10]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:42.345258  7920 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:42.375434  7816 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:42.376070  7816 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:42.377151  7816 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.378688  7816 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:42.382697  7920 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.383433  7920 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:42.384754  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2453) took 2588 ms (client timeout 1544 ms). Trace:
W20260709 12:24:42.385160  1803 rpcz_store.cc:269] 0709 12:24:39.796205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.797194 (+   989us) service_pool.cc:224] Handling call
0709 12:24:42.384458 (+2587264us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b540026f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:42.384618 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":375699328}
I20260709 12:24:42.386590  7920 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:42.400864  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.552s	user 0.044s	sys 0.048s
W20260709 12:24:42.401350  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.552s	user 0.044s	sys 0.048s
W20260709 12:24:42.401906  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 1.55758e+06us
W20260709 12:24:42.418936  2412 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.476s	user 0.012s	sys 0.021s
I20260709 12:24:42.419521  2412 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
W20260709 12:24:42.419718  1844 connection.cc:256] RPC call timeout handler was delayed by -1.1051637069999742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2s for a response
I20260709 12:24:42.420192  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1133) took 1487 ms. Trace:
W20260709 12:24:42.420429  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.420619  2412 rpcz_store.cc:276] 0709 12:24:40.932617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.933184 (+   567us) service_pool.cc:224] Handling call
0709 12:24:42.420113 (+1486929us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.425148  7920 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.425989  7920 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:42.429874  7920 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:42.443442  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3243) took 3330 ms. Trace:
I20260709 12:24:42.444267  1877 rpcz_store.cc:276] 0709 12:24:39.113295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.165995 (+ 52700us) service_pool.cc:224] Handling call
0709 12:24:39.191811 (+ 25816us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.795318 (+603507us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.441694 (+2646376us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b540021acd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:42.443322 (+  1628us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1347929472}
W20260709 12:24:42.446727  8153 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.492s	user 0.009s	sys 0.009s
I20260709 12:24:42.448964  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3257) took 2388 ms. Trace:
I20260709 12:24:42.449450  1872 rpcz_store.cc:276] 0709 12:24:40.060235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.245047 (+184812us) service_pool.cc:224] Handling call
0709 12:24:40.245388 (+   341us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:40.314988 (+ 69600us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.315037 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:40.510832 (+195795us) log.cc:844] Serialized 52 byte log entry
0709 12:24:40.600745 (+ 89913us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:40.614021 (+ 13276us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.618887 (+  4866us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.618980 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.445995 (+1827015us) spinlock_profiling.cc:243] Waited 5.79 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:42.448486 (+  2491us) raft_consensus.cc:1685] finished
0709 12:24:42.448641 (+   155us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.448872 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72064,"prepare.queue_time_us":72988,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":7453,"raft.queue_time_us":99723,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":9564,"spinlock_wait_cycles":13321600,"thread_start_us":58410,"threads_started":2,"wal-append.queue_time_us":372143}
I20260709 12:24:42.457463  7826 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:24:42.457650  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.4597e+06us
I20260709 12:24:42.458220  7826 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:42.459342  7826 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:42.469485  1727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3382) took 2293 ms (client timeout 2101 ms). Trace:
W20260709 12:24:42.473093  1696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.544s (SENT) [suppressed 16 similar messages]
I20260709 12:24:42.478454  8070 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.484113  8070 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:42.484702  8070 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:42.487430  1727 rpcz_store.cc:269] 0709 12:24:40.175901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.314100 (+138199us) service_pool.cc:224] Handling call
0709 12:24:42.469126 (+2155026us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b54002d19d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:42.469316 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":512485120}
I20260709 12:24:42.488973  1727 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:24:42.489588  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3367) took 3494 ms. Trace:
I20260709 12:24:42.489877  1727 rpcz_store.cc:276] 0709 12:24:38.995693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.488424 (+3492731us) service_pool.cc:224] Handling call
0709 12:24:42.488751 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.489331 (+   580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.489527 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.495419  8070 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.504175  1727 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 14 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:42.504944  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2481) took 3323 ms. Trace:
I20260709 12:24:42.505247  1727 rpcz_store.cc:276] 0709 12:24:39.181632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.490569 (+3308937us) service_pool.cc:224] Handling call
0709 12:24:42.503987 (+ 13418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.504661 (+   674us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.504878 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.508478  8021 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:42.509096  8021 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:42.510867  1727 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:42.514400  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2482) took 3253 ms. Trace:
I20260709 12:24:42.514874  1727 rpcz_store.cc:276] 0709 12:24:39.260611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.510355 (+3249744us) service_pool.cc:224] Handling call
0709 12:24:42.510671 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.511349 (+   678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.514320 (+  2971us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.516362  1727 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 10. (term mismatch)
I20260709 12:24:42.516659  1727 pending_rounds.cc:85] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 9
I20260709 12:24:42.516863  1727 pending_rounds.cc:107] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:24:42.517081  1727 raft_consensus.cc:2889] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:42.517724  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2483) took 3112 ms. Trace:
I20260709 12:24:42.518007  1727 rpcz_store.cc:276] 0709 12:24:39.405261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.515692 (+3110431us) service_pool.cc:224] Handling call
0709 12:24:42.515977 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.517487 (+  1510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.517671 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:42.520776  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.514452  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1121) took 3673 ms. Trace:
I20260709 12:24:42.527101  2410 rpcz_store.cc:276] 0709 12:24:38.840632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:38.840809 (+   177us) service_pool.cc:224] Handling call
0709 12:24:38.841117 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:39.673765 (+832648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.514046 (+2840281us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x7b5400488d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:42.514335 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1913532160}
W20260709 12:24:42.529316  2418 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1185) took 1844 ms (client timeout 1834 ms). Trace:
W20260709 12:24:42.529928  2418 rpcz_store.cc:269] 0709 12:24:40.684943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.754868 (+ 69925us) service_pool.cc:224] Handling call
0709 12:24:42.529011 (+1774143us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54004745d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:42.529205 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":258603392}
W20260709 12:24:42.534371  8042 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.554s	user 0.013s	sys 0.009s
I20260709 12:24:42.510159  8021 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:42.535791  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2541) took 3334 ms (client timeout 1540 ms). Trace:
W20260709 12:24:42.543619  1875 rpcz_store.cc:269] 0709 12:24:39.201416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.396301 (+194885us) service_pool.cc:224] Handling call
0709 12:24:42.535446 (+3139145us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b5400117454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:42.535639 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1754085248}
I20260709 12:24:42.545248  8021 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:42.546479  1728 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:24:42.549808  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2490) took 2595 ms. Trace:
I20260709 12:24:42.550238  1728 rpcz_store.cc:276] 0709 12:24:39.954519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.545613 (+2591094us) service_pool.cc:224] Handling call
0709 12:24:42.546118 (+   505us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.549523 (+  3405us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.549745 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.551777  1728 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: [19.9-19.9]
I20260709 12:24:42.552399  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3376) took 2382 ms. Trace:
I20260709 12:24:42.552716  1728 rpcz_store.cc:276] 0709 12:24:40.169935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.551080 (+2381145us) service_pool.cc:224] Handling call
0709 12:24:42.551481 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.552157 (+   676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.552350 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:42.555752  1846 connection.cc:256] RPC call timeout handler was delayed by -1.7315953470001659s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260709 12:24:42.556576  1846 connection.cc:256] RPC call timeout handler was delayed by -1.5821953470003791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
W20260709 12:24:42.563051  1846 connection.cc:256] RPC call timeout handler was delayed by -1.5578953470003398s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
W20260709 12:24:42.563532  1846 connection.cc:256] RPC call timeout handler was delayed by -1.4858953470002234s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
W20260709 12:24:42.563962  1846 connection.cc:256] RPC call timeout handler was delayed by -1.4786953470002118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
W20260709 12:24:42.564298  1846 connection.cc:256] RPC call timeout handler was delayed by -1.3931953470000735s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20260709 12:24:42.564603  1846 connection.cc:256] RPC call timeout handler was delayed by -1.3823953470000561s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
W20260709 12:24:42.564942  1846 connection.cc:256] RPC call timeout handler was delayed by -1.3733953470000415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19510000000000002s for a response
W20260709 12:24:42.565251  1846 connection.cc:256] RPC call timeout handler was delayed by -1.3184953469999527s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20120000000000002s for a response
W20260709 12:24:42.565569  1846 connection.cc:256] RPC call timeout handler was delayed by -1.303195346999928s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2029s for a response
W20260709 12:24:42.565914  1846 connection.cc:256] RPC call timeout handler was delayed by -1.2564609440000822s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17700000000000002s for a response
W20260709 12:24:42.566229  1846 connection.cc:256] RPC call timeout handler was delayed by -1.2527953470003013s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20850000000000002s for a response
W20260709 12:24:42.567734  1846 connection.cc:256] RPC call timeout handler was delayed by -1.1879953470001965s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2157s for a response
W20260709 12:24:42.568112  1846 connection.cc:256] RPC call timeout handler was delayed by -1.1443062660000578s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2s for a response
I20260709 12:24:42.536384  1727 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:42.568845  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2485) took 3060 ms. Trace:
I20260709 12:24:42.569226  1727 rpcz_store.cc:276] 0709 12:24:39.508311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.518690 (+3010379us) service_pool.cc:224] Handling call
0709 12:24:42.534900 (+ 16210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.568568 (+ 33668us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.568789 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:42.569957  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2387) took 2315 ms (client timeout 1514 ms). Trace:
W20260709 12:24:42.570535  1805 rpcz_store.cc:269] 0709 12:24:40.254258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.256195 (+  1937us) service_pool.cc:224] Handling call
0709 12:24:42.569680 (+2313485us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:42.569843 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":585981184}
I20260709 12:24:42.570533  1727 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
W20260709 12:24:42.581051  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 107982us
I20260709 12:24:42.582132  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3378) took 2411 ms. Trace:
I20260709 12:24:42.576381  1723 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
W20260709 12:24:42.587754  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.58429e+06us
I20260709 12:24:42.587947  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3384) took 2407 ms. Trace:
I20260709 12:24:42.588279  1723 rpcz_store.cc:276] 0709 12:24:40.180269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.575583 (+2395314us) service_pool.cc:224] Handling call
0709 12:24:42.576070 (+   487us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.587644 (+ 11574us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.587885 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.589003  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:42.587101  1727 rpcz_store.cc:276] 0709 12:24:40.170659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.569984 (+2399325us) service_pool.cc:224] Handling call
0709 12:24:42.570302 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.581749 (+ 11447us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.582047 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.590248  1723 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:42.584698  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:42.587141  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 93389.9us
I20260709 12:24:42.591586  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 880) took 1952 ms. Trace:
I20260709 12:24:42.591996  1723 rpcz_store.cc:276] 0709 12:24:40.639397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.589782 (+1950385us) service_pool.cc:224] Handling call
0709 12:24:42.590082 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.591217 (+  1135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.591467 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.592741  8045 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:42.589259  8161 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:42.593281  8045 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:42.593729  8045 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:42.594103  8161 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.595187  8120 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:42.595674  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:42.595261  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:42.596410  2412 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:42.597055  2410 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:42.595209  2418 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 12:24:42.598206  2418 pending_rounds.cc:85] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 14
I20260709 12:24:42.598002  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:42.598477  2418 pending_rounds.cc:107] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 12:24:42.598734  2418 raft_consensus.cc:2889] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:42.595609  1723 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:42.601287  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 882) took 1961 ms. Trace:
I20260709 12:24:42.601605  1723 rpcz_store.cc:276] 0709 12:24:40.639954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.592801 (+1952847us) service_pool.cc:224] Handling call
0709 12:24:42.595315 (+  2514us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.601058 (+  5743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.601237 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.602744  1723 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:42.603307  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 884) took 1962 ms. Trace:
I20260709 12:24:42.603595  1723 rpcz_store.cc:276] 0709 12:24:40.640691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.602295 (+1961604us) service_pool.cc:224] Handling call
0709 12:24:42.602591 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.603091 (+   500us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.603254 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.605523  1723 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:42.605852  1723 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:42.608713  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:42.606810  1723 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.612979  1723 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:42.613255  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:42.607834  8042 raft_consensus.cc:2764] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:24:42.612428  8070 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:42.621238  8070 raft_consensus.cc:2764] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260709 12:24:42.622892  7597 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.596s	user 0.015s	sys 0.009s
I20260709 12:24:42.615732  1846 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:42.631282  7995 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.688s	user 0.014s	sys 0.009s
I20260709 12:24:42.632056  1725 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:42.632607  8003 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.633066  8003 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:42.634214  1723 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:24:42.631418  8157 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.787s	user 0.010s	sys 0.000s
I20260709 12:24:42.636417  8000 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:42.636858  8000 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:42.637740  8000 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.9, Last appended by leader: 9, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:42.639617  8000 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:42.641218  1874 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:24:42.638449  8045 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.647101  8042 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:42.649072  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:42.650249  8045 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.649614  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 888) took 1918 ms. Trace:
I20260709 12:24:42.651278  1723 rpcz_store.cc:276] 0709 12:24:40.731209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.605124 (+1873915us) service_pool.cc:224] Handling call
0709 12:24:42.605419 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.634597 (+ 29178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.634840 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:42.653123  8014 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:42.655912  8021 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:42.657290  8012 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:24:42.657636  8122 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Append to log took a long time: real 2.613s	user 0.002s	sys 0.011s
I20260709 12:24:42.659039  8042 raft_consensus.cc:2764] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:24:42.657675  8012 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:42.660588  8012 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.662928  7694 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:42.663244  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:41.005905  7994 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:42.664008  7994 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:42.664449  7994 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:42.664798  7694 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:42.667516  7826 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:42.667896  7826 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:42.668804  7826 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.669716  7826 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:42.650496  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.674683  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08003236a0 after lost signal to thread 1664
I20260709 12:24:42.675427  8045 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:42.678422  8042 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.679041  8042 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.681742  8042 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.686852  1805 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:24:42.687368  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:42.688109  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:42.688987  1802 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:42.689152  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:42.690034  1881 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:42.683384  8045 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:42.683748  7597 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:42.690599  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:42.691300  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:42.684458  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:42.691627  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:42.692237  1878 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:42.685824  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:42.692167  1875 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:42.691311  1880 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:24:42.693614  8133 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:24:42.689865  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:42.694334  1723 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:42.694707  1803 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:42.693167  1872 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:24:42.690613  8045 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.693511  8014 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
I20260709 12:24:42.696496  8014 consensus_queue.cc:1243] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:24:42.695468  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:42.698064  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:42.682366  7995 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:42.705835  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:42.698643  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:42.688148  1877 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:42.699908  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:42.710358  1804 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:42.709121  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:42.700342  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:42.701192  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:42.701581  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:42.712114  1805 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:42.702030  8045 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.714139  8045 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:42.714504  8045 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:42.714910  8045 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:42.712141  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:42.715917  1804 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:42.701710  8014 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:42.716681  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:42.717211  8014 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:42.712431  1798 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:42.717933  1799 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:42.719334  1730 raft_consensus.cc:606] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:42.719875  1730 raft_consensus.cc:614] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:42.718715  8014 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.711972  1802 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 21 because replica has last-logged OpId of term: 19 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 12:24:42.723836  1803 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:42.724355  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:42.710952  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:42.726478  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40200 (request call id 50) took 3958 ms. Trace:
I20260709 12:24:42.727046  1730 rpcz_store.cc:276] 0709 12:24:38.766456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.515668 (+749212us) service_pool.cc:224] Handling call
0709 12:24:42.722746 (+3207078us) spinlock_profiling.cc:243] Waited 3.18 s on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd2853 00007f16b0da060a 00007f16aa3267d4 00007f16aa326762 00007f16aa326691 00007f16aa32660c 00007f16aa3258c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:42.724560 (+  1814us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7319593728}
I20260709 12:24:42.728677  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:42.730116  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:42.730772  1730 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:42.726465  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:42.733497  1803 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:42.737401  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:42.739030  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:42.715823  7627 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:42.740185  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:42.737946  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:42.741780  1723 raft_consensus.cc:2376] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20. Re-sending same reply.
I20260709 12:24:42.742769  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:42.742897  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:42.743901  1723 raft_consensus.cc:2368] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14.
I20260709 12:24:42.744520  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:42.744822  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:42.742643  1730 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:42.746146  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:42.746440  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:42.746879  1723 raft_consensus.cc:2376] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13. Re-sending same reply.
I20260709 12:24:42.747648  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:42.747882  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:42.748607  1723 raft_consensus.cc:2376] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18. Re-sending same reply.
I20260709 12:24:42.749094  1730 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:24:42.749895  1723 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: [17.14-17.14]
I20260709 12:24:42.755983  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:42.756592  8184 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:42.740619  8133 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
I20260709 12:24:42.722968  8045 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.761166  7826 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:42.709710  7994 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.762434  8045 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.754395  7995 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:42.753734  7947 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:42.758594  8184 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.008s
I20260709 12:24:42.767634  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:42.785646  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 79996.9us
I20260709 12:24:42.786968  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:42.770798  1880 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:42.788371  8184 consensus_queue.cc:1243] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 18.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:42.788913  8184 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:42.789268  8184 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:42.790096  7994 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:42.785626  7995 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.784477  1877 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:42.805574  8191 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:24:42.808068  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55384.9us
I20260709 12:24:42.790071  8184 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.813376  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:42.836541  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:42.785305  8045 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:42.839437  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:42.841723  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:42.843201  8192 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:42.845269  7611 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:42.845811  7611 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.846201  7611 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:42.802554  7995 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:42.846940  8000 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:42.847157  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 873) took 4250 ms. Trace:
I20260709 12:24:42.800753  8133 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:42.831346  7988 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:42.847622  1729 rpcz_store.cc:276] 0709 12:24:38.597281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.001206 (+2403925us) service_pool.cc:224] Handling call
0709 12:24:42.678595 (+1677389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.679018 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:42.679305 (+   287us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.706854 (+ 27549us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:42.707973 (+  1119us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.708166 (+   193us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:42.708205 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.708279 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.846877 (+138598us) raft_consensus.cc:1685] finished
0709 12:24:42.846905 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.847080 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":2693,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":3609,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":3754,"thread_start_us":27084,"threads_started":1,"wal-append.queue_time_us":76241}
I20260709 12:24:42.849766  8191 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.850104  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:42.850443  8191 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.851362  7920 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:42.851884  7920 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.852300  7920 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:42.853495  8191 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:42.856478  7987 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:42.854601  8045 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:42.875137  8188 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.006s
I20260709 12:24:42.875759  1802 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 12:24:42.880124  1802 pending_rounds.cc:85] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 8
I20260709 12:24:42.856854  8174 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.017s
I20260709 12:24:42.880376  1802 pending_rounds.cc:107] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:24:42.880708  1802 raft_consensus.cc:2889] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:42.874117  8045 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:42.887645  7947 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:42.864385  7988 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:42.888692  7988 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:42.889635  7988 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:42.863994  8184 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:42.874428  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:42.895532  1723 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:42.861187  7994 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:24:42.860296  8192 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.017s
I20260709 12:24:42.900161  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:42.901798  1844 leader_election.cc:400] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14.
I20260709 12:24:42.904002  1844 leader_election.cc:403] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:24:42.911424  8189 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:42.912207  8189 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:42.913445  8189 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:42.857831  8191 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:24:42.922186  1730 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:42.897104  7947 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:24:42.898115  8188 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:24:42.921969  8185 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 12:24:42.942207  1723 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:42.952284  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:42.956801  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 869) took 4424 ms. Trace:
I20260709 12:24:42.957224  1731 rpcz_store.cc:276] 0709 12:24:38.531968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.857950 (+2325982us) service_pool.cc:224] Handling call
0709 12:24:40.894606 (+ 36656us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.954394 (+ 59788us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:40.954432 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.954497 (+    65us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:40.954556 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.956480 (+2001924us) spinlock_profiling.cc:243] Waited 59.4 ms on lock 0x7b54003cdcd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:42.956553 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.956738 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1278,"raft.queue_time_us":541,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":19203,"spinlock_wait_cycles":136698752}
I20260709 12:24:42.960950  8051 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:42.989081  7987 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:42.991914  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.994416  2383 connection.cc:256] RPC call timeout handler was delayed by -1.4412275829999999s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
W20260709 12:24:42.996155  2383 connection.cc:256] RPC call timeout handler was delayed by -1.4016275829999358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15940000000000001s for a response
W20260709 12:24:42.996430  2383 connection.cc:256] RPC call timeout handler was delayed by -1.2513275830001476s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1761s for a response
W20260709 12:24:42.996663  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1856275830000413s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
W20260709 12:24:42.996898  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1856275830000413s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1834s for a response
W20260709 12:24:42.997136  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1604275830000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
W20260709 12:24:42.997376  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1604275830000006s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
I20260709 12:24:42.999677  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:43.000469  2418 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:43.003921  2410 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:43.006065  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.008282  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.012454  2410 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:43.017323  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:43.018899  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.020354  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:43.021620  8199 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:43.022490  8199 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:43.019511  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:43.023795  8199 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:43.029137  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 127470us
I20260709 12:24:42.995579  7994 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.034685  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:43.040176  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:43.013603  7920 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:43.055513  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.040714  8191 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:43.103777  8204 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:43.040170  7995 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:43.112287  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.124519  2410 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:43.126029  2418 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:43.126600  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:43.132447  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.133209  1724 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:43.035437  8042 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 12:24:43.037082  7994 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.098218  7987 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.086892  8207 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:43.136973  8051 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14.
W20260709 12:24:43.138518  7729 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 2.131s	user 0.013s	sys 0.012s
W20260709 12:24:43.150890  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63627us
I20260709 12:24:43.152762  8216 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:43.152838  8191 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:43.153661  8216 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:43.154863  8191 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.156432  8216 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.183641  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:43.185329  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.165973  7729 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:43.203559  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.120s (SENT)
W20260709 12:24:43.213200  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.148897  7987 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.217425  8207 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:43.200229  8206 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:43.228736  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 102273us
I20260709 12:24:43.244745  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:43.248342  8206 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:43.263581  8206 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:43.263877  8207 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.273353  7611 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:43.278527  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2564) took 3050 ms. Trace:
I20260709 12:24:43.286476  1874 rpcz_store.cc:276] 0709 12:24:40.228499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.705418 (+476919us) service_pool.cc:224] Handling call
0709 12:24:40.705704 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.641734 (+1936030us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.278147 (+636413us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54001f7f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:43.278434 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4450035456}
I20260709 12:24:43.287606  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.288940  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.290259  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.281967  8022 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:43.284258  7987 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:43.284701  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.285483  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:43.311968  2415 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:43.315407  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.316143  2413 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:24:43.329061  1726 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 2.486s	user 0.012s	sys 0.008s
I20260709 12:24:43.329651  1726 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:43.330274  1729 raft_consensus.cc:606] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:24:43.330755  1729 raft_consensus.cc:614] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:43.347116  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 116518us
I20260709 12:24:43.320614  7987 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.301138  7627 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.352165  8216 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:43.313189  7994 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:43.365576  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 872) took 4780 ms. Trace:
I20260709 12:24:43.366226  1732 rpcz_store.cc:276] 0709 12:24:38.585285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.938410 (+2353125us) service_pool.cc:224] Handling call
0709 12:24:40.938983 (+   573us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:40.951983 (+ 13000us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:40.952023 (+    40us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:40.955539 (+  3516us) log.cc:844] Serialized 52 byte log entry
0709 12:24:43.293346 (+2337807us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b1800289300. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:43.354544 (+ 61198us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:43.354605 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.354891 (+   286us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:43.365097 (+ 10206us) raft_consensus.cc:1685] finished
0709 12:24:43.365129 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:43.365480 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11773,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":1811,"spinlock_wait_cycles":31637248,"thread_start_us":58634,"threads_started":1,"wal-append.queue_time_us":2403902}
I20260709 12:24:43.369613  1732 pending_rounds.cc:85] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
I20260709 12:24:43.369972  1732 pending_rounds.cc:107] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:24:43.370326  1732 raft_consensus.cc:2889] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:43.351099  7611 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:43.348235  7987 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.354811  7660 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:43.392050  7987 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:43.353154  7627 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:43.396613  8188 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:43.397104  8188 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:43.398036  8188 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:43.355005  8222 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.399183  8188 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:43.373466  8216 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:43.407790  8220 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
W20260709 12:24:43.417804  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.417634  1723 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
W20260709 12:24:43.407408  8045 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.520s	user 0.007s	sys 0.005s
I20260709 12:24:43.397109  7627 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:43.421774  8219 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:24:43.430658  8216 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.410342  8222 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.432266  8045 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.448644  8045 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:43.446436  8222 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:43.454583  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:43.455464  2413 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.467938  8003 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:43.473372  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.477486  8195 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.537s	user 0.001s	sys 0.000s
I20260709 12:24:43.487892  2413 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:43.489562  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:43.491218  8042 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:43.491823  8042 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:43.492770  8042 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:43.497495  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.506428  8197 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.565s	user 0.002s	sys 0.000s
I20260709 12:24:43.516883  7826 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:43.518337  7826 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:43.519085  8199 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:43.520393  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 852) took 3297 ms. Trace:
I20260709 12:24:43.520907  1801 rpcz_store.cc:276] 0709 12:24:40.223082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.223188 (+   106us) service_pool.cc:224] Handling call
0709 12:24:40.223494 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.672451 (+448957us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.520082 (+2847631us) spinlock_profiling.cc:243] Waited 432 ms on lock 0x7b5400067054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:43.520318 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":993912192}
I20260709 12:24:40.997846  7960 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.520457  7826 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:43.526566  7826 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.528290  7826 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:43.529764  7891 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:24:43.530921  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 1.12812e+06us
W20260709 12:24:43.531783  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 942752us
I20260709 12:24:43.532649  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:43.534348  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:43.536304  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:43.537561  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.538148  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.548663  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:43.549477  1880 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:43.551075  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:43.546837  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:43.553550  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:43.554358  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:43.559919  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:43.560583  1872 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.535408  7891 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:43.563321  7891 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:43.572285  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:43.531075  7913 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:24:43.582063  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:43.540110  8161 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:43.544741  8042 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:43.545538  8188 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:43.592034  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44206.7us
I20260709 12:24:43.561537  8111 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.553388  8116 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.583009  1881 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
W20260709 12:24:43.603613  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71065.2us
I20260709 12:24:43.604024  1873 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:43.604736  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:24:43.605106  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68442.4us
W20260709 12:24:43.603665  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 67431.6us
I20260709 12:24:43.606724  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:43.611574  1872 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:43.616632  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:43.617534  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86581.3us
W20260709 12:24:43.619213  1845 connection.cc:256] RPC call timeout handler was delayed by -4.0039912980000736s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19430000000000003s for a response
W20260709 12:24:43.619640  1845 connection.cc:256] RPC call timeout handler was delayed by -3.9904912980000518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1958s for a response
W20260709 12:24:43.620638  1845 connection.cc:256] RPC call timeout handler was delayed by -3.8647433070004809s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
W20260709 12:24:43.621294  1845 connection.cc:256] RPC call timeout handler was delayed by -3.2781178230002297s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
W20260709 12:24:43.621670  1845 connection.cc:256] RPC call timeout handler was delayed by -3.0441178230003061s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1774s for a response
W20260709 12:24:43.622015  1845 connection.cc:256] RPC call timeout handler was delayed by -2.8938178230000631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19410000000000002s for a response
W20260709 12:24:43.622372  1845 connection.cc:256] RPC call timeout handler was delayed by -2.77141782300032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2077s for a response
W20260709 12:24:43.622736  1845 connection.cc:256] RPC call timeout handler was delayed by -2.4171324400003869s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30870000000000003s for a response
I20260709 12:24:43.595680  8022 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:43.634907  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:43.635967  1805 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:43.613418  8120 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:43.635967  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.623932  7358 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:43.637012  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.645990  1805 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:43.647131  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:43.647680  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.648288  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.637655  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.649042  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.649765  1804 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:43.650187  1798 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:24:43.607273  8111 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:43.628433  7816 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:43.653988  7816 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:43.654456  7816 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:43.655625  8232 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:24:43.633947  8022 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:43.660014  1801 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:43.667536  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49377.1us
W20260709 12:24:43.627518  1845 connection.cc:256] RPC call timeout handler was delayed by -1.7872912980001274s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.4406s for a response
I20260709 12:24:43.669992  1872 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:43.672024  1872 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:43.672098  1805 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:43.601413  8188 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:43.675657  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:43.676323  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.679029  1804 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:43.682976  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:43.682073  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:43.613143  8116 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:43.704016  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.705404  7970 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:43.707185  8076 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:43.707554  8076 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:43.708410  8076 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:43.709429  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:43.709631  8076 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:43.710628  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:43.711659  8112 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:43.712129  8112 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:43.713037  8112 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:43.715570  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:43.628530  8165 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:43.717619  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:43.659853  7358 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:43.719202  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:43.719154  1881 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:43.629154  8161 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.072s
I20260709 12:24:43.734339  1872 raft_consensus.cc:1217] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Deduplicated request from leader. Original: 18.14->[19.15-19.15]   Dedup: 19.15->[]
I20260709 12:24:43.743296  1872 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:24:43.731844  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.645876  7811 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:43.747320  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:43.736552  1876 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:43.749897  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:43.750502  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 82238.6us
I20260709 12:24:43.751307  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.764343  1804 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:43.699715  8067 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:43.718880  8088 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:43.733325  8120 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:43.771597  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.720000  8165 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:43.771600  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:43.736384  7358 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.773260  1878 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:24:43.746749  7891 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:43.777182  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59667.7us
W20260709 12:24:43.778998  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.779789  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.780139  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.781689  2412 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:24:43.783674  1804 raft_consensus.cc:2368] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17.
I20260709 12:24:43.717052  8199 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 12:24:43.784922  1798 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:43.778074  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:43.788107  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.768595  8067 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.791282  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.796849  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.797130  2411 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:24:43.800179  2413 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: [18.15-18.15]
I20260709 12:24:43.774081  7959 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:43.808710  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.777854  8161 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:43.774319  8165 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:43.811136  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:43.823597  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:43.812332  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:43.794811  8067 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:43.859750  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.811959  8120 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.823233  8161 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:43.810906  7811 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:43.866282  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.874675  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:43.967519  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:44.026685  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.038146  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:43.858956  7959 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.171s
I20260709 12:24:44.040323  8255 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.087848  8255 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.090253  8255 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:44.099268  8240 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.426s	user 0.000s	sys 0.001s
I20260709 12:24:44.100802  8188 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.039667  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:43.874176  7811 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.871964  8120 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.107364  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:44.072687  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2474) took 1305 ms. Trace:
I20260709 12:24:44.073055  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.113746  1805 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:44.115142  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:44.116130  1804 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:44.113936  1803 rpcz_store.cc:276] 0709 12:24:42.767362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.768313 (+   951us) service_pool.cc:224] Handling call
0709 12:24:42.768770 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:43.545438 (+776668us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:43.545469 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:43.570761 (+ 25292us) log.cc:844] Serialized 26 byte log entry
0709 12:24:43.737209 (+166448us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:43.737250 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.737321 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.072420 (+335099us) raft_consensus.cc:1685] finished
0709 12:24:44.072444 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.072611 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":776208,"raft.queue_time_us":798790,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":10402,"thread_start_us":134067,"threads_started":1,"wal-append.queue_time_us":224944}
I20260709 12:24:44.117198  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:44.118117  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:44.118486  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:44.119410  1801 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:44.050269  7826 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.521s	user 0.008s	sys 0.006s
W20260709 12:24:44.070864  8247 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.281s	user 0.001s	sys 0.000s
I20260709 12:24:44.120591  1803 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
W20260709 12:24:44.051708  8246 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.182s	user 0.000s	sys 0.000s
I20260709 12:24:44.035857  8249 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:44.047776  8252 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260709 12:24:44.073372  7959 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 12:24:44.132656  1880 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.583s	user 0.013s	sys 0.000s
I20260709 12:24:44.059248  8116 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.113018  7811 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:44.140969  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.142103  1881 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:24:44.137815  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:44.139384  1880 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:24:44.139961  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:44.143765  1872 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:44.144608  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 529819us
I20260709 12:24:44.147092  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:44.142578  8250 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.270s	user 0.001s	sys 0.000s
I20260709 12:24:44.148021  1801 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
W20260709 12:24:44.155143  1770 connection.cc:256] RPC call timeout handler was delayed by -1.2065234470001087s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20260709 12:24:44.157068  1770 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 2.149s (SENT)
I20260709 12:24:44.143990  8116 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.161424  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.166352  8257 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260709 12:24:44.166433  8258 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:44.167642  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2473) took 1432 ms. Trace:
I20260709 12:24:44.168049  1799 rpcz_store.cc:276] 0709 12:24:42.735165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.741125 (+  5960us) service_pool.cc:224] Handling call
0709 12:24:42.767101 (+ 25976us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:43.608814 (+841713us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:43.608864 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:43.624373 (+ 15509us) log.cc:844] Serialized 52 byte log entry
0709 12:24:43.755160 (+130787us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:43.755209 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.755282 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.167400 (+412118us) raft_consensus.cc:1685] finished
0709 12:24:44.167425 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.167580 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":853204,"raft.queue_time_us":855945,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":114,"thread_start_us":107693,"threads_started":1,"wal-append.queue_time_us":126055}
I20260709 12:24:44.172046  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.172745  1873 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:44.172896  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:44.174238  8188 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:44.173379  8165 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.175264  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.179140  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:44.159160  8255 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.206265  1872 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 19 because replica has last-logged OpId of term: 18 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:24:44.138732  8249 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:44.190564  8198 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.001s	sys 0.009s
I20260709 12:24:44.160194  8199 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:24:44.217005  1806 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:44.207974  8188 raft_consensus.cc:2764] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election decision vote started in defunct term 20: won
W20260709 12:24:44.215005  8198 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.001s	sys 0.009s
I20260709 12:24:44.226799  1804 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 12:24:44.214867  7816 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.560s	user 0.007s	sys 0.005s
I20260709 12:24:44.181728  8116 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.232290  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:44.236819  2410 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:44.247776  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:44.249210  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:44.250013  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:44.255389  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 89765.4us
I20260709 12:24:44.209676  8264 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:24:44.209929  8255 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:44.257467  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49296.7us
W20260709 12:24:44.248531  7960 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.725s	user 0.014s	sys 0.000s
I20260709 12:24:44.242770  8120 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.235453  7816 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:44.271859  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.273876  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:44.273665  7911 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:44.252779  8076 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:44.282341  7960 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.290370  7960 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.292158  1800 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:44.293891  7743 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:44.294471  7743 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:44.257368  8165 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.295490  7743 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.277100  7970 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:44.296397  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3366) took 5323 ms. Trace:
I20260709 12:24:44.296784  7970 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:44.296846  1724 rpcz_store.cc:276] 0709 12:24:38.973442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.306819 (+3333377us) service_pool.cc:224] Handling call
0709 12:24:42.307130 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:43.133666 (+826536us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.296150 (+1162484us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b540044aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:44.296339 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1853252352}
I20260709 12:24:44.299397  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 897) took 1439 ms. Trace:
I20260709 12:24:44.299844  1730 rpcz_store.cc:276] 0709 12:24:42.860219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.923822 (+ 63603us) service_pool.cc:224] Handling call
0709 12:24:42.924147 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.936811 (+ 12664us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.937099 (+   288us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:42.937610 (+   511us) log.cc:844] Serialized 26 byte log entry
0709 12:24:43.063515 (+125905us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:43.063571 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.063652 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.292106 (+1228454us) spinlock_profiling.cc:243] Waited 27.8 ms on lock 0x7b2400186cc8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:24:44.298993 (+  6887us) raft_consensus.cc:1685] finished
0709 12:24:44.299026 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.299325 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":122,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":4698,"spinlock_wait_cycles":63856512,"thread_start_us":86218,"threads_started":1,"wal-append.queue_time_us":122759}
I20260709 12:24:44.308442  1730 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:44.309430  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2508) took 1147 ms. Trace:
I20260709 12:24:44.309826  1730 rpcz_store.cc:276] 0709 12:24:43.162404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.301586 (+1139182us) service_pool.cc:224] Handling call
0709 12:24:44.308171 (+  6585us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.309069 (+   898us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.309348 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:44.298210  7970 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 9, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.321308  7911 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:44.331626  8249 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:44.333554  8259 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:24:44.335199  1727 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:44.336410  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 881) took 3696 ms. Trace:
I20260709 12:24:44.336792  1727 rpcz_store.cc:276] 0709 12:24:40.639701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.591079 (+1951378us) service_pool.cc:224] Handling call
0709 12:24:42.591365 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.335631 (+1744266us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.336176 (+   545us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b54002d0ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:44.336351 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4009569664}
I20260709 12:24:44.338243  1727 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:24:44.340754  1799 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:24:44.342425  7911 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:44.344185  8200 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.403s	user 0.005s	sys 0.007s
I20260709 12:24:44.345971  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2510) took 1182 ms. Trace:
I20260709 12:24:44.347165  8270 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:44.346531  1727 rpcz_store.cc:276] 0709 12:24:43.163622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.337679 (+1174057us) service_pool.cc:224] Handling call
0709 12:24:44.338023 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.338583 (+   560us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.338771 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:44.349397  1727 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 16. Ops: []
I20260709 12:24:44.349994  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2511) took 1185 ms. Trace:
I20260709 12:24:44.350373  1727 rpcz_store.cc:276] 0709 12:24:43.164185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.348777 (+1184592us) service_pool.cc:224] Handling call
0709 12:24:44.349193 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.349734 (+   541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.349915 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:44.347859  8270 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:44.362942  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3399) took 3578 ms (client timeout 2149 ms). Trace:
W20260709 12:24:44.409534  1726 rpcz_store.cc:269] 0709 12:24:40.784682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.784972 (+   290us) service_pool.cc:224] Handling call
0709 12:24:44.362706 (+3577734us) spinlock_profiling.cc:243] Waited 20.8 ms on lock 0x7b540030aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:44.362859 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":47786752}
I20260709 12:24:44.366886  1728 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 12:24:44.354090  7358 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.569s	user 0.007s	sys 0.003s
I20260709 12:24:44.377630  7945 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:44.412339  7945 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:44.413337  7945 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.417239  8255 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.418722  8184 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:44.419189  8184 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:44.420083  8184 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.421125  8184 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:44.421047  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:44.421852  1727 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:44.423763  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:44.384135  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08001c5980 after lost signal to thread 1756
I20260709 12:24:44.425976  1726 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:44.392604  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:44.427248  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2504) took 1274 ms. Trace:
I20260709 12:24:44.405997  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2476) took 1439 ms. Trace:
I20260709 12:24:44.427727  1731 rpcz_store.cc:276] 0709 12:24:43.152619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.214646 (+ 62027us) service_pool.cc:224] Handling call
0709 12:24:44.392494 (+1177848us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3eee 000000000036eb39 00007f16b0d271e0 00007f16b0df00c4 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:24:44.427180 (+ 34686us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2775168}
W20260709 12:24:44.397944  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 981266us
I20260709 12:24:44.429234  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:44.366329  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40200 (request call id 51) took 1593 ms. Trace:
I20260709 12:24:44.430543  1729 rpcz_store.cc:276] 0709 12:24:42.772835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.849915 (+ 77080us) service_pool.cc:224] Handling call
0709 12:24:44.366125 (+1516210us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b540030aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd2853 00007f16b0da060a 00007f16aa3267d4 00007f16aa326762 00007f16aa326691 00007f16aa32660c 00007f16aa3258c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:44.366255 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1092248832}
I20260709 12:24:44.428206  1802 rpcz_store.cc:276] 0709 12:24:42.966817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.967000 (+   183us) service_pool.cc:224] Handling call
0709 12:24:42.989181 (+ 22181us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:43.545486 (+556305us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:43.570590 (+ 25104us) log.cc:844] Serialized 14 byte log entry
0709 12:24:44.346827 (+776237us) spinlock_profiling.cc:243] Waited 4.57 ms on lock 0x7b48001c0158. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:24:44.362007 (+ 15180us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b180034c960. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:44.400704 (+ 38697us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b180034c960. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:24:44.401002 (+   298us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.401619 (+   617us) log.cc:844] Serialized 26 byte log entry
0709 12:24:44.405532 (+  3913us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:44.405577 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.405665 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.405725 (+    60us) raft_consensus.cc:1685] finished
0709 12:24:44.405743 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.405918 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":555942,"prepare.queue_time_us":25547,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":586674,"raft.run_cpu_time_us":11704,"raft.run_wall_time_us":767873,"spinlock_wait_cycles":46528256,"thread_start_us":15735,"threads_started":1,"wal-append.queue_time_us":29316}
I20260709 12:24:44.434258  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:44.453562  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:44.457258  1872 raft_consensus.cc:2376] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17. Re-sending same reply.
I20260709 12:24:44.389595  7920 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.459153  1877 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:44.410291  8112 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:44.452687  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 144991us
I20260709 12:24:44.461764  1873 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:44.462258  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:44.463543  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:44.463903  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:44.364192  7816 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.398129  8206 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:44.360364  8116 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.427085  7945 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:44.479828  8000 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:44.480707  8000 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:44.487787  8215 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.225s	user 0.000s	sys 0.003s
I20260709 12:24:44.390867  7995 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.395268  8270 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.391939  8185 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:44.489652  8270 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:44.490239  8270 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:44.491370  8270 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.494400  8230 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.495026  8230 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.490098  8185 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.470166  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.500087  8206 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.366461  8067 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.411851  7358 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.504977  7358 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.485921  7811 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.474733  8255 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:44.507901  1803 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:44.518234  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.519186  7816 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:44.520849  7811 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.520354  7816 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:44.521992  7816 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:44.522070  8116 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:44.523157  8230 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.443454  7988 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:44.523303  7816 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.524725  7816 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:44.524633  7988 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.521338  8255 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:44.526454  8255 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.527434  8206 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:44.519625  8185 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:44.529907  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.522716  8112 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:44.530969  8112 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:44.533177  8161 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.535301  7988 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.535948  7743 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:44.537196  8123 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:44.539830  8012 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:44.542122  8112 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.465937  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:44.550525  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 84655.2us
I20260709 12:24:44.444677  1727 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
W20260709 12:24:44.647516  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 216849us
I20260709 12:24:44.648670  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2507) took 1486 ms. Trace:
I20260709 12:24:44.706602  1724 rpcz_store.cc:276] 0709 12:24:43.161806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.298227 (+1136421us) service_pool.cc:224] Handling call
0709 12:24:44.298588 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.424314 (+125726us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.424347 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.424900 (+   553us) log.cc:844] Serialized 26 byte log entry
0709 12:24:44.647437 (+222537us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:44.647633 (+   196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.647830 (+   197us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.647977 (+   147us) raft_consensus.cc:1685] finished
0709 12:24:44.648018 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.648345 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":122386,"raft.queue_time_us":165583,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":5710,"thread_start_us":221603,"threads_started":1,"wal-append.queue_time_us":162779}
I20260709 12:24:44.649295  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 905) took 1548 ms. Trace:
I20260709 12:24:44.711045  1723 rpcz_store.cc:276] 0709 12:24:43.101184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.425710 (+324526us) service_pool.cc:224] Handling call
0709 12:24:43.426130 (+   420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.403692 (+977562us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.403739 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.404358 (+   619us) log.cc:844] Serialized 26 byte log entry
0709 12:24:44.648582 (+244224us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:44.648660 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.648744 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.648788 (+    44us) raft_consensus.cc:1685] finished
0709 12:24:44.648801 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.649050 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":976997,"raft.queue_time_us":983362,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":243377,"threads_started":1,"wal-append.queue_time_us":186759}
W20260709 12:24:44.713016  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.021s	sys 0.036s
W20260709 12:24:44.713816  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.538s	user 0.021s	sys 0.036s
I20260709 12:24:44.650453  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:44.652871  1726 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:44.665408  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3377) took 4494 ms. Trace:
I20260709 12:24:44.551707  7743 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:44.769634  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.589979  8022 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.691306  7988 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:44.762509   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:44.771035  1879 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:44.594264  8112 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.721045  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:44.729802  1725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2488) took 5040 ms (client timeout 2050 ms). Trace:
I20260709 12:24:44.767436  1728 rpcz_store.cc:276] 0709 12:24:40.170344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.553385 (+2383041us) service_pool.cc:224] Handling call
0709 12:24:44.366421 (+1813036us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.411596 (+ 45175us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.664718 (+253122us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b540030aa50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:44.665097 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4168810240}
I20260709 12:24:44.775067  1804 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:44.777158  1725 rpcz_store.cc:269] 0709 12:24:39.639433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.795175 (+155742us) service_pool.cc:224] Handling call
0709 12:24:44.662533 (+4867358us) spinlock_profiling.cc:243] Waited 2.79 s on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:44.679370 (+ 16837us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6423554816}
I20260709 12:24:44.767740  1726 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:44.768975  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.702657  1727 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:44.783926  1729 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:44.788285  8111 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:44.785389  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.806596  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:44.807472  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.807870  1798 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:44.808306  1873 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:44.810689  1877 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
W20260709 12:24:44.817215  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 111127us
I20260709 12:24:44.820605  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.825371  8161 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:44.827111  7611 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:44.827816  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2469) took 2122 ms (client timeout 1798 ms). Trace:
I20260709 12:24:44.828156  7611 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.831238  8174 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.831481  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:44.832192  8174 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.835078  8230 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.836099  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2586) took 2071 ms. Trace:
I20260709 12:24:44.836038  8230 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.836812  1875 rpcz_store.cc:276] 0709 12:24:42.763918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.766902 (+  2984us) service_pool.cc:224] Handling call
0709 12:24:42.786419 (+ 19517us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.839267 (+ 52848us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.839304 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:42.839354 (+    50us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:42.839400 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.835274 (+1995874us) spinlock_profiling.cc:243] Waited 52.5 ms on lock 0x7b5400073354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:44.835387 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.835771 (+   384us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":120822528}
I20260709 12:24:44.839897  8214 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:44.840454  8214 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:44.842310  8214 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:44.843981  1661 debug-util.cc:398] Leaking SignalData structure 0x7b080018e7a0 after lost signal to thread 1831
I20260709 12:24:44.844942  8214 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:44.847527  1726 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.849330  1726 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:44.850786  2410 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:44.851392  1801 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:44.854954  7945 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.855687  7945 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.856259  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2505) took 1695 ms. Trace:
I20260709 12:24:44.856462  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:44.858161  2415 raft_consensus.cc:2410] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:24:44.859964  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:44.860522  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:44.861200  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:44.861546  2413 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:44.862381  2414 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:44.828440  1806 rpcz_store.cc:269] 0709 12:24:42.705075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.705609 (+   534us) service_pool.cc:224] Handling call
0709 12:24:44.827206 (+2121597us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540013a954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:44.827443 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3473775488}
I20260709 12:24:44.842437  7826 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:24:44.865078  2411 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 12:24:44.846531  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08002f0fa0 after lost signal to thread 1905
I20260709 12:24:44.866654  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:44.868037  8189 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:44.868660  7859 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:44.869375  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08002ac8c0 after lost signal to thread 2443
W20260709 12:24:44.869902  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:44.869951  1661 debug-util.cc:398] Leaking SignalData structure 0x7b080020b860 after lost signal to thread 2444
I20260709 12:24:44.869707  7859 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:44.871315  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08003e29e0 after lost signal to thread 7820
I20260709 12:24:44.871554  1724 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: [17.14-17.14]
W20260709 12:24:44.873034  1661 debug-util.cc:398] Leaking SignalData structure 0x7b080031c040 after lost signal to thread 8048
I20260709 12:24:44.869146  8189 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:44.873323  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:44.874749  8189 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.876159  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:44.876816  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.879825  8192 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.880313  8009 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:44.880710  8192 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.881803  7743 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:44.882344  7743 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:44.882645  8009 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:44.884027  8270 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:44.884616  8270 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:44.885462  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:44.887399  8192 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.883587  7743 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:44.889441  8230 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.889655  7743 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:44.891086  1879 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:44.885964  8270 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:44.888391  7611 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.856779  1732 rpcz_store.cc:276] 0709 12:24:43.160471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.658097 (+497626us) service_pool.cc:224] Handling call
0709 12:24:43.658450 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:43.658710 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:43.659057 (+   347us) log.cc:844] Serialized 14 byte log entry
0709 12:24:44.741815 (+1082758us) log.cc:844] Serialized 14 byte log entry
0709 12:24:44.818183 (+ 76368us) spinlock_profiling.cc:243] Waited 23.1 ms on lock 0x7b4800041c58. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:24:44.855312 (+ 37129us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:44.855393 (+    81us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:44.855536 (+   143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.855861 (+   325us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.856060 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":640901,"prepare.queue_time_us":6611,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":45,"raft.queue_time_us":682832,"raft.run_cpu_time_us":3679,"raft.run_wall_time_us":351443,"spinlock_wait_cycles":53173632,"thread_start_us":440810,"threads_started":2,"wal-append.queue_time_us":145993}
W20260709 12:24:44.893438  1661 debug-util.cc:398] Leaking SignalData structure 0x7b0800147700 after lost signal to thread 8050
W20260709 12:24:44.899001  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 80251.2us
W20260709 12:24:44.900154  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08003c9460 after lost signal to thread 8061
I20260709 12:24:44.840312  8174 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:44.905270  7611 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:44.926678  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 159328us
I20260709 12:24:44.904099  1724 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:44.934293  1726 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:44.926947  7611 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:44.936942  1804 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:44.942076  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:44.938443  8009 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:44.962821  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:44.953649  8009 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.969022  8009 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:44.969585  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.970950  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2509) took 1742 ms. Trace:
I20260709 12:24:44.971475  1730 rpcz_store.cc:276] 0709 12:24:43.163023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.312474 (+1149451us) service_pool.cc:224] Handling call
0709 12:24:44.313559 (+  1085us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.652632 (+339073us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:44.663080 (+ 10448us) log.cc:844] Serialized 14 byte log entry
0709 12:24:44.866810 (+203730us) log.cc:844] Serialized 14 byte log entry
0709 12:24:44.867179 (+   369us) log.cc:844] Serialized 14 byte log entry
0709 12:24:44.867364 (+   185us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.868069 (+   705us) log.cc:844] Serialized 26 byte log entry
0709 12:24:44.868242 (+   173us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:44.868288 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.871587 (+  3299us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.897727 (+ 26140us) raft_consensus.cc:1685] finished
0709 12:24:44.897749 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.905350 (+  7601us) inbound_call.cc:177] Queueing success response
0709 12:24:44.967456 (+ 62106us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b4800041c58. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
Metrics: {"mutex_wait_us":157606,"prepare.queue_time_us":268054,"prepare.run_cpu_time_us":130,"prepare.run_wall_time_us":106,"raft.queue_time_us":176150,"raft.run_cpu_time_us":2761,"raft.run_wall_time_us":510632,"spinlock_wait_cycles":49932160,"thread_start_us":341026,"threads_started":3,"wal-append.queue_time_us":230194}
I20260709 12:24:44.974043  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:44.975021  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:44.975875  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47997.2us
I20260709 12:24:44.976891  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:44.977177  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:44.977813  1875 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:44.977981  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:44.982038  7611 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:44.984880  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 157351us
I20260709 12:24:44.975140  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1284) took 1156 ms. Trace:
I20260709 12:24:44.986135  8184 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.987131  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:44.985749  2419 rpcz_store.cc:276] 0709 12:24:43.781273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.782264 (+   991us) service_pool.cc:224] Handling call
0709 12:24:43.782599 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:43.970416 (+187817us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:43.970448 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.565277 (+594829us) spinlock_profiling.cc:243] Waited 2.82 ms on lock 0x7b1800455260. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a
0709 12:24:44.720473 (+155196us) spinlock_profiling.cc:243] Waited 63 ms on lock 0x7b2c00090838. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:24:44.770755 (+ 50282us) log.cc:844] Serialized 26 byte log entry
0709 12:24:44.899638 (+128883us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:44.899675 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.930800 (+ 31125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:44.934093 (+  3293us) raft_consensus.cc:1685] finished
0709 12:24:44.934111 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.938045 (+  3934us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":249304,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":4573,"spinlock_wait_cycles":151504896,"thread_start_us":284392,"threads_started":2,"wal-append.queue_time_us":122238}
I20260709 12:24:44.994153  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:44.995277  8270 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:44.996004  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:44.997051  8112 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:44.997730  8112 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.998164  8112 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:45.000164  8281 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:45.001328  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:45.002306  1804 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:45.003677  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:45.005080  1874 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:44.957804  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:44.997287  8042 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.989189  1732 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:44.974774  1730 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:44.949127  7945 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:45.006276   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:45.006973  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:44.989127  8266 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:44.966374  8067 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.025439  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:45.000176  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:45.028988  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:45.034750  1879 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:45.040323  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:45.041858  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:45.041419  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:45.044930  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 55058.7us
I20260709 12:24:44.943760  8230 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.084s
I20260709 12:24:45.053409  1874 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:44.987066  8184 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.023363  8266 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:45.064371  1875 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:45.066002  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:45.067317  8012 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.527s	user 0.016s	sys 0.015s
I20260709 12:24:45.069259  8133 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.073491  8133 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:45.074376  2416 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:45.075357  7694 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:45.075593  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.076000  7694 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:45.076728  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.033174  8231 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:24:45.030718  7859 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:45.086432  1874 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:45.031785  8207 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:45.080103  8174 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.019441  8042 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:45.096117  1880 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:45.079157  8266 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:45.101751  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:45.065841  8175 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:45.109356  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47280.1us
I20260709 12:24:45.127462  1804 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:45.115526  1807 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:45.128387  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 115901us
I20260709 12:24:45.104537  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:45.124246  8042 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:45.125005  8174 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:45.125946  1881 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:45.112535  1723 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:24:45.140882  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.143664  1798 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:45.145313  8014 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:45.148507  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 854) took 4924 ms. Trace:
I20260709 12:24:45.149039  1807 rpcz_store.cc:276] 0709 12:24:40.223617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.223805 (+   188us) service_pool.cc:224] Handling call
0709 12:24:45.115013 (+4891208us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:45.146898 (+ 31885us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.148144 (+  1246us) spinlock_profiling.cc:243] Waited 4.89 s on lock 0x7b540026f550. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:45.148379 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11247779200}
I20260709 12:24:45.159858  8120 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:24:45.160301  2416 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1146) took 1363 ms (client timeout 1796 ms). Trace:
I20260709 12:24:45.160678  8175 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:45.161005  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.161362  8175 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:45.160897  2416 rpcz_store.cc:269] 0709 12:24:43.796481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.796597 (+   116us) service_pool.cc:224] Handling call
0709 12:24:45.160007 (+1363410us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54006009d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:45.160181 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2935580416}
I20260709 12:24:45.160562  8120 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.165820  8207 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:45.171933  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 195039us
I20260709 12:24:45.143705  8184 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:45.168905  8120 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:45.160918  8296 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.167s
I20260709 12:24:45.160809  7816 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.159s
W20260709 12:24:45.179812  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 237768us
I20260709 12:24:45.182962  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:45.191249  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:45.184865  1801 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:45.190258  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:45.191216  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 150770us
W20260709 12:24:45.192940  8000 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.711s	user 0.011s	sys 0.013s
I20260709 12:24:45.194607  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:45.196151  2419 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:45.207168  2419 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:45.202172  1881 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:45.209702  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:45.214624  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.235611  2410 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:24:45.236195  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.238963  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:45.224874  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:45.241127  2416 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
W20260709 12:24:45.240370  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45893.9us
W20260709 12:24:45.244920  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62069us
I20260709 12:24:45.243258  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:45.245814  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:45.252568  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:45.239255  8230 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.253773  7826 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:45.282301  7826 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.282501  8174 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:45.282756  7826 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:45.257936  8192 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:45.284298  8192 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.284688  8192 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:45.281111  2418 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 15 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:24:45.281828  8175 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:45.211138  8000 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.215627  8184 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:24:45.295542  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:45.291666  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:45.239989  8009 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:45.296712  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:45.306239  8297 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.090s
W20260709 12:24:45.307713  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 48108us
I20260709 12:24:45.309149  7743 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:45.297020  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:45.263074  8067 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
W20260709 12:24:45.311075  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46254.2us
I20260709 12:24:45.323385  1725 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:45.303369  1799 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
W20260709 12:24:45.342310  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.347673  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 95709.3us
I20260709 12:24:45.327369  8175 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:45.305188  8161 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:24:45.348156  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:45.344595  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51572.8us
I20260709 12:24:45.301934  8230 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.331634  8014 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:24:45.380497  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:45.337852  8067 consensus_queue.cc:1243] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:45.339951  8297 consensus_queue.cc:1243] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 19.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:24:45.340540  8012 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:24:45.387147  8111 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:45.406137  8264 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.392253  7743 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.449769  8022 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.455962  8029 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:45.457998  8042 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:24:45.469890  7694 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:45.091176   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:45.455722  8022 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:45.495225  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 168243us
W20260709 12:24:45.501240  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47727.8us
I20260709 12:24:45.507527  8042 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:45.508976  8123 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:45.509507  8123 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:45.511305  7743 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.510660  8123 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:45.512164  2418 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1138) took 2227 ms (client timeout 2227 ms). Trace:
I20260709 12:24:45.512614  8123 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:45.512358  1625 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:24:45.512844  2418 rpcz_store.cc:269] 0709 12:24:43.284223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.285214 (+   991us) service_pool.cc:224] Handling call
0709 12:24:45.511698 (+2226484us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b5400840154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:45.512027 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4554304896}
W20260709 12:24:45.514729  7913 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.622s	user 0.013s	sys 0.030s
W20260709 12:24:45.515187  7913 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.741s	user 0.013s	sys 0.031s
I20260709 12:24:45.515956  8067 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.516590  8067 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:45.519218  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 159750us
I20260709 12:24:45.520399  8022 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.520740  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:45.525677  8000 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.527154  8000 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.527724  8000 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:45.528091  2419 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.529258  2419 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:45.531621  8300 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.077s	user 0.000s	sys 0.002s
W20260709 12:24:45.533774  2383 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.550s (SENT)
I20260709 12:24:45.534286  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:45.426810  8111 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:45.529947  8000 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:45.543248  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.545785  1625 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:45.551668  8230 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.553545  8009 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:45.554492  8255 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:45.553918  8111 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:45.565745  1624 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 257) took 28411 ms (client timeout 15000 ms). Trace:
I20260709 12:24:45.560149  8255 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:45.568701  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:24:45.603287  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.555410  7826 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.454277  8264 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.585186  8042 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:45.615389  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 215542us
I20260709 12:24:45.591899  8207 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:45.620466  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.590416  8014 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:45.601081  8000 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:45.622815  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89116.3us
I20260709 12:24:45.624135  8067 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.626596  7816 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.631717  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:45.640132  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.643927  8067 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.645061  8067 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.646901  8009 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:45.647537  8009 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:45.647291  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:45.648150  8009 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:45.653153  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.657680  1873 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:24:45.575819  8123 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.679364  1881 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 21 index: 9. (index mismatch)
W20260709 12:24:45.680523  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 65093.1us
I20260709 12:24:45.615943  8231 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:45.643903  7947 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.655678  8270 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:45.680732  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:45.581844  8192 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:45.697714  2383 leader_election.cc:336] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.594s (SENT)
W20260709 12:24:45.699257  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 140633us
I20260709 12:24:45.639989  8000 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:45.710187  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:45.653846  7816 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:45.641881  8042 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.694588  8270 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:45.655838  8029 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:24:45.691394  8207 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:45.724339  2411 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
W20260709 12:24:45.721709  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 196973us
W20260709 12:24:45.726347  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.726723  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.704109  7988 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:45.680328  8123 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:45.729796  1725 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
W20260709 12:24:45.739446  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 162936us
I20260709 12:24:45.731876  8270 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:45.763067  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:45.704902  7947 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:45.770757  8298 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:24:45.757784  7694 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:45.781153  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:45.781513  7826 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:45.781885  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:45.782326  1803 raft_consensus.cc:2410] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 19 because replica has last-logged OpId of term: 18 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 15.
I20260709 12:24:45.785658  8133 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:45.712543  8231 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:45.788486  8299 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
W20260709 12:24:45.791476  8306 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.324s	user 0.002s	sys 0.002s
I20260709 12:24:45.768119  8207 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.794507  8192 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:45.794684  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 51106.4us
I20260709 12:24:45.772056  8297 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:45.795918  7816 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:45.796214  8207 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:45.797279  8305 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:24:45.799172  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:45.801203  7913 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:45.801854  7913 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:45.802170  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:45.721673  8255 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.803037  2412 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:45.804081  7913 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:45.770820  8312 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:24:45.805838  8312 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:24:45.806028  8112 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.807s	user 0.004s	sys 0.012s
I20260709 12:24:45.806222  8312 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:45.806484  8112 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:45.807073  8308 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.777988  1845 leader_election.cc:400] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17.
I20260709 12:24:45.808442  1845 leader_election.cc:403] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:45.795996  8297 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:45.777230  1804 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:24:45.777187  1798 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:45.801854  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:45.794689  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 104349us
W20260709 12:24:45.571144  1624 rpcz_store.cc:269] 0709 12:24:17.153823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.154231 (+   408us) service_pool.cc:224] Handling call
0709 12:24:45.565388 (+28411157us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:36.723131 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:36.724049 (+   918us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:36.724137 (+    88us) write_op.cc:435] Acquired schema lock
0709 12:24:36.724172 (+    35us) tablet.cc:662] Decoding operations
0709 12:24:36.726030 (+  1858us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:36.726162 (+   132us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:36.726191 (+    29us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:24:36.783284 (+ 57093us) tablet.cc:701] Row locks acquired
0709 12:24:36.783327 (+    43us) write_op.cc:260] PREPARE: finished
0709 12:24:36.783559 (+   232us) write_op.cc:270] Start()
0709 12:24:36.783740 (+   181us) write_op.cc:276] Timestamp: P: 1783599876783474 usec, L: 0
0709 12:24:36.783767 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:24:37.037683 (+253916us) log.cc:844] Serialized 5600 byte log entry
0709 12:24:37.608635 (+570952us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd3e28 00007f16af80e581 00007f16af80dd58 00007f16af8129f1 00007f16af8129a9 00007f16af812939 00007f16af812901 00007f16af811b2d 0000000000388634 0000000000388479
0709 12:24:38.471179 (+862544us) spinlock_profiling.cc:243] Waited 8.13 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:38.644960 (+173781us) op_driver.cc:464] REPLICATION: finished
0709 12:24:38.955571 (+310611us) write_op.cc:301] APPLY: starting
0709 12:24:38.955730 (+   159us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:39.881154 (+925424us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:39.887059 (+  5905us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:40.168564 (+281505us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:40.168909 (+   345us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=28,mrs_lookups=0
0709 12:24:40.168967 (+    58us) write_op.cc:312] APPLY: finished
0709 12:24:40.197165 (+ 28198us) log.cc:844] Serialized 129 byte log entry
0709 12:24:40.460944 (+263779us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:40.463911 (+  2967us) write_op.cc:454] Released schema lock
0709 12:24:40.464017 (+   106us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:40.512068 (+ 48051us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:40.522360 (+ 10292us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1811554,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":251560,"apply.run_cpu_time_us":53629,"apply.run_wall_time_us":1577449,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62703,"num_ops":14,"prepare.queue_time_us":286694,"prepare.run_cpu_time_us":72230,"prepare.run_wall_time_us":885818,"raft.queue_time_us":170642,"raft.run_cpu_time_us":33196,"raft.run_wall_time_us":103968,"replication_time_us":1861019,"spinlock_wait_cycles":69446144,"thread_start_us":1006937,"threads_started":5,"wal-append.queue_time_us":702236,"wal-append.run_cpu_time_us":41420,"wal-append.run_wall_time_us":3624119}]]}
I20260709 12:24:45.807349  8312 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.832852  2419 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:45.837323  2415 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:45.790457  8231 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.837783  8312 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:45.771270  8214 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:24:45.685037  1625 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 10 to 11, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 11 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:45.866990  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:45.827934  8230 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:45.835816  8067 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.840648  8207 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17.
I20260709 12:24:45.807925  8308 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:45.879683  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 97968.1us
I20260709 12:24:45.825412  8270 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:45.883249  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 218174us
I20260709 12:24:45.883616  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:45.883608  1803 raft_consensus.cc:1217] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Deduplicated request from leader. Original: 14.10->[16.11-18.12]   Dedup: 18.12->[]
I20260709 12:24:45.870609  8230 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:45.894559  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 125117us
I20260709 12:24:45.888243  1803 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: [16.11-18.12]
I20260709 12:24:45.832782  8175 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
I20260709 12:24:45.838701  8133 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:24:45.908840  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:45.838114  8029 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.061s
W20260709 12:24:45.909200  8266 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.757s	user 0.015s	sys 0.000s
I20260709 12:24:45.908407  8230 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:45.927994  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.898366  7947 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:45.929553  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 86393.7us
W20260709 12:24:45.935678  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 168248us
W20260709 12:24:45.936955  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 57656.4us
I20260709 12:24:45.922222  8067 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.933988  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 913) took 1282 ms. Trace:
I20260709 12:24:45.921162  8000 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:45.938833  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.949919  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 106085us
I20260709 12:24:45.944381  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:45.921500  8207 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:45.918390  8112 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:45.950920  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:45.952277  1874 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:45.929060  8012 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:45.958241  1728 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:45.948333  1772 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:45.880894  8297 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:45.956460  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 209709us
I20260709 12:24:45.976681  8310 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 12:24:45.977059  8009 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.977942  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:45.978734  8311 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:45.958153  8022 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20260709 12:24:45.981019  8318 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:45.980741  1625 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 12 to 14, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 14 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:45.958737  8067 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.984081  8270 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.984859  8067 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.974805  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:45.982802  7987 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:45.987213  8009 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:45.957397  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:45.953145  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:45.993069  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 44835.4us
I20260709 12:24:45.970858  1877 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:24:45.973879  1876 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:45.974316  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:45.995802  1878 raft_consensus.cc:2410] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
W20260709 12:24:45.999109  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.969878  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:45.971942  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:46.000123  1879 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:45.975261  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:45.974891  2415 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:45.964115  8012 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:45.980190  8314 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:24:46.007599  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 77286.7us
I20260709 12:24:45.990996  1625 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:46.008512  8042 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:46.016132  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.009080  8324 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:46.009898  8308 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:46.010198  1873 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:46.011511  1880 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:46.025035  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:46.013831  8313 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
W20260709 12:24:45.976157  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.834s (SENT)
I20260709 12:24:45.972587  8123 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:45.975116  8120 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.763s	user 0.017s	sys 0.000s
I20260709 12:24:45.947103  1729 rpcz_store.cc:276] 0709 12:24:44.650682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.819516 (+168834us) service_pool.cc:224] Handling call
0709 12:24:45.032140 (+212624us) spinlock_profiling.cc:243] Waited 19.7 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:24:45.032564 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:45.033089 (+   525us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:45.059289 (+ 26200us) log.cc:844] Serialized 14 byte log entry
0709 12:24:45.666834 (+607545us) log.cc:844] Serialized 14 byte log entry
0709 12:24:45.899039 (+232205us) log.cc:844] Serialized 14 byte log entry
0709 12:24:45.910103 (+ 11064us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:45.910155 (+    52us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:45.910218 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.910282 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:45.932682 (+ 22400us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58535,"prepare.queue_time_us":71622,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":134,"raft.queue_time_us":65589,"raft.run_cpu_time_us":510,"raft.run_wall_time_us":25653,"spinlock_wait_cycles":45256448,"thread_start_us":660163,"threads_started":3,"wal-append.queue_time_us":547033}
W20260709 12:24:46.043716  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.044003  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:45.987377  7987 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.012158  8012 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.002734  8009 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.060s
I20260709 12:24:46.023697  8029 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:46.019604  8324 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:46.071041  8248 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:46.062307  8067 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.054253  8133 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:24:46.090765  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 147384us
W20260709 12:24:46.079161  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 134060us
I20260709 12:24:45.979420  8311 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:45.931245  8266 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.094476  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:46.071062  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 95085.5us
I20260709 12:24:46.100646  8264 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.103247  2419 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
W20260709 12:24:46.117583  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 103702us
I20260709 12:24:46.107380  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:46.119138  2414 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:46.110816  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:46.115098  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53508.2us
W20260709 12:24:46.132318  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.133028  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 155196us
I20260709 12:24:46.087471  8324 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.131945  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:46.136819  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:46.137588  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.862s (SENT)
I20260709 12:24:46.059636  8120 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.138173  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.094076  8133 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.140020  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:46.141655  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:46.145401  2411 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 16 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:46.145817  2415 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:46.092589  8112 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:46.146361  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.147579  8323 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.010s
I20260709 12:24:46.147902  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.148907  8264 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:46.149528  8321 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:46.152774  1876 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
W20260709 12:24:46.150146  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.915s (SENT)
I20260709 12:24:46.145118  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.077149  8308 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 12:24:46.164094  1726 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:46.166852  1806 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:46.092464  8067 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.174535  1877 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:46.164727  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:46.152765  8133 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:46.177841  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56332.6us
I20260709 12:24:46.192296  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:46.193094  1729 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:46.194316  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 99955.5us
I20260709 12:24:46.199175  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:46.144990  8029 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
W20260709 12:24:46.208567  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58483.5us
I20260709 12:24:46.187741  7913 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:24:46.213346  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101315us
I20260709 12:24:46.212572  1726 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:24:46.219940  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.194756  7988 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.226594  8304 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:46.187098  8112 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.233718  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.235934  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:46.246389  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.247170  2415 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:24:46.224596  8318 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:46.249598  8332 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:46.250285  8332 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:46.251696  8332 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.253413  8332 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.255635  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.256371  2418 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:46.231668  8304 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:46.263873  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 107794us
I20260709 12:24:46.263760  2419 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:46.265422  1879 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:46.267844  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:46.240015  8321 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:46.239585  8325 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.278981  1725 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:46.279266  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:46.277689  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 140344us
I20260709 12:24:46.277772  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:46.272797  8214 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:46.280872  1803 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.280701  1798 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:46.280766  8214 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:46.283686  8337 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:24:46.284598  8214 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.285296  7987 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.285890  8214 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:46.236281  7988 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.228969  8175 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:24:46.297765  2414 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:46.270669  8304 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:46.302593  1728 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:46.240509  7816 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:24:46.273949  8000 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:46.306279  2383 leader_election.cc:336] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.412s (SENT)
W20260709 12:24:46.315734  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.318655  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:46.310765  8327 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.221s	user 0.001s	sys 0.001s
I20260709 12:24:46.350327  1729 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:46.356143  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:46.322391  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:46.331270  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 51144.9us
I20260709 12:24:46.359632  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:46.361119  8321 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:46.361557  8321 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:46.361961  8321 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:46.362244  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:46.363149  8189 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:46.365572  8189 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:24:46.333623  8326 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.218s	user 0.000s	sys 0.001s
I20260709 12:24:46.367089  8029 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:24:46.335283  8334 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.367817  8029 consensus_queue.cc:1243] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 12:24:46.336915  8333 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:24:46.368263  1727 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:46.337675  8331 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:24:46.369143  1731 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: [12.11-13.12]
I20260709 12:24:46.339767  8067 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.340976  8112 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.344833  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:46.345388  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.368238  8334 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.371948  8029 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:46.372184  8189 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:46.372646  8334 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:46.375085  8029 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:46.316565  8000 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:46.318267  8266 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:46.320557  8284 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:46.395331  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.396211  2419 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:46.298489  8325 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:46.385668  8312 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.517s	user 0.010s	sys 0.004s
I20260709 12:24:46.228078  1625 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 9 to 10, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 10 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:46.354364  8120 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:46.401973  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:46.415216  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 109011us
I20260709 12:24:46.416642  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:46.397934  8284 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.397753  8000 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:46.419030  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:46.444018  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:46.425542  8297 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.418589  8230 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.451702  2419 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:46.451916  7988 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:46.452320  1803 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:46.452209  8312 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:24:46.453118  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.447595  7694 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:24:46.446204  8184 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:46.445186  7945 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:24:46.455458  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:46.483130  8312 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:46.483592  8312 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:46.484714  8192 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:46.484531  8312 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.485198  8192 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:46.431577  1625 catalog_manager.cc:5697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 8 to 10, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 10 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:24:46.456314  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.487882  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.488042  7694 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.487607  1625 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 7 to 11, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 11 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:46.488626  7694 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.488654  7945 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:46.489928  7945 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:46.435286  8262 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:46.491236  1732 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 12:24:46.490980  7945 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.491967  8332 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:46.419116  8284 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:46.492650  8332 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:46.493587  8230 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.466696  2418 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:46.422075  8325 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.472352  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.486436  8192 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.455451  8321 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.492274  7945 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:46.515646  7694 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.425741  8012 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.533830  2418 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:46.533962  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:46.546636  2419 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:46.493948  8332 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.560451  8012 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:46.562124  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.563007  2418 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:24:46.576244  2416 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:46.563586  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.568339  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:46.523806  8321 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:46.506815  1625 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:46.594249  1625 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 4 to 12, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 12 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:46.596320  1625 catalog_manager.cc:5697] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 9 to 10, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 10 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:46.581502  8192 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:46.602104  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:46.590260  7694 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:46.617730  1877 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:46.610455  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:46.598747  1625 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:46.609261  8000 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.624500  8000 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.625196  1724 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:46.633181  1881 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:46.634802  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.636089  1877 raft_consensus.cc:2410] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:24:46.647003  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.645721  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.609339  8009 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:46.627657  7743 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:46.618818  8042 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:46.629307  7694 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:46.736804  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.737675  1879 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:46.603665  8207 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:24:46.704787  2412 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:24:46.717125  2417 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:46.717125  2410 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:46.718210  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:46.740617  1881 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:24:46.633270  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:46.724007  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:46.725273  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:46.731595  8350 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
W20260709 12:24:46.733348  8356 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:24:46.745107  1873 raft_consensus.cc:2410] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
I20260709 12:24:46.745584  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:46.750766  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.751753  1881 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:46.755010  8207 consensus_queue.cc:1243] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:24:46.756003  8357 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:46.766393  8346 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:46.767143  8346 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.768426  8311 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:46.769577  8346 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:46.769174  8304 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:46.770370  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 494869us
I20260709 12:24:46.770584  8343 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:46.771144  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 368381us
I20260709 12:24:46.771318  7913 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:46.772365  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 876) took 1081 ms. Trace:
W20260709 12:24:46.749833  8324 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.594s	user 0.013s	sys 0.000s
W20260709 12:24:46.776811  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.777237  8009 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.778658  7694 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:46.782982  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:46.792048  1724 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:46.783587  8022 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:46.797904  2411 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
W20260709 12:24:46.786823  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 44475.5us
W20260709 12:24:46.790252  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 160513us
I20260709 12:24:46.799093  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:46.788765  1799 rpcz_store.cc:276] 0709 12:24:45.691211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.698968 (+  7757us) service_pool.cc:224] Handling call
0709 12:24:45.736795 (+ 37827us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:45.793657 (+ 56862us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:45.918325 (+124668us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.203532 (+285207us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:46.233570 (+ 30038us) log.cc:844] Serialized 52 byte log entry
0709 12:24:46.234364 (+   794us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:46.234694 (+   330us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.492993 (+258299us) spinlock_profiling.cc:243] Waited 1.82 ms on lock 0x7b180015e0e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf 00007f16abdf2038
0709 12:24:46.493210 (+   217us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.771880 (+278670us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.772051 (+   171us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.772098 (+    47us) raft_consensus.cc:1685] finished
0709 12:24:46.772111 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.772273 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1060964,"prepare.queue_time_us":473627,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":104,"raft.queue_time_us":448222,"raft.run_cpu_time_us":545,"raft.run_wall_time_us":13311,"spinlock_wait_cycles":4178688,"thread_start_us":325767,"threads_started":3,"wal-append.queue_time_us":258409}
I20260709 12:24:46.783011  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 877) took 1078 ms. Trace:
I20260709 12:24:46.800093  1801 rpcz_store.cc:276] 0709 12:24:45.704315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.704531 (+   216us) service_pool.cc:224] Handling call
0709 12:24:45.718155 (+ 13624us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:45.760124 (+ 41969us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:45.760171 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:45.779328 (+ 19157us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.932263 (+152935us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:46.145606 (+213343us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.782428 (+636822us) spinlock_profiling.cc:243] Waited 3.73 ms on lock 0x7b5400408920. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd53733 00007f16abdd8dbf 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c
0709 12:24:46.782478 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.782646 (+   168us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.782696 (+    50us) raft_consensus.cc:1685] finished
0709 12:24:46.782827 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.782955 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20321,"prepare.queue_time_us":76061,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":29,"raft.queue_time_us":105896,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":15214,"spinlock_wait_cycles":8567680,"thread_start_us":254226,"threads_started":2,"wal-append.queue_time_us":170100}
I20260709 12:24:46.807474  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.808409  2418 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:46.806222  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:46.802271  8296 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:24:46.802731  8207 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:46.813225  1844 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:46.815605  8348 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:46.817034  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.819840  1877 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:46.824024  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.824926  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:46.798211  8133 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.601s	user 0.008s	sys 0.004s
I20260709 12:24:46.831743  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:46.816429  8348 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:46.835906  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.835811  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:46.835757  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:46.788614  8343 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.814509  8009 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:46.834409  1803 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:46.836412  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:46.838809  1801 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:46.790624  8042 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:46.843477  1799 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:24:46.843799  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:46.842650  8296 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:46.848207  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.831408  8231 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260709 12:24:46.828374  8029 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.021s
I20260709 12:24:46.799186  8324 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.837047  8161 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:24:46.852183  1729 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:46.852401  8161 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:46.849475  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:46.852823  8161 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:46.853457  1798 raft_consensus.cc:2368] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14.
I20260709 12:24:46.853787  8161 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.854295  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:46.841073  8343 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:46.833545  7913 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.843425  8307 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20260709 12:24:46.855999  7913 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.856415  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:46.838451  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:46.762529  8284 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.858992  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:46.867007  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 62441us
I20260709 12:24:46.874033  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.877714  1731 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
W20260709 12:24:46.878268  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.882728  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:46.851707  8324 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.883692  1799 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:46.887475  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.888270  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:46.889135  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:46.889775  1729 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:46.864933  8307 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:46.895606  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:46.899539  8360 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.094s	user 0.002s	sys 0.000s
I20260709 12:24:46.900504  8349 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.111s
I20260709 12:24:46.903117  8338 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:46.903749  8338 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:46.904716  8338 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.907176  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:46.909296  8304 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:46.911664  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.912663  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 638015us
I20260709 12:24:46.875301  8161 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:46.894660  1798 raft_consensus.cc:2376] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18. Re-sending same reply.
I20260709 12:24:46.893294  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:46.842281  8133 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:46.935263  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.913478  8307 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.912817  7987 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:46.951900  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:46.954824  7859 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.958088  1727 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:46.961985  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 77603.3us
I20260709 12:24:46.965308  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:46.967474  1803 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:46.968467  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48196us
I20260709 12:24:46.919890  8123 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
W20260709 12:24:46.972342  8361 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:24:46.975929  8255 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:46.979203  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:46.980504  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.980746  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:46.987042  8322 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.987763  8322 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:46.990108  8322 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:46.991483  8296 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:46.991995  8296 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.992341  8296 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:46.999315  1800 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:46.999452  1806 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:47.002602  1801 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
W20260709 12:24:47.014905  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.950685  8307 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:47.000216  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.041873  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.044543  1728 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:24:47.039176  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:46.956035  8184 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:47.047809  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.063620  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.064178  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:47.064798  1804 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:47.065673  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.069633  1798 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:47.071069  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:47.071887  1872 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:47.074565  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.075382  1726 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:47.077653  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:46.998256  8312 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:47.080361  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:46.970005  8000 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:47.037771  7913 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:47.084714  1730 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:47.048647  8357 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:24:47.084626  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 144037us
I20260709 12:24:47.088192  1724 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:47.091640  2415 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
W20260709 12:24:47.094551  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55506.7us
I20260709 12:24:46.970161  7987 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.098090  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:46.969834  7859 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.079984  8325 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:47.108896  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:46.955628  8042 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:47.096127  8123 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:47.086704  8310 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:47.119341  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:24:47.125061  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.127799  1727 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 13 index: 13. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:24:47.128948  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.130651  1731 pending_rounds.cc:85] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 6
I20260709 12:24:47.131042  1731 pending_rounds.cc:107] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:24:47.131345  1731 raft_consensus.cc:2889] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:47.134065  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.135066  1798 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
W20260709 12:24:47.137449  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63401.1us
I20260709 12:24:47.102594  8357 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:47.139106  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:47.146896  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:47.150522  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47532.2us
I20260709 12:24:47.134182  8231 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.162223  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.135766  8284 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:47.159796  8214 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:47.168985  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.179822  1724 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:47.118039  8000 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:47.191143  1804 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
W20260709 12:24:47.197182  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.198511  1728 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:47.143869  8029 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:24:47.144104  8321 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:24:47.217118  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.222955  1724 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:47.225409  1726 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 21 because replica has last-logged OpId of term: 19 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:24:47.194097  8184 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:47.160838  8310 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:47.236301  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.193341  8380 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:47.243165  1805 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:47.244230  1724 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:47.247977  7859 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:47.176430  8231 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.250313  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:47.250862  7987 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:47.252228  7987 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:47.252969  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:47.253038  8318 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:47.254058  7987 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:47.263511  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49615.5us
I20260709 12:24:47.266508  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.271072  1806 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:24:47.278194  1872 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:47.231346  8133 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:47.254006  8370 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.308s	user 0.002s	sys 0.000s
I20260709 12:24:47.239192  8367 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.285336  1804 pending_rounds.cc:85] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 7
I20260709 12:24:47.302598  1804 pending_rounds.cc:107] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 15.8
I20260709 12:24:47.303068  1804 raft_consensus.cc:2889] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:47.297142  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.294659  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:47.306751  8375 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:24:47.311203  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.312144  1724 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:47.295214  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.319281  8000 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:47.291235  7987 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:47.327519  8385 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:47.330010  8284 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:47.333784  8231 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:47.337194  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.330535  8284 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:47.339854  8284 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:47.340507  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2528) took 1226 ms. Trace:
I20260709 12:24:47.341027  1732 rpcz_store.cc:276] 0709 12:24:46.114260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.122813 (+  8553us) service_pool.cc:224] Handling call
0709 12:24:46.138253 (+ 15440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:46.491665 (+353412us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.340208 (+848543us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b54002b7ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:47.340438 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":811080064}
I20260709 12:24:47.338716  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:47.345602  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.319316  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.348722  1805 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:47.349892  1806 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:47.287837  8324 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:47.345212  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.363442  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.363929  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.365032  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.365051  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:47.371160  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.371104  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.374027  1800 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:47.341372  8284 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:47.384819  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.385190  1806 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:47.385304  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.284642  8304 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.079s
I20260709 12:24:47.387459  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:47.387701  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 241185us
I20260709 12:24:47.348049  8340 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.393208  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.396338  1872 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:24:47.399986  1725 raft_consensus.cc:2410] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:24:47.364135  7611 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:47.411018  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.413962  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:47.414003  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.414649  2419 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:47.413676  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.418329  1800 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 13.
I20260709 12:24:47.326529  8296 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.388769  8324 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:47.427279  1732 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:47.427750  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.351943  8367 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.434513  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:47.406327  8297 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.428375  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.413584  8270 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 12:24:47.440961  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.288118  8357 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.412326  8340 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:47.453474  8340 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:47.454514  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.456554  1732 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:47.460309  1724 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:47.461222  1730 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:47.465085  1875 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:47.467428  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.392736  8347 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:24:47.426635  8184 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:47.491285  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.492373  1876 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
W20260709 12:24:47.478318  1770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 2.177s (SENT) [suppressed 39 similar messages]
I20260709 12:24:47.493916  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:47.467226  8123 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:47.504870  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:47.508628  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.512295  1723 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:47.512611  1881 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:47.513499  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.430759  7611 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.529804  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:47.530637  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:47.495651  8184 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:47.536056  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.457293  8398 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.082s
I20260709 12:24:47.540045  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.543478  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.554148  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.557070  1724 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:47.535197  8231 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:47.479570  8297 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.570407  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:47.565297  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 170862us
W20260709 12:24:47.552153  8390 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:24:47.556861  8325 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:47.575989  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:47.506296  8270 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:24:47.577788  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.579123  1872 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:47.582033  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.558164  8310 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:47.567296  8255 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.591s	user 0.014s	sys 0.000s
I20260709 12:24:47.560961  8312 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:47.592146  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:47.609241  1799 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:24:47.611963  2415 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:47.577356  8398 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:47.613863  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.614061  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.614374  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:47.551111  8324 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:47.595625  8347 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:24:47.627363  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.600350  8312 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.633510  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.611615  8352 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.635896  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.646721  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.619623  8000 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:24:47.634532  7988 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:47.665813  8000 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:47.616150  8398 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:47.618444  8255 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:47.664644  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:47.666383  7988 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.669787  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.619403  8175 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:24:47.620708  8214 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:47.635342  8312 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:47.682751  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:47.688417  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 232917us
W20260709 12:24:47.688548  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.672221  8352 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.673655  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.700696  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.671576  8380 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:47.666407  8000 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:47.664069  8133 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:47.684168  8214 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:47.706735  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.663389  8207 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.712075  1805 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:47.673214  8403 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.697916  8318 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:47.725219  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 125842us
I20260709 12:24:47.728147  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:47.740070  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.740631  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.701944  8029 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:47.737285  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:47.732919  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 147637us
I20260709 12:24:47.706103  1724 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:47.712627  8123 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:47.750180  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.734207  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.745155  7859 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:47.767882  1723 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:24:47.753116  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:47.806421  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58865.5us
I20260709 12:24:47.730376  8174 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:47.813621  1727 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:47.748260  7987 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:47.821213  1805 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:47.745968  7988 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:47.823670  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:47.828447  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75410.9us
I20260709 12:24:47.803113  8347 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:47.747611  8397 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 12:24:47.812464  8406 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:47.819416  8307 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.743s	user 0.006s	sys 0.017s
W20260709 12:24:47.826583  8404 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:24:47.826164  8296 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:47.813341  7611 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:47.831123  8399 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.294s	user 0.002s	sys 0.000s
I20260709 12:24:47.837065  8123 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:47.862989  8123 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.863456  8123 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:47.820499  8417 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.058s
I20260709 12:24:47.867753  8417 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:47.868216  8417 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:47.870127  8161 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.930s	user 0.012s	sys 0.001s
I20260709 12:24:47.870492  8352 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:47.871001  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.872246  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.873276  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.873888  1727 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:47.836393  8297 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:47.876010  8408 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:24:47.876874  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.878059  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:47.844751  8405 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:24:47.745965  8403 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.743224  8398 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:47.847760  8343 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.992s	user 0.013s	sys 0.002s
I20260709 12:24:47.848701  8347 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.744084  8207 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.759882  8029 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:47.830281  8358 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:24:47.852730  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:47.854969  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:47.844813  8341 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:47.849959  7743 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:47.903692  7743 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:47.904786  7743 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:47.905889  7743 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:47.857434  7660 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:47.858275  8407 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260709 12:24:47.858541  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:47.880925  1802 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:47.874565  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.852461  8406 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:47.885795  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:47.901443  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48829.2us
I20260709 12:24:47.908924  1807 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:47.902726  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.929015  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:47.909539  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:47.932451  1805 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:47.932324  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.911504  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.933974  1732 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:47.934082  1802 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:47.912421  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.920195  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.935712  1727 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:47.936363  1723 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 21 because replica has last-logged OpId of term: 19 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:24:47.906773  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:47.990375  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.914294  8029 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:47.940937  8408 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:24:48.010861  8297 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.028474  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:47.912822  8207 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.021456  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:47.852586  8307 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.023038  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:48.042196  1723 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 20 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:24:47.907696  7660 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:48.024849  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:48.053370  1727 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:48.019299  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:48.032004  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:48.034312  1879 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:48.016546  8312 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.035302  1872 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:48.035754  8403 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:47.996935  8318 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:48.060734  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:48.052158  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:48.061558  1802 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:47.941401  8270 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
I20260709 12:24:48.058563  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:47.910763  8214 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.061746  1726 raft_consensus.cc:2376] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16. Re-sending same reply.
I20260709 12:24:48.071177  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:48.075428  1802 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:48.076444  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:47.907044  8341 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:47.919587  8343 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:48.065565  2384 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.006s	sys 0.007s
W20260709 12:24:48.078480  2384 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.568s	user 0.006s	sys 0.007s
W20260709 12:24:48.080374  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.083900  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:48.087850  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:48.065608  8321 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.007s	sys 0.005s
W20260709 12:24:48.093333  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.060647  8318 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.088879  1802 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:48.099404  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:48.100463  1723 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
W20260709 12:24:48.102874  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.104234  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.105470  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:48.105909  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:48.107388  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:48.108388  8307 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.071060  8338 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:48.109230  8338 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.110175  8318 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:48.098253  8396 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:48.112212  8396 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.106040  2414 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:48.121409  8341 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.122632  1803 raft_consensus.cc:2368] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17.
I20260709 12:24:48.122947  1798 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:48.124286  8343 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.114478  8396 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.127303  8214 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:48.128868  8318 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.099956  1802 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:48.130641  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:48.131294  1802 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:48.131716  1798 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:48.077907  7660 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:48.139528  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:48.096544  8321 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.008s	sys 0.005s
I20260709 12:24:48.145174  2414 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:48.136729  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:48.050387  8297 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.149680  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:48.110693  8000 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.146713  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:48.146991  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:48.154954  1804 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:48.148685  1805 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:48.160714  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.147734  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.083096  8264 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:48.163764  1874 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:48.164644  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:48.164908  1805 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:48.165421  2418 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:48.166184  1730 raft_consensus.cc:2368] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
I20260709 12:24:48.161540  2411 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:48.168165  2419 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
W20260709 12:24:48.169992  1770 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.579s (SENT)
I20260709 12:24:48.171561  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.173117  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.174402  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 13 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.175714  8312 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:48.176983  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:48.176879  7660 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.177644  2417 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:48.177863  8297 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.183351  8406 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.183673  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.186472  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:48.187342  2412 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:48.188257  8367 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:48.189286  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.190091  2412 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:48.195484  8402 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.187194  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:48.200302  2417 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:48.203702  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:48.201988  1807 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:48.207806  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:48.210005  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48256.1us
W20260709 12:24:48.211436  1770 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.591s (SENT)
I20260709 12:24:48.196169  8402 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:48.213910  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66308.5us
I20260709 12:24:48.217875  1804 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:48.222664  8133 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:48.163404  8396 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.223815  8133 consensus_queue.cc:1243] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:24:48.224331  8417 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.225917  1877 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:24:48.227166  8000 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.228688  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2415) took 1338 ms. Trace:
I20260709 12:24:48.229157  1799 rpcz_store.cc:276] 0709 12:24:46.890059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.890263 (+   204us) service_pool.cc:224] Handling call
0709 12:24:46.920910 (+ 30647us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.609768 (+688858us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.228413 (+618645us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b5400133154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:48.228621 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1581839744}
I20260709 12:24:48.230973  7859 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:48.232296  8402 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:48.237332  1799 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:48.241128  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.242172  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.240906  1697 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.263409  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:48.236049  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.264833  1728 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:48.274260  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.280439  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.283398  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:48.287688  1802 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:48.288888  1804 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [16.9-18.10]
I20260709 12:24:48.294031  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:48.296459  1728 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:48.295332  1726 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:48.269739  8022 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:48.239830  7987 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:48.243872  8380 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.299985  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:48.302630  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 91211.5us
I20260709 12:24:48.301190  1799 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:24:48.300002  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 64031.2us
I20260709 12:24:48.300467  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:48.312975  1802 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:48.283787  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:48.284006  8184 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.732s	user 0.013s	sys 0.000s
W20260709 12:24:48.315950  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.270354  8000 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:48.198365  8264 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.334326  1799 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:48.224037  8396 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.350816  1725 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:48.265993  8307 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 12:24:48.351675  1728 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 9. Preceding OpId from leader: term: 21 index: 9. (term mismatch)
I20260709 12:24:48.302729  8419 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.302284  8175 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.277235  7859 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.063s
I20260709 12:24:48.355161  1728 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 8
I20260709 12:24:48.367655  1728 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 19.9
I20260709 12:24:48.368016  1728 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:48.321002  8340 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.333784  8133 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.373011  1802 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
W20260709 12:24:48.376266  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 92515.2us
I20260709 12:24:48.350243  8022 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:48.298974  8425 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:48.328715  8400 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.335950  8324 raft_consensus.cc:2764] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260709 12:24:48.373970  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 74080.3us
I20260709 12:24:48.363585  1727 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:48.384511  1728 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:48.386199  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.397303  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:48.335628  8184 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.403399  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.405505  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2500) took 1074 ms. Trace:
I20260709 12:24:48.408357  1697 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.413134  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:48.411899  1801 rpcz_store.cc:276] 0709 12:24:47.330736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.384754 (+ 54018us) service_pool.cc:224] Handling call
0709 12:24:47.402725 (+ 17971us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.408167 (+  5442us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:47.414677 (+  6510us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.234356 (+819679us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b18003a6c00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:24:48.405101 (+170745us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.405168 (+    67us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:48.405234 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.405307 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.405443 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25116,"prepare.queue_time_us":2316,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":29572,"spinlock_wait_cycles":35170048,"thread_start_us":145314,"threads_started":1}
W20260709 12:24:48.428467  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47241us
I20260709 12:24:48.430612  1728 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:48.431165  1728 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:48.432505  8436 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.073s	user 0.001s	sys 0.001s
I20260709 12:24:48.432072  1728 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.433633  1728 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:48.352813  8423 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.075s
I20260709 12:24:48.383116  8175 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
W20260709 12:24:48.411111  8284 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.021s	user 0.016s	sys 0.000s
I20260709 12:24:48.406391  8174 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:48.442584  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.409615  7859 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:48.412334  7987 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.377856  8340 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.459033  2412 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: [13.10-15.11]
I20260709 12:24:48.443264  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:48.370371  8422 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.069s
I20260709 12:24:48.435902  7988 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.470654  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2509) took 1028 ms. Trace:
I20260709 12:24:48.472484  8448 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:48.494623  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51465.3us
I20260709 12:24:48.496084  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.494230  1807 rpcz_store.cc:276] 0709 12:24:47.441855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.205183 (+763328us) service_pool.cc:224] Handling call
0709 12:24:48.205588 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:48.277137 (+ 71549us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:48.277174 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:48.281596 (+  4422us) log.cc:844] Serialized 26 byte log entry
0709 12:24:48.405498 (+123902us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:48.414862 (+  9364us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.415098 (+   236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.415172 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.470383 (+ 55211us) raft_consensus.cc:1685] finished
0709 12:24:48.470406 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.470563 (+   157us) inbound_call.cc:177] Queueing success response
0709 12:24:48.473905 (+  3342us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b48001c0158. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
Metrics: {"mutex_wait_us":118375,"prepare.queue_time_us":9088,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":71831,"raft.run_cpu_time_us":3581,"raft.run_wall_time_us":76860,"spinlock_wait_cycles":6505344,"thread_start_us":68026,"threads_started":1,"wal-append.queue_time_us":151165}
I20260709 12:24:48.475638  8429 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:48.475138  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.416987  8400 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.480559  8184 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.428946  8133 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:48.488857  1801 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:48.509835  8307 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:48.511979  2418 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:48.491458  1728 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
W20260709 12:24:48.447533  8123 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.584s	user 0.013s	sys 0.000s
I20260709 12:24:48.513809  8123 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:48.444721  8324 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:48.516533  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:48.517256  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2507) took 1076 ms. Trace:
I20260709 12:24:48.499363  1729 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:48.446148  7859 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:48.457545  8264 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:48.461506  8318 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.515311  1723 raft_consensus.cc:2368] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
I20260709 12:24:48.510198  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:48.487493  8022 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.523985  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:48.508028  8284 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 12:24:48.518024  1805 rpcz_store.cc:276] 0709 12:24:47.441288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.165716 (+724428us) service_pool.cc:224] Handling call
0709 12:24:48.197229 (+ 31513us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.219596 (+ 22367us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:48.312805 (+ 93209us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.516499 (+203694us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.516713 (+   214us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.516765 (+    52us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:48.516826 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.516906 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.517191 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7401,"prepare.queue_time_us":54335,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":12467,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":119,"thread_start_us":130630,"threads_started":1}
I20260709 12:24:48.506590  8422 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:48.458667  8415 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:24:48.528728  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:48.504029  8429 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:48.519131  7859 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:48.525605  7743 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.619s	user 0.014s	sys 0.000s
I20260709 12:24:48.510172  8406 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:24:48.525081  7660 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:48.524441  8324 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:48.594832  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66135.5us
I20260709 12:24:48.595463  1876 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:48.537062  8423 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.598239  8123 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:48.598671  8437 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:48.602828  1728 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:48.593611  8393 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.606405  8430 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:48.605336  7660 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:48.608119  8400 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:48.620827  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 145771us
I20260709 12:24:48.610146  8396 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.615448  8393 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.627183  8324 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.622687  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.610368  7743 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:48.636528  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:48.624167  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.609303  8437 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.642686  8437 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:48.643388  8393 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.649030  8123 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:48.649799  8123 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:48.651678  8123 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.653019  8340 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.660923  8133 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:48.616194  8430 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.664987  7988 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:48.678090  7988 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:48.679220  7988 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.680212  7859 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.666615  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:48.682956  1802 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:48.623731  8439 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:48.686681  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2431) took 1316 ms. Trace:
W20260709 12:24:48.669726  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45666.1us
I20260709 12:24:48.688912  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.672670  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:48.669023  1805 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260709 12:24:48.690351  1805 pending_rounds.cc:85] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 10
I20260709 12:24:48.690289  1801 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:48.691054  1805 pending_rounds.cc:107] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 12:24:48.691592  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:48.673977  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.691742  1805 raft_consensus.cc:2889] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:48.697165  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.687088  1806 rpcz_store.cc:276] 0709 12:24:47.376219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.402664 (+ 26445us) service_pool.cc:224] Handling call
0709 12:24:47.403056 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:47.414710 (+ 11654us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:47.414753 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:47.433344 (+ 18591us) log.cc:844] Serialized 26 byte log entry
0709 12:24:47.576635 (+143291us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:47.576663 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.576710 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.668242 (+1091532us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b24002b7918. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:24:48.686347 (+ 18105us) raft_consensus.cc:1685] finished
0709 12:24:48.686371 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.686601 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1545,"raft.queue_time_us":41011,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":52872448,"thread_start_us":136027,"threads_started":1,"wal-append.queue_time_us":159156}
W20260709 12:24:48.671476  8388 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.285s	user 0.026s	sys 0.019s
W20260709 12:24:48.771142  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74095.6us
I20260709 12:24:48.669749  8340 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:48.669169  8430 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.702497  8352 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:48.775431  1801 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
W20260709 12:24:48.785171  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.778644  1802 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:24:48.695753  8133 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:48.772046  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.782032  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:48.670267  7743 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.695943  8318 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:48.746865  7859 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.747365  8439 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:48.762907  8341 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:48.792093  8123 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:48.790835  8022 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:48.670141  7660 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.792766  8430 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:48.824553  8123 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:48.830185  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 58212.8us
I20260709 12:24:48.831753  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.793651  8133 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.840384  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.823755  8341 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:48.845104  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.818773  8318 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.844283  8022 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:48.845995  8439 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:48.835055  8340 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:24:48.863704  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:48.857571  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2432) took 1443 ms. Trace:
I20260709 12:24:48.833200  8430 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.873310  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.875275  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:48.866140  1800 rpcz_store.cc:276] 0709 12:24:47.414059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.434036 (+ 19977us) service_pool.cc:224] Handling call
0709 12:24:47.454640 (+ 20604us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:47.455195 (+   555us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:47.608616 (+153421us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.682165 (+ 73549us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.682351 (+   186us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:47.903832 (+221481us) spinlock_profiling.cc:243] Waited 1.71 ms on lock 0x7b1800418960. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:48.674317 (+770485us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b1800418960. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:48.750407 (+ 76090us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b1800418960. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a
0709 12:24:48.751536 (+  1129us) log.cc:844] Serialized 52 byte log entry
0709 12:24:48.751867 (+   331us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:48.847046 (+ 95179us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.847921 (+   875us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.849417 (+  1496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.849508 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.857316 (+  7808us) raft_consensus.cc:1685] finished
0709 12:24:48.857344 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.857512 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56200,"prepare.queue_time_us":14057,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":105,"raft.queue_time_us":120182,"raft.run_cpu_time_us":694,"raft.run_wall_time_us":37902,"spinlock_wait_cycles":38613248,"thread_start_us":304620,"threads_started":3,"wal-append.queue_time_us":128273}
I20260709 12:24:48.868100  8406 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.884840  1730 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:48.871279  8415 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:48.879755  8423 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:48.892009  1770 leader_election.cc:400] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
I20260709 12:24:48.894133  1770 leader_election.cc:403] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:24:48.896178  8341 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:48.899982  8340 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
W20260709 12:24:48.902614  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.881186  8022 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:48.883026  8307 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:24:48.911192  2418 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:48.915462  8456 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:24:48.908546  7743 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.907562  8184 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:48.911269  8434 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:48.911628  8349 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:48.912119  8406 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.937071  2414 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:24:48.935717  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.944175  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.936321  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.941707  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.942862  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.922726  8123 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.948035  2410 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:48.952401  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:48.942001  8349 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:48.955480  8458 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.073s	user 0.001s	sys 0.001s
I20260709 12:24:48.948554  2419 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:48.958837  2412 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:48.934770  8338 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.960829  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:48.943053  8459 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:24:48.966737  8437 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:48.944908  8434 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.944675  8184 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.971277  8184 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:48.971423  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2561) took 1047 ms. Trace:
I20260709 12:24:48.972856  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:48.972003  1727 rpcz_store.cc:276] 0709 12:24:47.923767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.409337 (+485570us) service_pool.cc:224] Handling call
0709 12:24:48.436084 (+ 26747us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.528101 (+ 92017us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:48.596414 (+ 68313us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.684377 (+ 87963us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.684447 (+    70us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:48.684514 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.971109 (+286595us) spinlock_profiling.cc:243] Waited 91.7 ms on lock 0x7b54003cc8d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:48.971168 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.971345 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":68287,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":138,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":210866816,"thread_start_us":98784,"threads_started":2,"wal-append.queue_time_us":189598}
I20260709 12:24:48.974237  8437 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:48.974690  8437 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.975078  8437 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:48.964396  7743 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.003s
I20260709 12:24:48.979324  7743 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:48.979759  7743 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:48.980758  7743 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:48.981525  8318 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:48.982364  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.982942  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.982286  7743 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:48.984638  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.953233  8406 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.005184  8406 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:48.987376  8400 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:49.006522  8400 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.009816  8400 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:48.999310  8349 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:49.011463  8464 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:24:49.011530  8349 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.028849  8332 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:49.029534  8332 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:49.033980  8417 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:49.037966  8424 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.002s	user 0.002s	sys 0.002s
W20260709 12:24:49.040097  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.043649  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:49.039532  8423 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:49.044178  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:49.044821  1877 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:49.046106  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:49.046886  1881 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:49.054131  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2433) took 1595 ms. Trace:
I20260709 12:24:49.058586  8207 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:49.059195  8207 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:49.060284  8207 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.054689  1804 rpcz_store.cc:276] 0709 12:24:47.458642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.218720 (+760078us) service_pool.cc:224] Handling call
0709 12:24:48.288491 (+ 69771us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:48.289430 (+   939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.053894 (+764464us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b540013a950. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:49.054057 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37190016}
I20260709 12:24:49.030647  8332 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.063575  1881 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:24:49.068451  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:49.069020  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:49.069591  1805 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.069856  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.070374  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.071159  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.071218  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.072072  1806 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:49.072158  1807 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:49.061733  8207 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:49.071192  1802 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:49.079846  1805 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:49.079949  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.080838  1804 raft_consensus.cc:2376] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20. Re-sending same reply.
I20260709 12:24:49.083160  8192 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.083791  8192 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.086097  8192 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:49.084649  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:49.094832  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:49.094900  1877 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:24:49.100630  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.101408  1802 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:49.102926  1807 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:24:49.107175  8380 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260709 12:24:49.107261  8425 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:49.110378  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:49.073482  8349 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:49.109848  8425 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:49.113191  8264 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:49.116849  8425 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.110330  8184 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:49.109078  8192 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:49.123229  8349 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.128288  8349 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:49.129437  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2554) took 1398 ms. Trace:
I20260709 12:24:49.129971  1725 rpcz_store.cc:276] 0709 12:24:47.731531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.733371 (+  1840us) service_pool.cc:224] Handling call
0709 12:24:47.733742 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.360708 (+626966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.129110 (+768402us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b54002b84d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:49.129360 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1384811776}
W20260709 12:24:49.116225  8443 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.675s	user 0.000s	sys 0.004s
I20260709 12:24:49.138857  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.135459  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.140033  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.140622  1805 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:49.141208  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2556) took 1409 ms. Trace:
I20260709 12:24:49.136749  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.141593  1731 rpcz_store.cc:276] 0709 12:24:47.732261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.734907 (+  2646us) service_pool.cc:224] Handling call
0709 12:24:47.735292 (+   385us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:47.824481 (+ 89189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:47.824530 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:47.825319 (+   789us) log.cc:844] Serialized 52 byte log entry
0709 12:24:48.300270 (+474951us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:48.336168 (+ 35898us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.353129 (+ 16961us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.480446 (+127317us) spinlock_profiling.cc:243] Waited 25.9 ms on lock 0x7b540030a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:48.480517 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.082441 (+601924us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:49.127412 (+ 44971us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b1800443320. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:49.141000 (+ 13588us) raft_consensus.cc:1685] finished
0709 12:24:49.141020 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.141155 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18430,"prepare.queue_time_us":35764,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":5517,"raft.queue_time_us":64277,"raft.run_cpu_time_us":355,"raft.run_wall_time_us":22545,"spinlock_wait_cycles":78709760,"thread_start_us":446253,"threads_started":1,"wal-append.queue_time_us":540137}
I20260709 12:24:49.143325  1807 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:49.138857  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.144645  1802 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:49.145200  8398 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:49.146708  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.147418  8470 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:49.147558  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.147996  8470 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.148404  8400 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.149034  8348 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:49.149727  8348 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.150301  8429 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.151820  8429 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:49.152208  8348 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:49.152735  8412 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:49.152936  8348 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:24:49.154186  8412 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:49.137058  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.148880  8400 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.150225  8470 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:49.138056  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:49.156599  8400 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:49.156972  1731 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:24:49.158707  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:49.154423  8348 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:49.161643  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.165292  1732 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:49.167666  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.169384  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.169498  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:49.170596  1727 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.174932  1725 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
W20260709 12:24:49.183234  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:49.187242  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:49.188131  1845 connection.cc:256] RPC call timeout handler was delayed by -1.3917042269999911s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1515s for a response
W20260709 12:24:49.198983  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:49.200158  8471 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
W20260709 12:24:49.200441  1845 connection.cc:256] RPC call timeout handler was delayed by -1.3584042269999372s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
W20260709 12:24:49.201452  1845 connection.cc:256] RPC call timeout handler was delayed by -1.3557042269999329s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20260709 12:24:49.201941  1845 connection.cc:256] RPC call timeout handler was delayed by -1.3062042269998528s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16100000000000003s for a response
W20260709 12:24:49.202411  1845 connection.cc:256] RPC call timeout handler was delayed by -1.2387042269997437s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1685s for a response
W20260709 12:24:49.202891  1845 connection.cc:256] RPC call timeout handler was delayed by -1.2009042270001373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17270000000000002s for a response
W20260709 12:24:49.203392  1845 connection.cc:256] RPC call timeout handler was delayed by -1.176604227000098s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1754s for a response
W20260709 12:24:49.203863  1845 connection.cc:256] RPC call timeout handler was delayed by -1.1478042270000515s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1786s for a response
I20260709 12:24:49.204730  8412 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:49.205523  8412 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.210520  8412 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:49.165043  8348 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:49.212225  8348 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.213927  8348 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:49.215216  1727 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:49.216905  8437 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.221426  8437 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:49.216879  1845 leader_election.cc:400] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14.
W20260709 12:24:49.224313  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2426) took 2134 ms (client timeout 1610 ms). Trace:
W20260709 12:24:49.224992  1803 rpcz_store.cc:269] 0709 12:24:47.089727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.097834 (+  8107us) service_pool.cc:224] Handling call
0709 12:24:49.224014 (+2126180us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400132254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:49.224214 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2353916160}
I20260709 12:24:49.227149  1845 leader_election.cc:403] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:24:49.232086  8430 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:49.233829  8425 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:49.235987  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:49.239137  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:49.244890  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.245970  1727 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 21 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
I20260709 12:24:49.249049  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:49.249730  2417 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.162065  8042 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:49.273229  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.254036  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:49.298532  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:49.299363  1725 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:49.299434  1724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3421) took 1782 ms (client timeout 1664 ms). Trace:
W20260709 12:24:49.300264  1724 rpcz_store.cc:269] 0709 12:24:47.474204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.513194 (+ 38990us) service_pool.cc:224] Handling call
0709 12:24:49.255912 (+1742718us) spinlock_profiling.cc:243] Waited 42.8 ms on lock 0x7b54002aa654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:49.255993 (+    81us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98510720}
I20260709 12:24:49.302527  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:49.303184  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:49.303481  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.303951  2411 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:49.304675  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:49.305608  1877 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 14: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:49.303268  1881 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:49.307883  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.307989  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:49.309111  1728 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:49.309737  1878 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.310668  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:49.310976  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:49.311631  1876 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:49.312171  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:49.312897  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.312896  2412 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:49.311753  2414 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:49.325642  2417 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:49.325496  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:49.328250  1723 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:49.301179  7611 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.256773  8310 raft_consensus.cc:2764] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:24:49.303743  8304 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:24:49.340166  1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3426) took 1658 ms (client timeout 2132 ms). Trace:
I20260709 12:24:49.340399  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1236) took 1936 ms. Trace:
W20260709 12:24:49.346633  1729 rpcz_store.cc:269] 0709 12:24:47.682080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.682379 (+   299us) service_pool.cc:224] Handling call
0709 12:24:49.339878 (+1657499us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b540030be54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:49.340076 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1748048128}
I20260709 12:24:49.301975  8430 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 13. Current term is 14.
I20260709 12:24:49.345549  1726 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:24:49.346803  2415 rpcz_store.cc:276] 0709 12:24:47.404044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.404704 (+   660us) service_pool.cc:224] Handling call
0709 12:24:47.405057 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.616758 (+211701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.340050 (+1723292us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b54000e6d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:49.340318 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":468756352}
I20260709 12:24:49.355787  1727 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:49.300535  8042 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.360162  1881 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
W20260709 12:24:49.363754  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 147119us
I20260709 12:24:49.364907  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:49.356709  8304 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:49.370563  8042 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:49.375681  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.242939  8264 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:49.384480  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.385316  1727 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:24:49.389341  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.391047  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.391741  1728 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:49.392397  1723 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:49.359153  7611 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.394992  1799 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:49.371918  8401 raft_consensus.cc:2764] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260709 12:24:49.398885  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2434) took 1926 ms (client timeout 1924 ms). Trace:
W20260709 12:24:49.399478  1798 rpcz_store.cc:269] 0709 12:24:47.468827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.041118 (+572291us) service_pool.cc:224] Handling call
0709 12:24:49.394553 (+1353435us) spinlock_profiling.cc:243] Waited 57.7 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:49.394730 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":132682496}
I20260709 12:24:49.400660  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:49.404430  8264 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.411053  1874 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:49.411639  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:49.412341  1875 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
W20260709 12:24:49.415287  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50495.2us
I20260709 12:24:49.417112  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:49.419101  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.423146  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.423329  8042 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:24:49.424991  8304 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.434165  1725 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
W20260709 12:24:49.436398  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.401579  1872 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:49.440042  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.444334  1725 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:49.444902  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.445892  1798 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:49.450379  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.451156  1727 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:49.452512  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:49.382416   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:49.457686  1806 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:49.459345  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:49.455365  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:49.464709  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:49.464835  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.474725  7611 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:49.476879  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:49.490643  8332 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.492331  8133 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:49.500105  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2518) took 1898 ms. Trace:
I20260709 12:24:49.500694  1799 rpcz_store.cc:276] 0709 12:24:47.601928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.360858 (+758930us) service_pool.cc:224] Handling call
0709 12:24:49.394737 (+1033879us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.395589 (+   852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.499803 (+104214us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54002f2950. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:49.500009 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2376377728}
I20260709 12:24:49.504082  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:49.505040  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:49.505865  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:49.506711  8474 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:49.507145  8474 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:49.507491  8474 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.510998  8396 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:49.511476  8396 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.511886  8396 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:49.513123  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57901us
I20260709 12:24:49.513605  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:49.516340  1770 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 2.145s (SENT)
I20260709 12:24:49.516825  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:49.519320  1878 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:49.519964  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:49.520993  1876 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:49.521886  1723 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:24:49.522588  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:49.512565  8042 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:49.531054  7945 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:49.537567  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:49.511499  8192 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:49.538549  1877 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:49.528627  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:49.452524  8410 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 12:24:49.547271  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:49.540570  1804 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:49.537572  7945 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.522008  8296 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:49.462356  8401 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 12:24:49.564716  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:49.532445  7987 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:49.568264  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51972.4us
I20260709 12:24:49.560133  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:49.562644  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:49.570677  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:49.575268  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.539549  8416 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:49.527817  8184 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.040s
I20260709 12:24:49.582602  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:49.570449  8296 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:49.586087  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:49.566094  7945 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.567526  8207 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:49.570472  8429 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:24:49.570832  7743 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.592090  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:49.603044  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.574057  8437 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:49.615103  1729 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 12:24:49.596699  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.555709  8000 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.034s
I20260709 12:24:49.613015  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:49.617514  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:49.618038  1727 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:49.619251  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:49.623186  1728 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.586040  8352 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:49.580920  8410 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:49.591958  8417 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:49.630849  8410 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.635289  8417 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.635640  8410 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.635835  8417 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:49.591744  8296 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:49.636734  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.637532  2416 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:49.608150  7611 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:49.631841  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:49.583462  8412 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:24:49.608795  8175 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:24:49.624356  8123 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:49.655136  8123 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:49.655797  8123 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:49.630539  8352 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:49.659169  8352 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:49.589166  8449 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:49.664861  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:49.606896  8207 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:49.637408  1727 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:24:49.592271  8174 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 12:24:49.651141  2414 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:49.668004  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:49.670890  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3441) took 1805 ms. Trace:
I20260709 12:24:49.580965  7987 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:49.706732  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 108510us
W20260709 12:24:49.712044  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 191242us
I20260709 12:24:49.743829  1728 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:49.675897  1726 rpcz_store.cc:276] 0709 12:24:47.865759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.317402 (+451643us) service_pool.cc:224] Handling call
0709 12:24:49.340828 (+1023426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.349311 (+  8483us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.670569 (+321258us) spinlock_profiling.cc:243] Waited 987 ms on lock 0x7b540030be50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:49.670803 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2270648576}
W20260709 12:24:49.777513  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:49.784673  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 119941us
I20260709 12:24:49.785542  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:49.788777  2416 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:49.640936  8470 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:49.627355  8416 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:24:49.620113  7743 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.669564  8398 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:49.795112  1845 leader_election.cc:400] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17.
I20260709 12:24:49.799903  1845 leader_election.cc:403] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Cancelling election due to peer responding with higher term
I20260709 12:24:49.674144  7611 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:49.793025  8400 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:49.846799  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.856447  1728 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:24:49.802126  8349 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:49.834488  7913 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.860277  8474 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.869972  8474 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:49.797816  8133 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:49.884708  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 270713us
W20260709 12:24:49.858089  8480 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.064s	user 0.001s	sys 0.001s
I20260709 12:24:49.843647  8401 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election lost for term 16. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17.
I20260709 12:24:49.865365  8192 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:49.904577  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.905284  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.860633  8349 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.909042  7611 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:49.805476  8470 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:49.912226  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 117232us
I20260709 12:24:49.913875  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.914134  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:49.852201  8184 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 12:24:49.926849  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:49.931640  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:49.932556  1877 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:49.933816  8380 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:49.843308  8207 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.935523  8207 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:49.938984  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:49.939473  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:49.939922  1877 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:24:49.940213  1731 raft_consensus.cc:2368] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20.
I20260709 12:24:49.941951  8380 consensus_queue.cc:1243] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:24:49.943041  8380 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:49.943550  8380 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:49.937930  8487 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:49.944694  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:49.945518  1877 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
W20260709 12:24:49.949404  8479 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.299s	user 0.000s	sys 0.001s
I20260709 12:24:49.956517  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.957751  1728 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:49.916358  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:49.962826  1877 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 14 index: 12. (term mismatch)
I20260709 12:24:49.963826  1877 pending_rounds.cc:85] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 11
I20260709 12:24:49.964362  1877 pending_rounds.cc:107] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 13.12
I20260709 12:24:49.966006  1877 raft_consensus.cc:2889] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:49.908659  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 123612us
I20260709 12:24:49.820135  8410 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.919297  8042 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:24:49.891788  7913 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:49.923089  7743 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:49.913671  8349 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:50.005537  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:50.030387  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:50.006121  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:50.031507  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:50.032035  1725 raft_consensus.cc:2376] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14. Re-sending same reply.
I20260709 12:24:50.013352  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:50.013151  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:50.002178  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 354183us
I20260709 12:24:49.896548  8133 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.010658  8042 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:50.036931  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:50.010116  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:50.075294  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.031162  1731 raft_consensus.cc:2376] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14. Re-sending same reply.
I20260709 12:24:50.040834  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:50.045404  8111 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:50.045720  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:50.037794  8000 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.960525  8416 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:24:50.102197  1724 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:24:50.053994  8192 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.103163  7987 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:50.104099  8000 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.106415  8000 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:50.107328  7987 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.107908  7987 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.113242  8417 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.121206  8417 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:50.094506  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48860.1us
I20260709 12:24:50.095093  8423 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:50.094717  8042 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:49.947528  8380 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.142176  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:50.122279  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.122824  8491 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.094077  7945 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.092072  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:50.156381  1728 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:24:50.143426  1770 leader_election.cc:400] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20.
I20260709 12:24:50.157079  1770 leader_election.cc:403] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
W20260709 12:24:50.145020  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52249.2us
I20260709 12:24:50.118140  1723 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:50.163373  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:50.149276  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 260250us
I20260709 12:24:50.169808  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:50.093762  8393 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:50.168531  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:50.172533  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:50.179288  1872 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:50.184013  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:50.185086  1732 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:50.185568  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:50.193240  8430 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:50.195194  8123 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.539s	user 0.005s	sys 0.008s
I20260709 12:24:50.195534  8400 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:50.196291  8400 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:50.195868  8123 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.196820  8400 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:50.204660  7945 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:50.206395  7987 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:50.207345  7987 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:50.207882  7987 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:50.208078  8486 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:50.208948  7987 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.124078  8434 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:50.175215  8491 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:50.178596  8296 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.542s	user 0.011s	sys 0.007s
I20260709 12:24:50.221004  2413 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:50.222452  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 79149.9us
W20260709 12:24:50.223187  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:50.225756  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:50.191874  8352 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.532s	user 0.013s	sys 0.000s
I20260709 12:24:50.168318  8493 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:50.178843  8393 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:50.231069  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:50.232245  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 339528us
I20260709 12:24:50.241585  1875 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:50.251308  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:50.252262  1880 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:50.211375  8482 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.310011  1875 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:50.217206  8423 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20.
W20260709 12:24:50.311421  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80485.4us
W20260709 12:24:50.316177  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.011s	sys 0.042s
W20260709 12:24:50.319609  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.311054  1872 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:50.341646  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:50.345069  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.011s	sys 0.043s
W20260709 12:24:50.345654  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 687976us
I20260709 12:24:50.346501  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:50.349412  2382 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:50.224314  8347 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:50.227461  7945 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:50.365605  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.373057  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:50.377065  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:50.337108  8493 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:50.384657  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:50.346575  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 161122us
I20260709 12:24:50.394762  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:50.412305  7945 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:50.420742  8412 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:50.421293  8412 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:50.421695  8412 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:50.385200  8123 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:50.423771  8425 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.189s	user 0.014s	sys 0.003s
I20260709 12:24:50.335110  8255 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:24:50.247355  8352 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.380759  8495 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:50.429827  8352 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:50.230291  8296 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:50.432262  8396 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.920s	user 0.012s	sys 0.000s
I20260709 12:24:50.432554  8491 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:50.432813  8396 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.434401  2415 raft_consensus.cc:2368] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17.
W20260709 12:24:50.381865  8332 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.890s	user 0.010s	sys 0.004s
W20260709 12:24:50.435962  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51370.8us
W20260709 12:24:50.436185  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64216us
I20260709 12:24:50.414259  7945 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:50.418427  8207 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:50.383018  8343 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:50.504925  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.351673  8482 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.537528  8296 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:50.379611  8417 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:50.556838  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 115593us
W20260709 12:24:50.636775  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 258452us
I20260709 12:24:50.426083  8393 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.387970  8423 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:50.637626  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 193119us
I20260709 12:24:50.653698  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:50.644749  1731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.008s	sys 0.011s
W20260709 12:24:50.662451  1731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.008s	sys 0.011s
W20260709 12:24:50.583652  8304 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.002s	sys 0.053s
I20260709 12:24:50.439258  8495 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:50.666102  7611 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:24:50.678237  1806 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:24:50.680091  1799 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:24:50.642131  8192 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.002s	sys 0.018s
I20260709 12:24:50.441869  7913 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:50.451388  7945 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:50.664031  1725 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.018s	sys 0.029s
W20260709 12:24:50.652806  8398 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.845s	user 0.015s	sys 0.001s
I20260709 12:24:50.691655  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:50.693011  1798 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 19 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:24:50.657136  8490 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.696890  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:50.448015  1880 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:24:50.703125  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:50.660166  8396 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:50.688997  1725 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.019s	sys 0.029s
W20260709 12:24:50.442435  8348 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.228s	user 0.014s	sys 0.005s
I20260709 12:24:50.690824  1799 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:50.693732  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:50.735690  1803 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:50.670629  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:50.679767  8393 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:50.695302  2385 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.016s	sys 0.025s
W20260709 12:24:50.742030  2385 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.016s	sys 0.025s
I20260709 12:24:50.679605  8111 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:50.702018  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:50.664139  8123 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:50.657367  8498 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:50.680104  8304 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.002s	sys 0.053s
I20260709 12:24:50.708385  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:50.755784  1807 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
W20260709 12:24:50.709681  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.008s	sys 0.008s
W20260709 12:24:50.756340  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.009s	sys 0.008s
W20260709 12:24:50.757078  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 716266us
I20260709 12:24:50.648602  8417 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.559729  8207 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:50.688800  8192 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.002s	sys 0.018s
I20260709 12:24:50.652335  8484 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
W20260709 12:24:50.682654  2384 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.002s	sys 0.014s
W20260709 12:24:50.781137  2384 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.003s	sys 0.014s
W20260709 12:24:50.708675  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.008s	sys 0.007s
W20260709 12:24:50.782373  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.009s	sys 0.007s
W20260709 12:24:50.783233  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 613465us
I20260709 12:24:50.739948  1805 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:50.656091  8255 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
W20260709 12:24:50.688997  1727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.011s	sys 0.039s
W20260709 12:24:50.825685  1727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.012s	sys 0.040s
I20260709 12:24:50.679149  8489 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 12:24:50.694816  7945 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:50.748116  8400 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.551s	user 0.012s	sys 0.000s
I20260709 12:24:50.749059  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:50.691452  8423 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:50.704463  8398 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:50.695854  7611 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:50.775126  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:50.738811  8486 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.795698  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:50.797071  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:50.708001  8490 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.807548  8482 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:50.751097  8497 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.772475  8498 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.812511  8417 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:50.753795  8111 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.823477  2414 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:50.780601  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:50.865535  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:50.840104  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 490670us
I20260709 12:24:50.780159  8000 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.779306  8352 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.921378  8000 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.841291  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:50.842183  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47967.1us
I20260709 12:24:50.916409  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:50.871201  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:50.927773  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3467) took 1024 ms. Trace:
I20260709 12:24:50.922098  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:50.962061  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:50.778635  8488 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.964928  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:50.848868  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:50.837057  1873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.885s	user 0.011s	sys 0.015s
W20260709 12:24:50.970002  1873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.018s	user 0.011s	sys 0.015s
I20260709 12:24:50.774796  8296 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.966521  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:50.976785  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:50.964291  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.008s	sys 0.003s
W20260709 12:24:50.978155  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.008s	sys 0.003s
W20260709 12:24:50.978552  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 583852us
I20260709 12:24:50.972771  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3461) took 1117 ms. Trace:
I20260709 12:24:50.980412  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:50.974224  1806 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:24:50.962440  1724 rpcz_store.cc:276] 0709 12:24:49.903148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.904938 (+  1790us) service_pool.cc:224] Handling call
0709 12:24:50.927469 (+1022531us) spinlock_profiling.cc:243] Waited 196 ms on lock 0x7b54002d0fd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:50.927677 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":450135424}
I20260709 12:24:50.848444  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:50.933861  8349 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.918s	user 0.009s	sys 0.000s
W20260709 12:24:50.975407  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53367.7us
I20260709 12:24:50.983265  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:50.984221  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:50.990008  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:50.990731  1806 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:50.710969  8493 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.980850  1726 rpcz_store.cc:276] 0709 12:24:49.855104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.855309 (+   205us) service_pool.cc:224] Handling call
0709 12:24:49.887351 (+ 32042us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:49.887732 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:49.887763 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:49.888253 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:24:50.385994 (+497741us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:50.972004 (+586010us) log.cc:844] Serialized 14 byte log entry
0709 12:24:50.972344 (+   340us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.972445 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:50.972501 (+    56us) raft_consensus.cc:1685] finished
0709 12:24:50.972518 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.972670 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":53865,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":58,"thread_start_us":865451,"threads_started":2,"wal-append.queue_time_us":869341}
I20260709 12:24:50.986326  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:50.973771  1798 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:51.029364  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 248865us
W20260709 12:24:50.837482  8474 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.002s	sys 0.024s
I20260709 12:24:50.939913  8509 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:50.987725  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:50.990116  1728 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:51.039492  1802 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:50.866466  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:50.865394  8304 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.924649  8510 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.047859  2414 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:50.922016  8111 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:50.971268  8042 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.008s	sys 0.038s
I20260709 12:24:50.924885  8406 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.921914  8352 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:50.984720  7987 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.030s	sys 0.053s
I20260709 12:24:51.038668  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:51.033459  8437 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.196s	user 0.010s	sys 0.018s
I20260709 12:24:50.976013  8255 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.063282  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:50.931000  8207 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:50.865504  8400 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.050309  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.034121  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3477) took 1019 ms. Trace:
I20260709 12:24:51.050946  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:50.984905  8430 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:50.969767  2410 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:51.054024  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:51.054692  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.055617  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:51.033104  8474 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.864s	user 0.002s	sys 0.024s
W20260709 12:24:51.026576  8412 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.571s	user 0.012s	sys 0.000s
I20260709 12:24:51.037303  8482 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:51.024654  8349 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.923683  8496 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.209s
W20260709 12:24:51.065325  8437 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.255s	user 0.011s	sys 0.019s
I20260709 12:24:51.090436  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:51.062290  7987 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.030s	sys 0.053s
I20260709 12:24:51.174125  1878 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:50.988842  8296 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.059063  8111 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.129868  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:51.175531  2414 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:51.176705  8503 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:24:51.177294  8494 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:51.178689  8508 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:51.179330  8508 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:51.180869  8296 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:51.181823  8505 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:24:51.181840  8111 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.171278  8412 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.169968  8482 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:51.140559  1802 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:51.047298  8434 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:24:51.172112  8521 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:24:51.061673  8509 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.969043  8486 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:50.977325  8497 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:50.967962  8423 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:51.186403  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 200167us
I20260709 12:24:51.038295  8398 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:51.044255  8123 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.051498  8304 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.112850  8310 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:51.150296  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.162367  1877 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:51.165145  1880 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:51.113104  8489 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.043s
W20260709 12:24:51.193703  8507 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:24:51.166200  1728 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:51.059695  8406 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.172185  2418 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:51.155998  8417 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:51.050195  8042 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.008s	sys 0.038s
I20260709 12:24:50.968075  7611 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.213129  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:51.192514  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.251356  1723 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:51.194250  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3464) took 1303 ms. Trace:
I20260709 12:24:51.220372  1806 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:51.183326  1798 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:51.221376  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.257973  1724 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:24:51.201540  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.259302  1802 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:51.203056  2414 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:51.185916  1879 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 11. (term mismatch)
I20260709 12:24:51.259758  1879 pending_rounds.cc:85] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 10
I20260709 12:24:51.259994  1879 pending_rounds.cc:107] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 12:24:51.260303  1879 raft_consensus.cc:2889] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:51.195552  8519 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.105000  8255 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:51.231343  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 81120us
I20260709 12:24:51.223974  8521 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:51.267551  1877 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:24:51.219918  8398 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:51.208367  8497 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.196194  8449 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:51.199456  8482 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:51.187433  8111 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.194355  8133 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.190194  8349 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.190301  8400 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:51.226086  7987 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.046s
I20260709 12:24:51.188156  8352 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.188449  8528 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.982286  8488 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.234453  8487 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:51.263934  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.233295  1880 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:24:51.187881  8512 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:51.206272  8509 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:51.265259  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52290.8us
I20260709 12:24:51.295480  8522 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:51.195412  8490 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.269783  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.275622  1805 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.352926  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.219363  8123 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.360739  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.357277  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:51.284464  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 211461us
I20260709 12:24:51.286720  1798 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:51.290270  1802 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:24:51.182757  7945 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.177909  8494 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.270984  8398 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:51.232079  8310 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.198220  8412 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.238193  8526 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:51.374352  8526 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.374737  8526 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.376544  8494 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.324704  1877 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 10. (term mismatch)
I20260709 12:24:51.349953  8487 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.378975  1877 pending_rounds.cc:85] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 9
I20260709 12:24:51.379338  1877 pending_rounds.cc:107] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 12:24:51.379570  1877 raft_consensus.cc:2889] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:51.379770  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.372525  8489 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:51.380560  8310 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.380939  1877 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:51.381078  8489 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.318090  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3343) took 1967 ms. Trace:
I20260709 12:24:51.382315  1873 rpcz_store.cc:276] 0709 12:24:49.374059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.384048 (+  9989us) service_pool.cc:224] Handling call
0709 12:24:49.400623 (+ 16575us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:49.402904 (+  2281us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:49.402941 (+    37us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:49.403811 (+   870us) log.cc:844] Serialized 78 byte log entry
0709 12:24:49.952342 (+548531us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b18003fb320. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:24:50.970451 (+1018109us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:51.301783 (+331332us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.302203 (+   420us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.302304 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.302362 (+    58us) raft_consensus.cc:1685] finished
0709 12:24:51.317777 (+ 15415us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.318007 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2821,"prepare.queue_time_us":181524,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":1161,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":17819,"spinlock_wait_cycles":43322752,"thread_start_us":1285181,"threads_started":2,"wal-append.queue_time_us":1536017}
I20260709 12:24:51.252310  1727 rpcz_store.cc:276] 0709 12:24:49.890406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.904226 (+ 13820us) service_pool.cc:224] Handling call
0709 12:24:49.920969 (+ 16743us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:49.962147 (+ 41178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:49.962177 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:49.999558 (+ 37381us) log.cc:844] Serialized 26 byte log entry
0709 12:24:50.826130 (+826572us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:51.103609 (+277479us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.103808 (+   199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.103893 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.193977 (+ 90084us) raft_consensus.cc:1685] finished
0709 12:24:51.194004 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.194179 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63680,"prepare.queue_time_us":342244,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":9232,"raft.queue_time_us":90322,"raft.run_cpu_time_us":484,"raft.run_wall_time_us":43189,"thread_start_us":930551,"threads_started":2,"wal-append.queue_time_us":970014}
I20260709 12:24:51.267074  8498 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.276680  8430 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:51.387663  8430 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:51.388221  8430 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:51.262240  8519 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:51.391400  8123 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.392781  8519 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:51.065042  8510 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.277395  8449 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:51.336319  1724 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:51.332667  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:51.339864  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.341961  1770 leader_election.cc:336] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 2.232s (SENT)
I20260709 12:24:51.206616  8486 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.276542  8521 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.402760  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:51.291711  8497 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:51.404462  8264 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:51.413522  8264 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:51.413728  8537 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.102s	user 0.001s	sys 0.000s
I20260709 12:24:51.414039  8264 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.414289  8412 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:51.414707  8412 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.412420  1802 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:51.365101  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3479) took 1323 ms. Trace:
I20260709 12:24:51.368134  1879 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:24:51.331331  8000 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.267266  8417 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.418577  8417 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:51.396373  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63773.3us
I20260709 12:24:51.419566  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:51.354835  8528 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.421759  8528 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.397604  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:51.422940  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 81054.3us
I20260709 12:24:51.425432  8352 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:51.422704  8264 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.426151  8352 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.416558  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:51.372139  8493 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:51.180397  8508 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.386472  7945 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.400578  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.411891  1724 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:51.443502  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.444151  1724 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:51.445115  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.445641  1724 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:51.445731  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.415110  8412 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:51.415572  1731 rpcz_store.cc:276] 0709 12:24:50.042283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.115070 (+ 72787us) service_pool.cc:224] Handling call
0709 12:24:50.115655 (+   585us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:50.116098 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:50.116129 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:50.116690 (+   561us) log.cc:844] Serialized 26 byte log entry
0709 12:24:50.663321 (+546631us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:50.663385 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.663487 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.364860 (+701373us) raft_consensus.cc:1685] finished
0709 12:24:51.364884 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.365036 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1357,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":546118,"threads_started":1,"wal-append.queue_time_us":921383}
I20260709 12:24:51.325271  8296 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:51.370165  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.407591  8494 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:51.427559  1879 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:24:51.409102  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:51.348946  1880 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 14. Ops: []
I20260709 12:24:51.473016  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:51.473393  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64400.6us
I20260709 12:24:51.428381  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.475440  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.476907  8531 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:51.477530  8531 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.464608  8528 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:51.479393  8528 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.479795  8528 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:51.480391  8531 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.483180  1800 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:51.439970  1805 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:51.430626  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:51.485788  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55214.5us
I20260709 12:24:51.413389  8497 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:51.407351  8343 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.768s	user 0.010s	sys 0.003s
I20260709 12:24:51.447638  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3466) took 1548 ms. Trace:
I20260709 12:24:51.447357  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.447468  8434 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.362376  8042 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.067s
I20260709 12:24:51.451114  8264 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.498677  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:51.499176  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.499450  1880 raft_consensus.cc:2376] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19. Re-sending same reply.
I20260709 12:24:51.500092  1801 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:51.500495  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:51.501312  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.502053  1801 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:51.507282  1727 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:51.508419  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.441344  8509 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:51.509235  8509 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.452239  8304 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.367089  8522 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.481838  8296 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.513603  8522 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.455437  8510 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.455972  7611 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.451095  8449 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.498714  8526 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.517174  8526 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.517850  1723 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:51.518409  8449 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:51.521039  8488 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.464208  8512 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:51.521922  8512 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.522327  8512 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:51.457176  8255 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.426612  8352 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:51.491809  8494 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.524698  8352 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:24:51.462152  8423 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.405104  8489 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.525166  8494 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.495512  7945 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:51.463622  8406 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.464794  1728 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:51.528710  8497 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.465793  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.495471  8417 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:51.473851  1724 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:51.530185  1731 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:24:51.530323  8417 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:51.501691  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:51.491024  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 43758.3us
I20260709 12:24:51.530351  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.537670  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.538621  1873 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
W20260709 12:24:51.542851  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44593.5us
I20260709 12:24:51.544067  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.544942  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.548141  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.549036  1724 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:51.514506  8296 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.495608  8343 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:51.526595  7945 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:51.490535  1729 rpcz_store.cc:276] 0709 12:24:49.899614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.900597 (+   983us) service_pool.cc:224] Handling call
0709 12:24:49.901022 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:50.232825 (+331803us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:50.232855 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:50.233355 (+   500us) log.cc:844] Serialized 26 byte log entry
0709 12:24:50.754678 (+521323us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:51.159331 (+404653us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.447023 (+287692us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.447122 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.447189 (+    67us) raft_consensus.cc:1685] finished
0709 12:24:51.447207 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.447567 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":406458,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":80075,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":217,"thread_start_us":1372977,"threads_started":4,"wal-append.queue_time_us":617511}
I20260709 12:24:51.497016  8493 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:51.545600  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.555080  8264 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 12:24:51.560462  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.530891  8417 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:51.575654  1800 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:51.562078  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.565367  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.544593  8522 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:24:51.568020  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.582055  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:51.594602  1724 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:51.542659  8489 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:24:51.551548  7987 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:51.590952  1729 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.531176  8434 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.559321  8398 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.598737  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:51.595925  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 125995us
I20260709 12:24:51.588387  8400 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.590029  8343 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.610910  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.574208  8264 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:51.498591  8412 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.575016  8042 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.595371  8494 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:51.595798  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.621881  7611 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.611840  8508 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:51.592171  8493 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:51.604061  8175 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:51.628273  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.589464  8497 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:51.629601  1879 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:51.552772  8511 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.052s
I20260709 12:24:51.654223  1729 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:51.661608  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:51.662503  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.612316  7987 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:51.679093  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50911us
I20260709 12:24:51.624318  8264 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:51.660775  7611 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.665505  8494 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.623716  8491 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:51.613258  8255 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
I20260709 12:24:51.667805  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.673625  8398 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:51.680567  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:51.667436  8538 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:24:51.670086  8497 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:51.686986  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.711397  1724 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.702466  1800 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:24:51.677664  8508 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:51.623921  8400 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.687616  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.697283  8539 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:51.683911  8509 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.670449  8175 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.658195  8042 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.630815  8343 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.713300  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.712476  8496 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.697422  8393 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:51.696626  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.753016  1879 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 12.
I20260709 12:24:51.685439  7987 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:51.665643  7945 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.709959  8423 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:51.800068  8423 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:51.800572  8423 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:51.712522  8497 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:51.804853  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.728147  8434 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.733318  8400 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:51.713002  8541 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.810983  8541 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.813426  8541 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.736097  8175 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.790359  8482 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.740154  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:51.819932  1800 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:51.821316  8482 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.717665  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.744387  1724 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:51.161120  1725 rpcz_store.cc:276] 0709 12:24:50.014608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.032726 (+ 18118us) service_pool.cc:224] Handling call
0709 12:24:50.042242 (+  9516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:50.042617 (+   375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:50.042652 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:50.043201 (+   549us) log.cc:844] Serialized 26 byte log entry
0709 12:24:50.711630 (+668429us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:51.033244 (+321614us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.033729 (+   485us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.033807 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.033856 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:51.033869 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.033985 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6156,"prepare.queue_time_us":405399,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":5755,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":10838,"thread_start_us":953404,"threads_started":2,"wal-append.queue_time_us":897344}
W20260709 12:24:51.745508  8542 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:24:51.824039  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.824177  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.824980  1725 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:51.825352  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.826048  1724 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:51.745812  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.746320  8509 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:51.827630  8509 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.828035  8509 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:24:51.746505  8544 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:24:51.747623  8490 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:51.833050  8490 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:51.833762  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:51.748916  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 913) took 1824 ms (client timeout 1583 ms). Trace:
I20260709 12:24:51.719184  8539 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.763324  8494 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.835242  8539 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:51.835009  1876 rpcz_store.cc:269] 0709 12:24:49.926044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.926536 (+   492us) service_pool.cc:224] Handling call
0709 12:24:51.748707 (+1822171us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400117954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:51.748812 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2426070272}
I20260709 12:24:51.837675  8416 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.733558  8528 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.838454  8416 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.840478  8410 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.840924  8528 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.745513  8412 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:51.841755  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.843199  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.754541  8496 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.844636  8487 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.846110  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1167) took 2546 ms. Trace:
I20260709 12:24:51.766527  8470 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.846342  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.846724  2413 rpcz_store.cc:276] 0709 12:24:49.299340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.588132 (+288792us) service_pool.cc:224] Handling call
0709 12:24:49.588532 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:50.228539 (+640007us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.845824 (+1617285us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b5400489754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:51.846007 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1430587264}
I20260709 12:24:51.848343  8255 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:51.848863  8255 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:51.774480  8512 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.849243  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.849959  1727 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:51.850963  8512 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:51.851048  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.851873  1727 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:51.773252  8430 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:51.852192  8255 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.777964  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.856192  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:51.856855  1805 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:51.857023  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.857560  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:51.858451  1799 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:51.859141  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:51.859934  1873 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:51.860728  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.861342  1873 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:51.862247  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.862581  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:51.863217  1726 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:51.864224  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.864405  8528 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:51.864979  8528 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.865334  8528 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:51.865511  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.865705  8521 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.867458  1801 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:51.868182  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2654) took 1166 ms. Trace:
I20260709 12:24:51.868595  1877 rpcz_store.cc:276] 0709 12:24:50.702061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.381793 (+679732us) service_pool.cc:224] Handling call
0709 12:24:51.382027 (+   234us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:51.399261 (+ 17234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:51.399297 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:51.399348 (+    51us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:51.408339 (+  8991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.867924 (+459585us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b5400116a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:51.867976 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.868122 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":130,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":28464,"spinlock_wait_cycles":38706432}
I20260709 12:24:51.869534  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.870221  1801 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:51.870507  1877 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:51.751390  8264 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.783421  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 919) took 1828 ms. Trace:
I20260709 12:24:51.784379  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:51.885316  1725 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
W20260709 12:24:51.885619  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101315us
I20260709 12:24:51.886356  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:51.785967  8508 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.751591  8493 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.834033  8490 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:51.908823  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:51.846205  8496 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:51.913496  8434 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:51.914237  8434 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:51.772935  8343 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:51.848783  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.916865  1807 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:51.917529  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2459) took 1117 ms. Trace:
I20260709 12:24:51.871253  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 911) took 1953 ms. Trace:
I20260709 12:24:51.918987  1881 rpcz_store.cc:276] 0709 12:24:49.917622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.920923 (+  3301us) service_pool.cc:224] Handling call
0709 12:24:49.921208 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:50.001393 (+ 80185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:50.001431 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:50.001473 (+    42us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:50.001509 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.871006 (+1869497us) spinlock_profiling.cc:243] Waited 79.9 ms on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:51.871056 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.871194 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183774976}
I20260709 12:24:51.871387  1872 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:51.776496  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.879323  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.923519  1875 raft_consensus.cc:2410] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:24:51.879561  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.879709  1800 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:51.866727  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.779142  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.881274  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.882614  1724 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:24:51.926035  1803 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:51.926743  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.928006  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.928658  1724 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:51.884004  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.929558  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.901667  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.898353  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:51.884629  1878 rpcz_store.cc:276] 0709 12:24:49.954924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.999276 (+ 44352us) service_pool.cc:224] Handling call
0709 12:24:49.999661 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:51.174665 (+1175004us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.783158 (+608493us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400206cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:51.783355 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2700470144}
I20260709 12:24:51.926075  1873 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:51.922451  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.931807  1723 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:24:51.917891  1807 rpcz_store.cc:276] 0709 12:24:50.800060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.028517 (+228457us) service_pool.cc:224] Handling call
0709 12:24:51.848593 (+820076us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:24:51.917448 (+ 68855us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2315136}
I20260709 12:24:51.933279  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:51.946962  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:51.949405  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.960062  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:51.942667  1878 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:51.945564  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47296.2us
I20260709 12:24:51.944090  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.943607  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.982757  1875 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:51.985534  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.971410  1725 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:51.969645  1726 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
W20260709 12:24:51.979857  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.991755  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.981990  1877 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:51.977389  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.997880  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.999523  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.986409  1875 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:52.005973  1727 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:52.008705  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.009138  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.020872  1879 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:51.895988  1805 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:52.015703  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:51.935529  8184 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:24:51.896322  8412 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:51.929505  8512 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:51.997116  8400 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:52.042533  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:52.043246  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.046914  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.049626  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:52.051337  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.052533  1879 raft_consensus.cc:2410] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:24:52.055801  2413 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:52.056864  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:52.063905  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.064774  2418 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:24:52.014289  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.983374  8490 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:52.028584  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:51.928937  8410 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:51.981745  8343 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:51.983112  8380 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:51.990312  8349 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:51.960705  8518 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:52.023209  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:52.036396  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:52.084942  1805 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:52.079577  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.032886  8430 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:52.069176  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.028643  8184 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:52.086512  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 298935us
I20260709 12:24:52.088105  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:52.089255  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66187.9us
I20260709 12:24:51.961148  8416 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.090631  2418 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:51.929136  8491 consensus_queue.cc:1175] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 20 status { last_received { term: 20 index: 14 } last_committed_idx: 14 last_received_current_leader { term: 20 index: 14 } }
I20260709 12:24:52.099335  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.101831  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:52.094812  8470 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.064746  8521 raft_consensus.cc:2764] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:52.078660  8343 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.070835  8434 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.103965  8417 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.181252  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 959) took 1183 ms. Trace:
I20260709 12:24:52.181730  1731 rpcz_store.cc:276] 0709 12:24:50.997535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.476768 (+479233us) service_pool.cc:224] Handling call
0709 12:24:51.529773 (+ 53005us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:51.530613 (+   840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.181056 (+650443us) spinlock_profiling.cc:243] Waited 52.6 ms on lock 0x7b5400119ed0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:52.181189 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":121067392}
W20260709 12:24:52.165186  2415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1173) took 2497 ms (client timeout 1857 ms). Trace:
W20260709 12:24:52.183699  2415 rpcz_store.cc:269] 0709 12:24:49.667554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.667707 (+   153us) service_pool.cc:224] Handling call
0709 12:24:52.164945 (+2497238us) spinlock_profiling.cc:243] Waited 758 ms on lock 0x7b54003c7154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:52.165104 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1744445824}
I20260709 12:24:52.185631  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.146659  8539 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.187175  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.188025  1876 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:52.188246  8539 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:52.153434  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:52.188935  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.189417  1802 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:52.190341  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:52.191316  8488 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:52.191704  8488 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:52.152338  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 911) took 1175 ms. Trace:
I20260709 12:24:52.166563  8042 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:52.192795  8488 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.193261  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:52.193892  8488 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:52.162194  1728 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:52.105074  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:52.105012  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:52.107388  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:52.082103  8133 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.065s
I20260709 12:24:52.197757  8470 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.198274  8133 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:52.198661  8133 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:52.199869  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.200387  8470 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:52.201020  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:52.201715  1725 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:52.202765  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:52.039439  8412 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.173758  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:52.202889  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:52.101107  8496 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:52.207412  8496 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.207813  8515 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:52.208839  8515 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:52.209594  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:52.210265  1799 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:52.209905  8515 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.148660  8474 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:24:52.199692  8133 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.089812  8410 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.192530  1798 rpcz_store.cc:276] 0709 12:24:50.977001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.043424 (+ 66423us) service_pool.cc:224] Handling call
0709 12:24:52.152060 (+1108636us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b5400133154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:52.152246 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39819008}
I20260709 12:24:52.221982  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:52.203997  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:52.206039  1731 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:52.207834  8496 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:52.211205  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:52.111856  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.058960  8400 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.154219  1806 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
W20260709 12:24:52.213754  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 106462us
I20260709 12:24:52.254371  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:52.254410  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:52.256430  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.063792  8512 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:52.197767  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:52.151081  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:52.203079  8549 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 12:24:52.208089  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:52.219053  1725 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:24:52.278371  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 182354us
I20260709 12:24:52.273240  1801 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:24:52.212427  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:52.258657  1728 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:52.151868  8548 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:24:52.101202  8184 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:52.288453  1801 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:52.121729  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:52.111372  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.240312  1805 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:24:52.290938  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:52.251734  8430 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.103245  8349 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:52.195792  8396 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:52.272528  8123 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.272774  8417 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:52.284224  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71843.8us
I20260709 12:24:52.221740  8519 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:52.294317  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.299381  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2458) took 1499 ms. Trace:
I20260709 12:24:52.300544  1805 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:52.258180  8474 consensus_queue.cc:1243] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:24:52.252266  8470 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.374914  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.329720  2410 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:52.215446  8343 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:52.167572  7987 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.380995  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:52.350026  1807 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:52.387048  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 909) took 1585 ms. Trace:
I20260709 12:24:52.259205  1799 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:52.335741  1800 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20260709 12:24:52.285478  8398 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:52.286603  8133 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:52.171782  8509 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.366101  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:52.383261  8548 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:24:52.431160  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.432238  2415 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:52.369347  1801 rpcz_store.cc:276] 0709 12:24:50.799749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.287694 (+1487945us) service_pool.cc:224] Handling call
0709 12:24:52.288200 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.299012 (+ 10812us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.299303 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:52.433740  1723 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:52.369818  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2460) took 1569 ms. Trace:
I20260709 12:24:52.094548  8518 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.223394  8497 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:52.443640  8551 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:24:52.445185  8497 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.376298  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:52.446126  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69910.2us
I20260709 12:24:52.382221  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 907) took 1581 ms. Trace:
I20260709 12:24:52.448048  1804 rpcz_store.cc:276] 0709 12:24:50.801292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.347426 (+1546134us) service_pool.cc:224] Handling call
0709 12:24:52.381603 (+ 34177us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.381825 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.381857 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:52.381904 (+    47us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:52.381951 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.382006 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.382157 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:52.382604  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.388224  2413 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:52.456871  8518 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:52.461473  7987 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.351414  8396 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.493423  8396 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.389241  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 905) took 1588 ms. Trace:
I20260709 12:24:52.364492  8123 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:52.499545  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 398658us
I20260709 12:24:52.501338  2415 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:52.503491  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:52.268585  2417 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:52.510507  1847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.003s	sys 0.025s
W20260709 12:24:52.510954  1847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.004s	sys 0.025s
I20260709 12:24:52.512617  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:52.516016  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 488231us
I20260709 12:24:52.411441  1724 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:52.427032  1731 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:52.362699  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:52.519882  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.435042  1805 rpcz_store.cc:276] 0709 12:24:50.800368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.290479 (+1490111us) service_pool.cc:224] Handling call
0709 12:24:52.300295 (+  9816us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.369494 (+ 69199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.369753 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:52.433373  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.335199  8430 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.387496  1807 rpcz_store.cc:276] 0709 12:24:50.801885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.349254 (+1547369us) service_pool.cc:224] Handling call
0709 12:24:52.349747 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.386686 (+ 36939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.386977 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:52.472926  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.476914  8393 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:52.478873  8512 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.405980  8417 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:52.483925  8400 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:52.484779  1799 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:24:52.485224  8412 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:52.381435  8470 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.487112  7945 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:52.487874  2416 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:52.489126  8509 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.550010  1730 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 16 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:52.489907  2413 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:52.413025  8398 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.413547  8495 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:52.569808  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:52.527998  8175 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.714s	user 0.010s	sys 0.000s
W20260709 12:24:52.390090  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.428171  8539 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:52.521989  1805 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:24:52.495499  8349 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.538455  8343 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.561136  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2544) took 1483 ms. Trace:
I20260709 12:24:52.578517  1799 rpcz_store.cc:276] 0709 12:24:51.078271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.463391 (+1385120us) service_pool.cc:224] Handling call
0709 12:24:52.484331 (+ 20940us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.560818 (+ 76487us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.561071 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:52.561707  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.553136  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.554865  2411 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:24:52.580350  2413 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
W20260709 12:24:52.564975  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.565719  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:52.583657  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:52.585002  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.587183  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:52.566469  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.547349  8410 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:52.515717  8423 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.715s	user 0.006s	sys 0.006s
I20260709 12:24:52.497334  1800 rpcz_store.cc:276] 0709 12:24:50.800675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.334831 (+1534156us) service_pool.cc:224] Handling call
0709 12:24:52.335370 (+   539us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:52.388882 (+ 53512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.389177 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:52.572971  1726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.032s	sys 0.055s
I20260709 12:24:52.573544  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2545) took 1495 ms. Trace:
W20260709 12:24:52.589399  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.572273  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:52.590220  1726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.032s	sys 0.056s
I20260709 12:24:52.588984  1805 rpcz_store.cc:276] 0709 12:24:51.078579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.521189 (+1442610us) service_pool.cc:224] Handling call
0709 12:24:52.521668 (+   479us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:52.573220 (+ 51552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.573481 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:52.551890  8528 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.686s	user 0.008s	sys 0.005s
I20260709 12:24:52.591650  8539 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:52.591733  8528 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.592151  8496 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.593010  8539 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:52.592343  8490 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.507s	user 0.010s	sys 0.000s
I20260709 12:24:52.595724  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2455) took 1796 ms. Trace:
I20260709 12:24:52.596716  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.548419  7611 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:52.561749  7945 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:52.548491  8123 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:52.567973  8000 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.546984  8255 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:52.653649  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:52.547266  8393 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:52.570006  8398 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:52.600835  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:52.538692  8184 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.576951  8175 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.598992  8380 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:24:52.594233  8528 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.571244  8417 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.656028  7945 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:52.597748  8423 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.697862  8423 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.698570  8430 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:52.699066  1798 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:52.701996  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2457) took 1902 ms. Trace:
I20260709 12:24:52.660063  7611 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 12:24:52.704259  7611 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:52.704731  7611 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:52.577693  8495 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.707239  8495 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:52.709100  8184 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.685446  2413 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:52.663666  8000 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.673645  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3481) took 2331 ms. Trace:
I20260709 12:24:52.713404  1732 rpcz_store.cc:276] 0709 12:24:50.342568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.342810 (+   242us) service_pool.cc:224] Handling call
0709 12:24:50.343147 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:50.353100 (+  9953us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:50.353142 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:51.468506 (+1115364us) spinlock_profiling.cc:243] Waited 6.34 ms on lock 0x7b18002e3540. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a
0709 12:24:51.877566 (+409060us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b2c0005a338. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:24:51.878067 (+   501us) log.cc:844] Serialized 26 byte log entry
0709 12:24:52.285520 (+407453us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:52.285560 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.285633 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:52.673281 (+387648us) raft_consensus.cc:1685] finished
0709 12:24:52.673310 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.673547 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73895,"raft.queue_time_us":9959,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":8931,"spinlock_wait_cycles":39605376,"thread_start_us":407079,"threads_started":1,"wal-append.queue_time_us":723011}
I20260709 12:24:52.663056  8255 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.651396  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:52.655059  1807 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 13. Current term is 15. Ops: []
I20260709 12:24:52.656680  2410 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:52.657544  8508 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:52.600265  8490 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:52.584794  8509 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:52.596271  1806 rpcz_store.cc:276] 0709 12:24:50.798887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.255273 (+456386us) service_pool.cc:224] Handling call
0709 12:24:52.153930 (+898657us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:52.254025 (+100095us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.595464 (+341439us) spinlock_profiling.cc:243] Waited 898 ms on lock 0x7b5400133150. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:52.595647 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2065779456}
I20260709 12:24:52.597579  7987 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:52.702518  1798 rpcz_store.cc:276] 0709 12:24:50.799447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.256352 (+1456905us) service_pool.cc:224] Handling call
0709 12:24:52.256709 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.699651 (+442942us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.701593 (+  1942us) spinlock_profiling.cc:243] Waited 442 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:52.701889 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1016672000}
I20260709 12:24:52.594218  8496 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:52.735019  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 83715.1us
W20260709 12:24:52.735052  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 491879us
I20260709 12:24:52.726095  1731 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:52.742534  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:52.748339  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.750198  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:52.751883  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.738801  1798 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:52.679790  8264 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:52.756588  1729 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:52.756989  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:52.758263  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 386749us
I20260709 12:24:52.758354  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2558) took 1261 ms. Trace:
I20260709 12:24:52.761616  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:52.763777  1798 rpcz_store.cc:276] 0709 12:24:51.497433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.738145 (+1240712us) service_pool.cc:224] Handling call
0709 12:24:52.738574 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.754522 (+ 15948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.758281 (+  3759us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:52.760339  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:52.764474  1873 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:52.762882  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:52.760237  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.766811  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2547) took 1463 ms. Trace:
I20260709 12:24:52.767446  1807 rpcz_store.cc:276] 0709 12:24:51.303766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.532874 (+1229108us) service_pool.cc:224] Handling call
0709 12:24:52.533285 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.718974 (+185689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.766494 (+ 47520us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:52.766714 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279309056}
W20260709 12:24:52.761907  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.678398  8343 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.718016  8417 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.676951  8470 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:52.681094  8393 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.9, Last appended by leader: 9, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:52.770902  1802 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:52.779480  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2552) took 1431 ms. Trace:
I20260709 12:24:52.780072  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 915) took 1185 ms. Trace:
I20260709 12:24:52.779500  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.776298  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:52.781960  1874 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:52.772436  1846 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:52.777905  1872 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: [13.10-15.11]
I20260709 12:24:52.780500  1807 rpcz_store.cc:276] 0709 12:24:51.594198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.779230 (+1185032us) service_pool.cc:224] Handling call
0709 12:24:52.779582 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.779762 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.779785 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:52.779837 (+    52us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:52.779867 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.779925 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.780021 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:52.795822  1807 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:52.770498  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:52.890689  1798 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:52.705890  7611 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:52.774878  8410 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:24:52.796927  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 917) took 1202 ms. Trace:
I20260709 12:24:52.779860  1802 rpcz_store.cc:276] 0709 12:24:51.348009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.190092 (+842083us) service_pool.cc:224] Handling call
0709 12:24:52.770679 (+580587us) spinlock_profiling.cc:243] Waited 7.57 ms on lock 0x7b5400404d20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abddbe18 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:52.779414 (+  8735us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17406336}
I20260709 12:24:52.733913  8490 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.934958  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2463) took 1167 ms. Trace:
I20260709 12:24:52.959771  1802 rpcz_store.cc:276] 0709 12:24:51.767777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.933744 (+1165967us) service_pool.cc:224] Handling call
0709 12:24:52.934154 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.934434 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.934478 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:52.934542 (+    64us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:52.934604 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.934685 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.934879 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:52.753098  8380 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:52.740574  7987 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:52.936790  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 166375us
I20260709 12:24:52.962702  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2464) took 1194 ms. Trace:
W20260709 12:24:52.938736  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.971930  1802 rpcz_store.cc:276] 0709 12:24:51.768085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.961465 (+1193380us) service_pool.cc:224] Handling call
0709 12:24:52.961934 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.962204 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.962244 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:52.962319 (+    75us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:52.962387 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.962464 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.962627 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:52.973042  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.973440  1802 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:52.940382  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2555) took 1536 ms. Trace:
I20260709 12:24:52.974478  1806 rpcz_store.cc:276] 0709 12:24:51.404332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.734487 (+1330155us) service_pool.cc:224] Handling call
0709 12:24:52.939622 (+205135us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.939895 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.939941 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:52.940003 (+    62us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:52.940060 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.940137 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.940319 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:52.936846  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:52.946201  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 185954us
W20260709 12:24:52.954900  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.954845  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:24:52.957250  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 184920us
I20260709 12:24:52.933048  1807 rpcz_store.cc:276] 0709 12:24:51.594826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.794996 (+1200170us) service_pool.cc:224] Handling call
0709 12:24:52.795402 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.796285 (+   883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.796852 (+   567us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:52.753842  8255 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:52.774526  8175 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:52.996951  2414 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.005s	sys 0.031s
W20260709 12:24:52.997572  2414 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.005s	sys 0.031s
W20260709 12:24:52.953994  8434 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.011s	sys 0.026s
W20260709 12:24:52.933795  8437 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.838s	user 0.024s	sys 0.093s
I20260709 12:24:52.968078  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 913) took 1374 ms. Trace:
I20260709 12:24:53.030552  1798 rpcz_store.cc:276] 0709 12:24:51.593622 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.889775 (+1296153us) service_pool.cc:224] Handling call
0709 12:24:52.890383 (+   608us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.891625 (+  1242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.967954 (+ 76329us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:52.970491  2418 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.003s	sys 0.024s
I20260709 12:24:52.987718  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:53.000922  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.011s	sys 0.033s
I20260709 12:24:53.006232  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:53.032197  2418 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.004s	sys 0.024s
W20260709 12:24:53.032608  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45104.6us
I20260709 12:24:52.966388  8490 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:53.032646  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.011s	sys 0.033s
I20260709 12:24:53.029887  1724 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:52.775053  8509 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.225s
W20260709 12:24:53.014999  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.990829  7987 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:53.040125  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:52.970132  8518 raft_consensus.cc:2764] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:24:52.969674  8296 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:52.970455  8470 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:53.044190  8133 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.609s	user 0.008s	sys 0.004s
I20260709 12:24:53.038028  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:52.769236  8559 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:53.045574  1728 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 17. (index mismatch)
I20260709 12:24:52.964210  8521 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:53.020985  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:52.998482  8416 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.011s	sys 0.033s
I20260709 12:24:53.060120  1727 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
W20260709 12:24:53.026567  8434 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.012s	sys 0.027s
W20260709 12:24:53.068749  1803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.023s	sys 0.030s
I20260709 12:24:53.023926  7611 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:52.999632  8184 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.958333  8410 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:53.067170  8515 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.011s	sys 0.043s
W20260709 12:24:53.030062  8437 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.024s	sys 0.094s
I20260709 12:24:53.040218  8175 raft_consensus.cc:2764] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
W20260709 12:24:53.086236  1803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.023s	sys 0.031s
W20260709 12:24:53.089546  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.055s	sys 0.142s
W20260709 12:24:53.092036  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54111.7us
W20260709 12:24:53.095813  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.055s	sys 0.142s
I20260709 12:24:53.040328  8380 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:53.096230  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:53.096211  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 648702us
W20260709 12:24:53.097265  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.068432  8296 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:53.106953  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.024s	sys 0.031s
W20260709 12:24:53.109131  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.024s	sys 0.032s
I20260709 12:24:53.094731  8410 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:53.107759  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:53.041206  8562 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:24:52.993655  1731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:53.114461  1731 raft_consensus.cc:606] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.114926  1731 raft_consensus.cc:614] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:53.107846  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:53.112243  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 542467us
I20260709 12:24:53.068584  8417 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:24:53.120260  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.120597  8417 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:53.121076  8417 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:53.121815  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.084008  8559 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:53.123030  8559 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:53.124423  8562 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:53.124951  8562 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:53.125881  8562 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:53.126955  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.048s	sys 0.113s
W20260709 12:24:53.127417  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.048s	sys 0.113s
W20260709 12:24:53.127730  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 609039us
I20260709 12:24:53.072430  8470 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:53.097986  8515 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.011s	sys 0.044s
I20260709 12:24:53.121989  8417 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.106196  8518 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:53.150071  1800 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.020s	sys 0.008s
I20260709 12:24:53.097865  8430 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:53.093179  8521 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.111152  8296 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:53.158078  1800 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.020s	sys 0.008s
W20260709 12:24:53.130501  8512 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.023s	sys 0.036s
I20260709 12:24:53.099416  8568 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:53.083138  8343 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 21.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:24:53.174438  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.108693  8437 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:53.139624  8470 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:53.216087  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:53.134093  8434 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:24:53.113032  8380 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:53.179898  8512 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.027s	sys 0.036s
I20260709 12:24:53.106662  8184 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.241222  8343 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:53.241999  8343 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:53.242357  8343 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.151132  8495 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 12:24:53.245456  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1339) took 1007 ms. Trace:
I20260709 12:24:53.248126  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2543) took 2183 ms. Trace:
I20260709 12:24:53.244019  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1338) took 1031 ms. Trace:
I20260709 12:24:53.248248  2418 rpcz_store.cc:276] 0709 12:24:52.238499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.276847 (+ 38348us) service_pool.cc:224] Handling call
0709 12:24:52.277234 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:52.277583 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.277610 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:52.278120 (+   510us) log.cc:844] Serialized 26 byte log entry
0709 12:24:53.032969 (+754849us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:53.033039 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.033148 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:53.245230 (+212082us) raft_consensus.cc:1685] finished
0709 12:24:53.245254 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.245385 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":137436,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":4060,"spinlock_wait_cycles":62208,"thread_start_us":653499,"threads_started":1,"wal-append.queue_time_us":911725}
W20260709 12:24:53.247462  1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3494) took 1589 ms (client timeout 1899 ms). Trace:
W20260709 12:24:53.253047  1729 rpcz_store.cc:269] 0709 12:24:51.658501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.717426 (+ 58925us) service_pool.cc:224] Handling call
0709 12:24:53.247224 (+1529798us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b54002b84d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:53.247396 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2146165632}
I20260709 12:24:53.249562  2412 rpcz_store.cc:276] 0709 12:24:52.213111 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.213244 (+   133us) service_pool.cc:224] Handling call
0709 12:24:53.243698 (+1030454us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b54007ce554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:53.243919 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2267161472}
I20260709 12:24:53.229151  8417 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:53.229890  8528 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:53.184194  8430 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:53.230510  8569 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:24:53.235376  8566 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 12:24:53.236523  8380 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.242767  8343 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:53.237093  8521 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:53.275076  1873 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.510s	user 0.011s	sys 0.001s
I20260709 12:24:53.275684  1873 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:53.237893  8000 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:53.239460  8558 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
W20260709 12:24:53.218688  8539 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.000s	sys 0.028s
I20260709 12:24:53.212568  8518 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:53.248773  1803 rpcz_store.cc:276] 0709 12:24:51.064824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.410436 (+1345612us) service_pool.cc:224] Handling call
0709 12:24:52.415050 (+  4614us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:52.415530 (+   480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.415570 (+    40us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:52.427933 (+ 12363us) log.cc:844] Serialized 78 byte log entry
0709 12:24:53.095869 (+667936us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:24:53.095907 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.096072 (+   165us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:53.247855 (+151783us) raft_consensus.cc:1685] finished
0709 12:24:53.247881 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.248059 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":883,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":7259,"thread_start_us":652683,"threads_started":1,"wal-append.queue_time_us":800340}
I20260709 12:24:53.242594  8184 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.183751  8561 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:53.179011  8416 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 12:24:53.263003  8528 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:53.220887  8568 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:53.265879  8569 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:53.242480  7945 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.552s	user 0.009s	sys 0.000s
I20260709 12:24:53.251859  8437 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:53.304104  7945 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.304720  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.305580  8559 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.306406  7945 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.300635  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:53.309253  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 354544us
I20260709 12:24:53.304013  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.328749  2410 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:53.298772  8561 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:53.268401  8430 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:53.291211  8539 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.000s	sys 0.028s
I20260709 12:24:53.308171  2415 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:53.279505  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:53.304024  8577 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
W20260709 12:24:53.319397  8567 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.227s	user 0.000s	sys 0.001s
I20260709 12:24:53.332628  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:53.336613  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 57162.2us
I20260709 12:24:53.336732  8000 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:53.311141  8518 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:53.312621  8437 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:53.359164  8576 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:24:53.363447  8410 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.364024  8559 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.364825  8568 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.365950  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2594) took 1798 ms. Trace:
I20260709 12:24:53.366225  8482 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:53.366523  1723 rpcz_store.cc:276] 0709 12:24:51.568844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.932752 (+363908us) service_pool.cc:224] Handling call
0709 12:24:51.933173 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.434256 (+501083us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.365662 (+931406us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b540030a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:53.365884 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1142075648}
W20260709 12:24:53.368290  8570 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.183s	user 0.000s	sys 0.001s
I20260709 12:24:53.330950  8561 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:53.351049  8528 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.9, Last appended by leader: 9, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.376123  8528 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:53.379933  8437 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:53.384694  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 52172us
I20260709 12:24:53.384985  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2602) took 1807 ms. Trace:
W20260709 12:24:53.334014  8349 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.009s	sys 0.018s
W20260709 12:24:53.409611  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:53.410331  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:53.353470  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 137489us
I20260709 12:24:53.414077  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:53.354318  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 246757us
I20260709 12:24:53.358084  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.385502  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:53.417336  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:53.352108  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:53.396610  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1330) took 1349 ms. Trace:
W20260709 12:24:53.397466  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48292.7us
I20260709 12:24:53.400481  8430 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.400203  8184 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:53.400555  2410 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:53.418217  2414 rpcz_store.cc:276] 0709 12:24:52.046749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.080512 (+ 33763us) service_pool.cc:224] Handling call
0709 12:24:52.080991 (+   479us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:52.384681 (+303690us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.384724 (+    43us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:52.387168 (+  2444us) log.cc:844] Serialized 78 byte log entry
0709 12:24:52.998332 (+611164us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:53.326718 (+328386us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.335617 (+  8899us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.396135 (+ 60518us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.396259 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:53.396322 (+    63us) raft_consensus.cc:1685] finished
0709 12:24:53.396343 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.396534 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":99680,"raft.queue_time_us":98161,"raft.run_cpu_time_us":468,"raft.run_wall_time_us":118146,"thread_start_us":1041686,"threads_started":3,"wal-append.queue_time_us":837631}
I20260709 12:24:53.385351  1726 rpcz_store.cc:276] 0709 12:24:51.577247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.992925 (+415678us) service_pool.cc:224] Handling call
0709 12:24:51.995066 (+  2141us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:52.019397 (+ 24331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.019430 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:52.020047 (+   617us) log.cc:844] Serialized 52 byte log entry
0709 12:24:52.591431 (+571384us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:52.591508 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.594978 (+  3470us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:53.368770 (+773792us) raft_consensus.cc:1685] finished
0709 12:24:53.384794 (+ 16024us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.384931 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3344,"raft.queue_time_us":13584,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":1380,"thread_start_us":519232,"threads_started":1,"wal-append.queue_time_us":893470}
I20260709 12:24:53.425072  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:53.426364  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:53.401373  8123 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:53.404007  8296 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.405583  8490 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:53.407226  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.407708  8586 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:53.419456  1725 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:24:53.444164  1725 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:53.421274  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:53.446450  1723 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:53.421643  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:53.422582  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:53.448071  1732 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:53.366725  8482 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:53.424043  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72049us
I20260709 12:24:53.365914  8410 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.426890  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.356088  8000 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.429769  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:53.409336  8349 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.009s	sys 0.018s
I20260709 12:24:53.428304  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:53.431972  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:53.438805  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:53.504958  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63719.7us
I20260709 12:24:53.416652  8423 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:53.424925  8380 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.424108  8588 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:53.483628  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.523814  2414 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:53.508978  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82234.7us
I20260709 12:24:53.524865  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.512082  8000 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:53.423840  8521 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:53.527294  8521 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.527710  8521 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:53.532092  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:53.533164  2412 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:53.517187  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:53.517995  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.383495  8518 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.434125  8175 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:53.431061  8123 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:53.524068  2410 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
W20260709 12:24:53.534899  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 96243.2us
W20260709 12:24:53.538062  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.526458  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.484290  8133 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:53.544600  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2474) took 1273 ms. Trace:
I20260709 12:24:53.548249  1723 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
W20260709 12:24:53.507786  1770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.899s (SENT) [suppressed 16 similar messages]
W20260709 12:24:53.517288  8264 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.746s	user 0.009s	sys 0.006s
I20260709 12:24:53.444355  8597 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:53.543576  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2553) took 2178 ms. Trace:
I20260709 12:24:53.569156  1806 rpcz_store.cc:276] 0709 12:24:52.271666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.975797 (+704131us) service_pool.cc:224] Handling call
0709 12:24:52.976187 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.976459 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:53.311295 (+334836us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.544147 (+232852us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:53.544212 (+    65us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:53.544277 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.544375 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.544530 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":466878,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":706,"thread_start_us":519941,"threads_started":2}
W20260709 12:24:53.574395  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 60823.3us
I20260709 12:24:53.512794  8497 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:53.579259  2413 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:24:53.582090  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:53.588750  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 62376.6us
I20260709 12:24:53.589797  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.592340  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:53.540524  8519 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.552982  8417 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:53.550729  8175 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:53.621948  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:53.620311  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.568429  8410 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.577343  8593 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:53.632993  8593 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.634588  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2549) took 2325 ms. Trace:
I20260709 12:24:53.635432  8593 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.539439  8470 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.629412  2418 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:53.603196  8594 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.541522  8588 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:53.648818  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.628005  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.635102  1800 rpcz_store.cc:276] 0709 12:24:51.309295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.588938 (+1279643us) service_pool.cc:224] Handling call
0709 12:24:52.648751 (+ 59813us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:53.176494 (+527743us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:53.176526 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:53.218527 (+ 42001us) log.cc:844] Serialized 26 byte log entry
0709 12:24:53.421083 (+202556us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:53.421137 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.422889 (+  1752us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:53.634335 (+211446us) raft_consensus.cc:1685] finished
0709 12:24:53.634355 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.634515 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":58859,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":22070,"thread_start_us":684143,"threads_started":2,"wal-append.queue_time_us":375511}
I20260709 12:24:53.536096  8380 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.541188  8586 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.082s
I20260709 12:24:53.619527  8559 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:53.595340  8563 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.624070  8497 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:53.622900  8569 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:53.599450  8352 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 12:24:53.661784  8591 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:53.661980  7945 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:53.579463  8133 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.667642  8133 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.668303  1728 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:53.670483  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3475) took 3656 ms (client timeout 2232 ms). Trace:
W20260709 12:24:53.671058  1730 rpcz_store.cc:269] 0709 12:24:50.013940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.014129 (+   189us) service_pool.cc:224] Handling call
0709 12:24:53.670109 (+3655980us) spinlock_profiling.cc:243] Waited 2.47 s on lock 0x7b54002aa154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:53.670307 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5688433024}
I20260709 12:24:53.642894  8184 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:53.625402  8175 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:53.699764  2416 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:53.674427  1732 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:24:53.657914  8559 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:53.702477  8559 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:53.719357  8133 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:24:53.720580  2415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1186) took 1208 ms (client timeout 1537 ms). Trace:
I20260709 12:24:53.720937  1872 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:53.721014  2415 rpcz_store.cc:269] 0709 12:24:52.512148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.512317 (+   169us) service_pool.cc:224] Handling call
0709 12:24:53.720382 (+1208065us) spinlock_profiling.cc:243] Waited 795 ms on lock 0x7b5400580f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:53.720523 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1827403136}
I20260709 12:24:53.722805  1872 raft_consensus.cc:1217] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Deduplicated request from leader. Original: 14.10->[17.11-19.12]   Dedup: 17.11->[19.12-19.12]
I20260709 12:24:53.641609  8410 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.723016  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:53.723316  1872 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20260709 12:24:53.711869  8597 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:53.725170  8410 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:53.714468  8588 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.725481  8597 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.726733  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:53.727496  1872 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:24:53.727919  8608 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:24:53.727797  8343 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:53.678680  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 101427us
I20260709 12:24:53.655910  8594 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.687125  8591 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:53.731685  1880 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:53.731943  8591 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:53.732120  1880 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:53.733001  8591 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.641834  8614 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:53.688160  8615 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:53.689105  8616 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:53.740350  8616 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:53.740348  8483 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:53.741197  8616 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:53.740464  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.689654  8380 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.748596  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:53.749509  2417 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:53.750607  8123 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.575420  1799 rpcz_store.cc:276] 0709 12:24:51.364924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.600866 (+1235942us) service_pool.cc:224] Handling call
0709 12:24:52.661930 (+ 61064us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:52.665859 (+  3929us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.666027 (+   168us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:52.729643 (+ 63616us) log.cc:844] Serialized 52 byte log entry
0709 12:24:53.282535 (+552892us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:53.524629 (+242094us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.542342 (+ 17713us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.542604 (+   262us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:53.542690 (+    86us) raft_consensus.cc:1685] finished
0709 12:24:53.542745 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.543487 (+   742us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20407,"prepare.queue_time_us":236883,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":10636,"raft.queue_time_us":91166,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":17335,"thread_start_us":549040,"threads_started":2,"wal-append.queue_time_us":611746}
I20260709 12:24:53.752063  8123 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:53.753073  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:53.752750  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:53.690343  8000 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.754868  8616 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.757277  8616 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.772540  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:53.696709  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 114761us
I20260709 12:24:53.778769  8000 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:53.694399  7611 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.604s	user 0.008s	sys 0.000s
I20260709 12:24:53.704329  8417 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:53.779418  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.779728  8417 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.781917  8417 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.704900  8583 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.782532  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.783248  1872 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 13.
I20260709 12:24:53.783082  8583 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.785425  8583 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.603055  8482 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.786372  8482 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:53.786804  1630 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:53.706710  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:53.791275  1800 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:53.792646  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:53.793788  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:53.794345  1800 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:53.706635  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:53.795217  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 88708.7us
I20260709 12:24:53.796622  8515 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:53.798156  8511 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:24:53.705720  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.707823  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:53.799258  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.799768  8474 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:53.799901  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:53.797381  8515 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:53.800863  1806 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:53.800377  8491 consensus_queue.cc:1175] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" responder_term: 20 status { last_received { term: 14 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: [17.11-19.12]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:53.661336  8563 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.709043  1804 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:53.801463  8515 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.709679  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:53.782495  8417 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:53.798486  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:53.804675  8508 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:24:53.802592  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:53.805358  8508 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:53.805750  1800 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:53.711963  8400 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:53.808072  8400 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:53.808540  8400 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:53.809656  8400 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.712818  8470 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:53.714314  8569 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:53.811811  8569 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:53.812258  8569 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:53.813289  8569 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.715792  8493 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:53.814335  8563 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.815025  8493 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:53.815434  8493 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:53.816203  8493 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.817601  8493 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:53.716823  7987 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.642s	user 0.010s	sys 0.000s
I20260709 12:24:53.624001  8519 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.819430  7987 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.677342  1728 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:24:53.731150  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:53.822144  8519 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.719290  8490 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.733428  8594 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:53.687163  8398 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.979s	user 0.009s	sys 0.004s
I20260709 12:24:53.814419  8569 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:53.831923  8398 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.735242  1880 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.835180  1880 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:53.835517  7987 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.742224  8561 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:53.800323  8474 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:53.737432  1802 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:53.837280  1802 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:53.838613  1802 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.839903  1802 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:53.834685  8398 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:53.738433  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:53.734403  8591 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:53.834721  8400 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:53.847795  8474 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.849215  8474 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:53.852322  8614 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.853034  8614 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.856627  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:53.856814  8561 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:53.857402  2413 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:53.861676  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:53.862043  1727 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:53.863459  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:53.864188  1724 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:53.865857  1731 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:53.806463  8508 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.867246  7611 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:53.867897  7611 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.773504  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.774075  8496 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.872354  8496 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.873036  7611 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.795764  1630 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 5 to 7, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 7 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:53.876964  1630 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 6 to 9, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 9 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:53.776206  8343 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.773989  1877 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:53.777603  1799 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:24:53.758472  8521 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.752024  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:53.697992  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 81874.6us
I20260709 12:24:53.710374  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:53.820071  1874 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:53.700261  8528 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:53.935812  8175 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.938197  8175 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.940920  8521 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:53.942077  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 198911us
I20260709 12:24:53.943171  8483 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:53.944803  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:53.945541  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.826249  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:53.817593  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:53.947453  1873 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:53.827631  1876 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:53.948321  1728 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:53.827720  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45241.7us
I20260709 12:24:53.949322  1729 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:53.950565  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.831077  8417 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:53.947002  8416 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:53.735829  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:53.774999  8296 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.888797  8184 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.955621  8184 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:53.956115  8559 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:53.891438  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 463151us
I20260709 12:24:53.960110  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:53.962563  8621 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:53.963990  8563 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:53.964437  8563 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.964800  8563 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:53.966717  8496 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.967428  2417 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:53.958688  8184 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:53.901314  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:53.928493  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:53.975071  1724 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:53.929201  1730 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:53.929510  1727 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:53.930236  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:53.970047  2410 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:53.932502  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:53.934844  8491 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:53.979043  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.970125  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.981251  8583 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:24:53.900504  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.952980  1725 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:24:53.968583  8559 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:53.969302  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.972123  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:53.973404  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:53.989619  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:53.963460  8621 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:53.990190  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:53.969764  2417 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:54.002867  2413 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
W20260709 12:24:54.007206  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53498.2us
I20260709 12:24:54.127427  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:54.078294  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 570482us
W20260709 12:24:54.029320  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.093433  1801 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 17. (index mismatch)
W20260709 12:24:54.048738  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 97705.4us
W20260709 12:24:54.028005  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.135964  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.137631  1801 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:54.078722  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.139000  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:54.139339  1872 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:54.076957  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.078296  1802 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:54.077042  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.068552  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:54.142655  1725 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:54.144104  1731 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:24:54.146059  2418 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 17 because replica has last-logged OpId of term: 14 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
W20260709 12:24:54.075713  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.093096  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.061811  1630 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 10 to 11, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 11 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:54.149878  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:24:54.151825  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 178511us
I20260709 12:24:54.151525  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:54.151787  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.153239  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:54.154597  2416 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:54.155088  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:53.913408  8255 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:54.156610  2410 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:54.156672  1800 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:54.155169  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:54.158200  2417 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:54.158468  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.159416  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:54.159869  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:54.160665  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.161662  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.163901  1807 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:54.167685  1807 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:54.170336  1807 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:54.164288  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:54.177626  1697 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:54.179352  1725 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 12:24:54.074949  8347 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:24:53.974036  8597 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:54.188295  2410 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:54.188447  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:54.180884  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.183000  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.191267  1803 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:54.185318  1807 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:24:54.192365  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:54.190366  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.199172  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:54.076408  8417 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:54.075932  8614 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:54.203011  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:54.216863  1725 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:54.217566  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.217954  1874 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:54.219641  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:54.220465  1732 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:24:54.204972  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:54.223135  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:54.207870  1800 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:54.200683  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:54.210253  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.210698  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.224565  1877 raft_consensus.cc:2376] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17. Re-sending same reply.
I20260709 12:24:54.175827  8490 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:54.211943  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:54.211156  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.075963  8581 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.132s
I20260709 12:24:54.059334  8496 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.162s
W20260709 12:24:54.237165  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.174765  8583 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.211088  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:54.218418  8624 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:24:54.205013  8380 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:54.225488  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.262017  1877 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:54.175629  8616 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:54.177029  8559 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:54.230266  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.182857  8521 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:54.083825  8175 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.152s
I20260709 12:24:54.238602  8613 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:53.972002  7987 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.228s
I20260709 12:24:53.972057  8000 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.226s
I20260709 12:24:54.180583  8615 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:54.174765  8593 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:54.270382  8508 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:54.301823  8593 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:54.302392  8593 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:54.048592  7611 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.158s
I20260709 12:24:54.303263  7611 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:54.237469  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.303560  8042 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:54.304553  8042 consensus_queue.cc:1243] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 19.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:24:54.237195  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.305543  1881 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:54.307794  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:54.184628  8255 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.309263  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:54.247236  1800 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:24:54.063885  8563 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.254912  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.177233  8123 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:54.207647  8610 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:54.250670  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:54.267233  1727 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:54.217450  8519 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.218201  8594 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:54.271207  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.271682  1725 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 9. Preceding OpId from leader: term: 22 index: 10. (index mismatch)
W20260709 12:24:54.322403  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 99391.8us
I20260709 12:24:54.258177  8597 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 12:24:54.236660  8614 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:54.225442  8470 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:54.349238  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:54.226377  8588 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:54.231773  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:54.279943  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 94372.7us
I20260709 12:24:54.198580  8518 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.065s
I20260709 12:24:54.198813  8400 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.071s
W20260709 12:24:54.297266  1625 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 272) took 63025 ms (client timeout 15000 ms). Trace:
W20260709 12:24:54.298933  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.216032  8561 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:24:54.315812  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:24:54.425472  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 109712us
I20260709 12:24:54.425925  1875 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:24:54.315884  2415 raft_consensus.cc:1217] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Deduplicated request from leader. Original: 14.10->[17.11-20.13]   Dedup: 17.11->[18.12-20.13]
I20260709 12:24:54.273250  8559 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:54.273536  8521 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:54.234094  8417 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.328727  1727 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 12:24:54.318028  1878 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [20.16-20.16]
I20260709 12:24:54.381748  8416 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:54.445842  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:54.446378  1878 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:24:54.446612  1727 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:54.446813  1803 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:24:54.448151  1878 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:54.302106  8508 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.328967  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:54.303648  7611 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:54.329777  8519 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:54.333109  8594 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.345757  8483 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:24:54.353291  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:54.437579  8559 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:54.470157  1725 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:54.354377  1799 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 9. Preceding OpId from leader: term: 22 index: 10. (index mismatch)
I20260709 12:24:54.356179  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:54.317973  8539 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:54.264707  8583 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.324003  8581 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:24:54.406744  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51784.7us
W20260709 12:24:54.401881  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48959.5us
W20260709 12:24:54.411350  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.419284  1877 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:54.444351  8521 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:54.429469  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:54.330309  8123 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:54.434993  2415 pending_rounds.cc:85] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 11
I20260709 12:24:54.180154  8609 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.095s
I20260709 12:24:54.417896  8296 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:24:54.472199  8000 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:54.471263  8483 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:54.465962  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 137105us
I20260709 12:24:54.433943  8184 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
I20260709 12:24:54.468209  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:54.439527  8343 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:54.420006  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:54.591361  1731 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:24:54.369752  8470 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.418596  8493 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.068s
I20260709 12:24:54.482681  1725 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:24:54.602718  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.603875  1802 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:54.484879  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:54.488587  1877 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:54.606287  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.489593  1806 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:54.489681  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:54.491426  1872 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:54.492416  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63012.9us
I20260709 12:24:54.492803  2415 pending_rounds.cc:107] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:24:54.615051  2415 raft_consensus.cc:2889] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:54.547065  1727 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
W20260709 12:24:54.502767  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 146652us
I20260709 12:24:54.552425  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.552057  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.617280  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:54.595140  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:24:54.498359  8620 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.241s	user 0.000s	sys 0.001s
W20260709 12:24:54.634085  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.635475  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.636435  1805 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:54.523613  8000 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:54.628036  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.500424  8255 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:54.522064  8581 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:54.597167  8343 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:54.525264  8609 consensus_queue.cc:1243] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 19.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:24:54.626631  1803 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:54.521948  8496 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:54.642625  1872 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:54.536521  8417 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:54.492204  8539 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:54.650976  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 161367us
W20260709 12:24:54.651165  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 166361us
I20260709 12:24:54.502274  8561 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.146s
I20260709 12:24:54.660979  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:54.652375  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.662635  8583 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:54.662876  8508 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:54.642064  8352 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:54.595041  8123 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.663414  8518 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:24:54.525942  7611 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:54.665158  7611 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:54.646344  8474 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.797s	user 0.014s	sys 0.000s
I20260709 12:24:54.653590  8493 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:54.667550  8493 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:54.667645  1877 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:54.668040  8493 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:54.668548  8493 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:54.652523  8563 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:54.657182  8343 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:54.671819  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.660907  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:54.646672  8000 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:54.683647  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.684389  1803 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:54.687484  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:54.659245  8629 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.170s	user 0.000s	sys 0.001s
I20260709 12:24:54.695031  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.699046  1802 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:54.700240  2383 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.000s	sys 0.013s
W20260709 12:24:54.700630  2383 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.000s	sys 0.013s
W20260709 12:24:54.700939  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 522025us
I20260709 12:24:54.677994  1798 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:54.705041  8613 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:54.709264  8613 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:54.721419  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.683223  8352 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:54.742388  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.749956  1802 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:54.728574  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0124083539999447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25120000000000003s for a response
I20260709 12:24:54.688401  8496 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:54.675401  8416 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:24:54.625836  8483 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.656968  8593 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.720781  8613 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:54.689503  8437 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260709 12:24:54.783950  1803 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
W20260709 12:24:54.784880  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.788112  2417 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 12. Current term is 13. Ops: []
I20260709 12:24:54.788354  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:54.788882  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.790010  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:54.791051  2410 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:54.791122  1725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:54.792007  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:54.779788  8518 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:54.791901  1725 raft_consensus.cc:606] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:54.795614  1725 raft_consensus.cc:614] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:54.796545  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.797925  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.789698  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:54.789410  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.788934  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:54.770042  8609 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:54.708792  8539 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:54.816439  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.827068  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.828256  1802 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:24:54.841809  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.842000  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.786867  8347 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:54.848251  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.848883  1875 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:24:54.858256  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:54.859407  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.792936  8634 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.863545  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.872634  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:54.800642  8594 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:54.883188  1875 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:54.789384  8614 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
I20260709 12:24:54.789829  8380 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.086s
W20260709 12:24:54.894181  1630 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.017s	sys 0.013s
W20260709 12:24:54.890978  2413 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.003s	sys 0.028s
W20260709 12:24:54.909391  1630 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.018s	sys 0.013s
W20260709 12:24:54.909595  2413 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.003s	sys 0.029s
I20260709 12:24:54.844818  8518 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 12:24:54.900422  1875 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:24:54.816051  8474 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.074s
I20260709 12:24:54.862673  8347 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:54.832036  8563 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 12:24:54.868191  8483 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:24:54.833123  8508 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:24:54.833317  8591 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.062s
I20260709 12:24:54.890014  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:54.917762  1725 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6388b1a5243d463589750794e8ac5511"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:54.925241  1725 raft_consensus.cc:606] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:24:54.925735  1725 raft_consensus.cc:614] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:54.917865  1729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.005s	sys 0.022s
I20260709 12:24:54.900559  8470 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:54.845079  8609 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.900204  8583 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.927587  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:54.923166  1875 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:24:54.940744  1723 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:54.927145  1729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.005s	sys 0.022s
I20260709 12:24:54.890749  8352 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:54.960392  1881 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.003s	sys 0.023s
I20260709 12:24:54.918546  8588 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.896209  8496 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:54.889991  8634 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:54.966820  1728 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.008s	sys 0.012s
I20260709 12:24:54.968616  1730 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 12:24:54.969228  1881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.003s	sys 0.023s
I20260709 12:24:54.926724  8515 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:24:54.952595  8175 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:54.967342  8490 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.730s	user 0.014s	sys 0.000s
W20260709 12:24:54.932571  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.983182  1723 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
W20260709 12:24:54.990612  1728 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.008s	sys 0.012s
I20260709 12:24:54.954982  8583 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:54.998426  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:54.999157  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 371178us
W20260709 12:24:54.999719  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.998667  1723 pending_rounds.cc:85] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
W20260709 12:24:54.998586  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 371465us
I20260709 12:24:55.002012  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:54.979326  8470 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.004617  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:55.002875  1723 pending_rounds.cc:107] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:24:55.008519  1723 raft_consensus.cc:2889] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:55.005652  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:55.009496  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:55.004009  8423 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s	user 0.016s	sys 0.018s
I20260709 12:24:54.996114  8490 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:24:55.027307  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 198131us
I20260709 12:24:55.001993  8593 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.011543  8597 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:54.986377  8588 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.034808  1726 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
W20260709 12:24:55.040441  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 343816us
W20260709 12:24:55.029210  8423 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.945s	user 0.017s	sys 0.019s
I20260709 12:24:55.021641  8582 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:55.012059  8652 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:55.061389  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:55.064762  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 400170us
I20260709 12:24:55.067700  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:55.011330  8470 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:54.991613  8347 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.064275  8042 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:55.097026  8417 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:55.097280  8633 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:55.062332  8423 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.062s
W20260709 12:24:55.127588  8642 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.127s	user 0.001s	sys 0.001s
I20260709 12:24:55.128931  8347 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:55.130384  8417 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:55.123338  8593 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 12:24:55.139472  8609 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.143419  8597 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:55.139000  8656 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:24:55.172003  8653 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:55.172952  8588 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.174013  8634 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:55.181236  8646 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
W20260709 12:24:55.182135  8641 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:24:55.130571  8633 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:55.182736  8615 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.861s	user 0.000s	sys 0.010s
I20260709 12:24:55.190143  8653 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:55.200281  8628 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.723s	user 0.018s	sys 0.027s
I20260709 12:24:55.197108  8615 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:55.200202  8653 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:54.423578  1625 rpcz_store.cc:269] 0709 12:23:51.271754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.611507 (+339753us) service_pool.cc:224] Handling call
0709 12:24:54.297145 (+62685638us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:47.339178 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:47.340037 (+   859us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:47.340103 (+    66us) write_op.cc:435] Acquired schema lock
0709 12:24:47.340123 (+    20us) tablet.cc:662] Decoding operations
0709 12:24:47.341397 (+  1274us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:47.341520 (+   123us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:47.341549 (+    29us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:24:47.577565 (+236016us) tablet.cc:701] Row locks acquired
0709 12:24:47.596001 (+ 18436us) write_op.cc:260] PREPARE: finished
0709 12:24:47.632826 (+ 36825us) write_op.cc:270] Start()
0709 12:24:47.683553 (+ 50727us) write_op.cc:276] Timestamp: P: 1783599887619582 usec, L: 0
0709 12:24:47.697803 (+ 14250us) op_driver.cc:348] REPLICATION: starting
0709 12:24:47.887057 (+189254us) log.cc:844] Serialized 4739 byte log entry
0709 12:24:48.431788 (+544731us) op_driver.cc:464] REPLICATION: finished
0709 12:24:48.599089 (+167301us) write_op.cc:301] APPLY: starting
0709 12:24:48.599250 (+   161us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:48.983608 (+384358us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:48.988341 (+  4733us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:49.086688 (+ 98347us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:49.087160 (+   472us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=24,mrs_lookups=0
0709 12:24:49.087222 (+    62us) write_op.cc:312] APPLY: finished
0709 12:24:49.087684 (+   462us) log.cc:844] Serialized 113 byte log entry
0709 12:24:49.087907 (+   223us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:49.090067 (+  2160us) write_op.cc:454] Released schema lock
0709 12:24:49.090152 (+    85us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:49.125212 (+ 35060us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:49.125255 (+    43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":150729,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":166897,"apply.run_cpu_time_us":26784,"apply.run_wall_time_us":513893,"cfile_cache_hit":50,"cfile_cache_hit_bytes":54335,"num_ops":12,"prepare.queue_time_us":260718,"prepare.run_cpu_time_us":25380,"prepare.run_wall_time_us":874402,"raft.queue_time_us":150920,"raft.run_cpu_time_us":46270,"raft.run_wall_time_us":199277,"replication_time_us":713171,"thread_start_us":546829,"threads_started":4,"wal-append.queue_time_us":278890,"wal-append.run_cpu_time_us":31837,"wal-append.run_wall_time_us":1926460}]]}
I20260709 12:24:55.235251  1727 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:55.236344  1727 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:55.238833  1727 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:55.249214  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 187723us
W20260709 12:24:55.260211  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.300252  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3394) took 1086 ms. Trace:
I20260709 12:24:55.300469  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3383) took 1091 ms. Trace:
I20260709 12:24:55.260291  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:55.300782  1874 rpcz_store.cc:276] 0709 12:24:54.214015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.342807 (+128792us) service_pool.cc:224] Handling call
0709 12:24:54.343199 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.641681 (+298482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.641711 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.641745 (+    34us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:54.641773 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.299654 (+657881us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54001f7a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:55.299723 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.300022 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":685993856}
I20260709 12:24:55.303786  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:55.263243  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:55.310272  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3400) took 1037 ms. Trace:
I20260709 12:24:55.310856  1878 rpcz_store.cc:276] 0709 12:24:54.273203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.485372 (+212169us) service_pool.cc:224] Handling call
0709 12:24:54.485725 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.485984 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:54.937918 (+451934us) log.cc:844] Serialized 14 byte log entry
0709 12:24:55.309471 (+371553us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:55.309586 (+   115us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:55.309717 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.309853 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.310090 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":162529,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":79,"thread_start_us":783639,"threads_started":2,"wal-append.queue_time_us":242982}
I20260709 12:24:55.268677  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:55.319739  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3392) took 1106 ms. Trace:
I20260709 12:24:55.320333  1881 rpcz_store.cc:276] 0709 12:24:54.213558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.306196 (+ 92638us) service_pool.cc:224] Handling call
0709 12:24:54.306540 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.306958 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.306998 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:54.307557 (+   559us) log.cc:844] Serialized 26 byte log entry
0709 12:24:54.990990 (+683433us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:54.998135 (+  7145us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.998423 (+   288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.998488 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.319096 (+320608us) raft_consensus.cc:1685] finished
0709 12:24:55.319147 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.319535 (+   388us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":552,"prepare.queue_time_us":6861,"prepare.run_wall_time_us":2497,"raft.queue_time_us":4939,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":9187,"spinlock_wait_cycles":11205760,"thread_start_us":586804,"threads_started":1,"wal-append.queue_time_us":790034}
I20260709 12:24:55.265838  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:55.301134  1872 rpcz_store.cc:276] 0709 12:24:54.209025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.210875 (+  1850us) service_pool.cc:224] Handling call
0709 12:24:55.299896 (+1089021us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400026054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:55.300174 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":628467584}
I20260709 12:24:55.323367  1801 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:55.323719  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3391) took 1110 ms. Trace:
I20260709 12:24:55.324075  1880 rpcz_store.cc:276] 0709 12:24:54.213322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.304659 (+ 91337us) service_pool.cc:224] Handling call
0709 12:24:54.305059 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.351987 (+ 46928us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.352036 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:54.389066 (+ 37030us) log.cc:844] Serialized 26 byte log entry
0709 12:24:54.882010 (+492944us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:54.882066 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.318407 (+436341us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b54002058d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:55.318486 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.318597 (+   111us) raft_consensus.cc:1685] finished
0709 12:24:55.318609 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.323487 (+  4878us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15091,"raft.queue_time_us":23055,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":33643,"spinlock_wait_cycles":71896960,"thread_start_us":417209,"threads_started":1,"wal-append.queue_time_us":607569}
I20260709 12:24:55.274642  1799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:24:55.262221  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 924) took 1186 ms. Trace:
I20260709 12:24:55.327859  1798 rpcz_store.cc:276] 0709 12:24:54.076021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.078388 (+  2367us) service_pool.cc:224] Handling call
0709 12:24:55.261768 (+1183380us) spinlock_profiling.cc:243] Waited 340 ms on lock 0x7b54003b74d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:55.261966 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":781057920}
I20260709 12:24:55.306896  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:24:55.331424  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 329261us
I20260709 12:24:55.332621  1799 raft_consensus.cc:606] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:24:55.307932  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 960) took 1333 ms. Trace:
I20260709 12:24:55.264246  1806 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:24:55.270054  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2493) took 1127 ms. Trace:
W20260709 12:24:55.332835  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.308959  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:55.321878  1805 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:55.175695  8470 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.345285  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 931) took 1251 ms. Trace:
W20260709 12:24:55.189401  8559 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.700s	user 0.008s	sys 0.006s
I20260709 12:24:55.160192  8437 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:55.340801  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1346) took 1190 ms. Trace:
I20260709 12:24:55.342204  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1277) took 1026 ms. Trace:
I20260709 12:24:55.335222  1800 rpcz_store.cc:276] 0709 12:24:54.143037 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.158208 (+ 15171us) service_pool.cc:224] Handling call
0709 12:24:55.269695 (+1111487us) spinlock_profiling.cc:243] Waited 48.7 ms on lock 0x7b54001d4554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:55.269891 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":112022912}
I20260709 12:24:55.334419  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:55.347713  2413 rpcz_store.cc:276] 0709 12:24:54.150188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.150344 (+   156us) service_pool.cc:224] Handling call
0709 12:24:54.150752 (+   408us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:54.175049 (+ 24297us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.175085 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:54.189710 (+ 14625us) log.cc:844] Serialized 52 byte log entry
0709 12:24:54.910181 (+720471us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:54.924197 (+ 14016us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.924478 (+   281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.924570 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.339975 (+415405us) raft_consensus.cc:1685] finished
0709 12:24:55.340035 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.340470 (+   435us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15749,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":6484,"raft.queue_time_us":29267,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":27107,"thread_start_us":636395,"threads_started":1,"wal-append.queue_time_us":885367}
I20260709 12:24:55.335721  1876 rpcz_store.cc:276] 0709 12:24:53.973980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.974423 (+   443us) service_pool.cc:224] Handling call
0709 12:24:53.974755 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:53.975035 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:53.975064 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:53.975752 (+   688us) log.cc:844] Serialized 26 byte log entry
0709 12:24:54.149188 (+173436us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:54.149233 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.149314 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.199745 (+1050431us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:55.307097 (+107352us) raft_consensus.cc:1685] finished
0709 12:24:55.307151 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.307632 (+   481us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12364672,"thread_start_us":172899,"threads_started":1,"wal-append.queue_time_us":440784}
W20260709 12:24:55.461139  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 127141us
I20260709 12:24:55.345983  1804 rpcz_store.cc:276] 0709 12:24:54.093243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.093416 (+   173us) service_pool.cc:224] Handling call
0709 12:24:54.183402 (+ 89986us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:54.651685 (+468283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.651724 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:54.692277 (+ 40553us) log.cc:844] Serialized 52 byte log entry
0709 12:24:55.066364 (+374087us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:55.066433 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.066593 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.344633 (+278040us) raft_consensus.cc:1685] finished
0709 12:24:55.344680 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.345019 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5551,"raft.queue_time_us":22100,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":7414,"thread_start_us":749308,"threads_started":2,"wal-append.queue_time_us":465372}
I20260709 12:24:55.347141  8559 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.459817  1727 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:55.347898  2415 rpcz_store.cc:276] 0709 12:24:54.315239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.315390 (+   151us) service_pool.cc:224] Handling call
0709 12:24:54.315691 (+   301us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:54.615974 (+300283us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:54.616018 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:54.626343 (+ 10325us) log.cc:844] Serialized 52 byte log entry
0709 12:24:55.001118 (+374775us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:55.001158 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.001227 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.341500 (+340273us) raft_consensus.cc:1685] finished
0709 12:24:55.341543 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.341902 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30703,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":5465,"thread_start_us":374455,"threads_started":1,"wal-append.queue_time_us":556813}
I20260709 12:24:55.434260  8670 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:55.481508  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.487401  8653 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.487876  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:55.488534  8653 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.489573  2412 raft_consensus.cc:2410] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:24:55.468397  8437 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:55.491581  8437 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:55.492856  8664 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:55.494234  8652 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.495299  8652 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:55.513041  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51675.1us
I20260709 12:24:55.541888  8616 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:55.557663  8615 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.558902  8615 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:55.563515  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 495767us
W20260709 12:24:55.566766  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 232622us
I20260709 12:24:55.567570  8470 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:55.568665  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:55.569289  8470 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:55.570123  8400 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:24:55.570818  8400 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:55.571322  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:55.572429  8400 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.568801  8398 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:55.574505  8400 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:55.584005  8563 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:55.585211  8563 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:55.585812  8416 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:55.587435  8652 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:55.588753  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:55.589864  1729 raft_consensus.cc:606] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:24:55.590442  1729 raft_consensus.cc:614] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:55.596299  8683 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.553028  8662 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:55.597801  8416 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.599752  2418 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:55.599941  2416 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 13. (term mismatch)
W20260709 12:24:55.600376  8680 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.055s	user 0.002s	sys 0.000s
I20260709 12:24:55.600267  8380 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.552289  8632 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:55.465261  8597 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.558120  8670 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.576951  8398 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:55.637348  1771 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:55.638007  8398 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.637360  8653 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:55.639565  8398 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:55.625097  8521 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.103s	user 0.013s	sys 0.005s
I20260709 12:24:55.641605  8521 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.643873  8519 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:55.644223  8521 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.619333  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:55.620359  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:55.646206  8657 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:55.621868  1726 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:55.622681  1798 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:24:55.623453  8380 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.574980  8539 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:55.625237  8416 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:55.465253  8417 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.610005  2416 pending_rounds.cc:85] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 12
I20260709 12:24:55.631351  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:55.635618  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:55.636487  8615 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:55.614147  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46010.3us
I20260709 12:24:55.616416  8559 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:55.646559  1730 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:55.620782  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:55.646739  1732 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:55.566694  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:55.665369  8683 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.659801  8380 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.665196  8688 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:55.658746  8347 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.502s	user 0.007s	sys 0.007s
I20260709 12:24:55.681480  1727 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:24:55.668860  2416 pending_rounds.cc:107] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 12:24:55.689668  2416 raft_consensus.cc:2889] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:55.655768  8662 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.653839  8657 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:55.690905  1725 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:24:55.692497  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:55.693228  8343 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.999s	user 0.007s	sys 0.017s
I20260709 12:24:55.693486  1725 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:55.693939  8343 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.684907  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:55.683364  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46296.9us
I20260709 12:24:55.701702  1726 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:55.700327  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:55.643922  8423 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.694706  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:55.696260  8000 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.990s	user 0.013s	sys 0.016s
I20260709 12:24:55.719517  8000 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.721339  1731 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:24:55.725961  1730 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [18.13-20.14]
I20260709 12:24:55.727082  1873 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:24:55.727454  1725 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:24:55.718657  1726 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:24:55.707093  8519 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:55.729780  8423 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.734134  8343 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:55.735968  8657 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.739521  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:55.741374  8352 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:55.742280  8352 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.743903  8632 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.747040  8519 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.748025  8519 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.752051  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:55.752935  8470 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:55.729869  7987 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260709 12:24:55.754492  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:55.755808  1880 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:55.746186  8675 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.757864  8675 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.761952  8670 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:55.762825  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 454270us
I20260709 12:24:55.764168  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:55.764686  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:55.753852  8470 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:55.772215  8690 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:24:55.773488  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:55.776540  8613 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.949s	user 0.019s	sys 0.000s
I20260709 12:24:55.777222  8613 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.779052  1876 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:55.783742  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:55.784523  8670 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:55.784649  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68666us
I20260709 12:24:55.785290  8670 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.790867  8678 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:55.791723  8678 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:55.793197  8678 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.811209  8175 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:55.811985  8175 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:55.815352  8621 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:55.819638  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 253271us
I20260709 12:24:55.829144  8380 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:55.836439  8380 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.836788  8662 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:55.841252  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:55.841668  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 77874.4us
I20260709 12:24:55.843472  8657 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:55.861289  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:55.839900  8683 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:55.864956  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78970us
I20260709 12:24:55.852310  8352 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.864123  2416 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:24:55.877331  2412 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:55.877985  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 936) took 1091 ms. Trace:
I20260709 12:24:55.878485  1806 rpcz_store.cc:276] 0709 12:24:54.786738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.792887 (+  6149us) service_pool.cc:224] Handling call
0709 12:24:55.263288 (+470401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:55.334921 (+ 71633us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.877727 (+542806us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54003b74d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:55.877910 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1080661248}
I20260709 12:24:55.879330  8675 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:55.864282  8343 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:55.880625  8343 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.882293  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:55.872331  8662 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:55.885494  8662 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.888026  8695 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:55.888576  8662 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.890625  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:55.893214  8474 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:55.893828  8474 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:55.894371  8632 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:55.895103  8610 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:55.895705  8610 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:55.895115  8474 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.896914  8610 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.898890  8684 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:55.899152  8610 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:55.900218  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:55.901409  8662 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:24:55.901623  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47083.6us
I20260709 12:24:55.883191  8591 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:55.909206  8632 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:24:55.909619  8693 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.218s	user 0.003s	sys 0.000s
I20260709 12:24:55.910112  8632 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.883668  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:55.884032  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:55.884307  8519 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.879937  8423 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:55.882411  8493 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.213s	user 0.012s	sys 0.015s
W20260709 12:24:55.913945  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50589.6us
I20260709 12:24:55.911379  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:55.921429  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:55.922242  1880 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:55.920007  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:55.925542  8633 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.927093  8343 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:55.866546  8000 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.919919  2415 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:55.945571  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:55.945997  8519 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:24:55.946775  8519 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:55.948760  8613 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:55.949241  8684 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:55.948426  8519 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.955133  8569 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:24:55.956899  8684 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.959656  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:55.959214  8703 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.004s
I20260709 12:24:55.960915  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:55.961129  8653 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:55.961876  1877 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:55.965687  8675 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:55.936839  8493 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.966449  8675 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.966456  8684 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.968550  8675 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:55.970660  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:55.961771  8653 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:55.971980  8653 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.973268  8653 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:55.975000  8569 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:55.979313  8569 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:55.981164  8569 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:55.983672  8701 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:55.984102  8670 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:55.984735  8701 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.985492  8662 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:55.945607  7611 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.280s	user 0.019s	sys 0.007s
I20260709 12:24:55.986465  8662 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:55.965494  8437 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.009931  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2682) took 1382 ms. Trace:
I20260709 12:24:55.951979  8470 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.011889  8490 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:56.010536  1873 rpcz_store.cc:276] 0709 12:24:54.627840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.890709 (+262869us) service_pool.cc:224] Handling call
0709 12:24:54.915178 (+ 24469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:55.727729 (+812551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.001787 (+274058us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b5400072e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:56.009837 (+  8050us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385882880}
I20260709 12:24:56.010632  1881 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:56.014851  8490 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:56.015396  8490 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:56.010700  1874 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:24:56.017616  8437 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:55.968538  8493 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:56.054381  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 151749us
W20260709 12:24:56.054469  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 83976.6us
W20260709 12:24:56.064615  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89207us
I20260709 12:24:56.080555  8255 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:56.081598  8701 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:56.082577  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 67265.6us
I20260709 12:24:56.077399  8474 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.006s
I20260709 12:24:56.084507  8528 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:56.084985  8528 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:56.087019  8591 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:56.087882  8591 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:56.088954  8644 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.941s	user 0.002s	sys 0.000s
I20260709 12:24:56.081393  8255 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.047334  8688 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:56.103106  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:56.102042  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:56.108408  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47059.1us
I20260709 12:24:56.090548  8591 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.128499  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.098706  8380 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.086189  8528 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.135849  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:56.143802  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.143959  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.144663  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:56.151105  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.150935  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.152246  8175 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.152992  8632 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.153820  1728 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:24:56.146227  2410 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:56.139645  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.170511  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:56.097584  8633 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.042s
I20260709 12:24:56.177985  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.171453  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.173183  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.172328  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.172230  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.163650  1729 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 12:24:56.180156  1729 pending_rounds.cc:85] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 10
I20260709 12:24:56.177295  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:56.180460  1729 pending_rounds.cc:107] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260709 12:24:56.180860  1729 raft_consensus.cc:2889] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:56.178015  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.184612  1732 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:56.189224  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.170208  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.194595  1872 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:24:56.192137  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.206815  1876 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:56.210238  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 79794.3us
W20260709 12:24:56.211592  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57534.5us
I20260709 12:24:56.215824  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.230823  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:56.226390  1730 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
W20260709 12:24:56.210700  8483 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.170s	user 0.010s	sys 0.020s
I20260709 12:24:56.215706  1880 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:24:56.192936  8528 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:56.238020  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.259528  8583 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.260179  8609 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:56.261588  8482 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:24:56.332113  8609 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.332582  8609 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:56.263744  8175 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.264308  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:56.334645  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70441us
I20260709 12:24:56.335008  8175 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:56.335808  8175 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:24:56.267215  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 75566.9us
I20260709 12:24:56.267398  8437 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:56.268373  8651 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:56.268963  8491 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:56.269265  8594 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:56.270709  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.271767  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.272517  2412 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:56.247268  1723 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:24:56.272979  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.273790  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.274631  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.271799  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:24:56.276839  8398 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.637s	user 0.017s	sys 0.006s
I20260709 12:24:56.277989  1725 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:56.267313  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:56.280639  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 120160us
I20260709 12:24:56.281320  8614 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.282684  1875 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:56.284320  8255 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.160862  8688 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.072s
I20260709 12:24:56.191643  8613 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:56.289753  8682 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.289986  8423 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:56.144254  8701 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.073s
I20260709 12:24:56.294180  8539 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.295023  8703 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:56.296768  8591 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.413579  8591 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.416106  8591 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:56.301447  8343 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:56.416837  8343 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:56.417968  8343 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.419605  8343 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:56.306839  8633 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.423221  8633 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.307814  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:56.234232  8474 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:56.427778  8474 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.428366  8609 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.320358  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.320716  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:24:56.329353  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.333308  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.336953  8175 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.338312  8651 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:56.346864  1724 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:24:56.341148  2413 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:56.441673  1880 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:56.339746  1798 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:24:56.368482  1728 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:56.370676  1881 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:56.375900  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.379840  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:56.379922  8613 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.377842  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:56.384701  1872 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:56.376832  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:56.382455  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:56.387669  8400 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.781s	user 0.022s	sys 0.007s
I20260709 12:24:56.388036  8423 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.445703  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.347115  8594 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:56.394454  8688 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:56.394806  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:56.395044  8682 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.466073  1805 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:56.397847  1730 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:56.398703  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.400504  8671 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:56.373142  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:56.403182  1772 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:56.403414  8695 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.403787  8483 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:56.466445  1801 raft_consensus.cc:2376] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19. Re-sending same reply.
I20260709 12:24:56.405941  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.405763  8515 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:56.406952  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:56.407717  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.410346  1723 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:24:56.403234  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:56.412506  8662 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.429991  8474 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.447184  1803 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
W20260709 12:24:56.448243  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.450076  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.451979  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:56.453575  8657 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.584s	user 0.015s	sys 0.000s
I20260709 12:24:56.455274  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:56.457823  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78137.7us
I20260709 12:24:56.484309  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.459831  8613 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.461679  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:56.461946  8651 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.346944  8491 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.467902  1876 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:24:56.463186  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:56.405216  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:56.477473  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 74343.8us
I20260709 12:24:56.487289  1879 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:24:56.467226  8688 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:56.453593  8609 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.459864  8633 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.476020  8695 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.632616  8701 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:56.490556  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:56.513537  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:56.704595  2418 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:56.525385  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 158716us
I20260709 12:24:56.482416  8597 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:56.707945  8597 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.708395  8597 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:56.715272  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.716279  1872 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:56.716784  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:56.717170  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:56.718067  1872 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:56.718254  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:56.444011  8539 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.723489  8688 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.724773  8539 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.554126  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.581216  8594 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.467731  8490 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.727279  8594 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:56.728286  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:56.566864  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:56.564534  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:56.568327  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 116446us
I20260709 12:24:56.730023  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:56.586846  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.731361  1732 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:56.569731  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.490816  8652 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:56.471145  8519 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:56.631954  1878 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:24:56.489296  1880 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:56.645345  1879 pending_rounds.cc:85] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 8
I20260709 12:24:56.740190  1879 pending_rounds.cc:107] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:24:56.740547  1879 raft_consensus.cc:2889] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:56.373229  8614 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.705979  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.752503  1876 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:24:56.707271  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:56.728528  8490 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.731451  2418 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:24:56.491007  8352 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:56.726696  1874 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:56.735172  1799 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:56.648069  8651 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:56.751089  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.751444  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:56.760401  1732 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:56.760670  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:56.738654  1877 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
W20260709 12:24:56.762991  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 133689us
I20260709 12:24:56.764036  8491 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.764356  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:56.765060  8519 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:56.766026  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:56.767032  8519 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:56.771229  8718 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.294s	user 0.000s	sys 0.009s
I20260709 12:24:56.766480  8398 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:56.773627  8423 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.777387  8688 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:56.779675  2410 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:24:56.781068  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.736284  8583 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:56.782425  8583 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.783602  1798 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:24:56.784713  8583 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.785630  8614 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.759205  8609 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.786505  8609 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.788599  8609 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:56.790735  8695 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:56.791056  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 217914us
I20260709 12:24:56.792800  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:56.793753  8398 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:56.794313  8398 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.794698  8398 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:56.795297  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.795718  1803 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:24:56.795971  2416 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:24:56.799491  1798 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:56.806749  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55488.4us
I20260709 12:24:56.737591  8652 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:24:56.798213  8569 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.779s	user 0.019s	sys 0.000s
I20260709 12:24:56.828258  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:56.805608  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3410) took 1045 ms. Trace:
W20260709 12:24:56.824770  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94829.1us
I20260709 12:24:56.760300  8175 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:56.758247  8352 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.831065  1874 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:24:56.815864  8184 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:56.806010  8609 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:56.806075  8528 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.829389  1879 rpcz_store.cc:276] 0709 12:24:55.760671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.760858 (+   187us) service_pool.cc:224] Handling call
0709 12:24:55.761313 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.796826 (+1035513us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.798720 (+  1894us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b540021b6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:56.805543 (+  6823us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1637662976}
I20260709 12:24:56.798101  8682 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.816888  8591 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:24:56.838647  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 131565us
I20260709 12:24:56.841171  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:56.834173  8184 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.841625  1873 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 12:24:56.845170  8184 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:56.847769  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 81904us
I20260709 12:24:56.849462  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:56.834414  8352 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:56.845230  1873 pending_rounds.cc:85] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 8
I20260709 12:24:56.860570  1873 pending_rounds.cc:107] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:24:56.860917  1873 raft_consensus.cc:2889] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:56.856472  8678 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:56.863034  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:56.862916  8678 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.865577  8678 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:56.866082  8582 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 19.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:24:56.866765  8569 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.867597  8569 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:56.869990  8569 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:56.870647  8184 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.846882  8609 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:56.872519  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:56.865478  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:56.871075  8490 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:56.846109  8591 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.880005  1874 raft_consensus.cc:606] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:24:56.891515  1874 raft_consensus.cc:614] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:56.849496  8632 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:56.899276  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.892079  8614 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:56.929379  1727 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:56.854463  8528 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:24:56.934343  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.945799  1878 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 13.
I20260709 12:24:56.900890  8632 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:56.894559  8591 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:56.899108  8569 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.926277  8539 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:56.933553  1880 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:56.931571  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:56.898465  8582 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:56.951331  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:56.917383  8400 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.873730  8609 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.959874  1800 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 13.
I20260709 12:24:56.961061  1726 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:56.974359  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:56.975260  1727 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:56.977008  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.985330  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.986446  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:56.953624  8632 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:57.002904  8578 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.003252  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:57.003616  8578 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:57.004602  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:57.005175  1800 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:24:57.005786  8578 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:57.006075  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:57.007227  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 134906us
I20260709 12:24:57.008282  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:57.010103  8597 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:56.969354  8695 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:24:57.019078  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:57.022845  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:57.023874  8657 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:56.967139  8582 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:57.028091  8582 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.029301  8582 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:57.023969  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 181590us
I20260709 12:24:56.896277  8652 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.048231  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:57.049750  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:56.995215  8490 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:56.981652  8400 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:57.077600  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:57.042011  8633 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:57.084947  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:57.002928  8569 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:57.096560  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 203045us
I20260709 12:24:57.061862  8652 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:57.040565  8255 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
W20260709 12:24:57.102442  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59777.5us
W20260709 12:24:57.105551  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44177us
I20260709 12:24:57.104806  8662 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:57.056823  8578 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:24:57.085121  8597 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:57.064364  8539 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:24:57.112216  1879 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:24:57.098464  8519 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:57.114553  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:57.121260  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:24:57.118819  1727 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:57.125977  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:57.130213  1876 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
W20260709 12:24:57.119722  8722 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.273s	user 0.000s	sys 0.002s
I20260709 12:24:57.137547  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:57.139601  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:57.140305  1874 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:57.105382  8398 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.140394  1878 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:57.169339  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54939us
I20260709 12:24:57.170149  8591 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.141615  8688 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:57.171211  8688 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:57.171478  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:57.171701  8688 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:57.172479  8591 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:57.173417  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:57.141338  8255 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.034s
W20260709 12:24:57.176038  8727 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:24:57.141789  8519 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:57.177577  8519 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:57.179594  8296 consensus_queue.cc:1175] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "075704438992440697732181e49ccd09" responder_term: 21 status { last_received { term: 20 index: 15 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:24:57.181255  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67761.2us
I20260709 12:24:57.182281  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:57.165530  8729 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.059s	user 0.003s	sys 0.000s
I20260709 12:24:57.191406  8528 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:24:57.180413  8519 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:57.218726  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49366.5us
I20260709 12:24:57.177597  8671 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:57.218746  8482 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.822s	user 0.026s	sys 0.069s
W20260709 12:24:57.235082  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.217363  8657 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 12:24:57.217042  8682 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
I20260709 12:24:57.241039  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:57.225808  8725 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.107s	user 0.002s	sys 0.002s
I20260709 12:24:57.237861  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:57.222517  8255 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.216797  8662 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.016s
W20260709 12:24:57.263355  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:57.245352  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 459867us
I20260709 12:24:57.277895  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:57.283226  2410 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:57.239758  8482 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.026s	sys 0.070s
W20260709 12:24:57.285682  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 103549us
I20260709 12:24:57.297124  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:57.299052  8539 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:57.299664  8539 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.300169  8539 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:57.285459  8255 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:57.287447  8662 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:57.290007  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:57.290709  8724 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.523s	user 0.000s	sys 0.004s
I20260709 12:24:57.291049  8400 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:57.294615  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.235833  8594 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:24:57.297363  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 56121us
I20260709 12:24:57.235621  8671 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.302624  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:24:57.339609  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 474193us
I20260709 12:24:57.309514  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:24:57.305464  8519 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:57.306174  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:24:57.344247  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45529.2us
I20260709 12:24:57.345171  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:57.244256  8578 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.082s
I20260709 12:24:57.318214  8662 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:57.349921  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:57.356463  1804 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:24:57.374948  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:57.341452  8569 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:57.343338  8671 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:57.398391  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:57.351027  8296 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:57.352001  8597 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:57.377501  8519 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:57.377772  8482 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.063s
I20260709 12:24:57.403558  8703 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:57.510847  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:57.512557  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.512434  1878 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:24:57.516837  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:57.513592  1804 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
W20260709 12:24:57.518026  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:57.519582  8730 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260709 12:24:57.519402  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:57.523741  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3425) took 1189 ms. Trace:
I20260709 12:24:57.529273  1873 rpcz_store.cc:276] 0709 12:24:56.334680 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.334852 (+   172us) service_pool.cc:224] Handling call
0709 12:24:56.335226 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.897238 (+562012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.523429 (+626191us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b540001efd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:57.523674 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1153241344}
I20260709 12:24:57.538306  1804 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:57.539891  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:57.548249  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:57.549247  1800 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
W20260709 12:24:57.551025  8724 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:24:57.552556  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 946) took 1260 ms. Trace:
I20260709 12:24:57.552433  8184 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:57.417956  8597 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.559192  8597 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:57.560400  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:57.564364  8657 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.565115  8657 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.567178  8633 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:57.567668  8633 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:57.568256  8657 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:57.568706  8633 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.569818  8633 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:57.575132  8662 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.415832  8296 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:57.577533  8296 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:57.587471  8519 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:24:57.595204  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76512.8us
W20260709 12:24:57.597549  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.553089  1802 rpcz_store.cc:276] 0709 12:24:56.330376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.330546 (+   170us) service_pool.cc:224] Handling call
0709 12:24:56.330999 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:56.331242 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.331270 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:56.331858 (+   588us) log.cc:844] Serialized 26 byte log entry
0709 12:24:56.624219 (+292361us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:56.646989 (+ 22770us) log.cc:844] Serialized 14 byte log entry
0709 12:24:56.647688 (+   699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.647821 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:56.768995 (+121174us) spinlock_profiling.cc:243] Waited 5.24 ms on lock 0x7b1800271de0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:56.773631 (+  4636us) spinlock_profiling.cc:243] Waited 33.3 ms on lock 0x7b1800271de0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1
0709 12:24:56.774050 (+   419us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b1800271de0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:24:57.289969 (+515919us) spinlock_profiling.cc:243] Waited 31.9 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:24:57.552186 (+262217us) raft_consensus.cc:1685] finished
0709 12:24:57.552209 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.552337 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":61750,"prepare.queue_time_us":22230,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":208,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":47622,"spinlock_wait_cycles":209712128,"thread_start_us":291956,"threads_started":1,"wal-append.queue_time_us":337826}
I20260709 12:24:57.601222  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:24:57.612248  2415 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.004s
I20260709 12:24:57.612900  2415 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
W20260709 12:24:57.613600  1803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.011s	sys 0.038s
W20260709 12:24:57.614972  1803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.817s	user 0.011s	sys 0.038s
I20260709 12:24:57.509380  8398 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:57.603542  8721 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:24:57.598367  8735 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:57.603128  8695 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:24:57.646430  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2591) took 1247 ms. Trace:
W20260709 12:24:57.648698  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 477300us
I20260709 12:24:57.648919  8737 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:24:57.652375  8609 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.653121  8609 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.655293  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:57.655649  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54553.3us
I20260709 12:24:57.656395  8701 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.656644  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:57.658123  1731 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:57.659224  8678 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:57.633319  8491 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.663838  8739 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.663443  8491 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.664489  8739 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:57.656066  1727 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:57.658185  8737 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.629060  8515 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.675565  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 944) took 1384 ms. Trace:
I20260709 12:24:57.696888  2410 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:24:57.676631  8737 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.701453  8739 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:57.702912  8739 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:57.703488  8739 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:57.703595  8743 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:57.704059  8739 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:57.704695  8743 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.709863  1727 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:57.731336  8255 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:57.647095  1803 rpcz_store.cc:276] 0709 12:24:56.399429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.796504 (+397075us) service_pool.cc:224] Handling call
0709 12:24:56.796797 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.797091 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:56.797414 (+   323us) log.cc:844] Serialized 14 byte log entry
0709 12:24:57.630771 (+833357us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.645958 (+ 15187us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:57.646044 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.646136 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.646332 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25537,"prepare.queue_time_us":54349,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"thread_start_us":818343,"threads_started":1}
I20260709 12:24:57.758498  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:57.763648  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:57.681468  8490 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:57.657044  8701 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.680972  1806 rpcz_store.cc:276] 0709 12:24:56.291207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.292401 (+  1194us) service_pool.cc:224] Handling call
0709 12:24:56.292854 (+   453us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:56.492223 (+199369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.628515 (+136292us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:56.707054 (+ 78539us) log.cc:844] Serialized 52 byte log entry
0709 12:24:57.112928 (+405874us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:57.132362 (+ 19434us) log.cc:844] Serialized 14 byte log entry
0709 12:24:57.132599 (+   237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.667640 (+535041us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b5400133154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:57.667691 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:57.667771 (+    80us) raft_consensus.cc:1685] finished
0709 12:24:57.675350 (+  7579us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.675503 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43018,"prepare.queue_time_us":19177,"prepare.run_cpu_time_us":2940,"prepare.run_wall_time_us":27720,"raft.queue_time_us":6786,"raft.run_cpu_time_us":330,"raft.run_wall_time_us":37745,"spinlock_wait_cycles":436530560,"thread_start_us":400053,"threads_started":1,"wal-append.queue_time_us":603080}
I20260709 12:24:57.678283  8398 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.082s
I20260709 12:24:57.686976  8493 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.097s
W20260709 12:24:57.775677  8352 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.856s	user 0.011s	sys 0.005s
I20260709 12:24:57.782696  8493 consensus_queue.cc:1243] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:24:57.782823  8352 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:57.785274  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3426) took 1345 ms. Trace:
I20260709 12:24:57.785842  1877 rpcz_store.cc:276] 0709 12:24:56.440300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.762168 (+321868us) service_pool.cc:224] Handling call
0709 12:24:56.814704 (+ 52536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:56.832185 (+ 17481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.832233 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:56.852546 (+ 20313us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.348578 (+496032us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:57.348639 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.348734 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:57.785009 (+436275us) raft_consensus.cc:1685] finished
0709 12:24:57.785031 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.785207 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20329,"raft.queue_time_us":6210,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":12229,"thread_start_us":428671,"threads_started":1,"wal-append.queue_time_us":787589}
I20260709 12:24:57.788604  1803 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:24:57.786085  8352 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:57.685237  8515 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.741684  8737 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:57.794932  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:57.797359  8662 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:57.797499  8701 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:57.799052  1728 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:57.801395  1730 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:57.794327  8515 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:57.822309  2415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1287) took 1408 ms (client timeout 1876 ms). Trace:
I20260709 12:24:57.822698  8657 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:57.823699  8657 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:57.823143  2415 rpcz_store.cc:269] 0709 12:24:56.398960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.406561 (+  7601us) service_pool.cc:224] Handling call
0709 12:24:57.807333 (+1400772us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b540049eb54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:24:57.807516 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1430199680}
I20260709 12:24:57.834591  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:57.836493  1728 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:57.836659  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:57.815850  8652 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.710s	user 0.013s	sys 0.004s
I20260709 12:24:57.826315  1624 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 reported cstate change: term changed from 12 to 15, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 15 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:57.842877  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.019s	sys 0.053s
W20260709 12:24:57.843546  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.669s	user 0.019s	sys 0.054s
I20260709 12:24:57.842090  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:57.828387  8582 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.794s	user 0.013s	sys 0.000s
W20260709 12:24:57.835745  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 496821us
I20260709 12:24:57.850363  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:57.851291  1725 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:24:57.840641  8539 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.518s	user 0.014s	sys 0.000s
W20260709 12:24:57.841039  8688 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.669s	user 0.014s	sys 0.000s
I20260709 12:24:57.827369  8657 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:57.845654  2416 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.015s	sys 0.018s
W20260709 12:24:57.868650  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.855804  1803 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:24:57.874326  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:57.856523  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:57.899130  2416 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.015s	sys 0.019s
I20260709 12:24:57.855373  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:57.898965  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:57.914541  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:57.930316  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 75059.1us
I20260709 12:24:57.931078  1724 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:57.937318  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:57.844314  8400 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.907369  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:57.953244  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 779869us
W20260709 12:24:57.876984  8175 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.000s	sys 0.025s
I20260709 12:24:57.873275  8609 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:57.955751  1725 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:24:57.907785  8746 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
W20260709 12:24:57.917006  8741 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:24:57.969524  1725 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:57.950913  8255 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:57.971321  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:57.948563  8682 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:57.973413  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:57.974766  1728 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:57.949551  8632 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.902s	user 0.015s	sys 0.001s
I20260709 12:24:57.983413  1624 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:57.971451  8747 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
W20260709 12:24:58.002889  1798 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.004s	sys 0.034s
W20260709 12:24:58.023097  1798 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.004s	sys 0.034s
I20260709 12:24:58.024550  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2595) took 1605 ms. Trace:
W20260709 12:24:58.026120  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.017s	sys 0.036s
W20260709 12:24:58.003279  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.010s	sys 0.019s
W20260709 12:24:58.027300  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.017s	sys 0.036s
W20260709 12:24:58.004251  1799 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.019s	sys 0.010s
W20260709 12:24:58.009577  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:58.028479  1799 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.019s	sys 0.010s
W20260709 12:24:58.028226  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 154057us
I20260709 12:24:58.029807  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2586) took 1631 ms. Trace:
I20260709 12:24:58.029884  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:58.030464  1799 rpcz_store.cc:276] 0709 12:24:56.398175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.764867 (+366692us) service_pool.cc:224] Handling call
0709 12:24:56.765334 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.765638 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:57.247689 (+482051us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.029417 (+781728us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.029476 (+    59us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:58.029544 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.029613 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.029747 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":64901,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":2226,"raft.queue_time_us":103,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":144,"thread_start_us":1177067,"threads_started":2}
W20260709 12:24:58.027588  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.010s	sys 0.019s
W20260709 12:24:58.021930  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.014s	sys 0.017s
W20260709 12:24:58.039090  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.014s	sys 0.018s
I20260709 12:24:58.033533  1624 catalog_manager.cc:5697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 16 to 18, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 18 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:24:58.040189  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 695063us
I20260709 12:24:57.903021  8688 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:58.040596  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 74312.2us
I20260709 12:24:57.904573  8539 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:57.910418  8597 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:58.044803  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:58.047670  1805 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.007s	sys 0.048s
W20260709 12:24:58.049041  1805 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.007s	sys 0.048s
W20260709 12:24:58.051388  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.045433  1729 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
W20260709 12:24:58.041625  8578 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.009s	sys 0.023s
I20260709 12:24:58.041049  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:58.026178  8657 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:58.051999  8633 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:58.069638  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:24:58.070976  1804 raft_consensus.cc:2368] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:24:57.986411  8400 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.041656  8743 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:58.080173  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:57.995260  8632 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:58.042905  8491 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:58.085304  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.082706  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 951) took 1702 ms. Trace:
I20260709 12:24:58.091758  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:58.057585  8749 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:58.088900  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.096570  1729 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:58.098542  1725 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:24:58.072074  8528 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:58.095551  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:58.072507  8682 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.094264  1805 rpcz_store.cc:276] 0709 12:24:56.380169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.719403 (+339234us) service_pool.cc:224] Handling call
0709 12:24:56.719774 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.720123 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:57.055402 (+335279us) log.cc:844] Serialized 14 byte log entry
0709 12:24:57.459112 (+403710us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.050354 (+591242us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.050436 (+    82us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:58.070224 (+ 19788us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.070425 (+   201us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.082582 (+ 12157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55147,"prepare.queue_time_us":76573,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":86,"raft.queue_time_us":550603,"raft.run_cpu_time_us":393,"raft.run_wall_time_us":80233,"thread_start_us":1241944,"threads_started":3,"wal-append.queue_time_us":612656}
I20260709 12:24:58.025317  1798 rpcz_store.cc:276] 0709 12:24:56.418973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.827870 (+408897us) service_pool.cc:224] Handling call
0709 12:24:56.839193 (+ 11323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.855504 (+ 16311us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:56.878071 (+ 22567us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.023793 (+1145722us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.023902 (+   109us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:58.023983 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.024087 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.024298 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50645,"prepare.queue_time_us":21952,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":279966,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":7981,"thread_start_us":1048968,"threads_started":2,"wal-append.queue_time_us":863765}
I20260709 12:24:58.099578  1728 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:58.109900  1806 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:58.110636  1624 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 16 to 19. New cstate: current_term: 19 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:58.065312  8578 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.010s	sys 0.023s
W20260709 12:24:58.096418  8613 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.018s	sys 0.030s
W20260709 12:24:58.113889  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72962.9us
I20260709 12:24:58.115015  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:58.104216  8671 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.635s	user 0.010s	sys 0.001s
I20260709 12:24:58.093055  8493 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:58.094857  8657 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:24:58.121062  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 464464us
I20260709 12:24:58.116361  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:58.089433  8750 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:58.118180  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:58.108317  1807 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.005s	sys 0.012s
I20260709 12:24:58.126960  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2520) took 1130 ms. Trace:
I20260709 12:24:58.127419  1806 rpcz_store.cc:276] 0709 12:24:56.996996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.842162 (+845166us) service_pool.cc:224] Handling call
0709 12:24:57.872582 (+ 30420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.114043 (+241461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.126573 (+ 12530us) spinlock_profiling.cc:243] Waited 234 ms on lock 0x7b54003b88d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:58.126859 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":539340032}
W20260709 12:24:58.117085  8613 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.019s	sys 0.030s
I20260709 12:24:58.129524  1732 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:24:58.125694  1807 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.005s	sys 0.013s
W20260709 12:24:58.131770  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.008s	sys 0.020s
W20260709 12:24:58.132344  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.008s	sys 0.021s
I20260709 12:24:58.125828  8671 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.133788  8756 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:24:58.136274  8755 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:58.123903  8657 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.137668  8657 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:24:58.139187  1876 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.012s	sys 0.035s
W20260709 12:24:58.139846  1876 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.012s	sys 0.036s
I20260709 12:24:58.087219  8633 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.099886  8255 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:24:58.125131  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.153756  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:58.154925  2382 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:58.156486  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:24:58.123155  8749 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:24:58.158001  1801 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.678s	user 0.007s	sys 0.016s
W20260709 12:24:58.179535  1801 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.007s	sys 0.016s
I20260709 12:24:58.180522  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 952) took 1800 ms. Trace:
I20260709 12:24:58.178383  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:58.148247  1624 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 reported cstate change: term changed from 11 to 14, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 14 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:58.187305  1728 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:58.186159  8483 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:58.189050  8398 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:58.189595  8398 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.190043  8398 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:58.196126  1624 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 11 to 15, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 15 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:58.198367  1624 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 11 to 18, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 18 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:58.187252  2382 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:58.211988  1724 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:24:58.140331  8760 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:58.215534  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59140.6us
I20260709 12:24:58.181537  1801 rpcz_store.cc:276] 0709 12:24:56.380425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.721804 (+341379us) service_pool.cc:224] Handling call
0709 12:24:56.722217 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.722475 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:57.098883 (+376408us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.180051 (+1081168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.180128 (+    77us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:58.180196 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.180281 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.180453 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":52012,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":25,"raft.queue_time_us":253636,"raft.run_cpu_time_us":73,"raft.run_wall_time_us":73,"thread_start_us":1449149,"threads_started":3,"wal-append.queue_time_us":596289}
W20260709 12:24:58.217298  1723 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.019s	sys 0.031s
W20260709 12:24:58.252274  1723 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.019s	sys 0.031s
I20260709 12:24:58.120584  8682 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:58.220558  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 291915us
I20260709 12:24:58.230724  1624 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 11 to 16, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 16 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:58.141525  8493 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:58.217968  8743 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:58.248230  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:58.262329  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.199788  8491 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:58.212451  8352 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:58.273739  1729 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:24:58.274986  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:58.276089  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 129258us
I20260709 12:24:58.230527  8490 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:58.146488  8750 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:58.261459  8519 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.007s	sys 0.014s
I20260709 12:24:58.285987  8753 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:24:58.287914  2384 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.005s	sys 0.013s
I20260709 12:24:58.323081  1728 pending_rounds.cc:85] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 11
I20260709 12:24:58.323446  1728 pending_rounds.cc:107] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 12:24:58.289058  8400 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:58.289996  8398 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:58.295706  8743 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:58.296383  8482 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:58.296911  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:58.297540  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:58.298040  8352 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:58.266426  8578 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:58.298846  1624 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 11 to 18, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 18 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:58.302042  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2509) took 1466 ms. Trace:
I20260709 12:24:58.303292  1725 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:58.312122  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:58.284081  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:58.323671  1728 raft_consensus.cc:2889] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:58.238271  8703 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:58.326388  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:24:58.331346  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:58.325851  8519 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.007s	sys 0.015s
I20260709 12:24:58.383623  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:58.510875  1799 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
W20260709 12:24:58.385898  1630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 172) took 73582 ms (client timeout 15000 ms). Trace:
I20260709 12:24:58.382808  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:58.384385  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:58.272593  8491 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:24:58.319825  8739 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.615s	user 0.005s	sys 0.009s
W20260709 12:24:58.433058  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 108254us
I20260709 12:24:58.381527  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:58.328527  1727 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
W20260709 12:24:58.438319  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 342766us
I20260709 12:24:58.537895  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:58.538869  1725 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:58.539283  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:58.445935  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:58.446455  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:58.448058  1803 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:24:58.328542  8764 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:58.453022  1802 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.012s	sys 0.035s
W20260709 12:24:58.553103  1802 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.012s	sys 0.035s
I20260709 12:24:58.553741  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:24:58.464202  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 310387us
I20260709 12:24:58.477233  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2585) took 2079 ms. Trace:
I20260709 12:24:58.475176  1772 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:58.474488  2415 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 22 because replica has last-logged OpId of term: 20 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 12:24:58.460167  8400 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:24:58.518163  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 206118us
I20260709 12:24:58.473495  1800 rpcz_store.cc:276] 0709 12:24:56.835886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.549890 (+714004us) service_pool.cc:224] Handling call
0709 12:24:57.550280 (+   390us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:57.639273 (+ 88993us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.639326 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:57.675516 (+ 36190us) log.cc:844] Serialized 52 byte log entry
0709 12:24:58.123937 (+448421us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:58.123991 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.237951 (+113960us) spinlock_profiling.cc:243] Waited 59.6 ms on lock 0x7b54001680d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:58.238023 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.301715 (+ 63692us) raft_consensus.cc:1685] finished
0709 12:24:58.301745 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.301973 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":102341,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"spinlock_wait_cycles":137164160,"thread_start_us":420741,"threads_started":1,"wal-append.queue_time_us":613159}
W20260709 12:24:58.457226  8296 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.877s	user 0.005s	sys 0.016s
I20260709 12:24:58.535737  1872 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:58.544991  8778 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:58.553941  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2510) took 1717 ms. Trace:
I20260709 12:24:58.326581  8749 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:58.477619  8703 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:24:58.468729  2412 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:24:58.476986  8653 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:58.559059  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:58.559087  1624 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 18, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 18 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:58.561838  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.565600  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:24:58.566258  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 91193.8us
I20260709 12:24:58.515316  8582 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.515571  8482 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:58.536577  8491 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:58.561717  8764 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.578207  1802 rpcz_store.cc:276] 0709 12:24:56.836245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.607380 (+771135us) service_pool.cc:224] Handling call
0709 12:24:57.644088 (+ 36708us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.676656 (+ 32568us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.676695 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.680463 (+  3768us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.553541 (+873078us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:58.553582 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.553653 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.553723 (+    70us) raft_consensus.cc:1685] finished
0709 12:24:58.553734 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.553883 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":63713,"raft.run_cpu_time_us":60,"raft.run_wall_time_us":61,"thread_start_us":786272,"threads_started":1,"wal-append.queue_time_us":851866}
I20260709 12:24:58.585498  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.611204  1724 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:24:58.612126  1727 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:58.587193  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:58.516413  8175 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:58.488269  8578 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:58.614954  1726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.036s	sys 0.053s
W20260709 12:24:58.616677  1726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.037s	sys 0.053s
I20260709 12:24:58.636309  1624 catalog_manager.cc:5697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 14. New cstate: current_term: 14 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:58.564049  8779 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:58.562795  8613 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:58.640800  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 81879.7us
W20260709 12:24:58.641148  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 75730.9us
W20260709 12:24:58.649865  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 93581us
I20260709 12:24:58.650875  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:24:58.562140  8758 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:58.554947  8739 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:58.577816  8765 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:58.654768  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:58.652261  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:58.655803  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:58.656643  2413 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:24:58.650586  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:24:58.664587  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:58.665647  1727 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:58.663379  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.595988  8770 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:58.666709  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:24:58.685922  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 290645us
I20260709 12:24:58.583033  8400 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.583967  8296 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.663932  1624 catalog_manager.cc:5697] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:58.655097  8519 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:24:58.573906  8743 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.661055  8778 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:58.608644  8749 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:58.700433  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.694617  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:58.695802  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.691170  1731 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:24:58.691692  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.694985  2413 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:58.703055  1727 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:24:58.624655  8582 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.636624  8175 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:58.695469  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44699.1us
I20260709 12:24:58.718252  2418 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:58.720790  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:58.690770  8613 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:58.664124  8779 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:24:58.697624  8682 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:58.511672  1630 rpcz_store.cc:269] 0709 12:23:44.735384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.750996 (+ 15612us) service_pool.cc:224] Handling call
0709 12:24:58.281687 (+73530691us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:55.105350 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:55.172291 (+ 66941us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:55.172361 (+    70us) write_op.cc:435] Acquired schema lock
0709 12:24:55.172391 (+    30us) tablet.cc:662] Decoding operations
0709 12:24:55.173171 (+   780us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:55.173240 (+    69us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:55.173268 (+    28us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:24:55.173739 (+   471us) tablet.cc:701] Row locks acquired
0709 12:24:55.173753 (+    14us) write_op.cc:260] PREPARE: finished
0709 12:24:55.173937 (+   184us) write_op.cc:270] Start()
0709 12:24:55.174084 (+   147us) write_op.cc:276] Timestamp: P: 1783599895173881 usec, L: 0
0709 12:24:55.174109 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:24:55.195010 (+ 20901us) log.cc:844] Serialized 1736 byte log entry
0709 12:24:55.991875 (+796865us) op_driver.cc:464] REPLICATION: finished
0709 12:24:56.287266 (+295391us) write_op.cc:301] APPLY: starting
0709 12:24:56.287403 (+   137us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:56.293192 (+  5789us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:56.293332 (+   140us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:56.382164 (+ 88832us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:56.382456 (+   292us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=8,mrs_lookups=0
0709 12:24:56.382526 (+    70us) write_op.cc:312] APPLY: finished
0709 12:24:56.396195 (+ 13669us) log.cc:844] Serialized 49 byte log entry
0709 12:24:56.396417 (+   222us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:56.396584 (+   167us) write_op.cc:454] Released schema lock
0709 12:24:56.396653 (+    69us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:56.397196 (+   543us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:56.397239 (+    43us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1733700,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":294949,"apply.run_cpu_time_us":9559,"apply.run_wall_time_us":109903,"cfile_cache_hit":18,"cfile_cache_hit_bytes":20863,"num_ops":4,"prepare.queue_time_us":754776,"prepare.run_cpu_time_us":21186,"prepare.run_wall_time_us":487773,"raft.queue_time_us":244822,"raft.run_cpu_time_us":37255,"raft.run_wall_time_us":213522,"replication_time_us":777653,"thread_start_us":1052307,"threads_started":4,"wal-append.queue_time_us":343418,"wal-append.run_cpu_time_us":15477,"wal-append.run_wall_time_us":1870897}]]}
I20260709 12:24:58.735373  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:58.739886  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.740911  1727 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
W20260709 12:24:58.744674  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50146.5us
I20260709 12:24:58.720717  8493 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:58.723857  8778 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:58.764233  8778 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:24:58.769459  8662 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.971s	user 0.005s	sys 0.011s
I20260709 12:24:58.764415  8578 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:58.771508  8749 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:58.729521  8519 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.774720  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1027) took 1122 ms. Trace:
I20260709 12:24:58.776062  8582 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:58.757746  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:58.781517  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44086.3us
I20260709 12:24:58.832278  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:58.747133  8651 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:58.723138  8703 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.746979  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.831084  1730 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:58.799228  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:58.756397  8175 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:58.747466  8682 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.737124  8609 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:58.739876  1624 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 10 to 18, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 18 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:58.811046  1725 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
W20260709 12:24:58.840934  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 97111.1us
I20260709 12:24:58.816413  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:24:58.727397  8750 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:24:58.824539  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89266.5us
W20260709 12:24:58.827405  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 83978.6us
I20260709 12:24:58.759824  8764 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:58.761091  8296 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:58.765895  8778 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:58.742949  8528 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.625s	user 0.013s	sys 0.004s
I20260709 12:24:58.860929  8528 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:58.816543  1723 rpcz_store.cc:276] 0709 12:24:57.652577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.679658 (+ 27081us) service_pool.cc:224] Handling call
0709 12:24:57.692847 (+ 13189us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.262073 (+569226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.262106 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.303635 (+ 41529us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.611828 (+308193us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:58.611886 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.611956 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.774403 (+162447us) raft_consensus.cc:1685] finished
0709 12:24:58.774431 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.774612 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":83135,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":72,"thread_start_us":858292,"threads_started":2,"wal-append.queue_time_us":392090}
I20260709 12:24:58.837440  1727 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:58.845073  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:58.859937  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:58.861249  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:58.756217  8400 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:58.835573  8519 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:58.847084  8776 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:24:58.758881  8482 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.835613  8651 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:58.870019  1725 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:58.846894  8757 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.876933  1724 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:24:58.863233  8780 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:24:58.880987  2416 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:24:58.862993  8613 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:58.881716  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70739.2us
I20260709 12:24:58.836020  8491 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:58.863340  8772 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.312s	user 0.000s	sys 0.002s
I20260709 12:24:58.847600  8797 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:58.862329  8175 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:58.892305  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:58.894960  1624 catalog_manager.cc:5697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 10 to 18, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 18 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:58.896286  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:58.863144  8786 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.865252  8703 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:58.900107  1723 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: [19.15-19.15]
I20260709 12:24:58.866299  8765 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:24:58.875948  8760 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.567s	user 0.013s	sys 0.000s
I20260709 12:24:58.875376  8651 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:58.902647  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:58.845299  8682 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:58.902479  1724 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [18.13-20.14]
I20260709 12:24:58.880380  8400 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:58.906140  8779 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:58.907632  8792 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:24:58.908265  8760 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:58.909974  1832 heartbeater.cc:646] Failed to heartbeat to 127.0.140.126:43371 (4 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.140.126:43371 timed out after 15.000s (SENT)
I20260709 12:24:58.885353  8757 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.869163  8795 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:58.915341  8783 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
W20260709 12:24:58.915710  8777 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
I20260709 12:24:58.915818  8765 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.861999  8662 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:24:58.920714  8750 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:24:59.033243  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3443) took 1476 ms. Trace:
I20260709 12:24:58.930349  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:58.934671  8770 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:58.935799  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:58.936205  8519 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:58.936347  8491 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:58.886229  8613 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:58.895267  8797 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.933292  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:58.930774  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:58.899562  1727 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:24:58.932387  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:58.932898  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:58.941009  8400 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:58.904090  8786 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:58.904397  8739 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:24:59.084554  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.937523  1624 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 10 to 20, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 20 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:24:59.086375  1624 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 13 to 19, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 19 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:58.953307  1770 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:24:58.973655  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.090850  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:58.974004  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.092077  1873 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:59.093174  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.093899  1873 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:24:59.028095  8791 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:59.019274  8822 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.010s
I20260709 12:24:58.954949  1697 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:59.030217  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:59.030951  8766 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:58.989444  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:59.101578  1874 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:24:58.995626  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.033740  1876 rpcz_store.cc:276] 0709 12:24:57.556730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.562233 (+  5503us) service_pool.cc:224] Handling call
0709 12:24:57.562731 (+   498us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.563309 (+   578us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:24:58.145684 (+582375us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.475474 (+329790us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.476269 (+   795us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.476460 (+   191us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:24:58.476508 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.476603 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.033038 (+556435us) raft_consensus.cc:1685] finished
0709 12:24:59.033063 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.033184 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":590120,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":73146,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":249,"thread_start_us":892963,"threads_started":2,"wal-append.queue_time_us":469036}
I20260709 12:24:59.039803  1804 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
W20260709 12:24:58.950657  8657 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.812s	user 0.015s	sys 0.000s
I20260709 12:24:58.949836  8798 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:58.963294  8749 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:24:59.046818  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 97184.1us
I20260709 12:24:58.945358  8779 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.052285  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:58.963980  8817 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:59.057698  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:24:59.062829  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:24:59.079557  1803 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:59.081923  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 407403us
I20260709 12:24:59.122447  1872 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:24:59.082592  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:59.083282  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 180705us
I20260709 12:24:59.107497  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:59.124722  1727 raft_consensus.cc:606] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:24:59.127552  1727 raft_consensus.cc:614] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:59.124390  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66785.3us
I20260709 12:24:59.057008  8613 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:59.125984  1875 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:24:59.079952  8400 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:59.132369  1802 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:24:59.077036  8797 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:59.135632  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:59.103341  8802 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:24:59.141970  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2652) took 1150 ms. Trace:
I20260709 12:24:59.133543  1624 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 14 to 18. New cstate: current_term: 18 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:59.143638  1801 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:59.153129  2383 leader_election.cc:400] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:24:59.153671  2383 leader_election.cc:403] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:24:59.125253  8819 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.060s	user 0.003s	sys 0.000s
I20260709 12:24:59.174402  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:59.159693  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:24:59.183269  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 228372us
I20260709 12:24:59.124078  8779 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.124779  8749 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:59.178370  1803 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:24:59.120699  8799 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:59.201611  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:59.137714  8491 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:59.144416  1726 rpcz_store.cc:276] 0709 12:24:57.991271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.992042 (+   771us) service_pool.cc:224] Handling call
0709 12:24:58.026328 (+ 34286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.026771 (+   443us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.026827 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.045556 (+ 18729us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.617783 (+572227us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:58.617827 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.617912 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.141703 (+523791us) raft_consensus.cc:1685] finished
0709 12:24:59.141733 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.141906 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7635,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":2594,"thread_start_us":523557,"threads_started":1,"wal-append.queue_time_us":619455}
I20260709 12:24:59.217612  1624 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 9 to 15, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 15 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:24:59.237114  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:59.121425  8657 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.243207  1726 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:24:59.204841  8749 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:24:59.260111  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 133522us
I20260709 12:24:59.226467  1624 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 15 to 19, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 19 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:59.209206  8491 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.235081  8701 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:59.285141  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3444) took 1025 ms. Trace:
I20260709 12:24:59.285542  1881 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:24:59.149178  8817 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.292224  1877 rpcz_store.cc:276] 0709 12:24:58.259779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.259952 (+   173us) service_pool.cc:224] Handling call
0709 12:24:58.283865 (+ 23913us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.688101 (+404236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.688140 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.716206 (+ 28066us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.852956 (+136750us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:58.853014 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.853116 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.284872 (+431756us) raft_consensus.cc:1685] finished
0709 12:24:59.284893 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.285080 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11712,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":79400,"thread_start_us":457011,"threads_started":2,"wal-append.queue_time_us":173552}
I20260709 12:24:59.324447  8832 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:59.326993  1624 catalog_manager.cc:5697] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 10 to 13, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 13 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:59.283492  8491 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.291661  8828 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:59.292145  8701 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:59.330875  8808 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.287s	user 0.002s	sys 0.000s
I20260709 12:24:59.329864   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260709 12:24:59.329483  8823 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.400717  8682 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:59.331856  8770 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:59.392827  1624 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 13. New cstate: current_term: 13 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:59.410501  8811 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:24:59.412215  8838 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
W20260709 12:24:59.418386  8814 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
W20260709 12:24:59.419698  8810 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.243s	user 0.001s	sys 0.000s
I20260709 12:24:59.369647  8832 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:59.423378  8757 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:59.425941  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3559) took 1168 ms. Trace:
I20260709 12:24:59.404637  8701 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:59.244423  8799 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.393003  8828 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:59.281234  8798 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:59.409124  8822 raft_consensus.cc:2764] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:59.467006  1725 rpcz_store.cc:276] 0709 12:24:58.257378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.870919 (+613541us) service_pool.cc:224] Handling call
0709 12:24:58.885829 (+ 14910us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.905380 (+ 19551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.905422 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:58.919621 (+ 14199us) log.cc:844] Serialized 26 byte log entry
0709 12:24:59.094303 (+174682us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:59.094540 (+   237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.094827 (+   287us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:59.425624 (+330797us) raft_consensus.cc:1685] finished
0709 12:24:59.425655 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.425863 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6997,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":103,"thread_start_us":56473,"threads_started":1,"wal-append.queue_time_us":281935}
I20260709 12:24:59.479524  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:59.481959  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:59.482750  1727 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:59.483137  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:59.486433  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:59.489389  1730 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:24:59.470312  8703 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.566s	user 0.012s	sys 0.000s
I20260709 12:24:59.493710  8799 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:59.494541  8844 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:59.387336  8816 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.154s
I20260709 12:24:59.506822  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:59.496781  8832 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.507339  1624 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 16, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 16 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:59.509676  1624 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 11 to 17. New cstate: current_term: 17 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:59.507953  1730 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:59.472095  8837 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:59.518965  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.522915  1624 catalog_manager.cc:5697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 reported cstate change: term changed from 7 to 12. New cstate: current_term: 12 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:24:59.523433  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 286362us
I20260709 12:24:59.463502  8823 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.568730  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:24:59.578948  1730 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
W20260709 12:24:59.579198  1844 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.597s (SENT) [suppressed 6 similar messages]
I20260709 12:24:59.584766  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:59.581534  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 427014us
I20260709 12:24:59.547194  8831 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:59.587422  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:59.604291  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:24:59.581015  8750 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.008s	sys 0.018s
I20260709 12:24:59.613875  1724 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:59.614513  1724 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:59.609756  1732 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:24:59.607578  8753 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.874s	user 0.016s	sys 0.015s
I20260709 12:24:59.629791  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.637701  1873 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:24:59.635406  8657 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:59.647163  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.607882  8832 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:24:59.650995  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63762.2us
I20260709 12:24:59.520131  8844 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.654662  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:59.663774  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.665232  1873 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:24:59.668326  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:59.673977  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:24:59.650647  8753 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.922s	user 0.016s	sys 0.016s
I20260709 12:24:59.677030  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.572122  8837 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.678395  1873 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:24:59.653714  8817 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:24:59.675233  8651 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.738s	user 0.010s	sys 0.004s
I20260709 12:24:59.687261  1732 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:24:59.665498  8657 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:59.690492  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87785.8us
I20260709 12:24:59.671501  8662 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:59.588493  8828 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:59.695499  8830 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
W20260709 12:24:59.697606  8841 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:24:59.692734  8651 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.702070  8657 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.702925  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:59.703294  8784 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:24:59.703492  8651 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:59.587462  8798 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.709713  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:59.727164  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:59.731616  1728 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 22 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:59.681049  8801 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.735728  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:59.705397  8837 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:59.681097  8778 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:24:59.706019  8662 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:59.741657  8657 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:59.751581  1873 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:59.752015  8651 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:59.754595  8825 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.761669  8825 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:59.762274  8779 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.000s
I20260709 12:24:59.754590   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:24:59.763093  8834 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 12:24:59.764358   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:59.764492  8835 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
I20260709 12:24:59.754751  8652 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.011s
I20260709 12:24:59.753021  8662 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:59.766160  8652 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:24:59.766543  8652 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:24:59.767666  8652 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:59.759929  8843 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.170s	user 0.001s	sys 0.001s
I20260709 12:24:59.770959  8844 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:59.765337  8825 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:59.769079  8652 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:24:59.774741  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:59.779742  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:59.780607  1732 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:24:59.782198  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:59.787212  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.788141  1874 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:59.789072  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:59.789376  8749 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.504s	user 0.008s	sys 0.002s
I20260709 12:24:59.789891  8749 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:59.793614  8797 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.587s	user 0.012s	sys 0.000s
I20260709 12:24:59.794059  8797 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.794601  8786 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:59.762715  8779 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.796038  1878 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:59.761039  8851 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:59.797096  8779 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:59.760274  8849 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.798065  8849 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.800163  8688 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:59.800364  8849 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:59.760201  8829 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:59.800967  8811 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:24:59.801165  8829 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.801811  8829 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:59.802973  8758 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:24:59.803476  8758 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:59.806994  8857 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:24:59.760373  8823 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:59.808365  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1292) took 1525 ms. Trace:
I20260709 12:24:58.555913  1807 rpcz_store.cc:276] 0709 12:24:56.397894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.730019 (+332125us) service_pool.cc:224] Handling call
0709 12:24:56.730360 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.730615 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:24:57.064462 (+333847us) log.cc:844] Serialized 14 byte log entry
0709 12:24:57.459726 (+395264us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.476350 (+1016624us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.476668 (+   318us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.476731 (+    63us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:58.476793 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.477012 (+   219us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.477165 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75956,"prepare.queue_time_us":68973,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":80,"raft.queue_time_us":606031,"raft.run_cpu_time_us":459,"raft.run_wall_time_us":37376,"thread_start_us":1654042,"threads_started":4,"wal-append.queue_time_us":599798}
I20260709 12:24:59.808784  2416 rpcz_store.cc:276] 0709 12:24:58.283327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.283467 (+   140us) service_pool.cc:224] Handling call
0709 12:24:58.298205 (+ 14738us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.885163 (+586958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.808128 (+922965us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b5400560c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:59.808308 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1326854912}
I20260709 12:24:59.760118  8856 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:59.812762  8856 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:59.812914  8770 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:59.813196  8856 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:59.813460  8770 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:24:59.815897  8491 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.816720  8859 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:59.817196  8859 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:59.817540  8859 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:59.818344  8796 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:24:59.819005  8796 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.825081  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:59.830451  1873 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:24:59.830426  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:24:59.783123  2383 connection.cc:256] RPC call timeout handler was delayed by -1.5839661420000084s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19290000000000002s for a response
W20260709 12:24:59.836987  2383 leader_election.cc:336] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.557s (SENT)
I20260709 12:24:59.838099  1877 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:59.839550  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:59.840240  1732 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:24:59.841454  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:59.843040  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:59.844345  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:59.842180  2411 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:24:59.845604  2419 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:59.846683  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:24:59.847262  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:59.848017  2414 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
W20260709 12:24:59.839767  8400 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.662s	user 0.013s	sys 0.000s
I20260709 12:24:59.851272  8400 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.853911  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:59.859298  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:59.752779  8801 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.861344  8491 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:59.863276  8801 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:59.864241  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:59.847324  1723 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.868165  2414 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:59.865583  2416 raft_consensus.cc:2376] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20. Re-sending same reply.
I20260709 12:24:59.874470  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:24:59.876781  2413 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:24:59.871377  8701 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:24:59.892338  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45208.4us
I20260709 12:24:59.886330  8796 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:59.895897  2417 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 14 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
W20260709 12:24:59.893759  8613 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.717s	user 0.012s	sys 0.001s
I20260709 12:24:59.894063  8797 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:59.896659  8701 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:59.899874  8632 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:59.906116  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3445) took 1645 ms. Trace:
W20260709 12:24:59.904825  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.905308  1723 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:24:59.907449  1879 rpcz_store.cc:276] 0709 12:24:58.279484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.295121 (+ 15637us) service_pool.cc:224] Handling call
0709 12:24:58.295473 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.326910 (+ 31437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.326972 (+    62us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.327039 (+    67us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:58.327097 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.905847 (+1578750us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x7b54002058d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:24:59.905902 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:59.906047 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":71531392}
W20260709 12:24:59.908751  2383 connection.cc:256] RPC call timeout handler was delayed by -1.200942644999941s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19640000000000002s for a response
I20260709 12:24:59.909454  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:24:59.909428  8839 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:59.907816  8613 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.912458  8839 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.914173  8613 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:59.909773  8823 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:59.915803  8823 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.920260  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:24:59.922320  8779 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:59.922951  8779 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:24:59.923501  8823 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:59.923933  8779 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.925945  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:24:59.931425  2413 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:24:59.936167  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:59.933950  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:24:59.937489  8849 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:59.938182  8790 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:59.938971  8790 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:59.939565  8790 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:59.945397  8859 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.949466  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:59.951988  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.952299  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.954092  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.954381  1873 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:24:59.954914  1875 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:24:59.955780  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:24:59.956480  1880 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:24:59.952044  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:24:59.957614  1877 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:59.958303  8662 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:24:59.921592  8797 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.039s
I20260709 12:24:59.933365  8671 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:24:59.966173  8839 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:24:59.967893  8829 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:24:59.971431  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.01811e+06us
I20260709 12:24:59.971316  8800 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:24:59.981473  8823 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:24:59.985388  8823 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:24:59.987897  8823 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:24:59.990162  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:24:59.990906  1807 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:59.991406  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:59.992794  1803 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:59.994356  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:24:59.996369  1801 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:24:59.997802  8791 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:24:59.998106  1805 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:00.007265  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.000712  1802 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:25:00.004839  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.009308  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.009812  1798 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 20 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 14.
W20260709 12:25:00.012112  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.011902  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.013379  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:00.017750  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45111.2us
I20260709 12:25:00.021036  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.021670  1877 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:00.021874  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:00.018193  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.026937  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.029683  1803 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 20 index: 15. (term mismatch)
I20260709 12:25:00.030253  1803 pending_rounds.cc:85] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 14
I20260709 12:25:00.030542  1803 pending_rounds.cc:107] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 19.15
I20260709 12:25:00.030992  1803 raft_consensus.cc:2889] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:00.031769  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:00.034047  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:00.035622  1732 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:25:00.027940  1880 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:25:00.041646  8633 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.039915  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:00.046653  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.010545  8849 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:59.998776  8791 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.057696  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:00.054718  8528 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:00.023258  8823 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:00.058264  8824 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.011979  8856 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.051126  8671 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:00.074229  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.065649  8796 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:00.065922  8823 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.020157  8786 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:25:00.077361  8796 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:00.078254  8786 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:00.078727  8786 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.079059  8796 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.079716  8859 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.080210  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:00.060597  8825 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.068296  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.081712  1729 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:25:00.081552  8825 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.068320  8824 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.083988  8825 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.084692  8824 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.085453  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.085628  8519 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:00.086081  8519 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.087203  8519 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.088291  8519 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.057122  8849 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.086129  1729 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:00.091952  1807 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:00.089462  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:00.071662  8755 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:00.017370  8613 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 12:25:00.051745  8806 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:00.050066  8703 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:00.066879  8829 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.060763  8657 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:00.075698  1723 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.077327  8823 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.089389  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:00.075332  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:00.024955  8753 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:25:00.044042  8688 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:25:00.106057  1805 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:00.098680  8703 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.087217  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:00.107128  8703 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:00.094673  2418 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:25:00.109387  8539 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.097085  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:00.110368  2412 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:00.043888  8352 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:00.100216  8613 consensus_queue.cc:1243] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:25:00.100905  8597 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:00.113906  8749 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.100121  8757 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.114928  8757 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.115814  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.091081  8662 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.117077  8757 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:00.094238  8849 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.101646  2413 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:00.102605  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:00.103045  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.104398  8800 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:00.055179  8632 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 12:25:00.117137  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.124439  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:00.124922  1732 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:00.120070  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.122045  1729 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:00.127094  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.127920  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:00.128265  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.129000  1801 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:00.125914  1802 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:00.130251  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.122920  8657 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.130210  8597 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.133833  8597 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.134187  8493 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:00.134630  8493 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.136283  8597 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.135901  8493 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.136699  8829 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:25:00.137137  8829 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:00.137424  8493 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:00.137519  8829 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:00.079711  8786 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.139407  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.132259  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.126645  8767 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:00.121258  8539 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:00.133224  8800 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.147004  8539 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.132126  8749 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.142020  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:00.142038  1732 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:00.143041  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.130206  8662 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:00.121090  8779 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:00.150646  1723 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:00.151214  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:00.123494  8757 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:00.177165  8757 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.178975  8854 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:00.179670  8757 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:00.179615  8854 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.181252  8398 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.182052  8854 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:00.155786  1877 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:00.182404  8806 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:00.194322  8849 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.158099  8801 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:00.155350  8779 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:00.197055  8801 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.197280  8779 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.197608  8801 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.158701  8657 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.152230  8767 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.160739  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1294) took 1876 ms. Trace:
I20260709 12:25:00.143539  8758 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.163136  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.140522  8786 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:00.171454  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:00.217224  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45882.5us
I20260709 12:25:00.217367  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:00.172124  8750 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:25:00.218081  1877 raft_consensus.cc:2368] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20.
I20260709 12:25:00.174808  1728 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:00.153208  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.219664  1732 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:00.220759  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.221608  1732 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:25:00.221992  8856 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.156702  8800 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.183203  1729 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:00.227437  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.228304  1729 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:00.183785  8797 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:00.187049  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:00.189383  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:00.189644  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.235939  2415 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:00.191526  1873 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:00.192226  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:00.192349  8352 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:00.193240  8749 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:00.195326  8806 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.196170  8662 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:00.200728  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.178108  1724 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 10, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:00.157635  8539 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.205744  1879 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:00.207017  1798 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:25:00.209585  2410 rpcz_store.cc:276] 0709 12:24:58.284781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.284938 (+   157us) service_pool.cc:224] Handling call
0709 12:25:00.160485 (+1875547us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b54005f05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:00.160660 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2134407936}
I20260709 12:25:00.222438  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.243710  1732 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.244946  1723 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:00.237090  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:00.236119  1801 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:00.190670  8829 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.239336  8797 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.238214  8657 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:00.254627  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.255652  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.260798  1724 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.261142  1877 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:00.262101  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.262694  1877 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:00.263398  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.264099  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:00.264515  1877 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:00.256175  1723 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:25:00.239857  8765 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.265879  8765 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.246913  8759 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.239085  8750 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.253928  8539 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:00.261014  1880 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:00.259797  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.274709  1801 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:00.268137  1877 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:25:00.275447  1798 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:00.267163  1732 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:00.276932  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.279044  8758 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:00.250389  8791 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.282723  8791 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.274049  8759 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.271891  8767 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:00.257972  8797 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:00.290882  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:00.259848  8657 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:00.291602  2410 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.271147  8703 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.293043  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.271816  8849 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.296708  1723 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:00.294275  8823 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.297178  1880 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:00.297768  1874 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:00.302632  1732 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:25:00.298473  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:00.303231  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.302083  1771 leader_election.cc:400] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20.
I20260709 12:25:00.325109  1771 leader_election.cc:403] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:25:00.296227  8657 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:25:00.411339  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 115796us
I20260709 12:25:00.413264  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:00.414180  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.414402  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.323762  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.259653  8790 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.427649  1724 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:25:00.431617  1724 pending_rounds.cc:85] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 11
I20260709 12:25:00.432193  1724 pending_rounds.cc:107] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 12:25:00.433913  1724 raft_consensus.cc:2889] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:00.299842  8749 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:00.439049  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.440378  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:00.442284  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:00.446907  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.447162  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.443461  2410 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:00.447963  1801 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:00.453450  8753 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:25:00.454815  8878 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:25:00.455106  2419 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:00.455796  8790 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:00.456504  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:00.439461  8756 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.018s
I20260709 12:25:00.457655  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.458161  8756 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:00.458531  8756 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:00.454819  8753 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:25:00.459715  8753 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:00.459486  8756 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.460733  8756 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.460638  8753 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.461894  8753 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:00.464620  8703 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:00.469625  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:00.478968  8873 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260709 12:25:00.480178  8597 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:00.480695  8597 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.481061  8597 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.439637  8757 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:00.484490  8823 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.484735  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.433648  8796 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.485193  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.485811  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.486325  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.486495  1807 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:00.487543  1806 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:25:00.487701  1800 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:25:00.489921  8765 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:00.490868  8739 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260709 12:25:00.491894  8582 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:00.492311  8582 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:00.492718  8582 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:00.436010  8539 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:25:00.500622  8539 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:00.501192  8539 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:00.415374  8798 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:25:00.504223  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.505235  1799 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 14: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:00.490959  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50615.8us
I20260709 12:25:00.507728  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.508966  8791 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:00.509487  8791 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:00.442672  8519 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.511300  8519 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.511467  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.513795  8828 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:00.514671  8829 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.517354  8769 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:00.517855  8769 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:00.518204  8769 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.520473  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.520993  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.416030  8779 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.525290  8779 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.525905  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.299671  8750 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.532589  1879 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20260709 12:25:00.533607  1879 pending_rounds.cc:85] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 11
I20260709 12:25:00.533982  1879 pending_rounds.cc:107] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260709 12:25:00.534349  1879 raft_consensus.cc:2889] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:00.543649  1880 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:00.548040  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.551388  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.552584  8854 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:00.552304  8801 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.553828  8854 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:00.554229  8854 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:00.549978  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.561618  1807 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:25:00.448294  8749 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:00.563300  8749 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.436806  8400 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.565707  8801 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.566476  8759 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:00.567212  8790 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:00.567728  8790 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:00.568091  8790 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.571130  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.572034  1799 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:25:00.572539  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:00.574198  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2636) took 2882 ms (client timeout 1597 ms). Trace:
W20260709 12:25:00.574615  1731 rpcz_store.cc:269] 0709 12:24:57.691691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.691895 (+   204us) service_pool.cc:224] Handling call
0709 12:25:00.573970 (+2882075us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b540030aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:00.574098 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2133693312}
I20260709 12:25:00.510414  8791 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.577427  8766 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:25:00.578099  8766 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:00.580567  8657 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.584481  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:00.585487  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:00.585794  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2670) took 1005 ms. Trace:
I20260709 12:25:00.586951  2416 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
W20260709 12:25:00.588773  1696 connection.cc:256] RPC call timeout handler was delayed by -1.8998841249999714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1505s for a response
W20260709 12:25:00.589206  1696 connection.cc:256] RPC call timeout handler was delayed by -1.808084124999823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1607s for a response
W20260709 12:25:00.589534  1696 connection.cc:256] RPC call timeout handler was delayed by -1.7810841249997793s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
W20260709 12:25:00.589874  1696 connection.cc:256] RPC call timeout handler was delayed by -1.7576841249997415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1663s for a response
W20260709 12:25:00.590143  1696 connection.cc:256] RPC call timeout handler was delayed by -1.6325841249999939s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18020000000000003s for a response
W20260709 12:25:00.590446  1696 connection.cc:256] RPC call timeout handler was delayed by -1.366184125000018s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2098s for a response
W20260709 12:25:00.590708  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2563841249998404s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
I20260709 12:25:00.591307  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:00.589414  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.592024  1806 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.592584  8846 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.593127  8846 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.513715  8519 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:00.590143  8854 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.618649  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2669) took 1038 ms. Trace:
I20260709 12:25:00.619338  1725 rpcz_store.cc:276] 0709 12:24:59.580147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.580304 (+   157us) service_pool.cc:224] Handling call
0709 12:24:59.600903 (+ 20599us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.601165 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:59.742952 (+141787us) log.cc:844] Serialized 14 byte log entry
0709 12:25:00.618060 (+875108us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:00.618151 (+    91us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:00.618229 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.618338 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:00.618542 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":875070,"prepare.queue_time_us":62439,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"thread_start_us":119324,"threads_started":1}
I20260709 12:25:00.622097  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:00.597241  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:00.603353  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44105.2us
I20260709 12:25:00.613734  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 988) took 1588 ms. Trace:
I20260709 12:25:00.586385  1726 rpcz_store.cc:276] 0709 12:24:59.580638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.583255 (+  2617us) service_pool.cc:224] Handling call
0709 12:24:59.583538 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.583744 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:59.603769 (+ 20025us) log.cc:844] Serialized 14 byte log entry
0709 12:25:00.584988 (+981219us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b1800370a20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:25:00.585206 (+   218us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:00.585310 (+   104us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:00.585389 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.585497 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:00.585680 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47871,"prepare.queue_time_us":20370,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":988356,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":135,"spinlock_wait_cycles":5858048}
I20260709 12:25:00.629076  1878 rpcz_store.cc:276] 0709 12:24:59.025387 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.090481 (+ 65094us) service_pool.cc:224] Handling call
0709 12:25:00.613459 (+1522978us) spinlock_profiling.cc:243] Waited 704 ms on lock 0x7b5400051754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:00.613657 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1619720960}
I20260709 12:25:00.628343  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.632927  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.636436  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:00.579576  8766 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.502334  8539 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.648851  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.651187  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.651978  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.652772  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.656177  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:00.658850  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:00.660770  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:00.670732  1803 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.671870  2417 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:00.632398  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.664115  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:00.664583  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 893110us
I20260709 12:25:00.618147  8831 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.674847  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:00.679417  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:00.685370  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:00.692818  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:00.634222  8759 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:00.702762  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:00.646260  8519 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:00.687021  8787 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:25:00.718031  8766 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:00.607076  8806 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:00.721892  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.722707  2413 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:00.722074  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:00.678021  8854 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:00.733263  1878 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:25:00.737630  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.688982  8493 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:25:00.763561  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.767974  1724 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.737478  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:00.768543  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.770573  1878 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:25:00.774741  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:00.794442  1804 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:25:00.775157  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.795279  1726 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:00.778362  1875 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:00.781713  1807 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:00.791819  2413 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:25:00.786976  1873 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:25:00.687955  8352 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:25:00.800038  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.720621  8582 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.766512  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.797658  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.810770  1725 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:00.767480  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:00.812350  1802 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:00.783915  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 985) took 1759 ms. Trace:
I20260709 12:25:00.813771  1881 rpcz_store.cc:276] 0709 12:24:59.024696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.037956 (+ 13260us) service_pool.cc:224] Handling call
0709 12:24:59.056345 (+ 18389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:59.292477 (+236132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.783494 (+1491017us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b5400073854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:00.783824 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":526464768}
I20260709 12:25:00.748162  2416 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:00.734858  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:00.734112  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:00.687095  8779 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.113s
I20260709 12:25:00.724050  8788 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:00.841169  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.842336  2410 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.857124  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:00.731083  8759 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.832026  8352 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:00.823024  8493 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.874254  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.878453  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:00.879307  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.883839  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.884622  1727 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:00.885039  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:00.885541  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.859709  1881 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:00.887746  1724 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:00.888590  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.889139  1724 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:00.836284  8788 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.819824  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.738220  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.895407  1732 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.822503  1878 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:00.723707  8853 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:00.767815  8519 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.904500  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.908164  2410 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:00.742224  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:00.889901  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.741209  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:00.899413  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:00.893438  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:00.862303  8786 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.638s	user 0.010s	sys 0.003s
W20260709 12:25:00.919452  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 194667us
I20260709 12:25:00.919507  1728 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 15. Current term is 16. Ops: []
W20260709 12:25:00.920414  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 179347us
I20260709 12:25:00.921756  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.921797  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:00.922032  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:00.922930  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.924269  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.863471  8849 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:00.890806  8837 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.893744  8829 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:00.878944  8175 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.880527  8490 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:00.929287  1732 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:00.926905  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.952935  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.955147  8840 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:00.955538  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:00.955938  8840 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:00.956419  8840 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:00.957950  8765 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:00.959124  8856 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:00.959713  8856 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.960667  8856 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.963323  8849 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.901415  8764 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.964452  8582 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:00.968111  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:00.909492  2417 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:00.974031  1806 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:00.939178  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:00.975569  1728 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:00.900638  8788 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:00.935488  8632 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:25:00.948052  8768 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.949077  1878 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:00.926863  8767 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:00.860538  8758 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 12:25:00.949493  8352 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.949865  8175 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:01.000336  8879 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:25:01.000144  8767 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:00.955230  1732 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:25:00.943296  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:00.967437  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:00.979487  1723 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.020s	sys 0.037s
I20260709 12:25:00.931428  8798 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:25:01.007507  1723 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.020s	sys 0.037s
W20260709 12:25:01.011385  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44063.2us
W20260709 12:25:00.989166  8483 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.027s	sys 0.033s
I20260709 12:25:00.930883  8836 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.067s
I20260709 12:25:00.944813  8800 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.073s
I20260709 12:25:00.985172  8829 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.966285  8519 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.967852  8837 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:00.993626  8753 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 9, Time since last communication: 0.035s
W20260709 12:25:01.019405  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 76218us
I20260709 12:25:01.044724  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:01.010887  8613 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.028476  8483 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.028s	sys 0.033s
I20260709 12:25:00.950625  8493 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:00.988636  8795 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:00.992393  8797 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.692s	user 0.012s	sys 0.000s
I20260709 12:25:01.028441  1725 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:25:00.964531  8849 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:01.076591  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.006158  8846 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:01.043102  8837 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:01.053881  8851 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:01.069782  1728 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:01.020690  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:01.031965  8798 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:00.990083  8703 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.039s
I20260709 12:25:01.058128  8806 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:01.053766  8769 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.535s	user 0.011s	sys 0.003s
I20260709 12:25:01.023831  8823 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:01.059384  1726 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
W20260709 12:25:01.137403  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.093549  8768 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:01.081029  8797 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:01.094849  8851 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:01.053695  8739 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:25:01.065491  8795 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:01.062012  8766 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
W20260709 12:25:01.325029  1803 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.653s	user 0.013s	sys 0.017s
I20260709 12:25:01.326948  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:01.327014  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.369184  1803 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:01.328143  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:01.328102  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:01.370589  8828 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:01.371132  1724 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:01.371279  8828 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:01.371762  1802 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.559s	user 0.011s	sys 0.000s
I20260709 12:25:01.372094  1802 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:25:01.372376  8828 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:01.375011  1728 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:01.054620  8613 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:01.366119  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:01.383942  1726 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:25:01.378746  8846 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:01.325899  8582 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.387722  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:01.378232  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:01.408579  1728 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
W20260709 12:25:01.381891  8756 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.921s	user 0.020s	sys 0.013s
I20260709 12:25:01.377712  8851 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:01.415544  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:01.418211  1724 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:25:01.471820  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56410.2us
W20260709 12:25:01.388824  8750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.010s	sys 0.019s
I20260709 12:25:01.376401  8823 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:01.475901  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:01.425431  1729 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.005s	sys 0.024s
I20260709 12:25:01.394826  8703 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:25:01.427605  1695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.029s	sys 0.069s
W20260709 12:25:01.478375  1695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.029s	sys 0.069s
I20260709 12:25:01.426630  8837 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:25:01.428862  1731 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.009s	sys 0.020s
W20260709 12:25:01.485610  1731 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.009s	sys 0.020s
I20260709 12:25:01.487051  1874 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:01.357856  8769 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:01.438968  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 481431us
I20260709 12:25:01.437325  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:01.174621  8829 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:01.528810  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:01.465725  8483 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:01.383905  8701 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:25:01.484754  8846 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:01.544371  2412 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:01.477195  1729 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.005s	sys 0.025s
I20260709 12:25:01.389317  8759 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:01.475772  1801 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:01.481948  1879 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:01.392721  8490 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.531016  8757 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.045s	sys 0.089s
W20260709 12:25:01.468086  8765 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.009s	sys 0.016s
I20260709 12:25:01.529812  1724 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:25:01.484098  8766 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.063s
I20260709 12:25:01.414528  8582 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:01.559180  8882 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:25:01.540927  8483 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:01.563200  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:01.475404  8750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.010s	sys 0.020s
I20260709 12:25:01.564523  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:01.544287  8846 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:01.473331  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:01.542474  1876 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:01.478152  8703 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:01.491832  8749 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.928s	user 0.010s	sys 0.003s
I20260709 12:25:01.410364  8798 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:01.546902  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 109665us
I20260709 12:25:01.592542  1877 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:25:01.554340  8759 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:01.593309  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:01.551398  8757 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.045s	sys 0.090s
W20260709 12:25:01.545066  8597 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.064s	user 0.005s	sys 0.019s
W20260709 12:25:01.511168  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.008s	sys 0.016s
W20260709 12:25:01.596282  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.009s	sys 0.017s
W20260709 12:25:01.561364  1727 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.001s	sys 0.021s
I20260709 12:25:01.531414  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:01.565461  1879 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:25:01.561719  8831 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.000s	sys 0.023s
I20260709 12:25:01.624773  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:01.556468  8765 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.010s	sys 0.017s
W20260709 12:25:01.568176  1798 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.003s	sys 0.011s
I20260709 12:25:01.638155  1875 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:01.638387  1798 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.006s	sys 0.012s
I20260709 12:25:01.583583  1728 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
W20260709 12:25:01.525012  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.792s	user 0.011s	sys 0.011s
W20260709 12:25:01.583614  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:01.563660  8880 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.830s	user 0.013s	sys 0.029s
W20260709 12:25:01.571910  8796 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.086s	user 0.014s	sys 0.000s
I20260709 12:25:01.553973  8490 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:01.638413  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.031s	sys 0.046s
W20260709 12:25:01.621966  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.014s	sys 0.019s
W20260709 12:25:01.640343  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.907s	user 0.012s	sys 0.013s
W20260709 12:25:01.592535  8519 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.007s	sys 0.019s
I20260709 12:25:01.666668  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:01.598696  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:01.670137  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2537) took 1369 ms. Trace:
I20260709 12:25:01.645126  2417 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:01.670701  1798 rpcz_store.cc:276] 0709 12:25:00.300287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.305212 (+  4925us) service_pool.cc:224] Handling call
0709 12:25:00.434173 (+128961us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.434375 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:00.450316 (+ 15941us) log.cc:844] Serialized 14 byte log entry
0709 12:25:00.966648 (+516332us) spinlock_profiling.cc:243] Waited 7.78 ms on lock 0x7b18000401e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf 00007f16abdf2038
0709 12:25:01.638856 (+672208us) log.cc:844] Serialized 14 byte log entry
0709 12:25:01.669628 (+ 30772us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:01.669707 (+    79us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:01.669793 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.669886 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.670058 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30452,"prepare.queue_time_us":47638,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":58,"raft.queue_time_us":131,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":12347,"spinlock_wait_cycles":68672640,"thread_start_us":1134368,"threads_started":2,"wal-append.queue_time_us":932115}
W20260709 12:25:01.642970  8831 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.000s	sys 0.026s
W20260709 12:25:01.648890  1727 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.001s	sys 0.021s
W20260709 12:25:01.646865  8880 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.934s	user 0.014s	sys 0.029s
W20260709 12:25:01.663789  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.031s	sys 0.046s
I20260709 12:25:01.589116  8846 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:01.664134  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.014s	sys 0.023s
W20260709 12:25:01.663877  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 930917us
I20260709 12:25:01.662482  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:01.669085  1881 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:25:01.597047  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 669342us
W20260709 12:25:01.537163  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.635237  8896 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:01.566870  8823 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:01.711974  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:01.646346  8765 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:01.712586  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 800617us
I20260709 12:25:01.567787  8837 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:01.666673  8519 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.008s	sys 0.019s
W20260709 12:25:01.677747  1723 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.668s	user 0.012s	sys 0.014s
I20260709 12:25:01.595208  8749 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:01.720611  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2675) took 2048 ms. Trace:
I20260709 12:25:01.676940  8889 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:01.598141  8597 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:01.757083  1723 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.016s	sys 0.018s
I20260709 12:25:01.676381  8897 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:25:01.690899  8885 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.305s	user 0.001s	sys 0.001s
I20260709 12:25:01.676844  8796 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20.
I20260709 12:25:01.778838  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1047) took 1753 ms. Trace:
I20260709 12:25:01.779181  1872 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:01.779718  1723 rpcz_store.cc:276] 0709 12:25:00.025164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.299158 (+273994us) service_pool.cc:224] Handling call
0709 12:25:00.410748 (+111590us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.411074 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:01.009787 (+598713us) log.cc:844] Serialized 14 byte log entry
0709 12:25:01.778259 (+768472us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:01.778348 (+    89us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:01.778416 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.778534 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.778735 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15884,"prepare.queue_time_us":999562,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":3545,"thread_start_us":1325178,"threads_started":2}
I20260709 12:25:01.786197  8490 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:01.786836  8490 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.787341  8490 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:01.789896  8597 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:01.794930  1800 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:01.795579  8790 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.227s	user 0.011s	sys 0.004s
I20260709 12:25:01.796020  8790 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:01.753284  8582 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:01.797042  1805 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
W20260709 12:25:01.724295  8882 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:25:01.677362  8703 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:01.832247  2415 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:01.836652  1723 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [14.8-16.9]
W20260709 12:25:01.798632  8883 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.307s	user 0.001s	sys 0.001s
I20260709 12:25:01.783180  8896 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:01.685668  8756 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.097s
I20260709 12:25:01.662911  8483 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:01.754204  1730 rpcz_store.cc:276] 0709 12:24:59.672514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.672706 (+   192us) service_pool.cc:224] Handling call
0709 12:24:59.673148 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:59.673452 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:59.683381 (+  9929us) log.cc:844] Serialized 14 byte log entry
0709 12:25:01.058035 (+1374654us) log.cc:844] Serialized 14 byte log entry
0709 12:25:01.720153 (+662118us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:01.720238 (+    85us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:01.720303 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.720384 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.720545 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":928008,"prepare.queue_time_us":835007,"prepare.run_cpu_time_us":154,"prepare.run_wall_time_us":9393,"raft.queue_time_us":957866,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":15790,"thread_start_us":1098114,"threads_started":2}
I20260709 12:25:01.778263  8797 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:01.838528  8884 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.295s	user 0.001s	sys 0.001s
I20260709 12:25:01.783903  8897 raft_consensus.cc:2764] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:25:01.903716  1723 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:25:01.796921  8739 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.772223  8889 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:01.920023  1723 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260709 12:25:01.923322  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 166318us
W20260709 12:25:01.911631  8795 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.533s	user 0.012s	sys 0.000s
I20260709 12:25:01.940347  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:01.911051  8352 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:01.949024  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 235327us
I20260709 12:25:01.919296  8896 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.928556  8483 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:01.956300  8701 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:01.963511  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 251453us
I20260709 12:25:01.970112  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:01.971361  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:01.972163  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:01.973930  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:01.945951  8910 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:01.993182  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3576) took 1117 ms. Trace:
I20260709 12:25:01.922474  8739 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.000078  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:01.938655  8797 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 12:25:01.995450  1727 rpcz_store.cc:276] 0709 12:25:00.875411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.969190 (+ 93779us) service_pool.cc:224] Handling call
0709 12:25:00.969537 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.969745 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:01.688595 (+718850us) log.cc:844] Serialized 14 byte log entry
0709 12:25:01.992611 (+304016us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:01.992724 (+   113us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:01.992801 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.992895 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.993102 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28942,"prepare.queue_time_us":573510,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"thread_start_us":969531,"threads_started":2}
I20260709 12:25:02.012782  1772 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:02.001570  8597 raft_consensus.cc:2764] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:25:02.013525  8917 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:01.941309  8757 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:25:01.961877  8352 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:01.960090  8766 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.943456  8795 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.017351  8831 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.033353  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:02.014180  8796 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 12:25:02.042088  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.043134  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.044709  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:02.034135  8897 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:01.923915  8765 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:01.978698  8890 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.357s	user 0.000s	sys 0.003s
I20260709 12:25:02.054665  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:02.036669  8908 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
W20260709 12:25:02.058736  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 118483us
W20260709 12:25:02.059827  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 397413us
I20260709 12:25:01.946141  8837 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.17, Last appended by leader: 17, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.065402  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:01.978477  8701 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.066165  1799 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:02.068090  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:02.075699  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 149615us
I20260709 12:25:02.037380  8911 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:02.053453  8897 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.076985  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:02.061185  8893 raft_consensus.cc:2764] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 12:25:02.079478  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:01.962445  8896 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.081178  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:02.055423  8597 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:02.038805  8926 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:02.064196  8757 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.055222  8888 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:02.028605  8910 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.049646  8749 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.038487  8800 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:25:02.078573  8790 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:02.082655  8891 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.307s	user 0.000s	sys 0.002s
I20260709 12:25:02.048321  8766 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.088059  8490 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.099750  8352 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.139147  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3577) took 1263 ms. Trace:
I20260709 12:25:02.085297  8916 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:02.085377  8897 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:02.147804  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2700) took 1319 ms. Trace:
I20260709 12:25:02.159374  1725 rpcz_store.cc:276] 0709 12:25:00.819291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.819526 (+   235us) service_pool.cc:224] Handling call
0709 12:25:02.128999 (+1309473us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b54002d05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:02.138422 (+  9423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":305650560}
I20260709 12:25:02.127595  8770 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.152940  1726 rpcz_store.cc:276] 0709 12:25:00.875681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.393763 (+518082us) service_pool.cc:224] Handling call
0709 12:25:01.411431 (+ 17668us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.411768 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.411941 (+   173us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.415809 (+  3868us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.768935 (+353126us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:01.769010 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.769109 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.138829 (+369720us) raft_consensus.cc:1685] finished
0709 12:25:02.138861 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.139058 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":352748,"threads_started":1,"wal-append.queue_time_us":507367}
I20260709 12:25:02.099887  8597 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.057919  8831 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:02.166873  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89976us
W20260709 12:25:02.159919  8906 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.124s	user 0.000s	sys 0.001s
W20260709 12:25:02.152735  8901 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.244s	user 0.000s	sys 0.005s
I20260709 12:25:02.143267  8922 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:02.169710  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 81886.2us
I20260709 12:25:02.088032  8911 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.155853  8916 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:02.108155  8757 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.188563  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:02.164283  8903 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
I20260709 12:25:02.164873  8352 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.185636  8862 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:02.139734  8797 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.188783  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2699) took 1387 ms. Trace:
I20260709 12:25:02.189453  1726 raft_consensus.cc:2368] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:25:02.169414  8597 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:02.182047  8846 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.194837  8933 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.219096  8933 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.221786  8933 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.185264  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:02.181499  8840 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.225s	user 0.012s	sys 0.004s
I20260709 12:25:02.200949  1725 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:02.168525  8490 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:25:02.181488  8832 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:02.213126  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2684) took 2397 ms. Trace:
I20260709 12:25:02.209062  1731 rpcz_store.cc:276] 0709 12:25:00.801586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.801900 (+   314us) service_pool.cc:224] Handling call
0709 12:25:00.824838 (+ 22938us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.825131 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:01.488292 (+663161us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.188156 (+699864us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:02.188243 (+    87us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:02.188346 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.188446 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.188701 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":651072,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":187292,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":6098,"spinlock_wait_cycles":37471616,"thread_start_us":1322740,"threads_started":3,"wal-append.queue_time_us":507956}
I20260709 12:25:02.216135  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:02.228760  1727 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:02.229794  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 159867us
I20260709 12:25:02.182580  8889 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.179584  8926 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:02.189674  8770 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:02.226325  1799 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:02.227830  1729 rpcz_store.cc:276] 0709 12:24:59.816162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.228781 (+412619us) service_pool.cc:224] Handling call
0709 12:25:00.245226 (+ 16445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.793591 (+548365us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:00.793642 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:00.809632 (+ 15990us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.545113 (+735481us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:01.545177 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.545299 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.212750 (+667451us) raft_consensus.cc:1685] finished
0709 12:25:02.212796 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.213033 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":573034,"raft.queue_time_us":553020,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":9417,"thread_start_us":698688,"threads_started":1,"wal-append.queue_time_us":826465}
I20260709 12:25:02.189744  8916 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:02.231024  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:02.230753  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:02.195695  8922 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:02.225817  8909 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.230s	user 0.000s	sys 0.002s
W20260709 12:25:02.245294  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.253734  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:02.262311  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:02.202521  8919 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:25:02.259502  1727 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:02.229522  8832 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.227512  8739 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.266453  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:25:02.272701  8788 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.276s	user 0.012s	sys 0.002s
I20260709 12:25:02.273300  8788 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.276188  8757 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.277063  8926 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:25:02.277737  8926 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:02.268944  8832 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:02.278872  8926 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:02.282155  8914 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20260709 12:25:02.207945  8352 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.227926  8920 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:02.235502  8927 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:02.263193  8854 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.469s	user 0.017s	sys 0.000s
I20260709 12:25:02.291472  2382 leader_election.cc:400] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:25:02.293900  2382 leader_election.cc:403] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:25:02.226104  8840 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.224042  8797 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.264103  8934 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:02.305609  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:02.306385  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.294401  8757 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:02.308007  8936 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.299850  8928 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:25:02.377912  8788 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:02.378798  8896 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.379122  8798 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.379518  8896 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:25:02.379683  8798 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:02.396720  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.345552  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3464) took 2064 ms. Trace:
I20260709 12:25:02.348052  8927 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:02.349493  8910 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.311731  8889 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
W20260709 12:25:02.353289  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 110358us
I20260709 12:25:02.359817  8766 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.360220  8352 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.361958  8701 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.362984  8657 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:02.303004  8888 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
W20260709 12:25:02.364099  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2544) took 1878 ms (client timeout 1612 ms). Trace:
I20260709 12:25:02.293744  8920 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.437705  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.361989  8831 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.367156  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:02.439950  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:02.441488  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:02.368149  1726 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:02.370102  8915 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20260709 12:25:02.370541  8846 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.371647  8756 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:02.327275  8922 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.421005  8493 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.399397  1872 rpcz_store.cc:276] 0709 12:25:00.278178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.537942 (+259764us) service_pool.cc:224] Handling call
0709 12:25:00.538383 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.783638 (+1245255us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.342464 (+558826us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400222254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:02.342758 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2852945280}
W20260709 12:25:02.401603  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 133492us
W20260709 12:25:02.354629  8931 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
W20260709 12:25:02.421372  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 125272us
I20260709 12:25:02.429153  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.395015  8934 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:02.432140  1805 rpcz_store.cc:269] 0709 12:25:00.485352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.486089 (+   737us) service_pool.cc:224] Handling call
0709 12:25:02.363831 (+1877742us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54001eaad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:02.363975 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3012833152}
I20260709 12:25:02.493291  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:02.433408  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.325035  8941 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:02.366148  1729 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:02.394383  8829 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.411038  8490 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.451721  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.411525  8823 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.455147  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:02.456530  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.457728  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.415468  8749 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.501808  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:02.458279  8756 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:02.373164  8840 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:02.428380  8483 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.469365  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.511031  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.511494  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.512022  1727 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:02.512215  1807 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:02.493006  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.494609  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:02.498045  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.501142  1872 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:02.410938  8936 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.459560  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.517881  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:02.518371  2413 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:02.518803  1874 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:02.519279  2418 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:02.502828  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.504534  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:02.428535  8910 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.060s
W20260709 12:25:02.502523  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47466.1us
I20260709 12:25:02.507090  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.467713  8926 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:02.523627  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:02.521978  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62722us
I20260709 12:25:02.520318  1872 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:02.524422  1874 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:02.492726  8846 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.498811  8795 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:02.529316  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:02.534229  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:02.530519  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:02.532974  1807 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:02.535646  1802 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:25:02.537406  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:02.537825  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.539398  1873 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:02.540349  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.506373  8831 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:02.517165  8840 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.508122  8790 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:02.545871  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.546504  2419 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:02.545083  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:02.505803  8490 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.490700  8493 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.549640  1876 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:02.551409  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.555693  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:02.559495  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.561406  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3579) took 1685 ms. Trace:
I20260709 12:25:02.538517  8798 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.508514  8749 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.535714  8934 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:02.502674  8829 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.537739  8927 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:02.561841  1724 rpcz_store.cc:276] 0709 12:25:00.876429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.562631 (+686202us) service_pool.cc:224] Handling call
0709 12:25:01.563258 (+   627us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.864552 (+301294us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.864604 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.915578 (+ 50974us) log.cc:844] Serialized 26 byte log entry
0709 12:25:02.037190 (+121612us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:02.037241 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.037335 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.353708 (+316373us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:02.556138 (+202430us) raft_consensus.cc:1685] finished
0709 12:25:02.556165 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.561320 (+  5155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4169,"raft.queue_time_us":128817,"raft.run_cpu_time_us":394,"raft.run_wall_time_us":8046,"spinlock_wait_cycles":7873280,"thread_start_us":314790,"threads_started":2,"wal-append.queue_time_us":101734}
I20260709 12:25:02.597731  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.527643  8788 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.033s
I20260709 12:25:02.614215  1876 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:02.547451  8888 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:25:02.601605  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:02.558504  8910 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:02.501340  8823 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.600054  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:02.525113  8483 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.560220  8837 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.546823  8945 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.065s
I20260709 12:25:02.620661  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.636154  8837 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.637383  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.561090  8800 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:02.538041  8846 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.638417  8837 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.623905  8490 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.562933  8756 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.624015  8934 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 22.10, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.624557  8483 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:02.625715  8947 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:25:02.625375  8832 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.617936  8910 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:02.627517  8943 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:25:02.628973  8823 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:02.629056  8954 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:25:02.631796  8493 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.633436  8829 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.545714  8926 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.556543  8831 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.603562  8352 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:25:02.640081  8846 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.556083  8840 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.640606  8597 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.638453  8800 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.601512  8927 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.641987  8916 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:02.604952  8851 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.118s	user 0.015s	sys 0.000s
I20260709 12:25:02.645793  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.684875  1726 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:02.602263  8889 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.644517  8749 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.610181  8923 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.699456  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:02.665158  8940 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
W20260709 12:25:02.665544  8921 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.438s	user 0.001s	sys 0.000s
W20260709 12:25:02.670109  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 126147us
I20260709 12:25:02.674579  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.676437  8832 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.662034  8945 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:02.662513  8910 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:02.682117  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.680744  8920 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.669160  8756 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:02.685227  1729 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:02.730770  1726 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:25:02.732196  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.694969  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:02.697490  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:02.551882  8897 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.700706  1874 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:02.684847  8927 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:02.703302  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.704430  1875 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:02.710345  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.712433  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:02.735831  1873 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:25:02.736146  8949 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.736760  8949 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.741025  8897 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.742988  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:02.743680  2413 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:02.745348  8597 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:02.713598  1845 connection.cc:256] RPC call timeout handler was delayed by -1.2625417889998971s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1708s for a response
I20260709 12:25:02.686549  8851 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.716997  8840 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.719218  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3582) took 1842 ms. Trace:
I20260709 12:25:02.705305  8941 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:02.721491  8911 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.700171  8829 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.711254  8923 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.768299  8851 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.769380  1729 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:02.770025  8923 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.723403  8831 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.724701  8926 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.772111  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:02.772842  1729 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:02.779655  1723 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:02.782233  8800 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.751524  8797 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.787379  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:02.725800  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.733081  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:02.748245  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:02.748791  1845 connection.cc:256] RPC call timeout handler was delayed by -1.2305020550002155s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1612s for a response
I20260709 12:25:02.788736  2418 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:02.789003  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55977.1us
W20260709 12:25:02.719774  8917 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.664s	user 0.014s	sys 0.000s
I20260709 12:25:02.753825  1728 rpcz_store.cc:276] 0709 12:25:00.877207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.653956 (+776749us) service_pool.cc:224] Handling call
0709 12:25:01.654607 (+   651us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.669879 (+ 15272us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.669934 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.689184 (+ 19250us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.969806 (+280622us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:01.969873 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.969987 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.644314 (+674327us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:02.718882 (+ 74568us) raft_consensus.cc:1685] finished
0709 12:25:02.718907 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.719150 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30806,"raft.queue_time_us":63988,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":34590,"spinlock_wait_cycles":27659008,"thread_start_us":210003,"threads_started":1,"wal-append.queue_time_us":444744}
I20260709 12:25:02.797147  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:02.700476  8889 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.764245  8175 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:02.766265  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:02.767719  8829 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.810405  8829 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:02.789769  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:02.790688  8926 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:02.790818  8797 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:02.811328  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:02.797878  1807 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:02.798063  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:02.820791  8790 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:02.821808  8790 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:02.801928  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:02.795105  8949 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.824750  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:02.826238  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:02.827396  1804 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:02.828589  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:02.829454  1804 raft_consensus.cc:2368] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:25:02.830449  1728 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:02.831393  2383 leader_election.cc:400] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:25:02.815552  8910 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.832037  2383 leader_election.cc:403] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:25:02.806078  8965 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:02.810858  8749 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 22.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:25:02.806248  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:02.806288  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:02.816097  8926 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.835210  8965 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.804477  2418 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
W20260709 12:25:02.838164  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:02.865130  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1387) took 2207 ms. Trace:
I20260709 12:25:02.860481  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:02.869544  1724 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:02.854192  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:02.868784  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:02.872975  1799 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:02.875429  1729 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:02.876278  1880 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 14.
I20260709 12:25:02.845434  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:02.877405  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:02.877379  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.878991  1880 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:02.880071  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:02.795131  8917 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:02.835788  8749 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:25:02.880613  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49547.2us
I20260709 12:25:02.858364  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3588) took 1823 ms. Trace:
I20260709 12:25:02.882230  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:02.882129  1730 rpcz_store.cc:276] 0709 12:25:01.052402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.902385 (+849983us) service_pool.cc:224] Handling call
0709 12:25:01.902820 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.945419 (+ 42599us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.945452 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:01.945508 (+    56us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:01.945552 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.858011 (+912459us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b540030be54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:02.858079 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.858282 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97500928}
I20260709 12:25:02.883669  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:02.883813  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.884589  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:02.885311  2413 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:02.874665  2415 rpcz_store.cc:276] 0709 12:25:00.657442 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.657619 (+   177us) service_pool.cc:224] Handling call
0709 12:25:00.657952 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.832949 (+1174997us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.864519 (+1031570us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400561b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:02.864981 (+   462us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2699962624}
I20260709 12:25:02.789235  8916 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:02.891362  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:02.884486  1730 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
W20260709 12:25:02.833201  8929 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.672s	user 0.003s	sys 0.000s
I20260709 12:25:02.879902  8926 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:25:02.982703  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 82687.9us
I20260709 12:25:02.891343  8756 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:02.974442  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:02.809684  8941 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.981925  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:02.984181  1878 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:02.846024  8770 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:02.820775  8968 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:02.854761  8539 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:02.851936  8175 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.899329  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:02.854206  8889 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.864041  8758 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:25:03.010278  8758 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:03.010417  8961 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.983739  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:02.851588  8933 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:02.936378  1728 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:02.986918  1880 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:02.898818  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.018059  1803 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
W20260709 12:25:02.891405  8886 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.503s	user 0.002s	sys 0.002s
I20260709 12:25:02.891973  8749 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:02.892167  8911 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:02.834352  8910 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:02.846272  8829 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:03.000146  8756 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:02.983418  8831 raft_consensus.cc:2764] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:25:02.983018  8493 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:02.898912  8296 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:25:03.041265  8913 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:03.041808  8913 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.042361  8913 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:02.986693  8955 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:25:03.049386  8749 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.10, Last appended by leader: 10, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.051618  8749 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:03.011418  8758 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.984601  8846 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:02.994951  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:03.019562  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:03.020507  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:02.993033  8770 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:03.001705  8923 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:25:02.880553  8743 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.822815  8790 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 19.17, Last appended by leader: 17, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:02.988332  8657 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 12:25:03.011085  8961 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:02.982383  8766 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:03.011767  8175 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.016986  8933 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:02.844929  8927 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.023121  8956 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:25:03.084067  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 89284.6us
I20260709 12:25:03.031138  8829 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:03.087988  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 67763.4us
I20260709 12:25:03.067268  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.090804  1730 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:03.064623  8493 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.096800  8823 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:25:03.085659  8923 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:03.146936  1728 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:25:03.086150  8846 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.152340  1729 raft_consensus.cc:2410] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:25:03.153865  1728 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:25:03.106205  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:03.155207  1728 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:03.116593  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:03.116804  1878 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:03.117794  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.118458  8770 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.120555  8493 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:03.107921  8766 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.019551  8968 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:03.163421  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46917.5us
I20260709 12:25:03.132011  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:03.135413  1726 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:03.137051  1806 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:25:03.086517  8831 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:25:03.155969  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49879.6us
I20260709 12:25:03.170639  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:03.166606  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:03.177381  1875 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:03.181584  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:03.187572  2413 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:03.150226  8846 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:03.176100  8910 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:03.138255  8958 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.192260  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:03.176991  8823 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:03.200623  2415 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:03.201862  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.202193  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:03.155146  8923 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:03.198573  1876 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:03.180888  8956 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:03.202692  1799 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:03.189183  8770 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:03.182622  8490 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.188920  8801 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:03.207154  8766 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:03.210479  8175 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:03.178480  8911 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:03.178121  8889 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.185842  8798 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:25:03.174494  8829 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:03.183575  8955 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:03.215555  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:03.193172  8632 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:25:03.185497  8539 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:03.227797  1725 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:03.212117  8836 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:03.194090  8910 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:03.094589  8743 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.196621  8849 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:03.236792  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.224339  8657 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:25:03.139016  8756 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.235198  8490 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.198983  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.242242  8889 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.235579  8801 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.220885  8823 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.212328  8958 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.262815  1725 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:03.276042  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.281090  1878 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:03.277029  1799 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:03.263818  8539 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.284695  8889 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:03.287292  8539 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.288931  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.289749  1878 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:03.289996  8490 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.290880  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.291865  8657 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:03.293108  8657 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:03.295141  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.255818  8968 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.259263  8798 consensus_queue.cc:1243] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:25:03.296932  1878 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:03.263854  8910 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:03.225672  8933 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.272970  8927 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.303835  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.309919  8743 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:03.329684  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.311671  8801 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:03.349609  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.352732  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.354657  8936 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:03.356096  1799 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:03.358304  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:03.369612  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:03.370489  1874 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:03.452193  8926 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.343776  8913 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.463451  8770 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.464329  8770 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.463546  1725 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:03.537335  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.553474  1799 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:03.558594  8770 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:03.620014  1874 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:03.623337  1802 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
W20260709 12:25:03.622586  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 979) took 1581 ms (client timeout 1947 ms). Trace:
W20260709 12:25:03.625164  1803 rpcz_store.cc:269] 0709 12:25:02.041464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.041636 (+   172us) service_pool.cc:224] Handling call
0709 12:25:03.622364 (+1580728us) spinlock_profiling.cc:243] Waited 933 ms on lock 0x7b5400133654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:03.622509 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2145491712}
I20260709 12:25:03.628525  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:03.629422  1803 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:03.628371  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.631255  1874 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 14.
I20260709 12:25:03.633967  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:03.634948  1799 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:03.636368  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.652164  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.652904  1873 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:03.654912  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.655735  1874 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:03.657413  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:03.658012  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.658510  1732 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:03.667546  1732 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 10. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
W20260709 12:25:03.669318  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 453842us
I20260709 12:25:03.675688  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:03.611161  8829 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.676478  1873 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:03.683010  1799 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:03.684870  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.660442  8926 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.692777  1803 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:03.689653  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:03.684499  8913 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.683566  8632 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.730260  8632 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:03.731288  8900 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.959s	user 0.001s	sys 0.014s
I20260709 12:25:03.732867  8632 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.734508  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.735808  8910 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.736173  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:03.699002  1873 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:03.699182  1803 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:25:03.699396  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:03.697767  8749 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.646s	user 0.012s	sys 0.003s
I20260709 12:25:03.710872  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2715) took 1062 ms. Trace:
I20260709 12:25:03.669973  8923 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.743110  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:03.744006  1874 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
W20260709 12:25:03.744251  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2540) took 3258 ms (client timeout 1708 ms). Trace:
W20260709 12:25:03.744771  1801 rpcz_store.cc:269] 0709 12:25:00.484159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.484433 (+   274us) service_pool.cc:224] Handling call
0709 12:25:03.742166 (+3257733us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:03.742300 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2277273856}
I20260709 12:25:03.730149  8965 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:03.729266  8493 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.540s	user 0.014s	sys 0.000s
I20260709 12:25:03.740304  1723 rpcz_store.cc:276] 0709 12:25:02.648013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.707490 (+ 59477us) service_pool.cc:224] Handling call
0709 12:25:03.710650 (+1003160us) spinlock_profiling.cc:243] Waited 43.8 ms on lock 0x7b54003cc8d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:03.710801 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":100721536}
W20260709 12:25:03.721949  8831 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.539s	user 0.012s	sys 0.000s
I20260709 12:25:03.747668  1801 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:03.747681  1879 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:03.761646  8829 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:03.743129  1873 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
W20260709 12:25:03.748090  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58523us
I20260709 12:25:03.764873  1801 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:25:03.767329  8923 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.752592  8749 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:25:03.771757  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.772881  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2721) took 1065 ms. Trace:
I20260709 12:25:03.756788  8493 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.775029  1873 rpcz_store.cc:276] 0709 12:25:02.707770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.742625 (+1034855us) service_pool.cc:224] Handling call
0709 12:25:03.742922 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.772539 (+ 29617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.772816 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:03.774240  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:03.776715  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.785612  1799 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:03.771222  8749 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:03.789531  1875 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:03.783352  8831 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:03.799539  1872 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:03.800305  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2731) took 1028 ms. Trace:
I20260709 12:25:03.800627  1872 rpcz_store.cc:276] 0709 12:25:02.771975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.799086 (+1027111us) service_pool.cc:224] Handling call
0709 12:25:03.799372 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.800067 (+   695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.800254 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:03.801460  8963 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:03.801852  8963 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:03.801997  1872 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:25:03.802549  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2733) took 1006 ms. Trace:
I20260709 12:25:03.799579  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.807063  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:03.805960  1723 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:03.830407  1798 raft_consensus.cc:2376] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18. Re-sending same reply.
I20260709 12:25:03.831049  1725 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:03.834127  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:03.828997  1872 rpcz_store.cc:276] 0709 12:25:02.796258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.801391 (+1005133us) service_pool.cc:224] Handling call
0709 12:25:03.801672 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.802321 (+   649us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.802501 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:03.813457  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:03.837453  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.838235  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:03.839448  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:03.840596  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.833459  8798 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.841241  8798 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.842402  1872 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:03.843324  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.844574  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:03.845198  1798 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:03.845525  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:03.819970  8950 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.863106  8823 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:03.869831  8923 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:03.860320  8829 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:03.870675  8923 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.871608  8829 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:03.872973  8923 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:03.859573  8749 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.880010  8493 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:03.884999  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 112336us
I20260709 12:25:03.888078  1798 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:25:03.892307  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1391) took 3208 ms. Trace:
I20260709 12:25:03.892722  2412 rpcz_store.cc:276] 0709 12:25:00.684444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.685013 (+   569us) service_pool.cc:224] Handling call
0709 12:25:00.734684 (+ 49671us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.548897 (+814213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.892027 (+2343130us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b54004560d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:03.892250 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1861556608}
I20260709 12:25:03.866925  8950 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:03.912842  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.802699  8963 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:03.893735  8910 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.884397  8978 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.028s
W20260709 12:25:03.942313  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74721.3us
W20260709 12:25:03.947969  8969 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.211s	user 0.002s	sys 0.000s
I20260709 12:25:03.948491  8950 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:03.950193  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1397) took 1426 ms. Trace:
I20260709 12:25:03.951016  8961 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:03.951296  8853 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:03.951786  8853 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:03.952706  8853 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:03.953799  8853 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:03.953946  8703 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:03.968490  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:03.969662  1879 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:03.984349  1879 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:03.985400  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1017) took 1185 ms. Trace:
I20260709 12:25:03.985865  1879 rpcz_store.cc:276] 0709 12:25:02.799825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.983521 (+1183696us) service_pool.cc:224] Handling call
0709 12:25:03.984145 (+   624us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.985114 (+   969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.985335 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:03.989784  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1018) took 1189 ms. Trace:
I20260709 12:25:03.950675  2414 rpcz_store.cc:276] 0709 12:25:02.523378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.526071 (+  2693us) service_pool.cc:224] Handling call
0709 12:25:02.548585 (+ 22514us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:02.560666 (+ 12081us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:02.560721 (+    55us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:02.569455 (+  8734us) log.cc:844] Serialized 52 byte log entry
0709 12:25:02.672738 (+103283us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:02.672771 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.672828 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:03.947169 (+1274341us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:03.949828 (+  2659us) raft_consensus.cc:1685] finished
0709 12:25:03.949854 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.950122 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":326,"raft.queue_time_us":10241,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":6016,"spinlock_wait_cycles":32931200,"thread_start_us":5607,"threads_started":1,"wal-append.queue_time_us":68754}
I20260709 12:25:03.944758  8963 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:03.996037  1879 rpcz_store.cc:276] 0709 12:25:02.800062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.988577 (+1188515us) service_pool.cc:224] Handling call
0709 12:25:03.989039 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.989299 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.989341 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:03.989406 (+    65us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:03.989458 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.989549 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.989709 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:03.951086  8613 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:03.974198  8950 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:03.983199  8400 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.019s
I20260709 12:25:03.991787  8961 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:25:03.983328  8823 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.019s
W20260709 12:25:04.009810  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.011426  1879 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:04.012874  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1019) took 1212 ms. Trace:
I20260709 12:25:04.013330  1879 rpcz_store.cc:276] 0709 12:25:02.800290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.998496 (+1198206us) service_pool.cc:224] Handling call
0709 12:25:04.011124 (+ 12628us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.012580 (+  1456us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.012815 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:04.014431  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:04.017237  1806 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 984) took 1970 ms (client timeout 2156 ms). Trace:
W20260709 12:25:04.017768  1806 rpcz_store.cc:269] 0709 12:25:02.046407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.058839 (+ 12432us) service_pool.cc:224] Handling call
0709 12:25:04.017022 (+1958183us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001676d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:04.017165 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2417376512}
W20260709 12:25:04.020931  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.021469  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.021927  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.027331  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:04.034404  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3459) took 3815 ms. Trace:
I20260709 12:25:04.037488  1877 rpcz_store.cc:276] 0709 12:25:00.210063 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.458472 (+248409us) service_pool.cc:224] Handling call
0709 12:25:00.463165 (+  4693us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.596676 (+1133511us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.025337 (+2428661us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:04.025549 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2588932992}
I20260709 12:25:04.037919  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:04.041136  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:04.041956  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:04.042213  1731 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.012522  8950 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.009680  8910 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:04.028569  8798 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:04.027030  8989 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:04.065219  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:04.035912  8923 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.035336  8831 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:04.033926  8767 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.065910  1877 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:04.033574  8759 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:04.067898  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.047832  8770 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.071638  2418 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: []
I20260709 12:25:04.047715  8926 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:04.072479  1806 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:04.073302  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:04.075939  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.076085  2417 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:04.073843  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.074319  2412 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:04.063153  8913 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:04.082548  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.084715  2413 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:04.088109  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:04.092223  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.069130  8989 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:04.098207  1731 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:04.061398  8910 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.100560  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:04.079962  8400 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:04.069057  8756 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:25:04.104990  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.105602  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:04.109452  8910 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:04.111975  8992 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:25:04.114004  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:04.123335  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.074985  8767 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.101328  8352 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:04.078600  8759 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.074862  8831 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:25:04.138718  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.099835  8989 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.072480  8923 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.080511  8749 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:04.141722  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.080080  8770 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.136117  8632 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:04.105053  8400 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:04.163444  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:04.164755  8831 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:04.166887  8923 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:04.106858  8978 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:25:04.171456  8983 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:25:04.173087  1725 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:04.141117  8796 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:04.164805  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.180678  8958 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:04.184520  8759 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:04.185977  1731 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:04.191298  1806 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.191373  1877 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:25:04.215792  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.178488  8767 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:04.141464  8352 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.221338  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.223805  1731 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:04.225169  1731 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:25:04.226471  1731 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:25:04.222119  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:04.227896  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:04.229677  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1023) took 1428 ms. Trace:
I20260709 12:25:04.185274  8749 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:04.190577  8632 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.230031  1877 rpcz_store.cc:276] 0709 12:25:02.801257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.228853 (+1427596us) service_pool.cc:224] Handling call
0709 12:25:04.229169 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.229361 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.229387 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.229431 (+    44us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:04.229471 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.229527 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.229622 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:04.237661  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 199826us
I20260709 12:25:04.238562  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:04.239722  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.239022  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.242569  1731 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:04.241211  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:04.231127  8352 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:04.249903  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:04.251667  1806 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:04.252300  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:04.254045  1877 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:25:04.236054  8632 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:04.256320  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.258371  8770 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:04.259037  8749 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.220826  8796 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.261910  8796 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:04.255659  8701 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:04.243167  8853 raft_consensus.cc:2764] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
W20260709 12:25:04.265370  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2722) took 1514 ms (client timeout 1639 ms). Trace:
W20260709 12:25:04.265852  1876 rpcz_store.cc:269] 0709 12:25:02.751911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.757572 (+  5661us) service_pool.cc:224] Handling call
0709 12:25:04.265198 (+1507626us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:04.265309 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":892559616}
I20260709 12:25:04.267373  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.243567  8400 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.269650  8400 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:04.270102  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3605) took 1526 ms. Trace:
I20260709 12:25:04.270574  1729 rpcz_store.cc:276] 0709 12:25:02.743705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.899001 (+155296us) service_pool.cc:224] Handling call
0709 12:25:04.269885 (+1370884us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:04.270029 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":344718080}
I20260709 12:25:04.272629  8795 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:04.276832  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2614) took 1471 ms. Trace:
I20260709 12:25:04.277535  8955 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:04.277972  8955 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:03.620352   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:04.279584  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.239552  8978 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:04.277349  1803 rpcz_store.cc:276] 0709 12:25:02.805046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.739521 (+934475us) service_pool.cc:224] Handling call
0709 12:25:03.741851 (+  2330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.742237 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.742287 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:03.742738 (+   451us) log.cc:844] Serialized 26 byte log entry
0709 12:25:03.887251 (+144513us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:03.991853 (+104602us) log.cc:844] Serialized 14 byte log entry
0709 12:25:03.993663 (+  1810us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.993758 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.276584 (+282826us) raft_consensus.cc:1685] finished
0709 12:25:04.276609 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.276761 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":148397,"prepare.run_cpu_time_us":152,"prepare.run_wall_time_us":4691,"raft.queue_time_us":2846,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":10181,"thread_start_us":212059,"threads_started":2,"wal-append.queue_time_us":212838}
I20260709 12:25:04.284816  1877 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:25:04.282593  8701 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:04.286753  8701 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.289117  8352 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.283830  8770 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:04.290534  8770 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:04.291947  8770 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.253026  8923 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:04.304028  1877 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:25:04.305897  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:04.308269  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2611) took 1511 ms. Trace:
I20260709 12:25:04.308696  1802 rpcz_store.cc:276] 0709 12:25:02.797005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.803376 (+  6371us) service_pool.cc:224] Handling call
0709 12:25:03.623039 (+819663us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.623992 (+   953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.308045 (+684053us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b5400133650. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:04.308195 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1883992320}
I20260709 12:25:04.309909  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.310247  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:04.310479  1802 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:25:04.307821  8853 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:04.312851  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.313378  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.313613  1806 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:04.314162  1802 raft_consensus.cc:2410] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:25:04.315044  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:04.301698  8923 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.302476  8913 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:25:04.332022  8923 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:04.317452  8986 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:25:04.340132  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2556) took 3707 ms (client timeout 1664 ms). Trace:
W20260709 12:25:04.342635  1800 rpcz_store.cc:269] 0709 12:25:00.632493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.632627 (+   134us) service_pool.cc:224] Handling call
0709 12:25:04.339880 (+3707253us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001eb4d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:04.340056 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2657360256}
I20260709 12:25:04.344098  1800 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:25:04.340791  1804 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:25:04.283260  8955 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.341894  9005 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:04.350880  8910 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:25:04.334336  8919 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.029s	user 0.004s	sys 0.015s
I20260709 12:25:04.347592  8978 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:04.369223  1724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3599) took 1876 ms (client timeout 2402 ms). Trace:
W20260709 12:25:04.379431  1724 rpcz_store.cc:269] 0709 12:25:02.492476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.597523 (+105047us) service_pool.cc:224] Handling call
0709 12:25:04.364076 (+1766553us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b5400002154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:04.369134 (+  5058us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":553755008}
I20260709 12:25:04.316481  8798 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:04.338709  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:04.373591  1800 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:25:04.375425  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:04.366540  8758 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:04.384322  1800 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:04.382424  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43797.6us
I20260709 12:25:04.385512  1724 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:25:04.386101  1724 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:04.389207  8769 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:04.391440  1724 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.393070  1724 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:25:04.390609  8919 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.089s	user 0.004s	sys 0.016s
W20260709 12:25:04.400381  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2725) took 1630 ms (client timeout 2048 ms). Trace:
W20260709 12:25:04.400902  1878 rpcz_store.cc:269] 0709 12:25:02.770120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.290567 (+520447us) service_pool.cc:224] Handling call
0709 12:25:04.400168 (+1109601us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x7b5400072e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:04.400315 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11782528}
I20260709 12:25:04.390105  8986 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:04.402318  8986 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.402719  8986 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:04.406095  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:04.409183  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:04.410300  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:04.411438  1729 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:04.412678  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.396771  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.415982  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.411437  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:04.417474  1731 raft_consensus.cc:2368] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
I20260709 12:25:04.383615  8352 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:04.418691  1877 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:04.419888  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.420593  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.421736  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:04.423048  1731 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 14. Current term is 15. Ops: []
I20260709 12:25:04.424166  1729 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:04.424448  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:04.425431  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.426334  1800 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:25:04.428005  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:04.398835  8701 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:04.432169  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:04.436259  8994 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:04.436712  8994 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:04.437502  8994 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.17, Last appended by leader: 17, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.437757  8632 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.439002  8916 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:04.440210  8916 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:04.440584  8916 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:04.440786  8950 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:04.441246  8632 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:04.441447  8916 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.443159  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3603) took 1790 ms. Trace:
I20260709 12:25:04.443539  1726 rpcz_store.cc:276] 0709 12:25:02.652639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.169286 (+516647us) service_pool.cc:224] Handling call
0709 12:25:03.184111 (+ 14825us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.184579 (+   468us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:03.956798 (+772219us) spinlock_profiling.cc:243] Waited 3.49 ms on lock 0x7b18004d26c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf 00007f16abdf2038
0709 12:25:04.032759 (+ 75961us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.232439 (+199680us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:04.284604 (+ 52165us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.284762 (+   158us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:04.284793 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.284864 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.434322 (+149458us) raft_consensus.cc:1685] finished
0709 12:25:04.442935 (+  8613us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.443097 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":552417,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":3817,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":13131,"spinlock_wait_cycles":8027392,"thread_start_us":226176,"threads_started":2,"wal-append.queue_time_us":237132}
I20260709 12:25:04.445835  1726 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:04.448781  8989 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.451017  1879 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:04.434319  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:04.460115  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.461529  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:04.462627  1724 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:04.465693  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.430183  8769 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.394889  8965 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.040s
I20260709 12:25:04.474536  1807 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:04.434267  9005 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:04.452527  8994 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.433885  8928 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:04.457093  8916 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:04.431937  8798 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.076s
W20260709 12:25:04.506421  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.529708  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1016) took 1730 ms. Trace:
I20260709 12:25:04.530246  1872 rpcz_store.cc:276] 0709 12:25:02.799600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.835181 (+1035581us) service_pool.cc:224] Handling call
0709 12:25:03.835506 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.842733 (+  7227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.529318 (+686585us) spinlock_profiling.cc:243] Waited 6.58 ms on lock 0x7b54003525d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:04.529609 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15135104}
I20260709 12:25:04.531515  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:04.531740  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:04.536653  1770 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:04.430943  8657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.022s	sys 0.018s
I20260709 12:25:04.510999  1724 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:25:04.505930  9005 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.542230  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:04.542910  2418 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:04.507624  8769 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:04.520727  8974 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 1.304s	user 0.007s	sys 0.009s
W20260709 12:25:04.523524  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.524119  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3480) took 2016 ms. Trace:
I20260709 12:25:04.434172  8701 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.468056  8989 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:04.544049  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.551290  1881 rpcz_store.cc:276] 0709 12:25:02.507777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.508926 (+  1149us) service_pool.cc:224] Handling call
0709 12:25:02.528087 (+ 19161us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:02.544295 (+ 16208us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:02.668438 (+124143us) log.cc:844] Serialized 14 byte log entry
0709 12:25:02.988102 (+319664us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:04.259647 (+1271545us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b2c00009c38. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:25:04.260357 (+   710us) log.cc:844] Serialized 52 byte log entry
0709 12:25:04.260539 (+   182us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:04.260572 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.260660 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.516796 (+256136us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b18002b7020. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:04.523826 (+  7030us) raft_consensus.cc:1685] finished
0709 12:25:04.523852 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.524048 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2901,"prepare.queue_time_us":178565,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":96097,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":43165,"spinlock_wait_cycles":19812608,"thread_start_us":436775,"threads_started":3,"wal-append.queue_time_us":330128}
I20260709 12:25:04.549710  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:04.557878  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89355us
I20260709 12:25:04.557801  2417 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:04.553191  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.554733  2418 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:04.560635  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:04.561249  1724 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:25:04.561160  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:04.541543  8758 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:04.467988  8935 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.082s
I20260709 12:25:04.466773  9011 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.116s
I20260709 12:25:04.634464  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.549034  9006 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:25:04.545348  9005 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:04.587257  2382 leader_election.cc:400] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22.
W20260709 12:25:04.599030  9007 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:25:04.593502  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.594002  8997 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.524933  8823 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.046s
I20260709 12:25:04.557049  8352 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:04.595360  8400 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.010s
I20260709 12:25:04.557700  8701 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:04.585917  8963 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.085s
W20260709 12:25:04.605926  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45346.7us
I20260709 12:25:04.677794  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:04.609944  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.594185  8769 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:04.554994  8950 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:04.647095  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:04.585988  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:04.649573  9010 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:25:04.598367  8923 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:04.650095  9009 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
W20260709 12:25:04.648995  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84215.9us
I20260709 12:25:04.652099  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:04.632166  9003 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.418s	user 0.003s	sys 0.000s
I20260709 12:25:04.648823  8928 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:04.656165  9016 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:04.656474  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:04.657328  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:04.658618  8770 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:04.659756  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:04.663206  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:04.700018  2415 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:04.673534  8352 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:04.563058  8764 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.088s
I20260709 12:25:04.565451  8919 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.087s
W20260709 12:25:04.551148  8657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.866s	user 0.023s	sys 0.018s
I20260709 12:25:04.688731  8703 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:04.632211  8765 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:04.673913  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 266116us
W20260709 12:25:04.675585  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.599726  9008 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.089s
W20260709 12:25:04.696782  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44764.1us
W20260709 12:25:04.708751  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49103.2us
I20260709 12:25:04.667549  8796 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:25:04.673156  8997 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.758001  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:04.758111  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:04.762676  1873 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 14. Current term is 15. Ops: []
I20260709 12:25:04.768908  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:04.773720  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:04.704291  8767 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:04.703116  8352 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:04.699344  8950 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:04.790200  2415 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:04.742724  8910 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:04.710805  8765 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.712282  8770 consensus_queue.cc:1243] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 21.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:04.710470  8703 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:04.798899  8796 raft_consensus.cc:2764] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:25:04.707612  8958 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.820250  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 118263us
I20260709 12:25:04.704610  8935 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.094s
W20260709 12:25:04.826759  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68860.1us
I20260709 12:25:04.828920  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:04.741608  9023 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.102s
I20260709 12:25:04.837818  9023 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:04.838223  9023 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:04.802217  8765 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:04.839619  9014 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260709 12:25:04.839408  9023 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.840675  9023 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.842177  8997 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:04.843916  9005 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.846539  9005 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:04.825997  8910 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:04.851253  8910 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:04.835493  8935 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:25:04.852329  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.826462  8958 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.816725  8703 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:04.854360  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:04.855080  2417 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:04.855198  8958 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:04.856585  8923 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:04.857699  8919 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:04.858572  8919 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:04.858929  8919 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.859297  8919 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:04.859663  2419 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:04.861076  9008 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:04.861481  9008 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:04.862107  2418 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:25:04.862373  9008 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.847023  8770 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:04.852630  8935 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:04.863787  9008 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.864496  8935 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.852294  8910 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.865943  8935 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:04.867863  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:04.869624  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:04.816623  8928 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.861477  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2612) took 2064 ms. Trace:
I20260709 12:25:04.876116  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:04.876695  2418 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:04.875756  1804 rpcz_store.cc:276] 0709 12:25:02.797299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.874977 (+ 77678us) service_pool.cc:224] Handling call
0709 12:25:04.340496 (+1465519us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.344742 (+  4246us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.861294 (+516552us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54001eb4d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:04.861420 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3369427200}
I20260709 12:25:04.847827  8950 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.833860  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:04.880724  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.889250  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.888541  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53764.7us
W20260709 12:25:04.891301  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.890739  9005 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.002s
I20260709 12:25:04.894096  8933 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:04.901696  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.902552  2416 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 15.
I20260709 12:25:04.915722  2417 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:04.919865  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:04.922681  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.887789  8758 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:04.925174  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.930424  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.887660  8657 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:04.925971  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 92180.1us
I20260709 12:25:04.928812  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:04.930102  1800 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:25:04.930652  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.903246  8970 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.936842  1805 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:25:04.890868  9011 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:04.920898  1804 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:04.942960  1803 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:25:04.943559  2418 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:04.919893  8935 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.946003  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.887231  8986 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.947139  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:04.952284  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2735) took 2091 ms (client timeout 1770 ms). Trace:
I20260709 12:25:04.953516  1799 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:25:04.953935  1799 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:04.939548  8632 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:04.936930  8923 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.944849  9011 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:04.952797  1875 rpcz_store.cc:269] 0709 12:25:02.861071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.183699 (+322628us) service_pool.cc:224] Handling call
0709 12:25:04.952034 (+1768335us) spinlock_profiling.cc:243] Waited 512 ms on lock 0x7b5400026554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:04.952207 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1178358528}
I20260709 12:25:04.955679  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:04.921423  9005 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:25:04.966152  2410 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:04.965270  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.946067  9029 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.971349  2418 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:04.972618  8632 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:04.968644  1799 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.981794  8796 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:04.982391  8796 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:04.982903  8796 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:04.983747  8854 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:04.984093  9005 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:04.984190  8854 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:04.956393  9032 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:04.990157  1799 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:04.968988  9011 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:04.952422  8935 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:04.944547  8970 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:04.984921  9005 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:05.054514  2415 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:04.986549  9029 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:05.087590  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.091135  8935 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:05.096367  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 322890us
I20260709 12:25:05.106060  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:05.105971  1805 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:25:05.107070  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 178462us
I20260709 12:25:05.107425  2418 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:05.107510  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:05.108744  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3597) took 2603 ms. Trace:
I20260709 12:25:05.109301  1730 rpcz_store.cc:276] 0709 12:25:02.491848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.981448 (+489600us) service_pool.cc:224] Handling call
0709 12:25:02.984943 (+  3495us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.116177 (+131234us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.072521 (+1956344us) spinlock_profiling.cc:243] Waited 79.7 ms on lock 0x7b540030a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:05.094817 (+ 22296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":183362944}
I20260709 12:25:05.111243  1730 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: []
I20260709 12:25:05.115736  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:05.116825  1730 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:05.114070  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:05.117312  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.119961  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:05.124330  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:05.126039  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:05.127008  1799 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:25:05.127725  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2573) took 1355 ms. Trace:
I20260709 12:25:05.128027  1799 rpcz_store.cc:276] 0709 12:25:03.771947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.864535 (+ 92588us) service_pool.cc:224] Handling call
0709 12:25:03.886036 (+ 21501us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.953421 (+1067385us) spinlock_profiling.cc:243] Waited 511 ms on lock 0x7b5400409320. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:25:05.127429 (+174008us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.127659 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1175037696}
I20260709 12:25:05.135246  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:05.123435  8770 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:05.140781  9028 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:05.148837  9028 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:05.149335  9028 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:05.146067  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 377129us
I20260709 12:25:05.123750  8928 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.146262  9029 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.146858  9000 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:05.451300  8701 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.762s	user 0.018s	sys 0.000s
I20260709 12:25:05.157975  9030 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:05.118589  8854 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.254079  9005 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:05.454118  9008 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.590s	user 0.010s	sys 0.010s
I20260709 12:25:05.147944  8769 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:05.487643  9030 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:05.142812  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:05.455474  2383 connection.cc:256] RPC call timeout handler was delayed by -1.7503356789998179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
I20260709 12:25:05.455859  9044 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:05.398253  9037 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:05.148062  8757 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:25:05.154353  8632 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:05.458510  8919 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.599s	user 0.018s	sys 0.000s
W20260709 12:25:05.458882  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.546263  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.461844  9011 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.548633  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:05.549837  1875 raft_consensus.cc:2410] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 12:25:05.462996  8765 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.624s	user 0.010s	sys 0.009s
I20260709 12:25:05.551232  8765 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.120366  8935 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 12:25:05.554844  8935 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:25:05.463896  8970 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:05.464039  8703 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.610s	user 0.018s	sys 0.000s
I20260709 12:25:05.462634  8913 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:05.464561  8955 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:05.498070  8753 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:05.465770  9023 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.625s	user 0.015s	sys 0.002s
I20260709 12:25:05.152340  8770 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:05.466802  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.467406  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.613759  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.458364  8923 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:05.466104  9042 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 12:25:05.484920  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 376634us
W20260709 12:25:05.462585  8352 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.664s	user 0.013s	sys 0.002s
I20260709 12:25:05.619835  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:05.490125  8701 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:05.524187  8796 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.541s	user 0.014s	sys 0.005s
I20260709 12:25:05.637120  8796 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.529196  9044 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:05.530632  8926 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:25:05.640964  8926 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.498754  8769 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.641661  8926 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.642313  8769 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:05.644594  8926 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.532008  9037 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:05.532722  1807 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2628) took 2676 ms (client timeout 1507 ms). Trace:
W20260709 12:25:05.646044  1807 rpcz_store.cc:269] 0709 12:25:02.856237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.898464 (+ 42227us) service_pool.cc:224] Handling call
0709 12:25:05.532461 (+2633997us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54002f2e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:05.532613 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3347824384}
I20260709 12:25:05.533888  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1026) took 1387 ms. Trace:
I20260709 12:25:05.647020  1880 rpcz_store.cc:276] 0709 12:25:04.146004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.263874 (+117870us) service_pool.cc:224] Handling call
0709 12:25:04.274746 (+ 10872us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.275005 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.275030 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.275076 (+    46us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:04.275115 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.275170 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.533686 (+1258516us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b54002062d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:05.533802 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24057216}
W20260709 12:25:05.534355  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2745) took 1521 ms (client timeout 1971 ms). Trace:
I20260709 12:25:05.534861  1876 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []
W20260709 12:25:05.578927  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.649075  1879 rpcz_store.cc:269] 0709 12:25:04.013155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.015492 (+  2337us) service_pool.cc:224] Handling call
0709 12:25:05.534112 (+1518620us) spinlock_profiling.cc:243] Waited 408 ms on lock 0x7b5400073354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:05.534294 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":938657664}
I20260709 12:25:05.498574  9005 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:05.651695  2410 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:05.519606  8933 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.602s	user 0.016s	sys 0.000s
I20260709 12:25:05.494417  9045 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.653898  1801 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:05.654284  9045 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:05.655709  9025 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.964s	user 0.004s	sys 0.000s
W20260709 12:25:05.657735  9021 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.853s	user 0.000s	sys 0.002s
I20260709 12:25:05.504040  9028 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.555384  8935 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:05.659983  8935 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.661432  9028 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:05.662053  8935 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:05.118193  9033 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:05.610826  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:05.611749  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:05.499053  9030 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:05.615700  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:05.608935  8913 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:05.540297  9008 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:05.620376  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:05.620553  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:05.500851  2383 connection.cc:256] RPC call timeout handler was delayed by -1.6315356790000806s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
W20260709 12:25:05.501921  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.671939  2383 connection.cc:256] RPC call timeout handler was delayed by -1.537679842000216s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1726s for a response
W20260709 12:25:05.672238  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.672526  2383 connection.cc:256] RPC call timeout handler was delayed by -1.2526356789999227s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
I20260709 12:25:05.664191  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2729) took 2892 ms. Trace:
W20260709 12:25:05.673182  2383 connection.cc:256] RPC call timeout handler was delayed by -1.2526356789999227s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
I20260709 12:25:05.625432  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:05.619858  9050 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:05.610751  8703 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.633898  8352 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.634258  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:05.634690  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:05.611850  9011 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:05.635298  9047 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:05.635349  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:05.636540  8767 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.842s	user 0.017s	sys 0.000s
I20260709 12:25:05.668083  8765 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.488023  8928 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.618100  2413 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:05.668769  1875 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:05.619418  8753 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.669984  1881 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:05.668892  2415 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:05.678611  8933 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:05.664858  8994 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.107s	user 0.011s	sys 0.011s
I20260709 12:25:05.623454  9023 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.668396  8958 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.645407  8926 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:05.596999  8919 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:05.669265  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.705389  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.689471  8767 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:05.687481  9011 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:05.689352  9047 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:05.679540  9008 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.752065  8701 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.753023  8703 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.753384  9008 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:05.753593  9044 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:05.750728  8753 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.751353  9057 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.681594  9050 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.705389  9016 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:05.778388  9016 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.707190  9023 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.782960  9016 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.785076  9023 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:05.762765  9057 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.767014  9011 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.786909  9030 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.789309  9030 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.789319  9011 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:05.790004  9057 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.774109  9045 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:05.775602  9043 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:25:05.776753  9055 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.796110  9055 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.750265  9047 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.798622  9055 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:05.524935  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.834283  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 691200us
I20260709 12:25:05.839828  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:05.843604  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 687513us
I20260709 12:25:05.844050  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:05.845448  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:05.846081  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2738) took 1332 ms. Trace:
I20260709 12:25:05.846306  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:05.846673  1727 rpcz_store.cc:276] 0709 12:25:04.513314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.542069 (+ 28755us) service_pool.cc:224] Handling call
0709 12:25:04.542416 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.542635 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.542657 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:04.543279 (+   622us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.691300 (+148021us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:04.691336 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.691408 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:05.654962 (+963554us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:05.845239 (+190277us) raft_consensus.cc:1685] finished
0709 12:25:05.845297 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.845818 (+   521us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5106944,"thread_start_us":147612,"threads_started":1,"wal-append.queue_time_us":148059}
W20260709 12:25:05.849854  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.850103  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:05.851084  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 216273us
I20260709 12:25:05.851986  1800 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:05.673300  1873 rpcz_store.cc:276] 0709 12:25:02.771389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.779195 (+1007806us) service_pool.cc:224] Handling call
0709 12:25:03.791382 (+ 12187us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.771595 (+980213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.663735 (+892140us) spinlock_profiling.cc:243] Waited 971 ms on lock 0x7b5400074254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:05.664091 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2233137920}
I20260709 12:25:05.702378  8933 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.856302  8703 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:05.844563  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2743) took 1100 ms. Trace:
I20260709 12:25:05.867614  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:05.903367  9057 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:05.862921  8701 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.905417  8701 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.706561  8958 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.760646  8926 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:05.909793  8933 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.884270  9030 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:05.910429  9045 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:05.910758  8926 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.911633  9052 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:25:05.912815  9054 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:05.913707  9033 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:05.916180  8978 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.899757  8703 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:05.703899  8913 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.18, Last appended by leader: 18, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:05.862740  1845 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.587s (SENT) [suppressed 8 similar messages]
I20260709 12:25:05.909864  8928 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:05.932754  9028 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:05.933051  8701 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:05.933568  9028 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.934161  9028 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:05.932811  8703 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.936034  8978 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:05.939764  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:05.941090  8913 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:05.942379  8928 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:25:05.942938  8928 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.943281  8928 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:05.952234  9045 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:05.957759  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3602) took 3305 ms. Trace:
I20260709 12:25:05.958329  9057 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 12:25:05.964756  9057 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:05.965220  9057 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:05.965730  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1074) took 1552 ms. Trace:
I20260709 12:25:05.967198  9057 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.970376  9057 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:05.971417  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2579) took 2197 ms (client timeout 1587 ms). Trace:
W20260709 12:25:05.972013  1801 rpcz_store.cc:269] 0709 12:25:03.773628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.773990 (+   362us) service_pool.cc:224] Handling call
0709 12:25:05.970815 (+2196825us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b54002f2454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:05.971068 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4309168512}
I20260709 12:25:05.972422  9056 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:05.913024  1724 rpcz_store.cc:276] 0709 12:25:04.743715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.744000 (+   285us) service_pool.cc:224] Handling call
0709 12:25:04.744423 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.938495 (+194072us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.938525 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:04.965968 (+ 27443us) log.cc:844] Serialized 26 byte log entry
0709 12:25:05.470650 (+504682us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:05.470713 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.470846 (+   133us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:05.843583 (+372737us) raft_consensus.cc:1685] finished
0709 12:25:05.843674 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.844169 (+   495us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":67711,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":8282,"thread_start_us":653798,"threads_started":2,"wal-append.queue_time_us":558800}
I20260709 12:25:05.954317  9045 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:05.960255  1728 rpcz_store.cc:276] 0709 12:25:02.652322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.156525 (+504203us) service_pool.cc:224] Handling call
0709 12:25:03.156774 (+   249us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.198644 (+ 41870us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:03.678495 (+479851us) log.cc:844] Serialized 14 byte log entry
0709 12:25:03.836542 (+158047us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:03.836601 (+    59us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:03.836658 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.860847 (+1024189us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b1800161da0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419
0709 12:25:05.956904 (+1096057us) spinlock_profiling.cc:243] Waited 41.6 ms on lock 0x7b54002b6bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:05.957007 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.957394 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":898,"prepare.queue_time_us":538697,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":32,"raft.queue_time_us":204,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":106934272,"thread_start_us":636274,"threads_started":2,"wal-append.queue_time_us":214959,"wal-append.run_cpu_time_us":1579,"wal-append.run_wall_time_us":1055356}
W20260709 12:25:06.002194  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 76313.3us
I20260709 12:25:05.987510  8769 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:06.006947  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 144455us
I20260709 12:25:06.023008  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:06.023465  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:06.024353  9033 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:05.966320  1729 rpcz_store.cc:276] 0709 12:25:04.412771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.431079 (+ 18308us) service_pool.cc:224] Handling call
0709 12:25:04.431420 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.464941 (+ 33521us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:04.560653 (+ 95712us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.709795 (+149142us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.718955 (+  9160us) log.cc:844] Serialized 14 byte log entry
0709 12:25:05.964465 (+1245510us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b18004124e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287
0709 12:25:05.964718 (+   253us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:05.964841 (+   123us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:05.965011 (+   170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.965173 (+   162us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.965522 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54052,"prepare.queue_time_us":135588,"prepare.run_cpu_time_us":154,"prepare.run_wall_time_us":8112,"raft.queue_time_us":85412,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":59976,"spinlock_wait_cycles":169899520,"thread_start_us":193027,"threads_started":3,"wal-append.queue_time_us":309205}
W20260709 12:25:06.036492  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 186773us
I20260709 12:25:06.040661  1724 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:06.043552  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.044677  8759 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:06.044973  1728 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:25:06.045383  8759 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:06.046942  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.044745  1730 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:06.049247  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:06.050711  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.051388  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.053123  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:06.053714  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.054736  9060 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.055804  9060 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.058943  8978 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:06.061671  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.062891  1807 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:06.063915  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:06.064292  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.006s	sys 0.005s
W20260709 12:25:06.064718  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.007s	sys 0.005s
I20260709 12:25:06.064921  1798 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:25:06.065575  8753 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.066725  8753 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.069680  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1324) took 1091 ms. Trace:
W20260709 12:25:06.069876  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.027s	sys 0.045s
I20260709 12:25:06.070204  2418 rpcz_store.cc:276] 0709 12:25:04.977786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.979311 (+  1525us) service_pool.cc:224] Handling call
0709 12:25:04.979876 (+   565us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.108446 (+128570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.069075 (+960629us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540049eb54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:06.069435 (+   360us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":292803328}
W20260709 12:25:06.070353  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.028s	sys 0.045s
I20260709 12:25:06.025338  9033 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:06.077733  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63371.4us
I20260709 12:25:06.070180  8753 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.083341  9054 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:06.084250  9054 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:06.087285  2415 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:25:06.089653  1802 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 17. Preceding OpId from leader: term: 21 index: 18. (index mismatch)
I20260709 12:25:06.091536  9030 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:06.092566  9030 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.095036  9019 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:06.095587  9019 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:06.098572  9022 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.152s	user 0.001s	sys 0.000s
W20260709 12:25:06.100441  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.101845  9024 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.248s	user 0.000s	sys 0.002s
I20260709 12:25:06.102478  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.103669  8796 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.054855  1725 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
W20260709 12:25:06.106864  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.114048  9019 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.115659  9019 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:06.117580  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2593) took 1200 ms. Trace:
I20260709 12:25:06.118084  1800 rpcz_store.cc:276] 0709 12:25:04.917400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.936686 (+ 19286us) service_pool.cc:224] Handling call
0709 12:25:04.945782 (+  9096us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.879419 (+933637us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.117108 (+237689us) spinlock_profiling.cc:243] Waited 905 ms on lock 0x7b54001680d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:06.117477 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2082066688}
W20260709 12:25:06.119390  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 76001.9us
W20260709 12:25:06.122888  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.012444  9052 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:06.155310  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.105347  9060 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.160902  9037 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.162876  8753 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.142668  8759 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.168855  8933 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:06.170920  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.172000  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:06.115772  9062 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:06.194095  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61914.2us
I20260709 12:25:06.085906  9054 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:06.200760  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54515us
I20260709 12:25:06.106114  9033 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.198066  1730 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
W20260709 12:25:06.207039  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 173017us
I20260709 12:25:06.129359  9028 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.172111  9044 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.261354  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:06.237479  9030 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.261529  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:06.275343  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:06.278283  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.278442  1800 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:06.278837  1846 connection.cc:256] RPC call timeout handler was delayed by -1.1031714139999167s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17070000000000002s for a response
I20260709 12:25:06.280340  8400 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.281260  8400 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:06.279160  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.238615  9005 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.276100  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.283776  8759 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:06.299074  1800 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
W20260709 12:25:06.315984  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.239537  9062 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.298923  9066 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.317879  1724 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:06.324726  9005 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.326067  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2724) took 2301 ms. Trace:
I20260709 12:25:06.252200  8753 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:06.327601  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84505.8us
I20260709 12:25:06.323055  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:06.336668  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.340585  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:06.356137  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.359074  8853 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.317116  8796 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.387993  8796 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:06.393384  8935 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.730s	user 0.022s	sys 0.003s
W20260709 12:25:06.397413  8955 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.784s	user 0.010s	sys 0.010s
W20260709 12:25:06.334304  9026 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.309s	user 0.013s	sys 0.022s
W20260709 12:25:06.411132  9026 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.521s	user 0.014s	sys 0.023s
I20260709 12:25:06.411231  9052 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.414244  1874 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:25:06.416662  8769 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:06.361706  8753 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.264813  8933 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:06.370285  8853 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.419235  8928 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.420775  8853 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.371235  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:06.421926  2417 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:25:06.407182  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 86351.7us
I20260709 12:25:06.371742  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2580) took 2597 ms. Trace:
I20260709 12:25:06.422991  1806 rpcz_store.cc:276] 0709 12:25:03.773917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.333834 (+559917us) service_pool.cc:224] Handling call
0709 12:25:04.372570 (+ 38736us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.376196 (+  3626us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:04.380848 (+  4652us) log.cc:844] Serialized 14 byte log entry
0709 12:25:05.750436 (+1369588us) spinlock_profiling.cc:243] Waited 7.08 ms on lock 0x7b18004518a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:05.978519 (+228083us) spinlock_profiling.cc:243] Waited 7.09 ms on lock 0x7b18004518a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:25:06.167006 (+188487us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:06.167861 (+   855us) log.cc:844] Serialized 26 byte log entry
0709 12:25:06.168111 (+   250us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:06.168147 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.168245 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.371097 (+202852us) raft_consensus.cc:1685] finished
0709 12:25:06.371123 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.371541 (+   418us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2221,"prepare.queue_time_us":6867,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":15880,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":5250,"spinlock_wait_cycles":34003200,"thread_start_us":144360,"threads_started":1,"wal-append.queue_time_us":1943714}
I20260709 12:25:06.425316  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:06.372030  8400 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:06.372279  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68885us
I20260709 12:25:06.374833  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1321) took 1442 ms. Trace:
I20260709 12:25:06.379007  8493 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.426750  2413 rpcz_store.cc:276] 0709 12:25:04.932201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.932341 (+   140us) service_pool.cc:224] Handling call
0709 12:25:06.374375 (+1442034us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b5400573ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:06.374577 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1544059136}
I20260709 12:25:06.427922  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:06.428086  9028 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.427424  8400 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.427400  8493 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.429596  8400 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.431181  9047 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:06.431947  1726 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 18. Preceding OpId from leader: term: 21 index: 18. (term mismatch)
W20260709 12:25:06.431951  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45181.8us
I20260709 12:25:06.432816  1726 pending_rounds.cc:85] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 17
I20260709 12:25:06.433261  1726 pending_rounds.cc:107] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 20.18
I20260709 12:25:06.433578  1726 raft_consensus.cc:2889] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:06.379062  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.387136  1730 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:06.261970  9044 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:06.341665  9000 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.721s	user 0.001s	sys 0.024s
I20260709 12:25:06.287041  9045 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:06.372030  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.326550  1723 rpcz_store.cc:276] 0709 12:25:04.024186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.024419 (+   233us) service_pool.cc:224] Handling call
0709 12:25:04.026773 (+  2354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.110690 (+ 83917us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:04.111083 (+   393us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.156845 (+ 45762us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.156916 (+    71us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:04.156971 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.325445 (+2168474us) spinlock_profiling.cc:243] Waited 83.6 ms on lock 0x7b54002d19d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:06.325550 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.325853 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25050,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":12190,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":104,"spinlock_wait_cycles":192278272,"thread_start_us":33579,"threads_started":1,"wal-append.queue_time_us":44107,"wal-append.run_cpu_time_us":11678,"wal-append.run_wall_time_us":1312879}
I20260709 12:25:06.355821  1798 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:06.408895  8831 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:06.410640  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2735) took 1988 ms. Trace:
I20260709 12:25:06.328145  9005 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.368446  9066 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.383466  9062 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.314533  8978 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:06.458861  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.429103  9028 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:06.433573  8928 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.460126  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:06.466029  1807 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
W20260709 12:25:06.468433  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.440718  8493 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.493638  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:06.496865  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.447381  8769 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:25:06.465899  9000 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:06.503918  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.508284  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:06.509454  1724 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:06.518460  8796 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.519779  8400 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:06.520895  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:06.521744  2413 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:06.525209  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:06.526403  2411 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:06.527191  2414 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:25:06.528280  9066 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.528957  1802 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:06.446738  1731 rpcz_store.cc:276] 0709 12:25:04.421621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.453357 (+ 31736us) service_pool.cc:224] Handling call
0709 12:25:04.453634 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.453842 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:04.456118 (+  2276us) log.cc:844] Serialized 14 byte log entry
0709 12:25:06.355618 (+1899500us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b1800332b00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:25:06.409857 (+ 54239us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:06.409966 (+   109us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:06.410097 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.410181 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.410409 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38262,"prepare.queue_time_us":1179,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":6673,"raft.queue_time_us":39470,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":167,"spinlock_wait_cycles":10194304,"thread_start_us":50807,"threads_started":1,"wal-append.queue_time_us":57313,"wal-append.run_cpu_time_us":1212,"wal-append.run_wall_time_us":1001241}
I20260709 12:25:06.529459  9026 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:06.530452  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.531052  1873 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:06.531175  1731 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
W20260709 12:25:06.532481  8910 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.635s	user 0.005s	sys 0.041s
I20260709 12:25:06.532908  8352 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.533643  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:06.532935  8910 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.664s	user 0.005s	sys 0.041s
I20260709 12:25:06.537271  1732 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:06.540073  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.541213  9044 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.532323  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:06.548635  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 183516us
W20260709 12:25:06.553170  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 123517us
I20260709 12:25:06.484364  8955 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.497967  9062 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:06.562903  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:06.565248  2417 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:25:06.574208  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76689.3us
I20260709 12:25:06.500037  8913 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:06.583328  9037 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.583312  8913 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.584208  9037 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.587018  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:06.521168  8978 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.587598  1731 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:06.591611  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:06.496635  9045 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.560096  9062 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:06.573845  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.577013  8796 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.621064  8796 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.616067  9037 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.607105  8955 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.615252  8657 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.625223  8657 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.625576  8955 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.627864  9050 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.606007  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.628759  8657 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:06.630234  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.605911  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:06.631650  8978 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.631819  1875 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:06.634459  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:06.635186  1800 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:06.637647  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.613286  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.638631  1799 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:06.638921  1877 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:06.640009  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.640218  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.640735  1877 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:06.641134  1873 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:25:06.641755  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.641966  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:06.642401  1877 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:06.646083  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.663491  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:06.642493  1873 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:06.649986  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.652621  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1086) took 1036 ms. Trace:
I20260709 12:25:06.608170  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.644688  1878 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:25:06.650951  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2595) took 1732 ms. Trace:
I20260709 12:25:06.607461  1879 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:06.645653  1880 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:06.578557  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.647411  1802 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:06.623425  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.635228  1801 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:25:06.615906  2417 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:06.615569  8958 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:06.618044  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2751) took 2209 ms (client timeout 2647 ms). Trace:
I20260709 12:25:06.619933  2410 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:25:06.663955  1723 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:06.654055  8400 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:25:06.655411  8493 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
W20260709 12:25:06.649684  9057 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.679s	user 0.015s	sys 0.000s
I20260709 12:25:06.677959  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:06.649570  9058 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:06.677502  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.656809  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:06.682735  1800 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:06.684350  1798 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
W20260709 12:25:06.686427  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 124687us
I20260709 12:25:06.686591  9058 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:06.687134  9026 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:06.693868  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.655251  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:06.688429  1724 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:25:06.689569  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.666864  1732 rpcz_store.cc:276] 0709 12:25:05.615937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.616100 (+   163us) service_pool.cc:224] Handling call
0709 12:25:05.662589 (+ 46489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:06.537784 (+875195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.652324 (+114540us) spinlock_profiling.cc:243] Waited 874 ms on lock 0x7b54003cdcd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:06.652544 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2010272640}
W20260709 12:25:06.670921  1874 rpcz_store.cc:269] 0709 12:25:04.408704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.410091 (+  1387us) service_pool.cc:224] Handling call
0709 12:25:06.617697 (+2207606us) spinlock_profiling.cc:243] Waited 2 s on lock 0x7b540001ead4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:06.617943 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4605686144}
I20260709 12:25:06.690578  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:06.586170  8913 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.697151  1807 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:25:06.692786  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:06.700896  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:06.667410  1804 rpcz_store.cc:276] 0709 12:25:04.917966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.946421 (+ 28455us) service_pool.cc:224] Handling call
0709 12:25:04.946695 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.092472 (+145777us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.092503 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:05.092544 (+    41us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:05.092584 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.650484 (+1557900us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54001d5454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:06.650555 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.650756 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":104464,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":205,"spinlock_wait_cycles":334552320}
I20260709 12:25:06.703106  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:06.695706  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.697553  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:06.701550  1730 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:06.709446  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:06.706542  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:06.707314  2414 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
W20260709 12:25:06.662014  9046 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.048s	user 0.003s	sys 0.000s
I20260709 12:25:06.710976  1723 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:06.713445  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.717687  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.688585  9062 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:06.697826  9026 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.723675  1800 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:06.718483  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.705809  8958 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:06.725324  1799 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:06.724146  1806 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:06.658345  8978 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.058s
I20260709 12:25:06.657125  9006 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.063s
I20260709 12:25:06.734930  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.737046  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.738865  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:06.738768  9058 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.740188  9058 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:06.740778  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.741551  1877 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:25:06.743048  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.672443  8753 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 12:25:06.746501  8753 consensus_queue.cc:1243] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:25:06.749081  8854 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:06.749574  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.751148  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.742568  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:06.751955  1732 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:06.752305  1877 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:25:06.753520  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.754274  1877 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:06.754420  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:06.755199  1731 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:06.755348  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.756685  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.732235  8958 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:06.759074  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 103988us
W20260709 12:25:06.759828  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.732021  9057 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:06.761030  8955 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:06.762291  8955 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:06.658707  9055 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.065s
I20260709 12:25:06.710969  8913 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:25:06.763815  8955 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.767035  9045 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.767804  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.758196  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:06.769129  1872 raft_consensus.cc:2410] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:25:06.727526  9026 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:06.759747  8796 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:06.765143  9055 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:25:06.758204  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 180648us
I20260709 12:25:06.771497  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.771855  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:06.759158  8958 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:06.755005  8986 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:06.777181  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1028) took 2630 ms. Trace:
I20260709 12:25:06.779150  1874 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:06.779690  8823 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:06.782960  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:06.789439  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 161798us
I20260709 12:25:06.797569  1879 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:06.783737  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:06.785686  9074 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.428s	user 0.001s	sys 0.001s
I20260709 12:25:06.775808  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:06.788757  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.798592  1802 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 16: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:06.799203  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.790248  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.801712  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.791364  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:06.722606  8933 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:25:06.794708  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:06.778649  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:06.800022  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:06.802632  1872 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:06.794337  1876 rpcz_store.cc:276] 0709 12:25:04.146523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.266774 (+120251us) service_pool.cc:224] Handling call
0709 12:25:05.534558 (+1267784us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.649243 (+114685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.775136 (+1125893us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400073350. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:06.776916 (+  1780us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2914625920}
I20260709 12:25:06.810454  2417 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:25:06.810189  1877 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:06.815799  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:06.817492  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:06.815800  2414 pending_rounds.cc:85] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 9
I20260709 12:25:06.819293  2414 pending_rounds.cc:107] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 14.10
I20260709 12:25:06.819582  2414 raft_consensus.cc:2889] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:06.821668  1731 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:06.760934  8400 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.829419  1723 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:06.786883  9055 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:06.833112  1798 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:25:06.822382  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:06.836913  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58326.7us
I20260709 12:25:06.833891  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.831131  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:06.843245  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.841612  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.847769  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2594) took 1930 ms. Trace:
I20260709 12:25:06.848743  1724 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:06.850313  1724 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: [18.13-22.15]
I20260709 12:25:06.851293  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.852210  1798 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:25:06.858924  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.848282  1805 rpcz_store.cc:276] 0709 12:25:04.917672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.939376 (+ 21704us) service_pool.cc:224] Handling call
0709 12:25:04.945948 (+  6572us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.107054 (+161106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.847217 (+1740163us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:06.847569 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":367139584}
I20260709 12:25:06.846405  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.862141  1880 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:06.864564  1723 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.11-23.11]
I20260709 12:25:06.867363  1723 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 14. Current term is 15. Ops: []
I20260709 12:25:06.860363  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.870296  1878 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:06.836726  8933 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:25:06.872521  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 215546us
I20260709 12:25:06.872997  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:06.860663  8770 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:06.874693  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:06.880925  1875 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:06.883091  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:06.884611  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.885272  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.886929  9032 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 12:25:06.875015  8933 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.896819  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.890244  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:06.898260  1872 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:06.895557  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:06.894872  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.864001  8796 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:25:06.903012  8796 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:06.903077  8955 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:06.903581  8796 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:06.903950  8955 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:06.904855  8796 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.905176  8955 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.907070  9057 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:06.907387  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.907704  9094 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:06.908145  9094 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:06.908493  9094 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:06.908809  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:06.909286  9011 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.005s	sys 0.008s
I20260709 12:25:06.909921  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:06.910748  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 97460.7us
I20260709 12:25:06.911706  9055 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.912927  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:06.913233  9055 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:06.914079  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.914584  1799 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:06.914141  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.901225  1876 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:06.883869  8770 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:06.918124  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.917891  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:06.914390  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:06.923727  1806 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:06.887487  8769 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:06.887076  8400 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:06.931629  8400 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:06.935149  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.935874  1874 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 14.
I20260709 12:25:06.936779  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.938081  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:06.938315  2416 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:06.939203  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.940033  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.941198  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:06.941727  1876 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
W20260709 12:25:06.942130  9016 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.012s	sys 0.004s
W20260709 12:25:06.943029  9016 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.013s	sys 0.004s
I20260709 12:25:06.943627  9087 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:06.944617  9087 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:06.900923  8933 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:06.935258  1875 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:06.950186  1875 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:25:06.950414  9072 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.593s	user 0.001s	sys 0.000s
I20260709 12:25:06.906579  8796 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:06.925714  8823 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:06.951107  1876 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 12:25:06.957667  1873 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:06.980865  1876 pending_rounds.cc:85] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 10
I20260709 12:25:06.981169  1876 pending_rounds.cc:107] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:25:06.981952  1876 raft_consensus.cc:2889] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:06.969347  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.957581  1879 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:06.962638  1877 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:25:06.976002  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 61668.8us
I20260709 12:25:06.991346  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.926867  8770 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:06.994000  1879 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:06.996167  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:06.994184  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:06.999902  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.000293  1799 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:06.980480  8823 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.984215  8400 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:06.999729  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2596) took 1998 ms. Trace:
I20260709 12:25:06.967263  9052 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:07.000864  1802 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:07.011014  1876 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:25:06.993783  8831 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.008059  8400 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.017930  8400 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:07.017827  8831 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.019272  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:07.019538  1879 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:07.020524  8831 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:07.020812  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:07.021390  1873 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:25:06.987304  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:06.967972  9062 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:06.990680  9016 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:07.008548  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.008122  8823 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:06.907848  9057 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.027274  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.036921  1799 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 14.
I20260709 12:25:07.029708  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:07.008219  1803 rpcz_store.cc:276] 0709 12:25:04.918231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.943823 (+ 25592us) service_pool.cc:224] Handling call
0709 12:25:04.955870 (+ 12047us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:05.094541 (+138671us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.094576 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:05.096709 (+  2133us) log.cc:844] Serialized 26 byte log entry
0709 12:25:05.398003 (+301294us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:05.398058 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.405132 (+  7074us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.661020 (+1255888us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:06.881496 (+220476us) raft_consensus.cc:1685] finished
0709 12:25:06.881517 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.916974 (+ 35457us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8856,"raft.queue_time_us":84774,"raft.run_cpu_time_us":1244,"raft.run_wall_time_us":55850,"spinlock_wait_cycles":5459200,"thread_start_us":435502,"threads_started":2,"wal-append.queue_time_us":397584}
I20260709 12:25:07.014900  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:06.996506  8986 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:25:07.031980  1805 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:07.040191  1877 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:25:07.016641  9058 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:07.024370  9028 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:07.047243  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 156246us
I20260709 12:25:07.050896  8965 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:07.064455  9062 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:07.078728  9096 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:25:06.945962  9087 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.083170  1877 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:25:07.022230  8831 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:07.036491  9006 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:07.099815  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 79089.6us
W20260709 12:25:07.127398  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 61678us
I20260709 12:25:07.127799  1872 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:07.130854  1874 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:25:07.131029  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.132305  1799 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 19 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:25:07.139961  1873 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:25:07.144697  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:07.127192  9006 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.035791  9052 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.153731  1799 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:25:07.155177  8400 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.140646  8986 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:25:07.172793  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:07.173975  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.158946  2417 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:25:07.162400  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:07.139153  8993 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:07.167023  8999 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.186231  1731 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:25:07.186410  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 162193us
I20260709 12:25:07.188519  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:07.187798  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1079) took 2773 ms. Trace:
I20260709 12:25:07.160974  9006 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:07.195523  1726 rpcz_store.cc:276] 0709 12:25:04.414152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.446582 (+ 32430us) service_pool.cc:224] Handling call
0709 12:25:06.356312 (+1909730us) spinlock_profiling.cc:243] Waited 3.74 ms on lock 0x7b54002d1328. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0d7cebc 00007f16b0db208c 00007f16b0d9c93b 00007f16b0d9c3d2 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c
0709 12:25:06.380353 (+ 24041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:06.434131 (+ 53778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.185251 (+751120us) spinlock_profiling.cc:243] Waited 51.1 ms on lock 0x7b54002d14d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:07.187741 (+  2490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":126096640}
I20260709 12:25:07.177888  8997 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.206269  1730 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:07.185000  8993 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.187018  8955 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:07.216131  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.216181  1726 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:07.217185  8999 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.225299  8999 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:07.226727  9094 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.229260  9094 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:07.230095  8997 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.231099  1876 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:07.232105  9077 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:25:07.232590  9077 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.232730  8997 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:07.235894  9026 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.238361  9026 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:07.239194  1731 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:07.240666  8400 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:07.241681  8823 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.213984  8993 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:07.244338  8823 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:07.245301  1878 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:07.261577  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.262264  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:07.264072  8923 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:25:07.264567  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.267482  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:07.269114  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:07.269663  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 278415us
W20260709 12:25:07.270424  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64506.5us
I20260709 12:25:07.267217  9030 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:07.257712  9077 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:07.273819  8823 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.200742  9045 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:25:07.278502  8823 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.279150  8493 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:07.279587  8493 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:07.271879  9101 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.085s	user 0.001s	sys 0.001s
I20260709 12:25:07.286772  8933 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.289465  8933 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:07.291730  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:07.293296  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:07.294219  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:07.256059  8958 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.296104  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:07.297709  8958 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:07.301255  1873 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:07.301528  1806 raft_consensus.cc:2368] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19.
I20260709 12:25:07.304260  8400 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:07.308962  1876 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:25:07.297734  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:07.314170  1877 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:25:07.316489  1878 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:25:07.306524  8400 raft_consensus.cc:2764] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:25:07.316964  1878 pending_rounds.cc:85] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 13
I20260709 12:25:07.323772  1878 pending_rounds.cc:107] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:25:07.324183  1878 raft_consensus.cc:2889] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:07.325840  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53032.2us
W20260709 12:25:07.328383  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.264597  8923 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:07.329926  8935 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:25:07.378230  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 85044.5us
I20260709 12:25:07.347944  9062 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:25:07.381287  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.385524  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2758) took 1130 ms. Trace:
W20260709 12:25:07.389498  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 91921.7us
I20260709 12:25:07.386713  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.391602  1798 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:07.361786  8935 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:07.310645  8493 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.387656  1724 rpcz_store.cc:276] 0709 12:25:06.255403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.876687 (+621284us) service_pool.cc:224] Handling call
0709 12:25:06.884716 (+  8029us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:06.890227 (+  5511us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:07.221538 (+331311us) log.cc:844] Serialized 14 byte log entry
0709 12:25:07.377716 (+156178us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:07.377771 (+    55us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:07.385245 (+  7474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.385324 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:07.385462 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100792,"prepare.queue_time_us":197556,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":26816,"thread_start_us":333512,"threads_started":2}
I20260709 12:25:07.354092  8823 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:07.383882  9057 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:07.384106  9062 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.362649  8770 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.408161  9098 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.012s
W20260709 12:25:07.420995  9055 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.507s	user 0.012s	sys 0.000s
I20260709 12:25:07.383646  9016 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.422068  9016 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.401924  8935 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:07.426151  1846 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:07.419319  9061 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:07.433877  8493 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:07.442133  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:07.438658  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75524.6us
I20260709 12:25:07.443760  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:07.453272  9108 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:25:07.455374  8965 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:07.456581  9006 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.458366  8770 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:07.459969  9006 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:07.460988  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.463171  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.463951  1724 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:07.465062  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:07.465575  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.466898  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:07.423236  9062 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:07.412318  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.469135  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.466287  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.476322  1880 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:07.479825  1798 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:07.482990  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:07.483373  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.483687  2412 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:07.463235  9107 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:25:07.487335  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.485337  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:07.490386  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.492399  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.503352  1798 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:07.490480  9109 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:25:07.526247  1730 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: [21.11-22.12]
I20260709 12:25:07.530853  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.493988  9061 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:07.492326  8854 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:07.542059  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56826.9us
I20260709 12:25:07.547561  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:07.549664  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:07.552891  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:07.536938  9052 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:07.559059  8767 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:07.544823  8854 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.559576  8767 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:07.560017  8767 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:07.561549  8854 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:07.563071  9016 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:07.566632  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 124653us
I20260709 12:25:07.562352  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:07.567919  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.543069  8796 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:07.578244  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:07.579001  1803 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:25:07.488965  8923 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.580830  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.585526  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:07.568156  9044 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:07.586923  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.587790  1798 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:07.587456  9044 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.587082  8757 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:07.588900  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:07.590042  9044 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:07.590396  1798 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:25:07.593559  1798 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 12:25:07.599826  9061 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:07.601263  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:07.601940  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:07.603725  1798 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:07.606088  8919 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:07.602423  8757 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:07.609685  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.616216  8853 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:07.616757  8853 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:07.622004  2416 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1331) took 2002 ms (client timeout 2054 ms). Trace:
I20260709 12:25:07.622563  8757 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.624907  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1247) took 1467 ms. Trace:
I20260709 12:25:07.625367  2418 rpcz_store.cc:276] 0709 12:25:06.175023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.192168 (+ 17145us) service_pool.cc:224] Handling call
0709 12:25:07.623273 (+1431105us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:07.623678 (+   405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.623713 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:07.623763 (+    50us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:07.623801 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.623878 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:07.624710 (+   832us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54005230d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:07.624834 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3199442176}
W20260709 12:25:07.622584  2416 rpcz_store.cc:269] 0709 12:25:05.619936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.620090 (+   154us) service_pool.cc:224] Handling call
0709 12:25:07.621687 (+2001597us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54005230d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:07.621878 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2912605056}
I20260709 12:25:07.629006  1730 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:07.629334  8767 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.631479  9047 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:07.631930  8767 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:07.597128  9121 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:25:07.617954  8853 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.630584  1730 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260709 12:25:07.647559  1802 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:07.659752  1730 pending_rounds.cc:85] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 10
I20260709 12:25:07.660156  1730 pending_rounds.cc:107] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:25:07.645962  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2760) took 1313 ms. Trace:
I20260709 12:25:07.660501  1730 raft_consensus.cc:2889] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:07.636777  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.665712  9087 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.666407  9087 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.667296  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.669220  1802 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:07.669322  9087 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:07.670397  8767 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:07.671813  8703 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:07.672256  8703 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:07.672506  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:07.673103  8963 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:07.670656  1802 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:07.660813  1732 rpcz_store.cc:276] 0709 12:25:06.332262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.030698 (+698436us) service_pool.cc:224] Handling call
0709 12:25:07.031754 (+  1056us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:07.039411 (+  7657us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:07.199246 (+159835us) log.cc:844] Serialized 14 byte log entry
0709 12:25:07.383679 (+184433us) log.cc:844] Serialized 14 byte log entry
0709 12:25:07.645158 (+261479us) spinlock_profiling.cc:243] Waited 5.3 ms on lock 0x7b18000640c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:25:07.645431 (+   273us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:07.645508 (+    77us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:07.645578 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.645663 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:07.645865 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28128,"prepare.queue_time_us":122280,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":83,"raft.queue_time_us":289253,"spinlock_wait_cycles":26836736,"thread_start_us":242816,"threads_started":2,"wal-append.queue_time_us":182823}
I20260709 12:25:07.692234  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:07.695101  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 128321us
W20260709 12:25:07.698073  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 254391us
I20260709 12:25:07.707015  1798 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:07.673434  8703 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 23.11, Last appended by leader: 11, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.663981  9047 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:07.713203  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:07.690505  9044 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:07.689954  8853 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:07.693527  9033 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:07.732051  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:07.732452  8703 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:07.733767  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:07.734987  8493 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:07.735558  9061 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:25:07.740726  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:07.741334  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.721606  8767 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.020s
W20260709 12:25:07.759454  9123 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.111s	user 0.000s	sys 0.001s
I20260709 12:25:07.762511  8769 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:07.763162  8769 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:07.768975  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2745) took 1599 ms. Trace:
I20260709 12:25:07.775538  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2760) took 1137 ms. Trace:
I20260709 12:25:07.776178  1874 rpcz_store.cc:276] 0709 12:25:06.638361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.967224 (+328863us) service_pool.cc:224] Handling call
0709 12:25:06.990952 (+ 23728us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.137003 (+146051us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.775104 (+638101us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b540021b1d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:07.775413 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":321030400}
I20260709 12:25:07.779501  9047 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.725325  9044 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.783419  9044 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:07.769438  1725 rpcz_store.cc:276] 0709 12:25:06.169771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.192939 (+ 23168us) service_pool.cc:224] Handling call
0709 12:25:06.247687 (+ 54748us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:06.543687 (+296000us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:25:06.543724 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:06.544353 (+   629us) log.cc:844] Serialized 26 byte log entry
0709 12:25:06.786951 (+242598us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:06.786996 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.768553 (+981557us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b54001bf3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:07.768606 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:07.768651 (+    45us) raft_consensus.cc:1685] finished
0709 12:25:07.768674 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:07.768909 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51779,"raft.queue_time_us":58157,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":18230,"spinlock_wait_cycles":522438400,"thread_start_us":242105,"threads_started":1,"wal-append.queue_time_us":306907}
W20260709 12:25:07.803324  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.808131  1803 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:25:07.823581  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.824626  1803 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
W20260709 12:25:07.832831  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 119697us
I20260709 12:25:07.834102  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:07.720468  8796 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.105s
I20260709 12:25:07.841392  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:07.725507  8993 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.832304  8913 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:07.852331  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:07.797644  9016 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.045s
W20260709 12:25:07.854139  9075 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.490s	user 0.004s	sys 0.000s
I20260709 12:25:07.854837  8935 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.857349  8935 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:07.859529  8993 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:07.862923  8400 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:07.863765  9006 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:07.864599  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:07.865000  9006 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:07.865372  9006 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:07.865926  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:07.854372  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:07.877039  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.874066  2412 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:07.892258  1731 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:07.893673  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:07.894378  1727 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:07.895061  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.866259  9006 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.904994  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:07.850761  8913 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.853029  9052 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:07.902642  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:07.880888  9077 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.603s	user 0.014s	sys 0.000s
I20260709 12:25:07.905355  1732 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:07.900502  1802 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260709 12:25:07.898037  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.907109  1803 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:07.838229  9061 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.049s
I20260709 12:25:07.912801  2412 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:07.918570  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.918572  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:07.920804  2413 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:07.922060  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 217954us
I20260709 12:25:07.924404  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:07.925390  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:07.942499  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:07.940089  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:07.911639  8913 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:07.945461  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.946102  1731 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:07.914628  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:07.916333  1802 pending_rounds.cc:85] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 14
I20260709 12:25:07.884634  9033 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:07.960081  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:07.962292  9033 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:07.963852  9033 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.10, Last appended by leader: 10, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:07.951299  1802 pending_rounds.cc:107] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260709 12:25:07.966033  1802 raft_consensus.cc:2889] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:07.952211  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.945269  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:07.953234  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:07.953445  8935 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:25:07.954146  9087 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:07.977380  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.951010  9098 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:07.879716  8989 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.090s
I20260709 12:25:07.954365  2414 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:07.978590  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:07.950593  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:07.980934  1726 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:07.960168  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:07.996632  2413 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:07.912328  9052 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.003253  1727 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
W20260709 12:25:07.998920  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84354.2us
I20260709 12:25:07.947409  8933 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.008219  8493 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:08.016824  2419 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:25:08.019176  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:08.020599  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:08.013897  2414 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 11. (term mismatch)
I20260709 12:25:08.023500  2414 pending_rounds.cc:85] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 10
I20260709 12:25:08.023741  2414 pending_rounds.cc:107] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:25:08.023980  2414 raft_consensus.cc:2889] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:08.017903  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:08.025444  1725 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:08.013859  2410 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:07.979943  9087 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.019865  1727 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:07.916327  9066 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.093s
W20260709 12:25:08.028113  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 190332us
I20260709 12:25:08.028842  9029 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:08.029739  9026 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:25:08.031431  9127 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:25:08.014880  8493 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:08.032737  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.041585  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:07.924669  9077 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:07.999833  9064 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:07.947230  8770 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.068s
I20260709 12:25:08.004776  9006 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:08.006300  9052 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:08.056288  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:08.101040  1732 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:08.044413  9087 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:08.103716  2412 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:08.045418  8958 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:08.149655  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:08.092041  9064 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.021947  8933 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:08.047605  8963 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.104s
I20260709 12:25:08.145084  9135 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:08.189429  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 238924us
I20260709 12:25:08.155490  8958 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:08.198624  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.194880  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 138103us
I20260709 12:25:08.157394  9064 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:08.212451  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 252422us
I20260709 12:25:08.170342  9026 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:25:08.202808  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 247179us
W20260709 12:25:08.253352  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.260993  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 255292us
I20260709 12:25:08.154309  9133 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:08.215687  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:08.257730  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.264345  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:08.265374  1729 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:08.271667  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.277813  1729 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:08.164099  9134 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:25:08.296057  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80507.2us
I20260709 12:25:08.170338  9028 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.090s
I20260709 12:25:08.301489  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:08.305903  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:08.306833  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:08.307461  1803 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:08.257167  9047 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.259506  9026 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:08.309546  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:08.256768  8935 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:08.284808  9128 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.391s	user 0.001s	sys 0.001s
I20260709 12:25:08.313120  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:08.173790  8493 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:08.316933  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:08.202358  9135 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:08.299636  9062 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.815s	user 0.015s	sys 0.000s
I20260709 12:25:08.260830  8770 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 12:25:08.313583  8400 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:08.331238  9066 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.018s
I20260709 12:25:08.329948  8993 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:08.351626  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:08.366282  1803 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
W20260709 12:25:08.386070  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.343673  8400 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.352622  9098 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:08.352974  8989 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:08.319945  9047 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.351693  8928 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20260709 12:25:08.365157  8703 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:08.405390  9141 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:08.406018  8703 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:08.407429  9129 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.408s	user 0.000s	sys 0.002s
I20260709 12:25:08.407910  8933 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:08.408911  9028 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:08.409637  9028 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:08.415868  9138 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.201s	user 0.000s	sys 0.001s
I20260709 12:25:08.421506  9047 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:08.336238  9062 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:08.423669  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66707.3us
W20260709 12:25:08.426517  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 163730us
W20260709 12:25:08.437855  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.451928  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.397218  8400 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:08.444309  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:08.363867  8923 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.046s
I20260709 12:25:08.459266  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:08.404477  1725 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:08.466900  1725 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:08.469020  1732 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:08.478955  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:08.479965  1800 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:08.481299  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:08.404335  9077 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:08.355422  9144 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.093s
I20260709 12:25:08.407476  8770 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:08.522753  1874 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:25:08.494412  9006 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:08.515044  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:08.353467  8958 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:08.542534  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 225690us
W20260709 12:25:08.550889  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 120383us
I20260709 12:25:08.552024  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:08.556567  9137 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
W20260709 12:25:08.558439  9044 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.775s	user 0.009s	sys 0.004s
W20260709 12:25:08.558924  9136 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.298s	user 0.002s	sys 0.000s
W20260709 12:25:08.560367  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69024.9us
I20260709 12:25:08.552630  1800 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:08.561884  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:08.564085  9066 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:08.564747  8965 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:08.564877  9066 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:08.563285  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:08.460780  9141 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.113s
I20260709 12:25:08.540863  9006 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.548218  9149 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.576037  9029 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:08.597437  1845 leader_election.cc:400] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19.
I20260709 12:25:08.575366  9144 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:08.534422  9142 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 12:25:08.601171  1845 leader_election.cc:403] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:08.488706  8935 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.584594  8919 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:08.553346  8770 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.589287  9006 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:08.617288  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.016s	sys 0.006s
W20260709 12:25:08.617693  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.016s	sys 0.006s
I20260709 12:25:08.566507  9066 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.626078  9134 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:08.568526  9044 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:08.568806  8703 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.614652  8965 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:25:08.648522  1802 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:08.652089  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:08.652168  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 175407us
I20260709 12:25:08.654188  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:08.594127  9153 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:25:08.653558  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:08.635797  9135 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:08.600860  9149 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.666347  1874 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:08.605001  9161 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:25:08.657012  9162 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:08.613340  9144 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.667649  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:08.638981  8919 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:08.672430  9146 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:25:08.675755  8749 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:08.610477  9098 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.597939  9026 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.683758  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.668855  8539 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:08.667022  9028 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:08.682350  2417 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:08.685261  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:08.686609  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.691846  9044 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:08.692453  8769 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.929s	user 0.008s	sys 0.005s
I20260709 12:25:08.702579  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.637771  9077 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
W20260709 12:25:08.678757  9125 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.846s	user 0.003s	sys 0.000s
W20260709 12:25:08.684999  9130 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.950s	user 0.006s	sys 0.000s
I20260709 12:25:08.792387  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2617) took 1598 ms. Trace:
I20260709 12:25:08.792886  1807 rpcz_store.cc:276] 0709 12:25:07.193741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.612866 (+419125us) service_pool.cc:224] Handling call
0709 12:25:07.613371 (+   505us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.643472 (+ 30101us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.643517 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:07.644137 (+   620us) log.cc:844] Serialized 26 byte log entry
0709 12:25:07.681616 (+ 37479us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:25:07.681665 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.681746 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:07.780152 (+ 98406us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b48001c0158. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:25:08.684244 (+904092us) spinlock_profiling.cc:243] Waited 7.69 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:08.792128 (+107884us) raft_consensus.cc:1685] finished
0709 12:25:08.792164 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.792326 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15569,"raft.queue_time_us":39094,"raft.run_cpu_time_us":3401,"raft.run_wall_time_us":115729,"spinlock_wait_cycles":42200320,"thread_start_us":37035,"threads_started":1,"wal-append.queue_time_us":90597}
W20260709 12:25:08.772730  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.680220  9156 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.775921  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.835304  1873 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:08.765089  8986 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.779489  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:08.781956  9155 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.129s	user 0.001s	sys 0.001s
I20260709 12:25:08.679308  9149 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:08.784883  8657 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.786726  1881 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:08.788147  1728 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:08.797195  9077 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:08.799512  8923 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:08.799512  9158 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:08.801637  9142 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:08.803566  9175 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:25:08.803421  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.817787  9033 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:08.737686  9135 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:08.808074  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:08.808333  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:08.808126  1730 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:08.808629  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:08.765443  1872 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:08.810496  2419 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:08.926544  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:08.775146  8769 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.924886  2411 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:08.891027  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:08.708024  8989 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.685710  9062 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:08.836848  2414 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
W20260709 12:25:08.840828  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 187372us
I20260709 12:25:08.852867  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.932395  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:08.933018  1873 raft_consensus.cc:2376] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23. Re-sending same reply.
I20260709 12:25:08.933646  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:08.934198  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.935184  9058 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:08.935268  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.935972  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:08.936074  1873 raft_consensus.cc:2376] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19. Re-sending same reply.
I20260709 12:25:08.742292  8770 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:08.774116  9161 consensus_queue.cc:1243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:25:08.773681  9057 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:08.878350  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55897.7us
W20260709 12:25:08.884598  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 228507us
I20260709 12:25:08.944696  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:08.766546  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:08.886703  1879 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
W20260709 12:25:08.895721  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.949007  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:08.950959  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:08.951853  1807 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 15: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:08.933485  9045 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:08.898006  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:08.903128  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.913313  8829 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.918267  9154 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:08.852221  9005 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.954074  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.981316  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:08.982231  1872 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:25:08.852520  8749 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.859918  1874 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:08.853750  9077 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:08.942404  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:08.935866  9058 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:08.703203  8919 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:08.954685  1726 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:25:08.955525  2418 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:08.969503  1798 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:08.939460  9016 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:09.005652  9016 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.005746  1800 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:09.006140  9016 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:09.006469  9179 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.944595  8989 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.007095  9179 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.008864  8989 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:09.009275  9179 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.009830  9027 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.060s
W20260709 12:25:08.985522  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49606.6us
I20260709 12:25:09.011365  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:08.885279  9158 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.012768  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:09.014060  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:08.872686  8657 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:09.035259  8913 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.066s	user 0.017s	sys 0.000s
I20260709 12:25:09.035787  8913 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.036108  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:09.036757  9133 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.740s	user 0.008s	sys 0.003s
I20260709 12:25:09.036981  8657 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:09.037231  9133 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:08.953065  9174 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:09.038004  9174 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:09.038662  9133 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:09.039369  9174 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:08.991528  9052 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.839s	user 0.010s	sys 0.004s
I20260709 12:25:09.043157  9052 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.043910  9161 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:09.044450  1725 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:25:09.046075  9052 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:08.983026  8770 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:09.047381  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:09.047472  8770 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.048087  1725 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:09.049360  1725 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:09.049685  8770 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:08.679260  9134 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.995652  1873 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:25:09.050709  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:09.051092  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.051443  1725 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:08.992388  1802 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:25:09.051936  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:09.003613  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:08.852253  8986 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:08.854566  9156 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.014086  8400 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.015098  9181 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:08.994352  8749 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.002193  9058 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:08.993860  9005 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:08.980952  8829 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.022615  9144 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.029167  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.029919  8965 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 18. Current term is 19.
I20260709 12:25:09.030604  9061 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.888921  9142 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:08.988399  9154 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:09.031740  9087 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.881s	user 0.012s	sys 0.000s
I20260709 12:25:08.937819  9135 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.033493  8769 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.032711  9077 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:09.034524  9064 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.773s	user 0.008s	sys 0.004s
I20260709 12:25:08.937650  9006 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.038097  9171 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:09.005093  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.010602  9027 consensus_queue.cc:1243] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 20.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:25:09.055137  1726 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:09.053634  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.053092  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:09.048010  1727 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:09.060114  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:09.064200  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:09.050035  9016 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.071084  2413 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:09.072155  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.145519  1872 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:25:09.068434  2417 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:09.070041  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:09.067423  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 138516us
I20260709 12:25:09.073529  1876 raft_consensus.cc:2376] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17. Re-sending same reply.
I20260709 12:25:09.060635  9047 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:09.067719  9052 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.067s
I20260709 12:25:09.067955  9158 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:09.147447  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:09.057461  8770 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:09.108749  1880 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:09.102281  9181 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:09.140511  1874 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:09.147226  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:09.147851  1725 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:09.223109  1800 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:09.224061  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:09.148326  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 88281us
W20260709 12:25:09.150847  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 86737.7us
W20260709 12:25:09.068136  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 152231us
W20260709 12:25:09.240025  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.051930  8958 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.047s
W20260709 12:25:09.152465  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 99471.3us
I20260709 12:25:09.154592  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:09.125767  8989 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:09.253605  1878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.016s	sys 0.010s
I20260709 12:25:09.219151  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:09.255003  1878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.016s	sys 0.010s
I20260709 12:25:09.255669  1879 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:09.220112  1877 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:25:09.247951  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.248769  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:09.259930  1806 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:09.126685  8400 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:09.127720  8829 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:09.217818  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.261456  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.109408  9087 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.147006  9176 raft_consensus.cc:2764] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:25:09.153332  9178 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:09.140791  9064 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.289325  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.227149  8703 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:09.267589  9085 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:09.153223  9047 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.261348  8749 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:09.302969  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:09.332620  2416 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:09.124850  9061 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.249717  9158 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:09.224390  8913 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.153329  9161 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:09.263207  9077 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:09.317509  1800 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:09.324126  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.270404  9195 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:09.341516  1873 raft_consensus.cc:2368] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
W20260709 12:25:09.342712  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.267871  8989 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.303049  9029 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:09.348659  1875 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
W20260709 12:25:09.341799  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3490) took 2798 ms (client timeout 2156 ms). Trace:
W20260709 12:25:09.361653  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.361770  1872 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:09.288583  8829 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:25:09.383750  8829 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:09.384189  8829 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:09.385064  8829 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.366487  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:09.386215  1872 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:09.387516  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:09.387869  9193 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:25:09.388295  1800 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:09.389223  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3652) took 1107 ms. Trace:
I20260709 12:25:09.389830  1731 rpcz_store.cc:276] 0709 12:25:08.282005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.914372 (+632367us) service_pool.cc:224] Handling call
0709 12:25:08.914824 (+   452us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:08.932929 (+ 18105us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.932976 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:08.933656 (+   680us) log.cc:844] Serialized 52 byte log entry
0709 12:25:09.151482 (+217826us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:09.151531 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.151598 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.388801 (+237203us) raft_consensus.cc:1685] finished
0709 12:25:09.388924 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.389156 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17612,"raft.queue_time_us":18065,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":215932,"threads_started":1,"wal-append.queue_time_us":392424}
I20260709 12:25:09.387521  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.392818  1872 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:09.393738  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.394325  1872 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:09.318042  9197 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:09.395586  9197 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:09.396970  9197 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.406433  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:09.333042  9047 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:09.318529  9085 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:09.364982  8919 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.368851  8986 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:09.369978  8989 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:09.371928  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:09.372702  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:09.373169  9029 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:09.364550  1881 rpcz_store.cc:269] 0709 12:25:06.543199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.568582 (+ 25383us) service_pool.cc:224] Handling call
0709 12:25:09.341575 (+2772993us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b5400117454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:09.341729 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4845907328}
I20260709 12:25:09.375572  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:09.332175  8928 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:09.381345  9016 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.392082  9064 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:09.407513  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2782) took 1132 ms. Trace:
I20260709 12:25:09.516716  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:09.517002  1732 rpcz_store.cc:276] 0709 12:25:08.275168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.470373 (+195205us) service_pool.cc:224] Handling call
0709 12:25:08.496242 (+ 25869us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:08.878603 (+382361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.878649 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.879220 (+   571us) log.cc:844] Serialized 26 byte log entry
0709 12:25:09.149312 (+270092us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:09.149369 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.149468 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.407238 (+257770us) raft_consensus.cc:1685] finished
0709 12:25:09.407268 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.407423 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72435,"raft.queue_time_us":190089,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":17742,"thread_start_us":559271,"threads_started":2,"wal-append.queue_time_us":469095}
W20260709 12:25:09.410409  9128 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.016s	user 0.005s	sys 0.012s
I20260709 12:25:09.412444  9154 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:09.523202  9029 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.412998  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.524775  9029 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:09.415611  9152 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.825s	user 0.003s	sys 0.000s
I20260709 12:25:09.416639  9006 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:09.417316  1872 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:09.419628  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:09.527972  2410 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:09.389267  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:09.254743  8770 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.313520  8703 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:09.425716  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71369.1us
I20260709 12:25:09.428112  9061 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:09.429668  9135 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.435283  9087 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.339609  9158 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.442502  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1430) took 1561 ms. Trace:
I20260709 12:25:09.533480  9158 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:09.360260  9195 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:09.458016  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:09.453377  9198 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
W20260709 12:25:09.466094  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.539148  9195 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:09.472312  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 356490us
I20260709 12:25:09.545261  8770 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:09.546636  8703 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:09.539386  2418 rpcz_store.cc:276] 0709 12:25:07.881558 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.881732 (+   174us) service_pool.cc:224] Handling call
0709 12:25:07.882079 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.002900 (+1120821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.442139 (+439239us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540046a354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:09.442409 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2468152704}
I20260709 12:25:09.531629  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.548986  1881 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:09.549984  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:09.474174  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 319622us
I20260709 12:25:09.504536  9142 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:09.519522  2419 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:09.520716  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:09.521754  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:09.523779  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3647) took 1574 ms. Trace:
I20260709 12:25:09.414459  9181 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.489427  9156 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.531652  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.531886  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.532563  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:09.463107  8829 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:09.558380  1727 rpcz_store.cc:276] 0709 12:25:07.949066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.101359 (+152293us) service_pool.cc:224] Handling call
0709 12:25:09.523511 (+1422152us) spinlock_profiling.cc:243] Waited 897 ms on lock 0x7b54003cd2d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:09.523678 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2062523136}
I20260709 12:25:09.565377  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:09.566530  1872 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:09.569033  1878 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:09.568224  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:09.573994  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3638) took 2112 ms. Trace:
I20260709 12:25:09.572248  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2615) took 2558 ms. Trace:
I20260709 12:25:09.574613  1724 rpcz_store.cc:276] 0709 12:25:07.461398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.503338 (+ 41940us) service_pool.cc:224] Handling call
0709 12:25:07.503715 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.919244 (+415529us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:08.045172 (+125928us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.330231 (+285059us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.333533 (+  3302us) log.cc:844] Serialized 26 byte log entry
0709 12:25:08.333752 (+   219us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:08.819133 (+485381us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.860616 (+ 41483us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.860698 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:08.929733 (+ 69035us) spinlock_profiling.cc:243] Waited 9.96 ms on lock 0x7b4800041c58. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:25:09.415033 (+485300us) spinlock_profiling.cc:243] Waited 3.78 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:09.573670 (+158637us) raft_consensus.cc:1685] finished
0709 12:25:09.573706 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.573903 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":294587,"prepare.queue_time_us":140875,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":115,"raft.queue_time_us":313533,"raft.run_cpu_time_us":3272,"raft.run_wall_time_us":548019,"spinlock_wait_cycles":31588352,"thread_start_us":790693,"threads_started":3,"wal-append.queue_time_us":391883}
W20260709 12:25:09.559974  9134 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.055s	sys 0.052s
I20260709 12:25:09.514183  8919 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.575326  1805 rpcz_store.cc:276] 0709 12:25:07.014085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.605263 (+591178us) service_pool.cc:224] Handling call
0709 12:25:07.605804 (+   541us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.614675 (+  8871us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.614709 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:07.618474 (+  3765us) log.cc:844] Serialized 26 byte log entry
0709 12:25:07.703077 (+ 84603us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:07.725545 (+ 22468us) log.cc:844] Serialized 14 byte log entry
0709 12:25:07.725936 (+   391us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.726039 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.410010 (+1683971us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b24002b79a8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d2a09d 00007f16ab95c10e 00007f16ab97388c 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221
0709 12:25:09.565581 (+155571us) raft_consensus.cc:1685] finished
0709 12:25:09.572030 (+  6449us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.572192 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11754,"prepare.queue_time_us":22112,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":26744,"raft.run_cpu_time_us":332,"raft.run_wall_time_us":80712,"spinlock_wait_cycles":4913920,"thread_start_us":84171,"threads_started":1,"wal-append.queue_time_us":181788}
I20260709 12:25:09.609318  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:09.593279  9134 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.055s	sys 0.053s
I20260709 12:25:09.559660  8769 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:09.620502  8769 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.622946  8919 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.624240  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:09.626108  1727 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:25:09.576301  9150 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:09.627533  9150 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:09.628544  9150 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.593885  9085 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.639411  9085 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:09.640446  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:09.641127  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.641757  8703 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.644433  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.646453  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.649565  1874 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:25:09.652274  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.654325  1798 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 11. Preceding OpId from leader: term: 24 index: 11. (term mismatch)
I20260709 12:25:09.654724  1798 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 10
I20260709 12:25:09.654968  1798 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 23.11
I20260709 12:25:09.655208  1798 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:09.655282  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.628463  8919 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:25:09.655098  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.653272  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.657598  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.658186  1873 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:09.658504  1876 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:09.658762  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:09.653264  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.661791  8769 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:09.665247  1727 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:09.667233  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.667498  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.667999  1872 raft_consensus.cc:2376] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19. Re-sending same reply.
I20260709 12:25:09.611908  9142 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:09.669839  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.670485  9142 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.671391  1804 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:25:09.673430  9142 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:09.674261  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.675038  1873 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:09.675781  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:09.670935  1872 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:09.677057  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:09.677805  1873 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:09.680085  2410 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:25:09.680382  1874 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:25:09.693292  1876 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:09.694680  1872 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:09.695093  1800 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 18 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:25:09.696791  8703 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:09.697496  8703 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:09.702669  9066 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:09.704157  9026 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:09.717759  1877 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 12:25:09.718469  1877 pending_rounds.cc:85] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 11
I20260709 12:25:09.718773  1877 pending_rounds.cc:107] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:25:09.727494  1877 raft_consensus.cc:2889] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:09.729847  1877 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:09.738279  1873 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:09.726881  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1336) took 1039 ms. Trace:
I20260709 12:25:09.744671  2416 rpcz_store.cc:276] 0709 12:25:08.702199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.725739 (+ 23540us) service_pool.cc:224] Handling call
0709 12:25:09.726633 (+1000894us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54005f28d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:09.726818 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":866710272}
I20260709 12:25:09.725036  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:09.736141  9178 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:09.733351  1800 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
W20260709 12:25:09.740299  9045 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.751s	user 0.013s	sys 0.001s
I20260709 12:25:09.746968  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:09.767570  1726 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:09.754631  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:09.768774  1806 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:09.757673  1872 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:25:09.762549  9178 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:09.771586  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:09.774497  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 206371us
W20260709 12:25:09.775079  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.765107  8493 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:09.779441  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.738286  9183 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:25:09.795506  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 136801us
I20260709 12:25:09.796573  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:09.798341  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 122754us
I20260709 12:25:09.803928  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:09.804508  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:09.805866  2417 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:09.811312  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:09.817644  2410 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:25:09.812925  1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3642) took 2317 ms (client timeout 1846 ms). Trace:
W20260709 12:25:09.876662  1728 rpcz_store.cc:269] 0709 12:25:07.477701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.485515 (+  7814us) service_pool.cc:224] Handling call
0709 12:25:09.795072 (+2309557us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54002b66d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:09.795200 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2989816704}
I20260709 12:25:09.877929  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:09.811236  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:09.803876  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:09.819063  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:09.776991  9045 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:09.820712  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:09.832093  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:09.791499  8657 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:09.833956  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:09.834823  1805 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:25:09.885063  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 73937.1us
I20260709 12:25:09.808923  9061 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.921053  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2618) took 2720 ms. Trace:
I20260709 12:25:09.791889  9183 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:09.921788  1801 rpcz_store.cc:276] 0709 12:25:07.200286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.662970 (+462684us) service_pool.cc:224] Handling call
0709 12:25:07.663443 (+   473us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.665678 (+  2235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.665713 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:07.666386 (+   673us) log.cc:844] Serialized 26 byte log entry
0709 12:25:07.880597 (+214211us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:07.904695 (+ 24098us) log.cc:844] Serialized 14 byte log entry
0709 12:25:07.922993 (+ 18298us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.923081 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:08.787539 (+864458us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b180038ff20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:09.918123 (+1130584us) spinlock_profiling.cc:243] Waited 28.7 ms on lock 0x7b180038ff20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:09.920619 (+  2496us) raft_consensus.cc:1685] finished
0709 12:25:09.920659 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.920925 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11064,"prepare.queue_time_us":16407,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":3419,"raft.queue_time_us":46069,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"spinlock_wait_cycles":158229632,"thread_start_us":213763,"threads_started":1,"wal-append.queue_time_us":301371}
I20260709 12:25:09.923243  9183 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.924801  9183 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:09.925282  9134 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:09.815353  9033 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.926590  9134 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.927407  9033 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.928009  9134 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:09.816074  9159 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:09.931154  9176 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:09.934043  9176 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:25:09.935746  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 103715us
I20260709 12:25:09.937194  8958 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:09.830596  1726 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:09.843475  9087 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.848409  8400 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.824746  9133 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:09.825064  9016 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:09.825309  9157 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:09.794018  1872 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:25:09.851368  9195 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:09.851850  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.852118  9205 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.813207  8770 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.852897  8854 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:09.833400  9053 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.848475  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:09.866117  8986 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:25:09.880220  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 99401.5us
I20260709 12:25:09.881121  2416 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:09.884272  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 73452.7us
I20260709 12:25:10.113420  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:09.839564  1806 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:09.902961  9172 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:09.904631  9161 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.542s	user 0.006s	sys 0.004s
I20260709 12:25:09.910732  9026 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.124576  1881 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:09.915884  1732 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:09.917353  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:10.125571  2410 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:09.813922  9145 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:25:09.956817  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:09.986039  8400 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:09.986451  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:09.815033  9135 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:25:09.988909  9005 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:09.991447  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 170890us
I20260709 12:25:10.135574  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:09.991834  8958 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:09.881114  9158 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.921232  9061 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:09.882210  9027 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.993240  8657 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:09.995146  9033 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:10.000860  9058 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.923s	user 0.010s	sys 0.000s
I20260709 12:25:10.002655  9087 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.003150  2382 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:25:10.167827  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 164722us
I20260709 12:25:10.021541  1872 raft_consensus.cc:606] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:10.169075  1872 raft_consensus.cc:614] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:09.998536  1730 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
W20260709 12:25:10.074867  2411 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s	user 0.033s	sys 0.074s
W20260709 12:25:10.170866  2411 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.034s	sys 0.074s
W20260709 12:25:10.075172  1731 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.653s	user 0.008s	sys 0.022s
W20260709 12:25:10.172310  1731 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.012s	sys 0.022s
I20260709 12:25:10.041536  9157 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:10.082862  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.115099  2413 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.010s	sys 0.057s
W20260709 12:25:10.173720  2413 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.010s	sys 0.057s
I20260709 12:25:10.072292  8770 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.119029  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:10.123108  1697 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:10.019582  8853 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:10.181565  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.041834  9133 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:10.124581  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.005s	sys 0.010s
I20260709 12:25:10.137573  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:10.137054  1807 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.003s	sys 0.019s
W20260709 12:25:10.134754  9154 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.009s	sys 0.007s
I20260709 12:25:10.005255  9057 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.119s
I20260709 12:25:10.155794  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.127947  9045 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:10.128434  9161 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:10.041258  9016 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.259480  1725 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:10.243224  8657 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.254279  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3650) took 2304 ms. Trace:
I20260709 12:25:10.262972  9178 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.263621  1723 rpcz_store.cc:276] 0709 12:25:07.949909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.442127 (+492218us) service_pool.cc:224] Handling call
0709 12:25:08.442567 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:08.445666 (+  3099us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:08.445714 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:08.462930 (+ 17216us) log.cc:844] Serialized 26 byte log entry
0709 12:25:08.839238 (+376308us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:08.839289 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.839369 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.253439 (+1414070us) spinlock_profiling.cc:243] Waited 50.5 ms on lock 0x7b54003613a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4ddbe 00007f16abd58930 00007f16abd588d8 00007f16abd58858 00007f16abd58813 00007f16abd57a9c 00007f16b1c432f6 00007f16b1c3daa8 00007f16abd947cb 00007f16abd972c0
0709 12:25:10.253774 (+   335us) raft_consensus.cc:1685] finished
0709 12:25:10.253812 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.254184 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":168013,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":16951,"spinlock_wait_cycles":116231552,"thread_start_us":210142,"threads_started":1,"wal-append.queue_time_us":260342}
I20260709 12:25:10.265226  9178 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.180102  9064 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.266140  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.266593  9064 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.268631  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2781) took 1206 ms. Trace:
I20260709 12:25:10.242751  9005 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:10.269105  1881 rpcz_store.cc:276] 0709 12:25:09.062306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.549675 (+487369us) service_pool.cc:224] Handling call
0709 12:25:10.268392 (+718717us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b5400260a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:10.268564 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1312604416}
I20260709 12:25:10.270246  9005 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.272409  1874 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:10.273464  9005 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:10.177950  9058 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.277134  9058 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.268946  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.279989  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:10.281104  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.281678  1723 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:10.283342  2415 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:10.127574  9053 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.284368  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2784) took 2008 ms. Trace:
I20260709 12:25:10.284837  2415 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:25:10.269053  9064 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:10.285755  9053 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:10.164494  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:10.134896  8757 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.003s	sys 0.013s
I20260709 12:25:10.171747  1876 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:10.135912  9056 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:10.176614  8928 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:10.138957  8958 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:10.290213  8757 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.003s	sys 0.013s
I20260709 12:25:10.144197  8829 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:25:10.185592  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 62505.5us
I20260709 12:25:10.188974  9135 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:25:10.187871  1807 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.003s	sys 0.019s
W20260709 12:25:10.139684  1802 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.000s	sys 0.026s
W20260709 12:25:10.261400  9212 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.187s	user 0.003s	sys 0.003s
I20260709 12:25:10.190372  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:10.286700  2415 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:10.191479  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:10.073027  9205 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:10.201604  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.208640  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.166931  9027 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:10.304092  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.017s	sys 0.016s
I20260709 12:25:10.304461  2418 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 20 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
W20260709 12:25:10.199029  9154 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.012s	sys 0.010s
I20260709 12:25:10.243643  9195 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:10.274420  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3551) took 1235 ms. Trace:
I20260709 12:25:10.242669  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.316478  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.320207  1726 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:25:10.320700  1806 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:10.321466  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.297024  8958 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.262257  9157 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.319901  1874 rpcz_store.cc:276] 0709 12:25:09.039237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.705686 (+666449us) service_pool.cc:224] Handling call
0709 12:25:10.272199 (+566513us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.272865 (+   666us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.274202 (+  1337us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b5400260a50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:10.274349 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1301947776}
I20260709 12:25:10.289597  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:10.179076  9197 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.000s	sys 0.018s
I20260709 12:25:10.291627  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:10.118257  2412 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.006s	sys 0.009s
I20260709 12:25:10.244867  9004 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:10.295176  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:10.187518  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.005s	sys 0.011s
I20260709 12:25:10.176091  9033 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:10.367687  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 72660.7us
I20260709 12:25:10.195912  8853 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:10.298172  1802 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.953s	user 0.000s	sys 0.026s
W20260709 12:25:10.297969  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 465405us
I20260709 12:25:10.297981  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2630) took 1344 ms. Trace:
I20260709 12:25:10.384241  1807 rpcz_store.cc:276] 0709 12:25:08.953724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.958821 (+  5097us) service_pool.cc:224] Handling call
0709 12:25:08.992450 (+ 33629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.992744 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:25:09.287183 (+294439us) log.cc:844] Serialized 14 byte log entry
0709 12:25:10.297632 (+1010449us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:10.297691 (+    59us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:10.297741 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.297803 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.297926 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":389607,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":46,"raft.queue_time_us":50272,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":1184393,"threads_started":3,"wal-append.queue_time_us":540431}
W20260709 12:25:10.251667  9077 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.884s	user 0.011s	sys 0.000s
I20260709 12:25:10.260780  9133 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:10.260144  9226 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:25:10.395241  9004 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.396714  1879 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:10.264122  9026 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.242223  8770 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.302201  1872 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
I20260709 12:25:10.254806  9158 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:10.304633  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.017s	sys 0.016s
I20260709 12:25:10.301510  2417 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
W20260709 12:25:10.303272  1804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.002s	sys 0.025s
I20260709 12:25:10.246129  9087 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:10.252833  8400 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:10.284919  1729 rpcz_store.cc:276] 0709 12:25:08.275862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.631052 (+355190us) service_pool.cc:224] Handling call
0709 12:25:09.795220 (+1164168us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.795622 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.795664 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:09.795733 (+    69us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:09.795796 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.795866 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.284121 (+488255us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54002b66d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:10.284287 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2636479872}
W20260709 12:25:10.411008  1804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.002s	sys 0.025s
I20260709 12:25:10.274252  9005 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:25:10.300316  9047 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.786s	user 0.011s	sys 0.000s
I20260709 12:25:10.282485  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.328591  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.329284  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:10.413992  1726 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:10.414660  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.310446  8928 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:10.301399  9181 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.015s	sys 0.018s
I20260709 12:25:10.262013  8769 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.053s
W20260709 12:25:10.350551  2412 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.824s	user 0.006s	sys 0.010s
W20260709 12:25:10.362195  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 788799us
I20260709 12:25:10.362883  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.363157  1728 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 12:25:10.423337  1728 pending_rounds.cc:85] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 10
I20260709 12:25:10.423651  1728 pending_rounds.cc:107] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260709 12:25:10.423949  1728 raft_consensus.cc:2889] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:10.245332  9061 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:10.296799  9056 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:10.189678  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52219us
I20260709 12:25:10.372443  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:10.427014  1881 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:10.427152  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:10.368521  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:10.373462  9135 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:10.438849  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.439463  9213 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:10.440496  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:10.445134  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3658) took 1395 ms. Trace:
I20260709 12:25:10.450716  1727 rpcz_store.cc:276] 0709 12:25:09.049534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.666279 (+616745us) service_pool.cc:224] Handling call
0709 12:25:09.666562 (+   283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.682731 (+ 16169us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.682776 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:09.682869 (+    93us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:09.682930 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.444848 (+761918us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b54002a9254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:10.444916 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.445068 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36617088}
I20260709 12:25:10.397982  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2675) took 1018 ms. Trace:
I20260709 12:25:10.397318  9033 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.401298  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:10.362074  9197 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.001s	sys 0.019s
I20260709 12:25:10.454820  1798 rpcz_store.cc:276] 0709 12:25:09.379965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.656266 (+276301us) service_pool.cc:224] Handling call
0709 12:25:09.656576 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.657037 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.657065 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.657540 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.076186 (+418646us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:10.076244 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.076351 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.397653 (+321302us) raft_consensus.cc:1685] finished
0709 12:25:10.397683 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.397894 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":59449,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"spinlock_wait_cycles":1716608,"thread_start_us":322914,"threads_started":1,"wal-append.queue_time_us":388927}
I20260709 12:25:10.405197  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.463411  1872 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:10.311296  8757 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.094s
I20260709 12:25:10.417495  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.417743  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:10.431236  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.431679  9016 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.403769  9026 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:10.421934  9181 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.016s	sys 0.019s
I20260709 12:25:10.432998  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3659) took 1010 ms. Trace:
I20260709 12:25:10.414886  9077 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 12:25:10.435299  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1113) took 1553 ms. Trace:
I20260709 12:25:10.436684  9056 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.437518  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:10.378661  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.482564  1806 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:25:10.447438  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2762) took 3805 ms. Trace:
I20260709 12:25:10.397318  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.446214  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1267) took 1300 ms. Trace:
W20260709 12:25:10.448680  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 80238.2us
I20260709 12:25:10.426088  8958 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:10.453722  1805 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:25:10.453598  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3557) took 1406 ms (client timeout 1701 ms). Trace:
I20260709 12:25:10.483040  1875 rpcz_store.cc:276] 0709 12:25:06.641579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.980115 (+338536us) service_pool.cc:224] Handling call
0709 12:25:09.342194 (+2362079us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.364180 (+ 21986us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.447259 (+1083079us) spinlock_profiling.cc:243] Waited 2.35 s on lock 0x7b5400117450. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:10.447386 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5399652736}
W20260709 12:25:10.399559  1877 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.000s	sys 0.050s
W20260709 12:25:10.495368  1877 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.000s	sys 0.050s
I20260709 12:25:10.427858  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:10.451869  9161 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:10.455792  9135 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:10.456210  9183 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:10.417052  9047 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:10.463842  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62653us
W20260709 12:25:10.510667  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 534428us
W20260709 12:25:10.469682  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.406270  8493 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:25:10.471875  1726 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:10.463414  2414 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.012s	sys 0.048s
I20260709 12:25:10.474622  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:10.475677  1732 rpcz_store.cc:276] 0709 12:25:09.422456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.518908 (+ 96452us) service_pool.cc:224] Handling call
0709 12:25:10.432802 (+913894us) spinlock_profiling.cc:243] Waited 357 ms on lock 0x7b54002aa654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:10.432911 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":822019456}
I20260709 12:25:10.434545  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:10.467625  1807 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:25:10.455011  9004 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:10.480031  1725 rpcz_store.cc:276] 0709 12:25:08.881950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.322134 (+440184us) service_pool.cc:224] Handling call
0709 12:25:09.346296 (+ 24162us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.263117 (+916821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.435007 (+171890us) spinlock_profiling.cc:243] Waited 913 ms on lock 0x7b54000d2ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:10.435245 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2099419520}
I20260709 12:25:10.486848  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:10.493331  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:10.493611  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:10.531618  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44853us
W20260709 12:25:10.501447  1800 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.013s	sys 0.013s
W20260709 12:25:10.512024  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84309.3us
I20260709 12:25:10.532914  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:10.512981  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:10.533852  2416 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:10.471201  8400 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:10.534585  1872 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:10.535043  1800 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.013s	sys 0.013s
I20260709 12:25:10.516741  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:10.519321  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.538156  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:10.538276  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.520370  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:10.539224  1875 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:10.464421  1873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.009s	sys 0.021s
W20260709 12:25:10.526033  2414 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.012s	sys 0.048s
I20260709 12:25:10.523869  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:10.531847  2417 raft_consensus.cc:2410] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 16 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:25:10.483964  2411 rpcz_store.cc:276] 0709 12:25:09.145706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.145849 (+   143us) service_pool.cc:224] Handling call
0709 12:25:09.430900 (+285051us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.431444 (+   544us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.431473 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.431978 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.171383 (+739405us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:10.171441 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.171530 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.260800 (+ 89270us) spinlock_profiling.cc:243] Waited 43.2 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:10.396202 (+135402us) raft_consensus.cc:1685] finished
0709 12:25:10.396241 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.446042 (+ 49801us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b540048a3d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:10.446154 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3349,"raft.run_cpu_time_us":54,"raft.run_wall_time_us":52,"spinlock_wait_cycles":588398336,"thread_start_us":738931,"threads_started":1,"wal-append.queue_time_us":642260}
W20260709 12:25:10.494663  1879 rpcz_store.cc:269] 0709 12:25:09.040739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.256425 (+215686us) service_pool.cc:224] Handling call
0709 12:25:10.446998 (+1190573us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400117e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:10.447100 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2522304384}
I20260709 12:25:10.540773  2410 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
W20260709 12:25:10.541384  1873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.705s	user 0.009s	sys 0.021s
I20260709 12:25:10.536784  1874 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:10.543819  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:10.545001  2419 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:10.545282  1799 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:10.547360  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:10.547813  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.548345  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:10.550287  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:10.553803  8928 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.557646  9172 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:10.558202  9172 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:10.558935  9157 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:10.559407  9157 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:10.559825  9157 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:10.561854  9217 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:10.562966  8961 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:10.537165  8400 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:10.563855  9214 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:25:10.564113  8400 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:10.565143  8829 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:10.565485  8961 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:10.543555  8703 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:10.566133  8961 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:10.565619  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1270) took 1339 ms. Trace:
I20260709 12:25:10.568531  2412 rpcz_store.cc:276] 0709 12:25:09.225839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.504083 (+278244us) service_pool.cc:224] Handling call
0709 12:25:09.513558 (+  9475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.513841 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.513881 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.525838 (+ 11957us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.421408 (+895570us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:10.421466 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.421544 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.565337 (+143793us) raft_consensus.cc:1685] finished
0709 12:25:10.565365 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.565544 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":895132,"threads_started":1,"wal-append.queue_time_us":837856}
I20260709 12:25:10.546514  8853 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:10.571252  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.563045  9217 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:25:10.573262  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:10.574016  1805 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:10.574463  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:10.577713  9216 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
W20260709 12:25:10.579138  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1035) took 1915 ms (client timeout 1675 ms). Trace:
W20260709 12:25:10.579574  1880 rpcz_store.cc:269] 0709 12:25:08.663515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.663854 (+   339us) service_pool.cc:224] Handling call
0709 12:25:10.578968 (+1915114us) spinlock_profiling.cc:243] Waited 369 ms on lock 0x7b54001f7a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:10.579069 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":848807808}
I20260709 12:25:10.580715  9047 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.581022  1880 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:25:10.582576  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.584677  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:10.514410  9135 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.585435  9223 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:10.586229  9223 raft_consensus.cc:2764] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:25:10.586326  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3550) took 1547 ms. Trace:
I20260709 12:25:10.510150  9197 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.085s
I20260709 12:25:10.586707  1876 rpcz_store.cc:276] 0709 12:25:09.038964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.704662 (+665698us) service_pool.cc:224] Handling call
0709 12:25:09.705137 (+   475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.294795 (+589658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.586096 (+291301us) spinlock_profiling.cc:243] Waited 466 ms on lock 0x7b540021c5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:10.586268 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1072376576}
I20260709 12:25:10.518074  9183 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.589568  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:10.582469  1880 raft_consensus.cc:1217] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Deduplicated request from leader. Original: 18.15->[19.16-21.17]   Dedup: 19.16->[21.17-21.17]
I20260709 12:25:10.595424  1880 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [19.16-21.17]
I20260709 12:25:10.596829  1880 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:25:10.598230  1875 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:10.527052  9045 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.598999  9217 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:10.600421  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:10.601259  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.602007  1806 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:10.603155  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.603349  1872 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 24 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
W20260709 12:25:10.604187  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.606484  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:10.607168  1798 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:10.608291  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.639895  1806 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:10.641311  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:10.540210  9211 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.114s	user 0.001s	sys 0.001s
I20260709 12:25:10.620153  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:10.635687  1881 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
W20260709 12:25:10.636003  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.647790  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94953.5us
I20260709 12:25:10.518321  9215 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:10.614640  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64420.3us
I20260709 12:25:10.654533  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:10.594420  8400 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.636794  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.649389  1878 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:10.546586  9061 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:10.656451  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:10.616945  9064 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:25:10.653748  9134 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.724s	user 0.011s	sys 0.004s
I20260709 12:25:10.690608  9134 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.691675  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1268) took 1545 ms. Trace:
I20260709 12:25:10.692314  2413 rpcz_store.cc:276] 0709 12:25:09.146060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.428271 (+282211us) service_pool.cc:224] Handling call
0709 12:25:09.428817 (+   546us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.429356 (+   539us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.429400 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.430100 (+   700us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.174196 (+744096us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:10.174249 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.174346 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.442523 (+268177us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:10.691331 (+248808us) raft_consensus.cc:1685] finished
0709 12:25:10.691355 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.691605 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5166,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":123,"spinlock_wait_cycles":8547200,"thread_start_us":743548,"threads_started":1,"wal-append.queue_time_us":888306}
I20260709 12:25:10.693573  9134 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.694480  1730 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:10.621111  8919 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:10.697921  8919 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.700580  8919 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:10.686040  1807 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:10.702030  9026 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:10.708595  8400 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:10.710012  1879 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:10.688705  1628 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 5 to 13, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 13 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:10.689790  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2672) took 1321 ms. Trace:
I20260709 12:25:10.691773  9205 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:10.623363  9223 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:10.695302  9033 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.673300  9217 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:10.672039  8989 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:10.642463  1806 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:10.714368  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:10.732163  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59211.1us
I20260709 12:25:10.732587  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.716034  1798 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
W20260709 12:25:10.656921  9006 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.022s	sys 0.030s
I20260709 12:25:10.724334  1799 rpcz_store.cc:276] 0709 12:25:09.368027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.371627 (+  3600us) service_pool.cc:224] Handling call
0709 12:25:10.689432 (+1317805us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400132754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:10.689661 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2398395776}
I20260709 12:25:10.739576  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.727578  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.741101  1878 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:10.743827  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:10.744202  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.745499  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.747169  1875 raft_consensus.cc:2376] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22. Re-sending same reply.
I20260709 12:25:10.748662  1881 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 15: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:10.730161  8989 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.790639  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:10.750861  1872 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:10.804018  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:10.805457  1806 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:10.793954  1805 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:10.813190  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:10.813316  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:10.849387  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2792) took 1056 ms. Trace:
I20260709 12:25:10.726105  9077 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:10.816425  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:10.852049  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.817185  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:10.751353  8749 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.015s	sys 0.028s
I20260709 12:25:10.855358  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.856070  1875 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
W20260709 12:25:10.856745  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.010s	sys 0.023s
I20260709 12:25:10.820734  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:10.857137  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.010s	sys 0.023s
I20260709 12:25:10.730497  9217 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:10.825845  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.803383  8989 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:10.833791  9135 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.833974  9027 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.835638  9215 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:10.836741  9223 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:25:10.866087  8749 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.865s	user 0.016s	sys 0.028s
I20260709 12:25:10.839347  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2784) took 1776 ms. Trace:
I20260709 12:25:10.840437  9019 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:10.750411  9033 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:10.845018  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2676) took 1464 ms. Trace:
I20260709 12:25:10.845641  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:10.847410  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.849839  1879 rpcz_store.cc:276] 0709 12:25:09.792540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.572365 (+779825us) service_pool.cc:224] Handling call
0709 12:25:10.572839 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.710484 (+137645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.849127 (+138643us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400222754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:10.849325 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":314982784}
W20260709 12:25:10.751205  9006 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.023s	sys 0.031s
I20260709 12:25:10.859513  1805 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:10.861618  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:10.939383  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.940701  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.863541  1727 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:10.880481  1847 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.003s	sys 0.032s
W20260709 12:25:10.941907  1847 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.003s	sys 0.032s
W20260709 12:25:10.880048  1731 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.030s	sys 0.005s
W20260709 12:25:10.944345  1731 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.756s	user 0.030s	sys 0.005s
I20260709 12:25:10.903752  1724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6388b1a5243d463589750794e8ac5511"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:10.945452  1724 raft_consensus.cc:606] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:25:10.945904  1724 raft_consensus.cc:614] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:10.908908  1725 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:10.906548  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.837843  8919 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:10.912287  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.919492  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.916739  1877 rpcz_store.cc:276] 0709 12:25:09.063075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.737688 (+674613us) service_pool.cc:224] Handling call
0709 12:25:09.738144 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.786264 (+ 48120us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.786388 (+   124us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.795120 (+  8732us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.495738 (+700618us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:10.495779 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.495852 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.838548 (+342696us) raft_consensus.cc:1685] finished
0709 12:25:10.838575 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.839246 (+   671us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":111300,"raft.queue_time_us":48181,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":18233,"thread_start_us":651708,"threads_started":1,"wal-append.queue_time_us":1010078}
I20260709 12:25:10.866968  9217 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:10.924710  1804 rpcz_store.cc:276] 0709 12:25:09.380404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.673526 (+293122us) service_pool.cc:224] Handling call
0709 12:25:09.674017 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.674318 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.674360 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.676115 (+  1755us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.411568 (+735453us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:10.411627 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.411719 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.844764 (+433045us) raft_consensus.cc:1685] finished
0709 12:25:10.844787 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.844945 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":734926,"threads_started":1,"wal-append.queue_time_us":1166714}
I20260709 12:25:10.936699  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:10.948764  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.949146  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.949947  1723 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:10.889497  9077 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:10.952754  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:10.958557  1807 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:10.906148  9045 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:10.955323  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1116) took 2019 ms. Trace:
I20260709 12:25:10.904891  9224 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.960808  1731 rpcz_store.cc:276] 0709 12:25:08.935841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.392286 (+456445us) service_pool.cc:224] Handling call
0709 12:25:09.392854 (+   568us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:09.393412 (+   558us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.393444 (+    32us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:09.394382 (+   938us) log.cc:844] Serialized 78 byte log entry
0709 12:25:10.172844 (+778462us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:10.944835 (+771991us) log.cc:844] Serialized 14 byte log entry
0709 12:25:10.954569 (+  9734us) log.cc:844] Serialized 14 byte log entry
0709 12:25:10.954870 (+   301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.954989 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.955045 (+    56us) raft_consensus.cc:1685] finished
0709 12:25:10.955063 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.955252 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4763,"raft.queue_time_us":7128,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":4972,"thread_start_us":1533948,"threads_started":2,"wal-append.queue_time_us":876370}
I20260709 12:25:10.963948  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:10.918200  8400 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.965022  1798 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 23 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 15.
I20260709 12:25:10.968289  9158 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.968860  9158 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.970916  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1437) took 1825 ms. Trace:
I20260709 12:25:10.971158  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:10.971973  9223 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:10.972681  9223 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.973891  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:10.971494  2414 rpcz_store.cc:276] 0709 12:25:09.145054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.150521 (+  5467us) service_pool.cc:224] Handling call
0709 12:25:09.249517 (+ 98996us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.266608 (+ 17091us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.266643 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.888721 (+622078us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b2c00090838. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:25:09.889358 (+   637us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.540630 (+651272us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:10.540679 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.540769 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.970445 (+429676us) raft_consensus.cc:1685] finished
0709 12:25:10.970471 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.970830 (+   359us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1366,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":4411,"spinlock_wait_cycles":4849408,"thread_start_us":637308,"threads_started":1,"wal-append.queue_time_us":1006722}
I20260709 12:25:10.977341  9223 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:10.977511  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.978969  9232 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.973243  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:10.979568  9232 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.979408  9077 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:10.981688  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2785) took 1918 ms. Trace:
I20260709 12:25:10.974805  1725 raft_consensus.cc:2410] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:10.984941  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:10.985730  2414 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:10.986796  9158 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:10.987984  9223 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:10.988806  9005 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:10.987612  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51004.3us
I20260709 12:25:10.956315  9157 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.990214  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:10.972577  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:10.991593  9005 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:10.992512  9157 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:10.992884  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:10.993811  2419 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:10.994455  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:10.995352  8749 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.970943  9154 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:10.997272  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:10.997761  8400 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:10.923557  9033 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:11.007786  1800 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:25:10.982184  1873 rpcz_store.cc:276] 0709 12:25:09.063301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.741044 (+677743us) service_pool.cc:224] Handling call
0709 12:25:09.741403 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.786306 (+ 44903us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:09.786352 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.786926 (+   574us) log.cc:844] Serialized 26 byte log entry
0709 12:25:10.541997 (+755071us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:10.542074 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:10.542176 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.981395 (+439219us) raft_consensus.cc:1685] finished
0709 12:25:10.981433 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.981618 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103832,"raft.queue_time_us":49677,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":17992,"thread_start_us":705639,"threads_started":1,"wal-append.queue_time_us":1113470}
I20260709 12:25:11.021319  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:10.956775  8657 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.014s	sys 0.017s
I20260709 12:25:11.024750  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:11.026259  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:11.026626  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:11.035641  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:11.037165  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:11.042378  1802 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.004s	sys 0.011s
I20260709 12:25:10.955842  8958 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:10.993857  9005 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:11.000864  8400 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:11.003852  9033 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:11.042879  1802 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s	user 0.004s	sys 0.011s
I20260709 12:25:11.044478  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:11.068145  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 222586us
I20260709 12:25:11.068670  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1030) took 1753 ms. Trace:
I20260709 12:25:11.068574  1875 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:11.053601  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:11.076957  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:11.051740  9210 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:25:11.072369  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:10.960850  9057 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.011s	sys 0.018s
W20260709 12:25:11.113095  9057 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.854s	user 0.011s	sys 0.018s
I20260709 12:25:11.113163  9172 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:11.044835  9053 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.758s	user 0.000s	sys 0.015s
W20260709 12:25:11.114570  9053 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.000s	sys 0.016s
I20260709 12:25:10.982249  9232 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:11.003993  9230 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.077s
I20260709 12:25:11.081210  1879 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:11.035916  8657 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.015s	sys 0.017s
I20260709 12:25:11.076220  8493 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:11.067413  9004 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:11.076323  8919 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:11.040771  9006 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:25:11.003536  9045 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.095659  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:11.055502  9005 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.097234  8400 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:11.052006  9231 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:25:11.100425  8961 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.534s	user 0.009s	sys 0.007s
I20260709 12:25:11.104777  1875 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:11.109822  1723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:11.160676  1723 raft_consensus.cc:606] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:11.161108  1723 raft_consensus.cc:614] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:11.111455  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:11.053301  8958 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:11.124048  9016 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.004s
I20260709 12:25:10.970607  9224 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.163323  8961 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.167328  8958 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.168816  8958 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:11.171953  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:11.175271  9033 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.176824  1728 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:11.177875  9045 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:11.178611  9033 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:11.160895  8829 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:11.179805  1723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "908089442ca44109875257bfbc9d06c2"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:11.180562  1723 raft_consensus.cc:606] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:11.157059  9179 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.880s	user 0.008s	sys 0.015s
W20260709 12:25:11.182749  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71420.6us
W20260709 12:25:11.070322  9145 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.909s	user 0.007s	sys 0.008s
I20260709 12:25:11.077945  8703 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.046451  9197 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:25:11.115389  9150 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:11.114753  9172 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:11.069154  1802 rpcz_store.cc:276] 0709 12:25:09.315554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.315747 (+   193us) service_pool.cc:224] Handling call
0709 12:25:09.319716 (+  3969us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.319961 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:10.383480 (+1063519us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.068320 (+684840us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.068375 (+    55us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:11.068425 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.068483 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.068617 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":62681,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":63,"thread_start_us":1723083,"threads_started":2}
W20260709 12:25:11.162060  1770 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.740s (SENT) [suppressed 33 similar messages]
I20260709 12:25:11.133785  8493 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.159319  9005 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:11.187022  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:11.181339  9179 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.008s	sys 0.015s
I20260709 12:25:11.206228  9150 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:11.207615  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:11.228734  1731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "123095127f334445889c21a42ba1d41e"
 from {username='slave'} at 127.0.0.1:40200
W20260709 12:25:11.193997  9145 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.046s	user 0.008s	sys 0.009s
I20260709 12:25:11.218856  8493 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:11.223534  1802 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
W20260709 12:25:11.192584  9133 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.000s	sys 0.027s
I20260709 12:25:11.194389  8961 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:11.229708  1731 raft_consensus.cc:606] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Received request to transfer leadership to TS 123095127f334445889c21a42ba1d41e
I20260709 12:25:11.192266  9223 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:11.157150  8919 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:11.233511  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.029s	sys 0.031s
W20260709 12:25:11.233696  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49464.9us
W20260709 12:25:11.238765  1874 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.011s	sys 0.014s
W20260709 12:25:11.240521  1874 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.011s	sys 0.014s
W20260709 12:25:11.241914  1801 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.021s	sys 0.023s
W20260709 12:25:11.244539  1801 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.022s	sys 0.024s
I20260709 12:25:11.159216  8989 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:11.231774  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.018s	sys 0.012s
W20260709 12:25:11.240006  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.029s	sys 0.031s
W20260709 12:25:11.245366  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.018s	sys 0.013s
W20260709 12:25:11.224424  2385 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.008s	sys 0.014s
W20260709 12:25:11.245903  2385 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.008s	sys 0.014s
W20260709 12:25:11.245783  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 607932us
I20260709 12:25:11.246966  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:11.249384  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:11.235745  9133 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.839s	user 0.000s	sys 0.028s
I20260709 12:25:11.205729  8703 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.239496  9223 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:11.243752  8928 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.071s	sys 0.059s
I20260709 12:25:11.256785  9176 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.257074  9033 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.284698  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:11.274523  1807 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:25:11.291395  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:11.282346  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:11.296101  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.296367  1807 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:25:11.298271  1799 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: [24.11-24.11]
W20260709 12:25:11.281419  8928 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.697s	user 0.072s	sys 0.059s
W20260709 12:25:11.306180  1880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.006s	sys 0.018s
W20260709 12:25:11.306979  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 457192us
W20260709 12:25:11.322111  1880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.007s	sys 0.018s
I20260709 12:25:11.275555  9223 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:11.259557  8400 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:25:11.289933  8986 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:11.321496  1804 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:25:11.316753  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:11.328387  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:11.343475  1876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.008s	sys 0.008s
I20260709 12:25:11.378266  1727 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:11.378551  1876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.009s	sys 0.010s
I20260709 12:25:11.289562  9057 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:25:11.306741  9047 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.015s	sys 0.011s
I20260709 12:25:11.300863  9045 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.388736  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.386585  8986 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.392064  8928 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:11.393652  9061 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:11.399231  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.300518  9133 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.302390  9004 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:11.300731  9052 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.404944  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2795) took 1474 ms. Trace:
I20260709 12:25:11.304764  9154 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:11.300994  9134 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:11.403955  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:11.403085  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 79096.1us
I20260709 12:25:11.291308  9176 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.416972  2414 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:25:11.410055  9045 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.418198  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:11.410970  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:11.405874  9158 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:11.411110  9135 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:25:11.327852  9217 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.297050  9033 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.456038  9133 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.531917  9133 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:11.417649  9061 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:11.473631  9052 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.475718  9134 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.541443  9134 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:11.417817  9224 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:11.547755  1725 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:25:11.548722  1800 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:11.495157  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 206217us
I20260709 12:25:11.387303  9232 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:11.479744  1727 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:11.554001  2415 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:25:11.554392  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:11.480234  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:11.555833  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 75692.3us
I20260709 12:25:11.559504  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:11.562655  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 151964us
I20260709 12:25:11.563253  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:11.555194  1727 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:11.456534  1876 rpcz_store.cc:276] 0709 12:25:09.930076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.587798 (+657722us) service_pool.cc:224] Handling call
0709 12:25:10.588097 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.588298 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:10.588547 (+   249us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.403878 (+815331us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.404526 (+   648us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:11.404611 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.404712 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.404873 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22755,"prepare.queue_time_us":530,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":26387,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":48237,"thread_start_us":791812,"threads_started":1}
I20260709 12:25:11.565521  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:11.563665  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:11.476449  9241 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:25:11.563999  1872 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
W20260709 12:25:11.482295  9238 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.180s	user 0.000s	sys 0.000s
I20260709 12:25:11.571184  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:11.572005  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:11.598527  9205 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:11.600600  1876 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:11.568481  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.603150  9150 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.604408  9150 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:11.606629  9052 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:11.607280  9052 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:11.607847  9052 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.610133  9052 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:11.605369  9005 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.612032  9183 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:11.612700  9183 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:25:11.518576  8703 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:11.524570  9217 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:11.526854  9033 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:11.617426  9005 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:11.522372  9135 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:11.567287  9057 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:11.592815  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:11.562386  9232 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.572736  9004 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:11.567426  9236 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:11.493037  8986 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:11.595391  1628 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:11.485946  9045 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:11.596843  9134 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:11.597858  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:11.482193  9176 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:11.618594  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:11.643535  1725 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:11.620008  1727 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:11.610692  9052 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:11.645310  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:11.646075  1727 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:11.647272  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:11.623895  8853 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.022s	sys 0.024s
W20260709 12:25:11.630316  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 241688us
I20260709 12:25:11.635083  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.636041  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:11.638900  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.653931  1805 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:11.640027  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.654412  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.655313  1807 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:11.654289  1804 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
W20260709 12:25:11.637264  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 475208us
W20260709 12:25:11.651167  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53400.1us
I20260709 12:25:11.658218  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:11.659196  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:11.660538  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:11.662264  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:11.668874  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:11.670574  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.632488  9232 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:11.679710  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:11.668972  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2644) took 1158 ms. Trace:
I20260709 12:25:11.686172  1801 rpcz_store.cc:276] 0709 12:25:10.511104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.511263 (+   159us) service_pool.cc:224] Handling call
0709 12:25:10.537541 (+ 26278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.537790 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:25:11.245508 (+707718us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.668474 (+422966us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.668559 (+    85us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:11.668633 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.668728 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.668895 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96231,"prepare.queue_time_us":109897,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":3264,"raft.queue_time_us":27533,"thread_start_us":991144,"threads_started":2}
W20260709 12:25:11.635897  9077 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.040s	sys 0.029s
I20260709 12:25:11.702606  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.704097  1805 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:25:11.667093  8853 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.023s	sys 0.025s
I20260709 12:25:11.702641  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:11.715636  9205 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:11.718116  8400 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:11.673235  9087 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:11.720429  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:11.721263  1876 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:11.722826  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2791) took 1922 ms. Trace:
I20260709 12:25:11.723333  1874 rpcz_store.cc:276] 0709 12:25:09.792272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.543514 (+751242us) service_pool.cc:224] Handling call
0709 12:25:10.552469 (+  8955us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:10.552854 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.552892 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:10.553451 (+   559us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.241662 (+688211us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:11.669659 (+427997us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.714491 (+ 44832us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.714595 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.714643 (+    48us) raft_consensus.cc:1685] finished
0709 12:25:11.714657 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.714828 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44512,"raft.queue_time_us":60292,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":214,"thread_start_us":1094491,"threads_started":2,"wal-append.queue_time_us":852692}
I20260709 12:25:11.726646  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:11.727388  1876 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
W20260709 12:25:11.731564  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.733460  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.733737  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:11.668303  8986 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.685292  8928 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:11.673214  9158 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:11.701894  9005 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:11.734206  1801 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
W20260709 12:25:11.744361  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 151615us
I20260709 12:25:11.765982  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.767316  9145 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:11.767942  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.741905  1725 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:11.770089  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:11.771333  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:11.771992  1725 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
W20260709 12:25:11.773255  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.035s	sys 0.017s
W20260709 12:25:11.773643  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.035s	sys 0.017s
I20260709 12:25:11.686026  9134 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:11.775878  9134 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.681658  9236 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:11.706413  9077 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.040s	sys 0.030s
I20260709 12:25:11.784044  8958 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:11.784090  1881 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:11.785907  1771 leader_election.cc:400] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
I20260709 12:25:11.786372  1771 leader_election.cc:403] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:11.787900  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.704550  9217 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:11.789456  9217 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:11.789963  9217 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:11.734903  8769 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:11.796772  8769 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:11.797920  8769 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.799252  9220 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:11.800925  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.801767  9220 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:11.761550  9135 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.803184  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.781348  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.804487  9255 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:11.805858  9220 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:11.806623  8928 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:11.807646  8928 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.788265  9145 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:11.788717  8958 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
I20260709 12:25:11.745168  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:11.816057  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:11.817198  8958 raft_consensus.cc:2764] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:25:11.817773  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.818681  8958 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:11.804939  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:11.673182  9251 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 12:25:11.821938  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.820062  9217 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.823192  9145 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:11.824208  9145 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:11.824736  9252 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:25:11.824973  9255 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:11.824724  9145 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:11.826746  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:11.828915  9251 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:11.829479  9251 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.829867  9251 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:11.746299  9248 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:11.836598  9248 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.839934  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:11.841171  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.842695  9217 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:11.843593  9247 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:25:11.749193  9158 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:11.844588  9158 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:11.850029  2382 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:25:11.802520  8493 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.561s	user 0.004s	sys 0.003s
W20260709 12:25:11.858901  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 76169.6us
I20260709 12:25:11.862282  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:11.833890  9223 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.846908  9251 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.859799  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:11.765038  9005 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.873953  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:11.864462  9255 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.867627  9237 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:11.748184  8986 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.883322  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:11.883911  2417 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:11.897553  1879 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:25:11.885541  1878 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:25:11.897655  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:11.886358  1798 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:11.900456  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:11.900677  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 297727us
I20260709 12:25:11.901998  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:11.903155  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:11.872994  9027 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:11.863919  9220 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:11.887291  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 464457us
W20260709 12:25:11.874538  1806 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.024s	sys 0.041s
I20260709 12:25:11.881031  1628 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 10 to 14, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 14 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:25:11.894235  1726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.017s	sys 0.014s
I20260709 12:25:11.886209  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.872628  9230 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:11.906111  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46413.8us
I20260709 12:25:11.923014  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.906585  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.907383  1877 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:11.886488  2416 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.014s	sys 0.013s
W20260709 12:25:11.925025  1806 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.025s	sys 0.041s
W20260709 12:25:11.925827  2416 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s	user 0.014s	sys 0.013s
I20260709 12:25:11.748945  9057 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.876240  9005 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:11.942215  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:11.921229  1726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s	user 0.017s	sys 0.016s
I20260709 12:25:11.944607  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:11.946484  1879 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:11.916986  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:11.867020  8703 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:25:11.867213  9217 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:25:11.866544  9197 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:25:11.869270  9228 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:25:11.872056  8958 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:25:11.943490  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:11.897969  9237 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.865140  8493 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.947176  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:11.921948  1723 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.016s	sys 0.032s
I20260709 12:25:11.947824  1798 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
W20260709 12:25:11.990700  1723 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.017s	sys 0.032s
I20260709 12:25:11.940217  9179 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:11.993530  1798 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:25:11.948581  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:11.949432  2410 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.012s	sys 0.017s
W20260709 12:25:11.997859  2410 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.013s	sys 0.017s
I20260709 12:25:11.955224  1872 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
I20260709 12:25:11.956561  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.999629  1805 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:11.958194  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:11.914403  1880 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.011s	sys 0.013s
I20260709 12:25:11.930546  8749 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:11.985399  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:11.985793  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:12.005937  1801 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:11.867148  9004 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 22.10, Last appended by leader: 10, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:11.997134  1798 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:12.000751  1880 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.012s	sys 0.013s
I20260709 12:25:12.009418  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2797) took 2071 ms. Trace:
I20260709 12:25:12.010113  1880 rpcz_store.cc:276] 0709 12:25:09.938515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.597746 (+659231us) service_pool.cc:224] Handling call
0709 12:25:10.598076 (+   330us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:10.598814 (+   738us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.598848 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:10.599578 (+   730us) log.cc:844] Serialized 52 byte log entry
0709 12:25:11.324140 (+724562us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:12.008500 (+684360us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.008923 (+   423us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.009042 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.009104 (+    62us) raft_consensus.cc:1685] finished
0709 12:25:12.009141 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.009344 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":659343,"raft.queue_time_us":100,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":91,"thread_start_us":1343156,"threads_started":2,"wal-append.queue_time_us":887245}
I20260709 12:25:11.907023  9255 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:11.955677  9052 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:25:11.927953  9250 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:12.044042  1727 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:11.990970  9237 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:11.989693  8958 consensus_queue.cc:1243] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:25:11.944932  8853 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.081s
I20260709 12:25:12.031618  8749 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:11.994021  9220 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:11.939747  9230 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:12.012399  9064 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:12.099388  2415 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:12.003301  9179 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:12.095340  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:12.057089  9172 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.840s	user 0.011s	sys 0.003s
W20260709 12:25:12.059549  9061 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.526s	user 0.013s	sys 0.000s
I20260709 12:25:12.154320  9179 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.099985  9033 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 12:25:12.179929  1726 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
W20260709 12:25:12.153308  9254 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:25:12.058627  9236 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.127s
I20260709 12:25:12.098207  9220 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:12.199118  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 251423us
W20260709 12:25:12.194845  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 306904us
I20260709 12:25:12.123497  9154 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.104s
W20260709 12:25:12.213418  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 232901us
I20260709 12:25:12.216702  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:12.211768  9258 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.339s	user 0.000s	sys 0.001s
I20260709 12:25:12.184077  9061 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.210736  9223 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:12.224277  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:12.219235  9251 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:12.222911  9197 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:12.236217  9264 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.281s	user 0.001s	sys 0.001s
W20260709 12:25:12.266149  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 90366.9us
I20260709 12:25:12.266253  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:12.245479  9004 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:12.215152  8749 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:12.268051  8493 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:12.288040  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2685) took 1771 ms. Trace:
I20260709 12:25:12.288575  1799 rpcz_store.cc:276] 0709 12:25:10.516520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.963755 (+447235us) service_pool.cc:224] Handling call
0709 12:25:11.289484 (+325729us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.303415 (+ 13931us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.287788 (+984373us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b540013a6d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:12.287964 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":574042496}
W20260709 12:25:12.290428  9259 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
W20260709 12:25:12.291952  9270 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:25:12.294539  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2653) took 1545 ms. Trace:
I20260709 12:25:12.295006  1806 rpcz_store.cc:276] 0709 12:25:10.749742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.116223 (+366481us) service_pool.cc:224] Handling call
0709 12:25:11.116576 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.116967 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.116995 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.117468 (+   473us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.926477 (+809009us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:11.926580 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.926723 (+   143us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.294294 (+367571us) raft_consensus.cc:1685] finished
0709 12:25:12.294318 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.294479 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3717,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"thread_start_us":808412,"threads_started":1,"wal-append.queue_time_us":934272}
W20260709 12:25:12.297601  9260 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.236s	user 0.000s	sys 0.001s
W20260709 12:25:12.301432  9267 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:25:12.302986  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1446) took 1557 ms. Trace:
W20260709 12:25:12.270373  9263 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.279s	user 0.000s	sys 0.001s
I20260709 12:25:12.301362  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2818) took 1178 ms. Trace:
I20260709 12:25:12.263986  8986 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:12.275432  9265 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.284s	user 0.001s	sys 0.001s
I20260709 12:25:12.303368  2416 rpcz_store.cc:276] 0709 12:25:10.745865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.746765 (+   900us) service_pool.cc:224] Handling call
0709 12:25:10.811977 (+ 65212us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.262127 (+450150us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.262160 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.274896 (+ 12736us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.926700 (+651804us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:11.926765 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.926899 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.302718 (+375819us) raft_consensus.cc:1685] finished
0709 12:25:12.302747 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.302919 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":104005,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":25518,"thread_start_us":1052947,"threads_started":2,"wal-append.queue_time_us":736819}
I20260709 12:25:12.377420  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:12.316744  1730 rpcz_store.cc:276] 0709 12:25:11.123373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.123604 (+   231us) service_pool.cc:224] Handling call
0709 12:25:11.126590 (+  2986us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.126895 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.126919 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.138302 (+ 11383us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.773997 (+635695us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:11.774037 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.774100 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.296902 (+522802us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:12.300971 (+  4069us) raft_consensus.cc:1685] finished
0709 12:25:12.301000 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.301293 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":66705664,"thread_start_us":581075,"threads_started":1,"wal-append.queue_time_us":828799}
I20260709 12:25:12.381616  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
W20260709 12:25:12.313102  9257 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.322s	user 0.000s	sys 0.001s
I20260709 12:25:12.382211  1730 raft_consensus.cc:606] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:12.412591  1730 raft_consensus.cc:614] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:12.321807  9230 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:12.361461  9224 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:12.270953  9077 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.108s
I20260709 12:25:12.241398  9220 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:12.451763  1730 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:25:12.279119  9288 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.148s
W20260709 12:25:12.455133  9280 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
W20260709 12:25:12.456300  9279 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.075s	user 0.001s	sys 0.001s
W20260709 12:25:12.454006  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 510567us
W20260709 12:25:12.469074  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.006s	sys 0.009s
W20260709 12:25:12.481110  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.006s	sys 0.010s
W20260709 12:25:12.465147  8928 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.000s	sys 0.011s
W20260709 12:25:12.489363  8928 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.000s	sys 0.012s
I20260709 12:25:12.438009  9224 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:12.493239  9232 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.765s	user 0.011s	sys 0.000s
W20260709 12:25:12.462622  9266 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.409s	user 0.000s	sys 0.002s
I20260709 12:25:12.505213  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2698) took 1669 ms. Trace:
I20260709 12:25:12.501112  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2821) took 1275 ms. Trace:
I20260709 12:25:12.505828  1802 rpcz_store.cc:276] 0709 12:25:10.835363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.243804 (+408441us) service_pool.cc:224] Handling call
0709 12:25:11.248352 (+  4548us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.248594 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.248629 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.254509 (+  5880us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.709343 (+454834us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:11.709387 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.709476 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.504870 (+795394us) raft_consensus.cc:1685] finished
0709 12:25:12.504905 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.505115 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":434823,"threads_started":1,"wal-append.queue_time_us":701794}
I20260709 12:25:12.506129  1724 rpcz_store.cc:276] 0709 12:25:11.225755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.225972 (+   217us) service_pool.cc:224] Handling call
0709 12:25:11.226457 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.227168 (+   711us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.227218 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.227925 (+   707us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.687848 (+459923us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:11.687892 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.687961 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.500613 (+812652us) raft_consensus.cc:1685] finished
0709 12:25:12.500646 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.500956 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1104,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":3863,"thread_start_us":428624,"threads_started":1,"wal-append.queue_time_us":655033}
I20260709 12:25:12.507970  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:12.509114  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:12.512277  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:12.503297  1874 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260709 12:25:12.501397  9220 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:12.515101  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:12.515599  9293 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:25:12.516049  1874 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:12.517741  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.518226  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.500756  9297 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.490497  9224 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:12.503688  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:12.521443  1877 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:12.505348  9145 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.679s	user 0.012s	sys 0.000s
W20260709 12:25:12.527510  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 150172us
I20260709 12:25:12.493916  9232 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.531965  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2814) took 1574 ms. Trace:
I20260709 12:25:12.532759  1728 rpcz_store.cc:276] 0709 12:25:10.957254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.957443 (+   189us) service_pool.cc:224] Handling call
0709 12:25:10.957853 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.177363 (+219510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.531601 (+1354238us) spinlock_profiling.cc:243] Waited 219 ms on lock 0x7b54002b70d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:12.531877 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":502898944}
I20260709 12:25:12.516199  8919 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:12.522511  9297 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.539129  8493 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.557320  9223 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.549401  8919 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:12.527954  9145 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.585089  9285 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.585614  9300 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:12.588430  9283 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
W20260709 12:25:12.589346  9291 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:25:12.594923  1807 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:25:12.561590  9061 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:12.597048  9307 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:12.603109  9296 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:12.664580  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3666) took 1348 ms. Trace:
I20260709 12:25:12.605338  9248 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:12.607307  8757 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:12.583698  9223 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.664934  9296 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:12.665042  1729 rpcz_store.cc:276] 0709 12:25:11.315765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.315898 (+   133us) service_pool.cc:224] Handling call
0709 12:25:11.387019 (+ 71121us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.887869 (+500850us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.887901 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.916671 (+ 28770us) log.cc:844] Serialized 26 byte log entry
0709 12:25:12.234288 (+317617us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:12.234340 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.236197 (+  1857us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.664339 (+428142us) raft_consensus.cc:1685] finished
0709 12:25:12.664361 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.664515 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15508,"raft.queue_time_us":131385,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":116,"thread_start_us":734906,"threads_started":2,"wal-append.queue_time_us":357598}
I20260709 12:25:12.599541  9285 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.667618  9296 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:12.667618  9223 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:12.579356  8493 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.669694  9285 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:12.671653  8493 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:12.611799  9005 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.632s	user 0.011s	sys 0.002s
I20260709 12:25:12.672724  9005 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:12.673794  9301 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.112s	user 0.000s	sys 0.002s
I20260709 12:25:12.676986  9232 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:12.680797  8703 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:12.681373  8703 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:12.682336  8703 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.630086  8657 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:12.700009  8657 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:12.700939  8657 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:12.701897  8657 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:25:12.634392  9282 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.388s	user 0.001s	sys 0.000s
I20260709 12:25:12.622359  1798 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:12.648171  9053 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:25:12.647910  9158 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.803s	user 0.009s	sys 0.005s
I20260709 12:25:12.704890  9053 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:12.705066  9158 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.705788  9053 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:12.707468  9053 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:12.657171  9297 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:12.659081  9145 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:12.661394  9179 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.000s
I20260709 12:25:12.708554  9179 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.661839  9300 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.663973  8854 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:12.711087  8854 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:12.711980  8854 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.712455  9300 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:12.713007  8854 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:12.608843  1799 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:12.655457  9255 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:12.683753  9268 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.487s	user 0.006s	sys 0.001s
I20260709 12:25:12.697976  8989 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:12.693753  1876 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:12.723563  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 242775us
I20260709 12:25:12.725097  9302 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:12.726135  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2822) took 1111 ms. Trace:
I20260709 12:25:12.726562  1732 rpcz_store.cc:276] 0709 12:25:11.615141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.615335 (+   194us) service_pool.cc:224] Handling call
0709 12:25:11.615696 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.735060 (+119364us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.735095 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:11.735141 (+    46us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:11.735185 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.725901 (+990716us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54002b7fd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:12.725946 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.726075 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":273918208}
I20260709 12:25:12.731860  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2826) took 1112 ms. Trace:
I20260709 12:25:12.732478  1727 rpcz_store.cc:276] 0709 12:25:11.619463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.652386 (+ 32923us) service_pool.cc:224] Handling call
0709 12:25:11.652747 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.061051 (+408304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.731535 (+670484us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b540030be54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:12.731782 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":899352448}
I20260709 12:25:12.740309  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:12.728801  9005 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:12.745114  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:12.746138  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:12.746908  2419 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:12.747435  9045 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:12.748056  9045 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:12.726719  9183 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:25:12.754634  1725 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:12.755020  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:12.756067  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:12.756875  1726 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:12.756632  9250 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:12.720907  9172 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:12.757936  9172 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:12.757733  9250 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.758880  9172 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 24.11, Last appended by leader: 11, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:12.760354  9172 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:12.760504  9250 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:12.763769  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:12.764405  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:12.721515  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 516396us
I20260709 12:25:12.765239  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:12.716877  9135 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:12.765971  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:12.766213  9045 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.767308  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:12.664600  9307 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:12.720814  9134 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:12.755144  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:12.764655  2410 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:12.719796  9179 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:12.765789  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:12.722337  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3665) took 1408 ms. Trace:
I20260709 12:25:12.755995  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:12.765156  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:12.691288  9237 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.538s	user 0.012s	sys 0.000s
I20260709 12:25:12.776772  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:12.715875  1874 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:12.772225  1727 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:12.770246  1723 rpcz_store.cc:276] 0709 12:25:11.314454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.314604 (+   150us) service_pool.cc:224] Handling call
0709 12:25:11.314953 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:11.315237 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:11.382310 (+ 67073us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.218114 (+835804us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.682956 (+464842us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:12.721973 (+ 39017us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:12.722031 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:12.722080 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.722147 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.722280 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":512187,"prepare.queue_time_us":73797,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":1433,"raft.queue_time_us":1233994,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":191,"spinlock_wait_cycles":5622656,"thread_start_us":834420,"threads_started":2,"wal-append.queue_time_us":796582}
I20260709 12:25:12.785365  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:12.800051  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:12.800541  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:12.815989  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:12.816974  1879 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:12.818738  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:12.794698  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:12.801713  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:12.819622  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:12.820832  1803 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:25:12.823942  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:12.824803  1806 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:12.826979  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 298514us
I20260709 12:25:12.828464  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:12.781795  9052 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:12.848295  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:12.848942  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:12.868916  1801 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:12.851327  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:12.853704  1879 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:25:12.854898  9087 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:12.856945  1803 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
W20260709 12:25:12.874728  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 73080.9us
I20260709 12:25:12.879890  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:12.881134  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:12.881462  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:12.881995  1873 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:12.858355  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.861512  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:12.791419  9302 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:12.882001  1881 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:12.889766  1877 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:12.893297  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:12.894275  1805 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:12.896167  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:12.873910  9297 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:12.907567  9087 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:12.908249  9087 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:12.914189  1873 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
W20260709 12:25:12.915067  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53709.2us
I20260709 12:25:12.916317  9285 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:12.916867  9285 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:12.919299  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:12.920928  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:12.933313  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:12.933897  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:12.934732  2416 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:25:12.919142  1803 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:12.909663  9087 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:12.936296  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.938441  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89659.2us
I20260709 12:25:12.943795  1771 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:12.939028  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:12.938668  1880 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:12.852030  9179 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:12.947292  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:12.936421  2419 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:12.947570  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:12.791105  9237 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.975590  2410 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:12.917819  9285 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:12.967561  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46732.8us
W20260709 12:25:12.987080  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 123549us
I20260709 12:25:12.987949  1801 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:12.988605  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:12.990860  9310 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:12.972393  9085 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:12.991496  9310 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.991644  9085 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.992013  9310 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:12.992132  9085 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:12.968959  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:12.995020  9077 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:12.959522  2413 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:12.995621  9077 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:12.924706  9135 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:12.996695  9077 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:12.965590  9179 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.936044  9158 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:12.980525  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:12.865798  9300 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.107s
I20260709 12:25:12.949208  9027 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:12.826421  9248 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.123s
I20260709 12:25:12.998502  2417 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:12.986451  1799 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:13.009578  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:13.011399  1874 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:25:12.951349  1806 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:12.935086  9273 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.791114  9134 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.012717  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:13.136962  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.117777  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.150761  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.013217  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:13.012542  1873 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:13.156490  1877 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:13.157795  1873 raft_consensus.cc:1217] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Deduplicated request from leader. Original: 14.10->[16.11-19.12]   Dedup: 16.11->[19.12-19.12]
I20260709 12:25:13.158612  1873 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: [16.11-19.12]
I20260709 12:25:12.940245  9302 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:13.038672  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:12.960839  9064 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.168725  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:13.076095  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 66160.5us
I20260709 12:25:13.077636  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:13.171468  2419 raft_consensus.cc:2393] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:13.033094  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:13.172233  1806 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:13.001631  9296 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:13.187417  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:13.188400  1874 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:25:13.109169  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.152019  9005 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.153452  9087 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:12.947393  9220 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:13.002645  9307 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:13.079583  1880 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:25:12.998318  9179 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:13.003710  8853 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.172s
I20260709 12:25:13.004477  9217 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.172s
I20260709 12:25:13.003520  9027 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.038237  9197 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.167309  8986 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:13.176627  8958 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:13.177471  9019 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:13.038084  9053 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.182085  9064 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.198135  9302 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:13.209204  1724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.005s	sys 0.025s
I20260709 12:25:12.997750  9135 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.214051  1799 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
W20260709 12:25:13.269620  1724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.736s	user 0.006s	sys 0.025s
I20260709 12:25:13.270161  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:13.224108  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 64390.9us
I20260709 12:25:13.271870  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:13.001806  9223 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.196s
I20260709 12:25:13.007380  9231 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.204s
I20260709 12:25:12.997431  9224 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:13.002562  9228 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.215s
I20260709 12:25:13.004850  9158 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.213s
W20260709 12:25:13.255463  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.017s	sys 0.035s
I20260709 12:25:13.258692  1873 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
W20260709 12:25:13.264562  9272 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.019s	sys 0.018s
W20260709 12:25:13.283783  9272 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.019s	sys 0.019s
W20260709 12:25:13.283934  1770 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.629s	user 0.014s	sys 0.016s
I20260709 12:25:13.261641  1806 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 10. Preceding OpId from leader: term: 25 index: 11. (index mismatch)
W20260709 12:25:13.293325  1770 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.014s	sys 0.017s
I20260709 12:25:12.998238  9311 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.245s
W20260709 12:25:13.222374  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.007s	sys 0.024s
W20260709 12:25:13.300380  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.008s	sys 0.024s
I20260709 12:25:13.273787  1799 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:25:13.080973  9273 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.270941  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1138) took 1489 ms. Trace:
I20260709 12:25:13.279277  1881 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:13.263656  9296 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:13.184454  9237 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:13.288388  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.017s	sys 0.035s
I20260709 12:25:13.223587  9005 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:13.295776  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 640704us
I20260709 12:25:13.272173  9300 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.273998  9135 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:13.258546  9197 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:13.285691  9313 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:13.301506  9272 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 12:25:13.258102  8986 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:13.282054  9217 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:13.332037  9217 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.332557  9217 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:13.333009  9027 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:13.335441  9064 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:13.336992  8703 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:13.337337  9134 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:13.329730  9300 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:13.340335  9005 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:13.302416  9223 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:25:13.326776  1731 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.011s	sys 0.038s
W20260709 12:25:13.285125  8749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.018s	sys 0.010s
I20260709 12:25:13.306895  1724 rpcz_store.cc:276] 0709 12:25:11.781664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.510605 (+728941us) service_pool.cc:224] Handling call
0709 12:25:12.523490 (+ 12885us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.523884 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:12.532243 (+  8359us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.270440 (+738197us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:13.270520 (+    80us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:13.270628 (+   108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.270732 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.270886 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":263,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":47,"thread_start_us":737061,"threads_started":1}
I20260709 12:25:13.266964  9053 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:13.355983  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:13.314389  9228 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:13.359194  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:13.381067  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:13.360905  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:13.369815  2412 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
W20260709 12:25:13.326020  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.370913  1731 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.011s	sys 0.038s
W20260709 12:25:13.371304  1698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.021s	sys 0.031s
I20260709 12:25:13.287647  8958 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
W20260709 12:25:13.385682  1698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.021s	sys 0.031s
I20260709 12:25:13.386469  1799 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:13.338613  9134 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:13.312726  9296 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:13.390139  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:13.332566  9286 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:13.331234  9313 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:13.384076  8749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.019s	sys 0.010s
I20260709 12:25:13.396736  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:13.396777  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3663) took 2096 ms. Trace:
I20260709 12:25:13.413793  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:13.363171  9223 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:13.416687  1731 rpcz_store.cc:276] 0709 12:25:11.300263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.329448 (+ 29185us) service_pool.cc:224] Handling call
0709 12:25:11.375536 (+ 46088us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:11.384640 (+  9104us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.384686 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:11.385449 (+   763us) log.cc:844] Serialized 52 byte log entry
0709 12:25:11.989945 (+604496us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:12.025190 (+ 35245us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.710802 (+685612us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b180030a480. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:25:13.384608 (+673806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.384747 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:13.384820 (+    73us) raft_consensus.cc:1685] finished
0709 12:25:13.396527 (+ 11707us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.396715 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15465,"prepare.queue_time_us":39315,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":5739,"raft.queue_time_us":706847,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":16122,"spinlock_wait_cycles":6653824,"thread_start_us":1042515,"threads_started":2,"wal-append.queue_time_us":611229,"wal-append.run_cpu_time_us":2917,"wal-append.run_wall_time_us":1007169}
W20260709 12:25:13.395256  8657 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.693s	user 0.012s	sys 0.000s
I20260709 12:25:13.386104  9133 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:13.379865  9228 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:13.460670  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.409014  9296 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:13.431792  2418 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.003s	sys 0.020s
I20260709 12:25:13.396165  9027 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:13.390729  9019 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:25:13.430619  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 417857us
W20260709 12:25:13.470323  2418 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.003s	sys 0.020s
I20260709 12:25:13.413535  9286 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:13.500706  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:13.410720  9237 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.066s
I20260709 12:25:13.478106  9019 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.409281  9134 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:13.430096  8928 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:13.500523  9309 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
W20260709 12:25:13.502203  9052 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.623s	user 0.021s	sys 0.000s
I20260709 12:25:13.468668  8749 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:13.429854  8769 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.088s
I20260709 12:25:13.501156  9251 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:25:13.520291  9304 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:25:13.500707  9286 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:13.530167  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 252334us
I20260709 12:25:13.584000  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:13.587606  1798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.009s	sys 0.027s
W20260709 12:25:13.588168  1798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.009s	sys 0.028s
W20260709 12:25:13.590160  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 398520us
I20260709 12:25:13.518826  9300 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:13.520664  9255 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:25:13.608837  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.611127  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.614324  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.524834  9026 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:13.654989  8986 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:13.657557  9231 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:13.599159   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
W20260709 12:25:13.659600  9324 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 12:25:13.661190  9321 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
W20260709 12:25:13.667622  9314 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.191s	user 0.000s	sys 0.000s
W20260709 12:25:13.668918  9318 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:25:13.675055  9332 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:13.675968  9332 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:13.676895  9053 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:13.678335  9332 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:13.681638  9085 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.689s	user 0.005s	sys 0.005s
I20260709 12:25:13.682658  9320 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.683260  9320 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.684187  9255 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:13.685595  9320 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:13.557929  9224 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:13.685189  9255 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.675237  9197 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:13.708899  9183 raft_consensus.cc:993] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: : Instructing follower 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
W20260709 12:25:13.709074  9310 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.716s	user 0.009s	sys 0.005s
I20260709 12:25:13.709513  9183 raft_consensus.cc:1081] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Signalling peer 6084fbc6918e4e0b95a9bd0206a620d8 to start an election
I20260709 12:25:13.709563  9310 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.650038  9026 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:13.602936  8928 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.713392  8928 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:13.524636  9223 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:13.713999  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.715946  9319 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:13.716544  9319 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:13.597672  9052 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:13.723999  9134 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:13.724830  9273 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:13.593382  8749 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.727658  8749 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:13.731633  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.661087  9316 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260709 12:25:13.738245  2414 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:13.682333  9085 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:13.711072  9026 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:13.534976  9019 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:13.718535  9319 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:13.784111  1628 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 4 to 8, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 8 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:13.747882   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:13.791527  1873 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:13.714707  9223 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:13.806696  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.009s	sys 0.022s
W20260709 12:25:13.807513  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.010s	sys 0.022s
I20260709 12:25:13.847256  9310 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:13.803310  8853 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.014s	sys 0.017s
I20260709 12:25:13.551546  9228 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:13.741096  9332 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.058s
W20260709 12:25:13.888166  1801 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.002s	sys 0.031s
W20260709 12:25:13.888778  1801 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.703s	user 0.002s	sys 0.031s
W20260709 12:25:13.850272  8853 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.014s	sys 0.018s
I20260709 12:25:13.930330  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1286) took 1165 ms. Trace:
I20260709 12:25:13.931048  2418 rpcz_store.cc:276] 0709 12:25:12.764706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.766877 (+  2171us) service_pool.cc:224] Handling call
0709 12:25:12.767214 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.767611 (+   397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.767642 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:12.768161 (+   519us) log.cc:844] Serialized 26 byte log entry
0709 12:25:13.487493 (+719332us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:13.487552 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.487643 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:13.929883 (+442240us) raft_consensus.cc:1685] finished
0709 12:25:13.929920 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.930230 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2112,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":15167,"thread_start_us":714866,"threads_started":1,"wal-append.queue_time_us":1151358}
I20260709 12:25:13.941602  9286 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:13.948894  9135 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.603s	user 0.001s	sys 0.008s
W20260709 12:25:13.954579  9172 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.194s	user 0.014s	sys 0.001s
I20260709 12:25:13.956811  9085 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:13.900080  9179 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.648s	user 0.008s	sys 0.003s
W20260709 12:25:13.965268  1730 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.024s	sys 0.036s
W20260709 12:25:14.008687  1730 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.686s	user 0.024s	sys 0.036s
W20260709 12:25:13.984733  1872 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.004s	user 0.001s	sys 0.037s
W20260709 12:25:14.010756  1872 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.030s	user 0.001s	sys 0.037s
W20260709 12:25:13.989107  2415 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.013s	sys 0.023s
W20260709 12:25:13.971805  9323 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:25:14.010066  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1132) took 2370 ms. Trace:
W20260709 12:25:13.986994  9016 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.624s	user 0.003s	sys 0.034s
W20260709 12:25:13.995934  9217 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.663s	user 0.014s	sys 0.000s
I20260709 12:25:13.960106  9289 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
W20260709 12:25:14.011648  2415 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.013s	sys 0.023s
W20260709 12:25:14.030730  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.040128  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.044999  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1288) took 1225 ms. Trace:
I20260709 12:25:14.019471  1730 rpcz_store.cc:276] 0709 12:25:11.639630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.460150 (+820520us) service_pool.cc:224] Handling call
0709 12:25:12.460546 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.460820 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:12.461180 (+   360us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.652807 (+191627us) spinlock_profiling.cc:243] Waited 3.34 ms on lock 0x7b1800458c80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:25:14.009367 (+1356560us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.009471 (+   104us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:14.009554 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.009777 (+   223us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.009979 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":138,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":28344,"raft.queue_time_us":263860,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":60442,"spinlock_wait_cycles":7673728,"thread_start_us":1356106,"threads_started":2,"wal-append.queue_time_us":1014181}
W20260709 12:25:14.050729  1805 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.014s	sys 0.045s
I20260709 12:25:13.938189  9296 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.146374  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1454) took 1227 ms. Trace:
W20260709 12:25:14.030494  9016 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.003s	sys 0.034s
I20260709 12:25:14.034758  9275 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:14.004674  9135 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.046401  9085 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:14.145259  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.076129  9145 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:25:14.149163  1805 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.016s	sys 0.045s
I20260709 12:25:14.043303  9217 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.070429  9340 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.023011  9179 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.163292  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:14.208673  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.210876  1800 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:14.206802  1807 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:14.148480  8703 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:14.164522  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1059) took 1305 ms. Trace:
I20260709 12:25:14.220258  1805 rpcz_store.cc:276] 0709 12:25:12.859515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.895170 (+ 35655us) service_pool.cc:224] Handling call
0709 12:25:12.948064 (+ 52894us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.948328 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:13.348860 (+400532us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.349231 (+   371us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.079775 (+730544us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b18003f4e40. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:14.152164 (+ 72389us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b18003f4e40. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:14.164098 (+ 11934us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.164171 (+    73us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:14.164234 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.164318 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.164454 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1562957,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":91,"raft.queue_time_us":655758,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":27520,"spinlock_wait_cycles":111452800,"thread_start_us":1141852,"threads_started":2}
I20260709 12:25:14.149216  2414 rpcz_store.cc:276] 0709 12:25:12.919110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.938623 (+ 19513us) service_pool.cc:224] Handling call
0709 12:25:14.145961 (+1207338us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b54000e6d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:14.146290 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1767602560}
I20260709 12:25:14.169449  9085 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.195746  9135 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.225562  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:14.165486  9145 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:14.201028  9217 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:14.202105  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:14.203595  9336 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:25:14.203991  2417 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:14.153613  9176 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:14.198239  9026 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.240864  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1083) took 1207 ms. Trace:
W20260709 12:25:14.229852  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.808s	user 0.020s	sys 0.016s
W20260709 12:25:14.247949  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.020s	sys 0.016s
W20260709 12:25:14.249408  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 835636us
I20260709 12:25:14.247741  1880 rpcz_store.cc:276] 0709 12:25:13.033248 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.252325 (+219077us) service_pool.cc:224] Handling call
0709 12:25:13.252789 (+   464us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.253114 (+   325us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:13.264798 (+ 11684us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.268449 (+  3651us) log.cc:844] Serialized 14 byte log entry
0709 12:25:13.956500 (+688051us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b18003a9f60. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:14.231138 (+274638us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b18003a9f60. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16ab98ed70 00007f16ab982293 00007f16ab977cea 00007f16ab97823a 00007f16abde2a2a 00007f16abdf20bf 00007f16abdf2038
0709 12:25:14.240305 (+  9167us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.240394 (+    89us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:14.240475 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.240583 (+   108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.240780 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15585,"prepare.queue_time_us":26013,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":118,"raft.queue_time_us":11970,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":140,"spinlock_wait_cycles":34638592}
W20260709 12:25:14.238768  8958 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.810s	user 0.029s	sys 0.026s
W20260709 12:25:14.244148  9285 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.904s	user 0.009s	sys 0.023s
I20260709 12:25:14.247597  9176 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:14.212551  9340 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.298838  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:14.266536  9098 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.001s	sys 0.017s
W20260709 12:25:14.269356  2416 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.012s	sys 0.007s
I20260709 12:25:14.298527  9340 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:14.258611  8958 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.030s	sys 0.026s
W20260709 12:25:14.258579  9285 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.918s	user 0.009s	sys 0.023s
W20260709 12:25:14.267273  9077 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.807s	user 0.011s	sys 0.033s
I20260709 12:25:14.306605  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3584) took 1292 ms. Trace:
I20260709 12:25:14.316991  1877 rpcz_store.cc:276] 0709 12:25:13.014073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.157294 (+143221us) service_pool.cc:224] Handling call
0709 12:25:13.157692 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:13.183442 (+ 25750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:13.183478 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:13.184039 (+   561us) log.cc:844] Serialized 26 byte log entry
0709 12:25:13.589108 (+405069us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:13.591585 (+  2477us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.591700 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.306377 (+714677us) raft_consensus.cc:1685] finished
0709 12:25:14.306399 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.306543 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33606,"raft.queue_time_us":26612,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":28285,"thread_start_us":404664,"threads_started":1,"wal-append.queue_time_us":815126}
I20260709 12:25:14.226905  8703 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.260636  9341 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.325275  9341 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.328471  9341 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:14.263460  9333 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:25:14.330569  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2676) took 1524 ms. Trace:
I20260709 12:25:14.331035  1798 rpcz_store.cc:276] 0709 12:25:12.805759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.805956 (+   197us) service_pool.cc:224] Handling call
0709 12:25:12.806418 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.806952 (+   534us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.806999 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:12.864141 (+ 57142us) log.cc:844] Serialized 26 byte log entry
0709 12:25:13.588659 (+724518us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:13.588705 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.588793 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.330050 (+741257us) raft_consensus.cc:1685] finished
0709 12:25:14.330077 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.330493 (+   416us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58195,"raft.queue_time_us":710,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":27832,"thread_start_us":602517,"threads_started":1,"wal-append.queue_time_us":1055813}
W20260709 12:25:14.314919  1804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.735s	user 0.005s	sys 0.014s
W20260709 12:25:14.333369  1804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.754s	user 0.005s	sys 0.014s
W20260709 12:25:14.236631  9232 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.800s	user 0.019s	sys 0.016s
I20260709 12:25:14.303685  9087 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:14.319322  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.009s	sys 0.010s
W20260709 12:25:14.336175  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.751s	user 0.010s	sys 0.010s
W20260709 12:25:14.305327  9098 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.002s	sys 0.018s
I20260709 12:25:14.323244  8989 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.324524  8703 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:14.316939  9077 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.011s	sys 0.033s
W20260709 12:25:14.303445  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.816s	user 0.018s	sys 0.030s
W20260709 12:25:14.345453  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.019s	sys 0.030s
W20260709 12:25:14.345971  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 858900us
W20260709 12:25:14.306413  2416 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.804s	user 0.012s	sys 0.009s
W20260709 12:25:14.347361  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.962s	user 0.018s	sys 0.021s
W20260709 12:25:14.347715  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.963s	user 0.018s	sys 0.021s
I20260709 12:25:14.334298  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2683) took 1325 ms. Trace:
I20260709 12:25:14.344060  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:14.316111  9248 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.003s	sys 0.013s
W20260709 12:25:14.308243  9297 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.896s	user 0.005s	sys 0.014s
I20260709 12:25:14.348903  1880 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:14.339148  9232 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.019s	sys 0.017s
I20260709 12:25:14.347769  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:14.358533  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1289) took 1531 ms. Trace:
I20260709 12:25:14.358659  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:14.320739  9286 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.362053  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:14.393069  1879 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.972s	user 0.014s	sys 0.028s
I20260709 12:25:14.393981  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.393287  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.405104  1877 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:14.403162  1873 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:25:14.403734  2410 rpcz_store.cc:276] 0709 12:25:12.827354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.827607 (+   253us) service_pool.cc:224] Handling call
0709 12:25:12.965562 (+137955us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.983714 (+ 18152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.358296 (+1374582us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b540048a3d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:14.358471 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":315954432}
I20260709 12:25:14.368144  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2817) took 1487 ms. Trace:
W20260709 12:25:14.412266  1802 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.011s	sys 0.008s
W20260709 12:25:14.412736  1802 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.011s	sys 0.008s
W20260709 12:25:14.412686  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65024.3us
I20260709 12:25:14.413856  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:14.409821  1872 rpcz_store.cc:276] 0709 12:25:12.880413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.882186 (+  1773us) service_pool.cc:224] Handling call
0709 12:25:12.882513 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.882973 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.883019 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:12.883630 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 12:25:14.031789 (+1148159us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:14.031868 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.031986 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.367930 (+335944us) raft_consensus.cc:1685] finished
0709 12:25:14.367950 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.368079 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":965,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":39011,"thread_start_us":1030765,"threads_started":1,"wal-append.queue_time_us":1426657}
I20260709 12:25:14.420084  1880 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
W20260709 12:25:14.362366  9297 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.954s	user 0.005s	sys 0.014s
I20260709 12:25:14.423357  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:14.422322  1698 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.031s	user 0.014s	sys 0.032s
W20260709 12:25:14.425052  1698 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.034s	user 0.015s	sys 0.032s
I20260709 12:25:14.425235  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:14.353155  9248 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.003s	sys 0.013s
W20260709 12:25:14.403112  1879 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.015s	sys 0.028s
I20260709 12:25:14.427596  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:14.347615  9087 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:14.411175  9019 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.003s
I20260709 12:25:14.467446  9342 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:14.467298  9019 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.462947  9027 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:14.467944  1726 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:14.468519  9027 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.470826  9019 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.472102  9087 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.475100  2418 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:14.427464  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:14.447216  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:14.476540  1730 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:14.448009  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:14.477823  1723 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:14.448454  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:14.424105  9339 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:14.308403  9145 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:14.450004  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.988s	user 0.010s	sys 0.041s
W20260709 12:25:14.479517  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.018s	user 0.011s	sys 0.041s
I20260709 12:25:14.480783  8769 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:14.449239  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:14.482544  1731 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:14.483109  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:14.483600  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:14.484135  1723 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:25:14.367488  8703 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:14.484998  9339 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:14.485648  9339 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.485498  8703 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.488094  9339 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.487699  1628 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 7 to 11, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 11 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:14.489141  8703 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:14.489475  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:14.489704  8703 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:14.347143  8989 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.490294  9320 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:14.490963  9320 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:14.491420  9346 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:25:14.492205  8989 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:14.452231  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.008s	user 0.036s	sys 0.117s
I20260709 12:25:14.453080  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:14.453708  1728 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:14.493135  9320 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.354461  1804 rpcz_store.cc:276] 0709 12:25:13.009353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.009846 (+   493us) service_pool.cc:224] Handling call
0709 12:25:13.154182 (+144336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.154414 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:13.579305 (+424891us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.333882 (+754577us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.333949 (+    67us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:14.334014 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.334097 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.334239 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31133,"prepare.queue_time_us":554415,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":5558,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":6340,"thread_start_us":1091269,"threads_started":2}
W20260709 12:25:14.493809  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.049s	user 0.037s	sys 0.117s
I20260709 12:25:14.493716  9320 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:14.445765  9242 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:14.494936  9026 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.495252  9242 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:14.494570  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.09781e+06us
I20260709 12:25:14.497411  9242 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:14.496366  1725 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:14.499363  8961 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:14.499946  8961 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.502375  8961 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:14.367219  9217 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:14.424278  9135 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:14.422803  9296 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.461407  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:14.517086  1732 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.422991  9085 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.462069  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:14.393282  9310 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.435210  9286 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:14.421504  9341 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:14.483799  9297 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:14.408139  9158 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.006s	sys 0.025s
I20260709 12:25:14.490556  1628 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 6 to 15. New cstate: current_term: 15 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:14.526363  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98968.8us
I20260709 12:25:14.513044  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:14.520329  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:14.496044  9220 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:14.456640  9179 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:14.523809  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:14.527742  9340 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:14.510677  9027 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.524976  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:14.535063  8769 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:14.535717  8769 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.536103  9289 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:14.435202  8757 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.069s
I20260709 12:25:14.538084  8757 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:14.538177  8769 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.452319  9275 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:14.538712  8757 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.533808  1628 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 9 to 11, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 11 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:14.539115  9275 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.540786  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:14.541182  9275 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.349678  9176 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.535678  2410 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:25:14.543449  9223 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.694s	user 0.006s	sys 0.006s
I20260709 12:25:14.528486  9310 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.531028  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:14.545085  1728 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 22 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
W20260709 12:25:14.528817  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66840.3us
I20260709 12:25:14.545900  9310 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.546320  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:14.533182  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:14.498598  9300 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.534708  1799 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:14.549400  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1055) took 1743 ms. Trace:
I20260709 12:25:14.549813  1800 rpcz_store.cc:276] 0709 12:25:12.806168 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.823294 (+ 17126us) service_pool.cc:224] Handling call
0709 12:25:13.527614 (+704320us) spinlock_profiling.cc:243] Waited 3.17 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3eee 000000000036eb39 00007f16b0d271e0 00007f16b0df00c4 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:25:13.527904 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.211529 (+683625us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.549141 (+337612us) spinlock_profiling.cc:243] Waited 677 ms on lock 0x7b54002c0454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:14.549333 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1564233728}
I20260709 12:25:14.550772  9275 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:14.551306  9275 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:14.551132  9300 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.551535  9319 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:14.502512  9312 raft_consensus.cc:2764] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:25:14.541090  9172 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:14.551990  9319 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:14.552212  9172 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:14.534865  9340 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.553033  9319 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.553200  9172 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:14.529592  9158 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.007s	sys 0.027s
I20260709 12:25:14.554630  9172 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:14.528869  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:14.555644  2418 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:14.555879  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.556568  1875 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:25:14.557343  9335 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.410s	user 0.001s	sys 0.000s
I20260709 12:25:14.557509  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.547253  8757 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:14.550315  1880 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:25:14.521070  9320 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:14.456120  8958 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
I20260709 12:25:14.532747  1807 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:14.572034  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:14.521837  1877 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:14.540252  1628 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 10 to 14, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 14 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:14.559657  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.595597  1880 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:14.568234  1732 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
W20260709 12:25:14.589825  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.597795  1800 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:14.597812  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:14.600090  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:14.560178  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.600643  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:14.601190  1881 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:14.602124  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:14.602872  1881 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:14.467962  9342 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:14.600447  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:14.554534  9340 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.546602  9310 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:14.551798  9275 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:14.559834  1873 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:14.558640  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:14.599519  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1049) took 2658 ms. Trace:
I20260709 12:25:14.601315  1728 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:14.619060  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.009s	sys 0.009s
I20260709 12:25:14.593611  9320 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:14.554457  9319 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:14.606725  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.596221  9016 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.016s
I20260709 12:25:14.608117  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:14.609908  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.611217  9311 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:14.611876  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:14.614658  9342 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.612303  9242 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.612870  9286 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:14.587502  9272 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:14.560606  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.564334  9297 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.560901  9341 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.616748  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.616961  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:14.529698  9220 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:14.606102  2414 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:14.621337  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:14.622748  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:14.622833  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:14.619431  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.009s	sys 0.009s
I20260709 12:25:14.628418  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:14.618942  1802 rpcz_store.cc:276] 0709 12:25:11.940911 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.509931 (+569020us) service_pool.cc:224] Handling call
0709 12:25:12.510247 (+   316us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:13.030295 (+520048us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:13.030343 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:13.168038 (+137695us) log.cc:844] Serialized 52 byte log entry
0709 12:25:13.580061 (+412023us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:14.413360 (+833299us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.413637 (+   277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.413718 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.556688 (+142970us) spinlock_profiling.cc:243] Waited 2.25 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:14.592985 (+ 36297us) raft_consensus.cc:1685] finished
0709 12:25:14.599319 (+  6334us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.599464 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37969,"prepare.queue_time_us":128964,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":77571,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":4987,"spinlock_wait_cycles":5174144,"thread_start_us":1609776,"threads_started":3,"wal-append.queue_time_us":859036}
I20260709 12:25:14.633062  2412 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:14.604861  1799 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:14.612133  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:14.559185  8757 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 12:25:14.529040  9085 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.534528  9217 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:14.628015  9310 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:14.617695  9302 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.922s	user 0.012s	sys 0.014s
I20260709 12:25:14.721622  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:14.734211  1628 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 11 to 14, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 14 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:14.629978  9312 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:14.713804  1803 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.294s	user 0.006s	sys 0.030s
W20260709 12:25:14.679078  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:14.739702  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.490298  8703 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.729530  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:14.729951  1723 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:25:14.735494  1799 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
W20260709 12:25:14.643169  9255 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.004s	sys 0.022s
I20260709 12:25:14.723685  1875 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:14.748373  1628 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 11 to 13, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 13 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:14.742702  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:14.722923  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 110864us
I20260709 12:25:14.743695  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:14.743356  1803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.329s	user 0.006s	sys 0.030s
I20260709 12:25:14.773697  1799 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:14.728731  9339 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:14.776515  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:14.777068  1802 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:14.774973  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1061) took 1822 ms. Trace:
W20260709 12:25:14.741201  9302 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.046s	user 0.012s	sys 0.014s
I20260709 12:25:14.778518  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:14.748822  9255 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.010s	sys 0.022s
I20260709 12:25:14.788369  1803 rpcz_store.cc:276] 0709 12:25:12.952527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.952694 (+   167us) service_pool.cc:224] Handling call
0709 12:25:12.969486 (+ 16792us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.969773 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:13.392141 (+422368us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.774433 (+1382292us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.774513 (+    80us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:14.774593 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.774686 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.774896 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35609,"prepare.queue_time_us":738275,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":479264,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":7424,"thread_start_us":1759570,"threads_started":2}
I20260709 12:25:14.632978  9311 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:14.801079  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:14.778854  2411 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.496s	user 0.021s	sys 0.065s
I20260709 12:25:14.800001  9006 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:14.799341  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:14.740700  9272 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:14.805157  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:14.806277   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:25:14.808570  2411 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.526s	user 0.021s	sys 0.066s
I20260709 12:25:14.809746  1799 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:25:14.789263  8961 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:14.819353  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:14.819515  8961 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.820348  1799 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:25:14.821311  1799 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "908089442ca44109875257bfbc9d06c2"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:33802
I20260709 12:25:14.821897  1799 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:14.822238  1799 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:14.823243  1803 raft_consensus.cc:2410] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
W20260709 12:25:14.824090  9345 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:25:14.824733  1803 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:25:14.811805  9220 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:14.823158  1624 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 258) took 28247 ms (client timeout 15000 ms). Trace:
I20260709 12:25:14.825706  8989 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:14.827075  8989 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.829689  8989 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:14.809734  1628 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 10 to 13, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 13 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:14.648183  9286 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.839514  9272 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.840994  9286 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.815660  2414 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:25:14.823150  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:14.844259  1732 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:14.643599  9242 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:14.849565  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.851929  1802 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:14.855309  1730 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 22 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:14.855973  1628 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 6 to 7, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 7 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:14.738402  9085 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:14.839613  9357 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:25:14.861932  1799 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:14.746094  9312 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:14.826308  9220 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:14.822304  9363 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:25:14.794698  8853 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.067s
I20260709 12:25:14.811910  9006 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:14.848407  9300 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.849138  8961 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:14.863952  9341 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:14.915936  1799 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:14.924940  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.816524  9339 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:14.925884  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:14.944233  1730 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 11. Preceding OpId from leader: term: 25 index: 11. (term mismatch)
I20260709 12:25:14.970209  1730 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 10
I20260709 12:25:14.970618  1730 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 24.11
I20260709 12:25:14.946132  1807 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:25:14.950291  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:14.971846  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:14.961635  1803 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260709 12:25:14.912171  9363 consensus_queue.cc:1243] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:25:14.975970  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:14.976830  1803 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:14.970938  1730 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:14.972303  1731 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
W20260709 12:25:14.826241  1624 rpcz_store.cc:269] 0709 12:24:46.465747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.468091 (+  2344us) service_pool.cc:224] Handling call
0709 12:25:14.648794 (+28180703us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:59.946631 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:59.947379 (+   748us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:59.947436 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:24:59.947458 (+    22us) tablet.cc:662] Decoding operations
0709 12:25:00.072048 (+124590us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:00.072293 (+   245us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:00.072348 (+    55us) tablet.cc:685] Acquiring locks for 24 operations
0709 12:25:00.147244 (+ 74896us) tablet.cc:701] Row locks acquired
0709 12:25:00.147304 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:25:00.147496 (+   192us) write_op.cc:270] Start()
0709 12:25:00.147671 (+   175us) write_op.cc:276] Timestamp: P: 1783599900147424 usec, L: 0
0709 12:25:00.147697 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:25:00.190099 (+ 42402us) log.cc:844] Serialized 9463 byte log entry
0709 12:25:00.712854 (+522755us) op_driver.cc:464] REPLICATION: finished
0709 12:25:01.004400 (+291546us) spinlock_profiling.cc:243] Waited 9.18 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16a8cce0ac 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16af80eef0
0709 12:25:01.784712 (+780312us) write_op.cc:301] APPLY: starting
0709 12:25:01.784826 (+   114us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:02.631903 (+847077us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:02.631929 (+    26us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:02.773468 (+141539us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:02.773911 (+   443us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=48,key_file_lookups=48,delta_file_lookups=48,mrs_lookups=0
0709 12:25:02.774101 (+   190us) write_op.cc:312] APPLY: finished
0709 12:25:02.774666 (+   565us) log.cc:844] Serialized 212 byte log entry
0709 12:25:03.101033 (+326367us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:03.136844 (+ 35811us) write_op.cc:454] Released schema lock
0709 12:25:03.162203 (+ 25359us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:03.221488 (+ 59285us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:03.226930 (+  5442us) write_op.cc:454] Released schema lock
0709 12:25:04.520173 (+1293243us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
Metrics: {"mutex_wait_us":174792,"reported_tablets":50,"spinlock_wait_cycles":3968,"child_traces":[["op",{"apply.queue_time_us":1044732,"apply.run_cpu_time_us":105484,"apply.run_wall_time_us":1419613,"cfile_cache_hit":98,"cfile_cache_hit_bytes":104543,"num_ops":24,"prepare.queue_time_us":158532,"prepare.run_cpu_time_us":32520,"prepare.run_wall_time_us":312859,"raft.queue_time_us":352391,"raft.run_cpu_time_us":57566,"raft.run_wall_time_us":1074857,"replication_time_us":538947,"spinlock_wait_cycles":28062208,"thread_start_us":1623930,"threads_started":5,"wal-append.queue_time_us":490529,"wal-append.run_cpu_time_us":139833,"wal-append.run_wall_time_us":3598293}]]}
I20260709 12:25:15.019061  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:15.021210  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:15.035399  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.039054  1728 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 24 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
W20260709 12:25:15.041368  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.879664  9310 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.047430  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:15.048878  1728 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:25:15.051693  1802 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 20. Ops: []
I20260709 12:25:14.879374  9313 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.024714  9045 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:15.057557  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:15.062773  1799 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:25:15.056643  1878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.011s	sys 0.019s
W20260709 12:25:15.064965  1878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.012s	sys 0.020s
W20260709 12:25:15.069130  1874 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.002s	sys 0.009s
W20260709 12:25:15.069634  1874 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.002s	sys 0.009s
I20260709 12:25:15.070911  1803 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:15.074265  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:15.078610  1801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.180s	user 0.009s	sys 0.022s
W20260709 12:25:15.079123  1801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.190s	user 0.010s	sys 0.023s
I20260709 12:25:15.080624  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2684) took 2054 ms. Trace:
I20260709 12:25:15.083746  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:15.056640  9362 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:15.064327  9217 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:25:15.067952  8703 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:15.106506  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.021s	sys 0.027s
W20260709 12:25:15.118615  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.021s	sys 0.027s
W20260709 12:25:15.106123  2416 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.019s	sys 0.028s
W20260709 12:25:15.119561  2416 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.019s	sys 0.028s
I20260709 12:25:15.136314  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:15.137113  1803 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 15.
I20260709 12:25:15.136314  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:15.081547  9297 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:15.142609  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.089623  9006 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.137267  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1461) took 1721 ms. Trace:
I20260709 12:25:15.155982  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:15.096490  9362 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:15.136580  9158 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.145795  9251 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:15.154583  9366 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.157744  9341 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.175558  9297 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.196748  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:15.184855  9248 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.714s	user 0.012s	sys 0.078s
I20260709 12:25:15.180970  9242 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:15.201671  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:15.204800  9352 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
W20260709 12:25:15.194067  1724 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.005s	sys 0.008s
W20260709 12:25:15.193146  1806 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.006s	sys 0.007s
I20260709 12:25:15.190279  9085 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:15.209105  9365 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.294s	user 0.002s	sys 0.000s
I20260709 12:25:15.195199  9366 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.186705  9158 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:15.217165  1798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.018s	sys 0.026s
W20260709 12:25:15.251605  1798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.018s	sys 0.026s
I20260709 12:25:15.204435  9312 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:15.190550  9251 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:15.212157  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.012s	sys 0.002s
W20260709 12:25:15.259918  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.013s	sys 0.002s
W20260709 12:25:15.260347  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 730794us
W20260709 12:25:15.244053  2382 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.000s	sys 0.029s
W20260709 12:25:15.261354  2382 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.001s	sys 0.030s
I20260709 12:25:15.197789  9341 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:15.261951  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 542760us
I20260709 12:25:15.081259  1801 rpcz_store.cc:276] 0709 12:25:13.026139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.026395 (+   256us) service_pool.cc:224] Handling call
0709 12:25:13.148084 (+121689us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.148375 (+   291us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:13.889288 (+740913us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.079948 (+1190660us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:15.080031 (+    83us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:15.080090 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.080187 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.080473 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22137,"prepare.queue_time_us":560177,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":119,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":5680,"thread_start_us":1893370,"threads_started":2}
W20260709 12:25:15.239470  1806 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.006s	sys 0.008s
I20260709 12:25:15.271659  8853 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.272387  8853 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.273252  9341 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:15.258980  9223 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:15.275395  8853 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:15.239975  9087 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.000s	sys 0.014s
W20260709 12:25:15.276036  9358 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.411s	user 0.001s	sys 0.001s
W20260709 12:25:15.277026  9372 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 12:25:15.277403  9087 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s	user 0.001s	sys 0.015s
W20260709 12:25:15.214435  9248 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.743s	user 0.013s	sys 0.079s
W20260709 12:25:15.240159  1724 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.005s	sys 0.008s
I20260709 12:25:15.268299  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:15.322721  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54516.4us
W20260709 12:25:15.326427  2411 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.010s	sys 0.005s
W20260709 12:25:15.327154  2411 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.011s	sys 0.006s
W20260709 12:25:15.327548  1872 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.005s	sys 0.012s
W20260709 12:25:15.328464  1872 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.005s	sys 0.013s
I20260709 12:25:15.211440  9297 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.332947  9297 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:14.045718  2415 rpcz_store.cc:276] 0709 12:25:12.819396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.871617 (+ 52221us) service_pool.cc:224] Handling call
0709 12:25:12.872055 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.872342 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:13.384925 (+512583us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.044535 (+659610us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.044609 (+    74us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:14.044684 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.044786 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:14.044936 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19172,"prepare.queue_time_us":686098,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":5715,"thread_start_us":1115644,"threads_started":2}
W20260709 12:25:15.337085  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.009s	sys 0.009s
W20260709 12:25:15.337778  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.009s	sys 0.009s
W20260709 12:25:15.338070  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 542403us
I20260709 12:25:15.324134  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:15.333870  8958 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.013s	sys 0.024s
I20260709 12:25:15.318809  9135 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:15.339254  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:15.359594  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:15.361001  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:15.330004  9341 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:25:15.347247  8958 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.013s	sys 0.024s
I20260709 12:25:15.370411  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1085) took 2189 ms. Trace:
I20260709 12:25:15.376400  9312 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.350930  9342 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.707s	user 0.011s	sys 0.001s
W20260709 12:25:15.355371  9172 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.800s	user 0.007s	sys 0.006s
I20260709 12:25:15.350162  9135 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:15.218176  9362 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.376418  1879 rpcz_store.cc:276] 0709 12:25:13.180509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.319240 (+138731us) service_pool.cc:224] Handling call
0709 12:25:13.319705 (+   465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:13.320025 (+   320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:13.320066 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:13.331030 (+ 10964us) log.cc:844] Serialized 26 byte log entry
0709 12:25:14.460333 (+1129303us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:14.460373 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.460515 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.370061 (+909546us) raft_consensus.cc:1685] finished
0709 12:25:15.370126 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.370314 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1039356,"threads_started":1,"wal-append.queue_time_us":1415246}
I20260709 12:25:15.371582  9341 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.374425  9375 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.368177  9297 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:25:15.368804  9272 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.015s	sys 0.021s
I20260709 12:25:15.371786  9289 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:15.362855  9286 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.006s	sys 0.025s
W20260709 12:25:15.386591  9369 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
W20260709 12:25:15.493319  9378 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:25:15.368114  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:15.494310  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 126405us
W20260709 12:25:15.495518  9371 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:25:15.495941  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:15.447515  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.496847  9373 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.247s	user 0.000s	sys 0.001s
W20260709 12:25:15.458225  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.480554  9272 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.016s	sys 0.021s
I20260709 12:25:15.499068  9272 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:15.499888  9272 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:15.501502  9272 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:15.368989  9248 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.044s
W20260709 12:25:15.502712  9374 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:25:15.426465  9375 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.509271  9375 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:15.420301  9341 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:25:15.511879  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.530241  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.487051  9289 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.419013  9087 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.552858  9366 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:15.422367  9370 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:15.511929  9286 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.007s	sys 0.025s
I20260709 12:25:15.509853  9375 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:15.382494  9312 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.385087  9342 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.480571  9297 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:15.488044  9085 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.549444  9289 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:15.609989  1878 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.017s	sys 0.011s
W20260709 12:25:15.611954  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.595077  9275 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.966s	user 0.004s	sys 0.010s
W20260709 12:25:15.595077  9319 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.963s	user 0.013s	sys 0.001s
I20260709 12:25:15.617205  2418 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:15.509701  9311 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:15.612303  1878 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.018s	sys 0.011s
W20260709 12:25:15.640863  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.668145  1730 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
W20260709 12:25:15.670552  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 50867.8us
I20260709 12:25:15.702260  1723 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:15.596136  9087 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:15.690249  8958 consensus_queue.cc:1175] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "123095127f334445889c21a42ba1d41e" responder_term: 20 status { last_received { term: 19 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:15.636212  9077 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:15.719852  9339 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.748s	user 0.011s	sys 0.001s
I20260709 12:25:15.720089  9379 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:15.623637  9085 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:15.695410  9045 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.629s	user 0.011s	sys 0.002s
I20260709 12:25:15.724202  9297 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.673507  9272 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.683436  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2816) took 2803 ms. Trace:
I20260709 12:25:15.683211  9158 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:15.602996  9370 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.614553  9375 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.668583  9275 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:15.737946  9320 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.101s	user 0.009s	sys 0.006s
I20260709 12:25:15.732734  9248 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.746476  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.622395  1845 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.020s	sys 0.029s
W20260709 12:25:15.828821  1845 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.021s	sys 0.029s
I20260709 12:25:15.817782  1878 rpcz_store.cc:276] 0709 12:25:12.880119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.882094 (+  1975us) service_pool.cc:224] Handling call
0709 12:25:12.897909 (+ 15815us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.933663 (+ 35754us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:12.933704 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.074238 (+1140534us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x7b2c00009c38. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:25:14.157818 (+ 83580us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.065407 (+907589us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:15.635363 (+569956us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.635709 (+   346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.636609 (+   900us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.636671 (+    62us) raft_consensus.cc:1685] finished
0709 12:25:15.683200 (+ 46529us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.683379 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":487053,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":210675,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":22843,"spinlock_wait_cycles":6118784,"thread_start_us":1385381,"threads_started":2,"wal-append.queue_time_us":993882}
I20260709 12:25:15.809595  9339 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:25:15.848384  1874 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.026s	sys 0.062s
W20260709 12:25:15.849083  1874 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.027s	sys 0.062s
W20260709 12:25:15.850193  1729 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.018s	sys 0.018s
W20260709 12:25:15.852243  1729 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.019s	sys 0.018s
W20260709 12:25:15.851646  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 355745us
I20260709 12:25:15.838899  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2823) took 1283 ms. Trace:
I20260709 12:25:15.856606  1728 raft_consensus.cc:2410] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:25:15.854755  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:15.868564  1807 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:15.822629  9272 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.878698  9272 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:15.879880  9383 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
I20260709 12:25:15.881790  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1068) took 1371 ms. Trace:
I20260709 12:25:15.882295  1798 rpcz_store.cc:276] 0709 12:25:14.510228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.510374 (+   146us) service_pool.cc:224] Handling call
0709 12:25:14.516293 (+  5919us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.516624 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.516649 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.525305 (+  8656us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.252077 (+726772us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:15.252126 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.252227 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.881470 (+629243us) raft_consensus.cc:1685] finished
0709 12:25:15.881505 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.881704 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":27640,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"thread_start_us":645590,"threads_started":1,"wal-append.queue_time_us":963981}
I20260709 12:25:15.884306  9135 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.885800  9176 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:15.886906  9339 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:15.887449  9339 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.887818  9339 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:15.890082  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.848479  9158 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.899580  9158 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.900993  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:15.901672  1731 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:15.846801  9085 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:15.907270  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:15.907500  9382 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.284s	user 0.000s	sys 0.001s
I20260709 12:25:15.908555  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:15.909147  1730 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:15.910403  9353 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:15.911033  9353 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:15.912156  9353 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.913444  9353 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:15.915755  9381 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:15.916386  9381 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.919234  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2832) took 1297 ms. Trace:
I20260709 12:25:15.919687  1732 rpcz_store.cc:276] 0709 12:25:14.621928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.622297 (+   369us) service_pool.cc:224] Handling call
0709 12:25:15.919015 (+1296718us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b540030a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:15.919162 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279208704}
I20260709 12:25:15.920668  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:15.921137  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:15.918524  9381 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:15.926075  9315 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:15.926615  9315 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:15.927527  9315 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.928653  9315 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:15.935784  1726 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:15.815131  9379 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.815404  9077 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:15.943598  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:15.944334  1807 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:15.946699  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:15.947335  1807 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:15.950239  1807 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:15.953534  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:15.954162  1807 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:15.955668  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2337768859997595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
W20260709 12:25:15.956229  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0654768859999422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21389999999999998s for a response
I20260709 12:25:15.956533  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:15.832321  9320 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:15.964546  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.975852  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:15.978986  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:15.836860  9248 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:15.980080  1801 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:15.959731  1731 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:25:15.902630  9158 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:15.990093  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:15.936590  9242 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.994158  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:15.971114  9342 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:15.994093  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.019465  1807 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
W20260709 12:25:15.997426  9220 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.100s	user 0.012s	sys 0.003s
I20260709 12:25:16.021965  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:16.026401  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:15.977723  9386 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:15.988768  9045 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:16.023541  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:15.996996  9332 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:16.035210  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.035522  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:16.039846  1725 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:16.038609  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:16.046868  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:16.048619  9319 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:16.027729  9242 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.057615  9339 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:16.059667  9400 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:25:16.064308  1732 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:16.064913  9248 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.065986  9272 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:16.067116  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.069171  9087 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.074106  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:16.075534  2418 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:16.077087  9379 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.080142  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2690) took 1566 ms. Trace:
I20260709 12:25:16.081418  9339 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.080612  1806 rpcz_store.cc:276] 0709 12:25:14.513850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.514808 (+   958us) service_pool.cc:224] Handling call
0709 12:25:14.531308 (+ 16500us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.265933 (+734625us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.265982 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.326186 (+ 60204us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.816558 (+490372us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:15.816627 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.826503 (+  9876us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.079873 (+253370us) raft_consensus.cc:1685] finished
0709 12:25:16.079897 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.080075 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":86060,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":8414,"thread_start_us":1177638,"threads_started":2,"wal-append.queue_time_us":643670}
I20260709 12:25:16.019508  9399 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:16.022847  9402 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.085675  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:16.032397  8853 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:16.095546  1878 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:25:16.039718  9342 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:16.082062  9375 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.097227  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.098604  1880 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:16.103363  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.090546  9319 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:16.086905  9395 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:16.036717  9045 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.117033  9045 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.034021  9386 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.091794  8853 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:16.040418  9332 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.122138  9332 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.122844  8853 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:16.125900  9145 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:16.110097  9370 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.127507  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.128612  1877 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:16.094630  9402 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.091409  9399 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.136054  2418 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:16.136387  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.138814  1878 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:25:16.139242  2412 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:16.136026  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.137142  1875 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:25:16.142375  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.137446  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:16.143451  1881 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:16.140450  2415 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:25:16.136559  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.136508  9406 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.147893  9406 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.148152  9320 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.144264  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.153656  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.153738  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:16.160005  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.134476  9026 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:16.107402  9319 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:16.166517  9026 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:16.169281  9026 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.170369  9026 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:16.136859  9275 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.176843  9135 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.177953  9033 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.380120  9135 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.380306  9033 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.304472  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:16.350912  2382 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.785s (SENT) [suppressed 9 similar messages]
I20260709 12:25:16.326493  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.406340  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: true dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:16.315573  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.284288  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:16.292359  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 145461us
I20260709 12:25:16.383898  8703 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.387480  9158 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:16.370441  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.308215  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.314383  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:16.292222  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 452862us
I20260709 12:25:16.289888  1730 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:16.277665  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:16.161438  9289 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.167794  9319 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:16.223225  9342 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:25:16.187289  9341 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.598s	user 0.008s	sys 0.008s
I20260709 12:25:16.195041  9406 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.433947  1725 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:16.195844  9386 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.203900  9399 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:16.195063  9402 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:16.221796  9413 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
W20260709 12:25:16.297299  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.390957  8853 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:25:16.440042  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 162961us
I20260709 12:25:16.440873  8853 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:16.424527  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.441884  1877 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:25:16.425009  1727 raft_consensus.cc:2410] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:25:16.432452  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1158) took 1123 ms. Trace:
I20260709 12:25:16.425246  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:16.434898  9341 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.439347  8703 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.429083  1878 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 15.
I20260709 12:25:16.446223  9353 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:16.436209  9158 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:16.448763  1723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2836) took 1650 ms (client timeout 1654 ms). Trace:
I20260709 12:25:16.429584  9135 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.453372  9135 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:16.449635  1723 rpcz_store.cc:269] 0709 12:25:14.798424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.799035 (+   611us) service_pool.cc:224] Handling call
0709 12:25:16.448066 (+1649031us) spinlock_profiling.cc:243] Waited 577 ms on lock 0x7b54003ccdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:16.448434 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1328233344}
W20260709 12:25:16.433348  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 82651.7us
I20260709 12:25:16.444325  1875 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:16.443516  1730 rpcz_store.cc:276] 0709 12:25:15.305505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.908311 (+602806us) service_pool.cc:224] Handling call
0709 12:25:16.425988 (+517677us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.460052  9402 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:16.461881  9402 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:16.462482  9402 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:16.467715  9406 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.002s
I20260709 12:25:16.468891  9402 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.471923  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.473066  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:16.474225  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.474283  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:16.475198  9289 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.462211  9394 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:16.476385  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.476857  9394 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.477210  2417 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:16.478003  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:16.479077  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45294us
I20260709 12:25:16.455339  9158 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:16.479912  9347 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.484853  9347 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.487327  9158 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:16.482267  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.479866  9406 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:16.491707  9406 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.492214  9406 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:16.493721  8703 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.489607  9347 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:16.495805  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.496515  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:16.497933  9347 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:16.480295  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.498445  9341 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.499841  9402 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:25:16.483773  1881 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:16.477448  9394 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:16.499238  1729 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:16.513899  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:16.480693  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:16.515364  8703 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:16.515879  8703 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:16.516363  8703 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:16.517568  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.517704  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2847) took 1426 ms. Trace:
I20260709 12:25:16.528627  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1066) took 2022 ms. Trace:
I20260709 12:25:16.529209  1805 rpcz_store.cc:276] 0709 12:25:14.506508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.506871 (+   363us) service_pool.cc:224] Handling call
0709 12:25:14.507172 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.745321 (+238149us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.745358 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.745419 (+    61us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:14.745453 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.527882 (+1782429us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b54001e4454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:16.527991 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.528302 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":547181824}
I20260709 12:25:16.514122  9406 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:16.532325  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.069s	user 0.018s	sys 0.022s
W20260709 12:25:16.532869  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.070s	user 0.018s	sys 0.022s
W20260709 12:25:16.533301  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 2.07059e+06us
I20260709 12:25:16.543992  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:16.518328  1728 rpcz_store.cc:276] 0709 12:25:15.091654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.136004 (+ 44350us) service_pool.cc:224] Handling call
0709 12:25:16.517161 (+1381157us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b54002b75d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:16.517454 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1649540352}
I20260709 12:25:15.855310  1872 rpcz_store.cc:276] 0709 12:25:14.555112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.558349 (+  3237us) service_pool.cc:224] Handling call
0709 12:25:14.558670 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.558913 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:15.342136 (+783223us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.838308 (+496172us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:15.838397 (+    89us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:15.838478 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.838584 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.838813 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":993412,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":16086,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":9022,"spinlock_wait_cycles":1920,"thread_start_us":1256314,"threads_started":2,"wal-append.queue_time_us":525877}
W20260709 12:25:16.554766  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.556102  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.558085  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:16.534475  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:16.565946  9379 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.567049  9379 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.567531  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1457) took 3647 ms. Trace:
I20260709 12:25:16.568418  9255 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.569034  9255 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.568269  2411 rpcz_store.cc:276] 0709 12:25:12.920391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.920537 (+   146us) service_pool.cc:224] Handling call
0709 12:25:12.952623 (+ 32086us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.267851 (+315228us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:14.817843 (+1549992us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.348914 (+531071us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:15.348993 (+    79us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:15.349050 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.566746 (+1217696us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b5400561b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:16.566882 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.567256 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9351,"prepare.queue_time_us":418752,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":23064,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":12601,"spinlock_wait_cycles":724241664,"thread_start_us":2044601,"threads_started":2,"wal-append.queue_time_us":733239,"wal-append.run_cpu_time_us":1272,"wal-append.run_wall_time_us":1002716}
I20260709 12:25:16.569998  9319 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.572353  9370 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:16.572965  9370 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.573165  9319 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.573504  9370 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:16.578518  2412 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:16.498449  9347 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.592424  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:16.582365  9347 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:16.602551  9362 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:16.605216  9341 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:25:16.607527  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44758.7us
I20260709 12:25:16.606460  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:16.611557  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:16.610054  9367 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.694s	user 0.012s	sys 0.008s
I20260709 12:25:16.607254  9379 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.610491  9319 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:16.612098  9341 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:16.609659  8703 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.624859  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:16.630710  9394 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.634502  9394 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:16.618366  9406 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:16.635331  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 170104us
I20260709 12:25:16.636641  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:16.637805  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2829) took 2171 ms. Trace:
I20260709 12:25:16.638631  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:16.622957  9319 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:16.639611  9319 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:16.639765  9289 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:16.640483  9289 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:16.641894  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:16.641947  9289 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.642145  9406 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:16.645288  1730 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:16.645478  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:16.646288  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1169) took 1035 ms. Trace:
I20260709 12:25:16.646605  9406 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:16.647033  9135 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:16.647118  9406 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:16.648061  9406 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:16.648204  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.649458  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1156) took 1344 ms. Trace:
I20260709 12:25:16.649876  1728 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:16.649806  1723 rpcz_store.cc:276] 0709 12:25:15.304905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.647524 (+1342619us) service_pool.cc:224] Handling call
0709 12:25:16.649365 (+  1841us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.650612  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1150) took 1412 ms. Trace:
I20260709 12:25:16.651012  1728 rpcz_store.cc:276] 0709 12:25:15.238247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.649116 (+1410869us) service_pool.cc:224] Handling call
0709 12:25:16.649606 (+   490us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.650345 (+   739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.650519 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.638253  1724 rpcz_store.cc:276] 0709 12:25:14.466476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.466689 (+   213us) service_pool.cc:224] Handling call
0709 12:25:14.467208 (+   519us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.523949 (+ 56741us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.523983 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.528145 (+  4162us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.308359 (+780214us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:15.308440 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.637053 (+1328613us) spinlock_profiling.cc:243] Waited 48.7 ms on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:16.637128 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.637202 (+    74us) raft_consensus.cc:1685] finished
0709 12:25:16.637221 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.637557 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7560,"raft.queue_time_us":12200,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":35655,"spinlock_wait_cycles":111936768,"thread_start_us":772394,"threads_started":1,"wal-append.queue_time_us":951981}
I20260709 12:25:16.653409  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1154) took 1414 ms. Trace:
I20260709 12:25:16.653651  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:16.646837  1730 rpcz_store.cc:276] 0709 12:25:15.611175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.641421 (+1030246us) service_pool.cc:224] Handling call
0709 12:25:16.646191 (+  4770us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.655437  9135 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:16.655912  9135 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.656273  9135 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:16.656433  1730 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:16.657254  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1161) took 1350 ms. Trace:
I20260709 12:25:16.657625  1730 rpcz_store.cc:276] 0709 12:25:15.306770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.655703 (+1348933us) service_pool.cc:224] Handling call
0709 12:25:16.656165 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.656943 (+   778us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.657187 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.653918  1728 rpcz_store.cc:276] 0709 12:25:15.304071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.652233 (+1348162us) service_pool.cc:224] Handling call
0709 12:25:16.652748 (+   515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.653059 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.653087 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.653134 (+    47us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:16.653179 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.653229 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.653342 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.640513  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:15.157411  2416 rpcz_store.cc:276] 0709 12:25:13.415609 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.415987 (+   378us) service_pool.cc:224] Handling call
0709 12:25:13.465669 (+ 49682us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.465956 (+   287us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:14.346923 (+880967us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.136891 (+789968us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:15.136948 (+    57us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:15.136992 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.137058 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.137209 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14356,"prepare.queue_time_us":220859,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":15182,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":8722,"thread_start_us":1617526,"threads_started":2}
I20260709 12:25:16.661152  9315 raft_consensus.cc:2764] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20260709 12:25:16.660894  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:16.667125  2412 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 20
W20260709 12:25:16.669413  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52135.1us
I20260709 12:25:16.672537  9077 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:16.678607  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.686961  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.687912  2411 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:16.688103  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.689214  2410 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:16.689860  9223 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:16.681587  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 147345us
I20260709 12:25:16.672585  9332 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:16.702062  1728 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:25:16.697630  9223 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:16.701684  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:16.708190  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1163) took 1189 ms. Trace:
I20260709 12:25:16.708602  1728 rpcz_store.cc:276] 0709 12:25:15.516012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.668744 (+1152732us) service_pool.cc:224] Handling call
0709 12:25:16.698307 (+ 29563us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.703338 (+  5031us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.705378 (+  2040us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.703130  9332 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:16.720204  1730 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:25:16.723353  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1162) took 1336 ms. Trace:
I20260709 12:25:16.723788  1730 rpcz_store.cc:276] 0709 12:25:15.387392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.664331 (+1276939us) service_pool.cc:224] Handling call
0709 12:25:16.706269 (+ 41938us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.720677 (+ 14408us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.723289 (+  2612us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.715126  9379 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:16.727352  9077 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:16.727665  9379 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.717911  9332 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:16.732438  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:16.723091  9402 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:16.710767  9341 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.744381  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:16.746992  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 106681us
I20260709 12:25:16.747864  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.757725  9026 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:16.759351  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 150616us
W20260709 12:25:16.760114  1844 connection.cc:256] RPC call timeout handler was delayed by -2.1530816790000245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20990000000000003s for a response
W20260709 12:25:16.760797  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.750697  9379 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:16.769477  1844 connection.cc:256] RPC call timeout handler was delayed by -1.5627585540000837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21680000000000002s for a response
I20260709 12:25:16.752645  9402 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:16.770081  1844 connection.cc:256] RPC call timeout handler was delayed by -1.5360858240001107s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
I20260709 12:25:16.772054  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1170) took 1047 ms. Trace:
W20260709 12:25:16.775099  1844 connection.cc:256] RPC call timeout handler was delayed by -1.3907816789997014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29460000000000003s for a response
W20260709 12:25:16.775635  1844 connection.cc:256] RPC call timeout handler was delayed by -1.3668858240002919s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1738s for a response
W20260709 12:25:16.776005  1844 connection.cc:256] RPC call timeout handler was delayed by -1.1958858240000154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1928s for a response
I20260709 12:25:16.773962  1730 rpcz_store.cc:276] 0709 12:25:15.724373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.724607 (+1000234us) service_pool.cc:224] Handling call
0709 12:25:16.743351 (+ 18744us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.771806 (+ 28455us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.771825 (+    19us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.771851 (+    26us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:16.771874 (+    23us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.771918 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.772007 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:16.778909  9255 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.773065  8703 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:16.788964  9311 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:16.798285  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:16.805074  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2841) took 1834 ms (client timeout 1648 ms). Trace:
W20260709 12:25:16.809651  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 98526.9us
I20260709 12:25:16.790320  9379 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:16.812289  9315 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:16.810870  1726 rpcz_store.cc:269] 0709 12:25:14.970432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.970707 (+   275us) service_pool.cc:224] Handling call
0709 12:25:16.804708 (+1834001us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54000d20d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:16.804893 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1823333504}
I20260709 12:25:16.812835  9353 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:16.819160  9315 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.799492  9077 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:25:16.823936  9315 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:16.824609  9077 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:16.825083  9077 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:16.826305  9077 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.828042  9077 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:16.829696  2412 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:16.822705  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:16.797731  9319 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.853093  9319 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.842929  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:16.865307  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3590) took 2473 ms. Trace:
I20260709 12:25:16.866869  9402 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.869541  9242 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:16.871578  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 153311us
W20260709 12:25:16.872108  1727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1144) took 2424 ms (client timeout 1708 ms). Trace:
I20260709 12:25:16.872654  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:16.872560  1727 rpcz_store.cc:269] 0709 12:25:14.447891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.448046 (+   155us) service_pool.cc:224] Handling call
0709 12:25:16.871816 (+2423770us) spinlock_profiling.cc:243] Waited 1.94 s on lock 0x7b5400436cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:16.872002 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4455413632}
W20260709 12:25:16.874634  1725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1157) took 1569 ms (client timeout 2051 ms). Trace:
W20260709 12:25:16.875129  1725 rpcz_store.cc:269] 0709 12:25:15.305203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.085246 (+780043us) service_pool.cc:224] Handling call
0709 12:25:16.874397 (+789151us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54003cdcd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:16.874557 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413830400}
I20260709 12:25:16.878137  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1155) took 1573 ms. Trace:
I20260709 12:25:16.877915  9315 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.878994  8703 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:16.880971  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:16.881495  9251 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.878620  1724 rpcz_store.cc:276] 0709 12:25:15.304577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.652321 (+1347744us) service_pool.cc:224] Handling call
0709 12:25:16.876133 (+223812us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.876580 (+   447us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:16.876615 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.876659 (+    44us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:16.876702 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.876767 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.877926 (+  1159us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b5400436cd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:16.878041 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2651,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":122,"spinlock_wait_cycles":513505408}
I20260709 12:25:16.903854  2412 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:16.823243  9347 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.909968  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:16.913375  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.915024  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:16.924013  9228 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:16.924822  9347 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.865819  1874 rpcz_store.cc:276] 0709 12:25:14.391977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.392232 (+   255us) service_pool.cc:224] Handling call
0709 12:25:14.392736 (+   504us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.493626 (+100890us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.493654 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.494149 (+   495us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.070054 (+575905us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:15.325745 (+255691us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.849626 (+523881us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.864546 (+1014920us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54001f7a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:16.864628 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:16.864704 (+    76us) raft_consensus.cc:1685] finished
0709 12:25:16.864799 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.865063 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24050,"prepare.queue_time_us":482595,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":449324,"raft.run_cpu_time_us":373,"raft.run_wall_time_us":14242,"spinlock_wait_cycles":231152512,"thread_start_us":1323358,"threads_started":3,"wal-append.queue_time_us":785226,"wal-append.run_cpu_time_us":3275,"wal-append.run_wall_time_us":1172397}
I20260709 12:25:16.900390  9379 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:16.926707  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:16.933887  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.928005  9158 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:16.940450  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1153) took 1698 ms. Trace:
I20260709 12:25:16.912138  9223 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.932211  9251 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:16.952416  2383 connection.cc:256] RPC call timeout handler was delayed by -1.7039392130000124s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19610000000000002s for a response
I20260709 12:25:16.952741  8989 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:16.953423  8989 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.953915  8989 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:16.954833  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.958061  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.958891  9312 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.959640  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.959846  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 20 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.961061  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.961187  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:16.961436  9423 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 12:25:16.962654  1805 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:16.962944  9242 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:16.963753  9242 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:16.967162  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 159434us
I20260709 12:25:16.970525  9399 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:16.946815  8853 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:16.970537  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.974022  1725 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:16.973035  8853 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:16.975270  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 143098us
I20260709 12:25:16.958194  1724 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:16.962474  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.941022  1723 rpcz_store.cc:276] 0709 12:25:15.239139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.650625 (+1411486us) service_pool.cc:224] Handling call
0709 12:25:16.659780 (+  9155us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.676092 (+ 16312us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:16.785645 (+109553us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.926772 (+141127us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.926884 (+   112us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:16.937262 (+ 10378us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.937347 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.937480 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6796,"prepare.queue_time_us":157947,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":2829,"raft.queue_time_us":12655,"raft.run_cpu_time_us":847,"raft.run_wall_time_us":23830,"thread_start_us":208961,"threads_started":2}
I20260709 12:25:16.975492  8853 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:16.949600  9341 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.980347  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:16.987675  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.988097  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:16.989387  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:16.989977  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:16.990437  1726 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:16.994454  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:16.995724  1723 raft_consensus.cc:2376] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19. Re-sending same reply.
W20260709 12:25:17.000330  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46409.3us
W20260709 12:25:17.005046  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55670.1us
I20260709 12:25:16.966460  9242 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.006615  9319 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:17.007606  9319 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.007320  9242 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:17.008009  9319 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:17.010149  9370 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.011929  9313 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:17.012706  9370 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:16.980376  1727 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
W20260709 12:25:17.015290  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86761.8us
I20260709 12:25:17.016496  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:17.017037  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:17.017925  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:17.018317  9370 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:16.985251  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1149) took 1740 ms. Trace:
I20260709 12:25:17.006560  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:17.056152  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.052548  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.065722  1877 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:25:17.052874  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 151597us
I20260709 12:25:17.045966  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.044111  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.044945  1879 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:17.069336  1878 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:17.021999  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:17.053798  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.045475  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.073086  1881 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:17.073967  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:17.052712  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:17.070544  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68019.2us
I20260709 12:25:17.021100  9077 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:17.077754  9135 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.998556  9223 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:17.079327  1730 raft_consensus.cc:2368] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25.
I20260709 12:25:17.074213  1724 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:17.063745  1729 rpcz_store.cc:276] 0709 12:25:15.237950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.910682 (+1672732us) service_pool.cc:224] Handling call
0709 12:25:16.947488 (+ 36806us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.978274 (+ 30786us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.978309 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.978362 (+    53us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:16.978401 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.978460 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.978587 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:17.034472  1731 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:17.042726  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.087229  9179 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:17.048002  9390 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.316s	user 0.003s	sys 0.000s
I20260709 12:25:17.091763  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.093307  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.094365  1879 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:16.982496  9406 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.095335  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.028481  9379 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.052548  9006 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:17.096992  9406 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:17.082731  9332 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.087693  9255 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:17.051071  9033 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:17.099736  9255 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:16.976269  9251 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.103168  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.104082  1805 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:17.104511  9251 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.104867  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:17.105477  2412 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:17.106936  9135 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:17.108074  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:17.021888  9395 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:17.109200  9395 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.109615  9395 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:17.112699  1874 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:17.115005  9179 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.020117  8961 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:17.116384  9332 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.116998  8961 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:17.117692  1875 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:17.118485  8961 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.119331  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.120548  9310 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.121085  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.121168  9310 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.122002  1800 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
W20260709 12:25:17.090162  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 83815.9us
I20260709 12:25:17.122661  1806 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:17.035900  9315 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:17.129925  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:16.980382  9402 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:17.133248  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.135545  2412 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:17.136965  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:17.139781  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.140543  1880 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:17.142087  8989 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.128782  9251 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:17.144884  1872 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:17.146055  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 176303us
W20260709 12:25:17.148010  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57284.4us
W20260709 12:25:17.148582  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.151578  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:17.131925  9289 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.162202  8989 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:17.165235  9425 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.099s	user 0.002s	sys 0.000s
W20260709 12:25:17.166841  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149062us
I20260709 12:25:17.167979  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:17.147435  9251 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.168598  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.169035  9289 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.144047  9362 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:17.172391  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:17.157281  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1159) took 1851 ms (client timeout 1855 ms). Trace:
I20260709 12:25:17.162523  9310 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:17.182070  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.173664  1731 rpcz_store.cc:269] 0709 12:25:15.305828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.993762 (+687934us) service_pool.cc:224] Handling call
0709 12:25:17.147546 (+1153784us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540030af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:17.157197 (+  9651us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2343225088}
I20260709 12:25:17.159341  9135 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:17.188642  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.189157  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.196424  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.197000  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:17.146308  9402 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.174252  9251 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:17.204373  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:17.205348  2411 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:25:17.179107  9332 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:17.200605  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.201186  1881 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:17.202150  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.202908  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:17.169004  8989 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:17.207409  1806 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:17.210733  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.211398  1875 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:17.183332  9362 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:17.209638  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44987.4us
I20260709 12:25:17.215618  1724 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:17.215123  2416 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:17.178644  9026 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:25:17.185912  9399 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.027s
I20260709 12:25:17.196645  9033 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.209069  8989 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.227543  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:17.222286  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66290.4us
I20260709 12:25:17.224792  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:17.228487  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 83161.4us
I20260709 12:25:17.224359  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.213691  9362 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:17.230490  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.223183  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:17.233714  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1300) took 1939 ms. Trace:
I20260709 12:25:17.234174  2414 rpcz_store.cc:276] 0709 12:25:15.191650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.191829 (+   179us) service_pool.cc:224] Handling call
0709 12:25:15.192198 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.192534 (+   336us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:15.205206 (+ 12672us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.855400 (+1650194us) spinlock_profiling.cc:243] Waited 17 ms on lock 0x7b1800477580. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:25:16.945620 (+ 90220us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:16.977564 (+ 31944us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:17.079834 (+102270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.087833 (+  7999us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.130990 (+ 43157us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":12423,"prepare.run_cpu_time_us":96,"prepare.run_wall_time_us":1464,"raft.queue_time_us":743,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":4026,"spinlock_wait_cycles":39089920,"thread_start_us":78863,"threads_started":1,"wal-append.queue_time_us":1926630}
I20260709 12:25:17.238903  1731 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:17.242393  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1164) took 1692 ms. Trace:
I20260709 12:25:17.242859  1728 rpcz_store.cc:276] 0709 12:25:15.550227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.711826 (+1161599us) service_pool.cc:224] Handling call
0709 12:25:16.739797 (+ 27971us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.770717 (+ 30920us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.770739 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:16.796231 (+ 25492us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.945734 (+149503us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:16.945777 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.945876 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.242079 (+296203us) raft_consensus.cc:1685] finished
0709 12:25:17.242099 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.242299 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":107035,"threads_started":1,"wal-append.queue_time_us":313553}
I20260709 12:25:17.244975  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:17.246119  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:17.247066  9315 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:17.247520  9315 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.247841  9315 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:17.251022  9332 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:17.251451  9033 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.252223  9332 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.254235  9033 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.254820  9402 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.257835  9228 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:17.258352  9228 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.258697  9228 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:17.258922  9289 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:17.265753  1877 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:17.268607  1880 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:17.233392  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:17.275585  1878 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:25:17.276805  1874 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:17.276412  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:17.287340  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67976.5us
I20260709 12:25:17.288296  2411 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:17.292002  1879 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:17.298527  1872 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: [20.16-23.18]
I20260709 12:25:17.231503  8989 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:17.300117  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:17.308709  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:17.304883  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:17.310065  1724 raft_consensus.cc:2368] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20.
I20260709 12:25:17.312871  9310 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:17.313611  9310 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:17.314702  9310 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.315333  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.322333  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.276494  9135 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:17.297196  9311 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:17.327560  1874 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:17.326910  1873 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:17.334429  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.336584  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.323827  8961 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:17.320386  9255 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.295754  9395 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:17.344368  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 112278us
I20260709 12:25:17.349085  1875 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:17.306820  9432 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:25:17.329428  9319 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.354911  9223 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:17.355365  9179 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:17.330691  9026 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:17.356074  9135 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.340186  9158 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.358237  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.447273  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.330641  9311 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:17.366905  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.367209  1728 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
W20260709 12:25:17.450008  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 216735us
I20260709 12:25:17.375973  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.376729  1805 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:25:17.376588  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:17.378867  9315 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.381238  9251 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.455734  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:17.456689  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.383211  1872 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:17.386207  8853 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.453454  1806 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
W20260709 12:25:17.444762  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 156618us
I20260709 12:25:17.447463  1878 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:17.365567  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.447975  1875 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:17.372047  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.441792  9026 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:17.464535  1874 raft_consensus.cc:2368] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 19. Current term is 20.
I20260709 12:25:17.451815  9311 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:17.469055  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.476866  1805 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:17.469700  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.474998  1804 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:17.479108  9300 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.479679  9300 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.480165  9228 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.480527  9315 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.470094  8853 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:17.466305  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 195086us
I20260709 12:25:17.483844  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:17.485924  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:17.485782  9311 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.487084  9341 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:17.487947  9341 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:17.488440  9341 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:17.489571  9341 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.492137  9362 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.476994  9395 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.495038  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:17.485939  8989 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.505473  9228 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:17.506712  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:17.507304  8989 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:17.508309  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:17.510262  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:17.510932  2411 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:17.515475  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:17.508416  1877 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:17.518079  2411 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:17.518446  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:17.521463  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70229.5us
I20260709 12:25:17.541702  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.453924  9255 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:25:17.452023  9158 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.517493  1807 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:17.520642  9313 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.552016  1875 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:17.550961  8853 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.566118  9310 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:17.567353  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.544951  9395 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.551309  9228 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:25:17.572842  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78983.5us
I20260709 12:25:17.573426  1875 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:17.573168  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:17.543097  9315 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:17.580555  1807 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:17.618960  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.629307  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.634618  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.581290  9296 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:17.580152  9077 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:17.646068  1879 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:17.646040  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.647017  1881 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 20
W20260709 12:25:17.647395  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 142685us
I20260709 12:25:17.655838  1875 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:17.657358  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.657779  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:17.602151  9228 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:17.603232  9251 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:17.659073  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.664605  1879 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:17.556926  9026 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.600463  9313 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.659047  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:25:17.660497  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 97165.5us
I20260709 12:25:17.582957  9395 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.665082  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.601867  8853 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:17.657987  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.674916  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.658677  9296 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:17.680982  1875 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
W20260709 12:25:17.688359  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 84549.6us
I20260709 12:25:17.678215  9217 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:17.694039  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:17.694612  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 183892us
I20260709 12:25:17.696542  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:17.705440  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.717140  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.718165  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.718449  9228 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.719208  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.719578  1876 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:17.719937  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:17.720690  9450 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
I20260709 12:25:17.720038  1727 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:17.721436  2382 leader_election.cc:400] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25.
I20260709 12:25:17.719095  1730 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:17.721632  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.722965  9455 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:25:17.723315  9448 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:25:17.723655  9455 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:17.725207  1770 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:17.720954  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.718995  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: true dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:17.714411  9251 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:17.727195  9251 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:17.727427  1728 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:17.727594  9251 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:17.729331  9145 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:25:17.730528  9145 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:25:17.732203  9452 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:25:17.702608  9311 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:17.736846  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:17.737814  1731 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
W20260709 12:25:17.739097  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68988us
I20260709 12:25:17.756363  1879 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:25:17.756549  1801 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:17.758214  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.759074  1729 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:17.681213  9087 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.181s
W20260709 12:25:17.743664  9439 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.222s	user 0.000s	sys 0.002s
I20260709 12:25:17.721823  2382 leader_election.cc:403] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:25:17.746852  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1148) took 2509 ms. Trace:
I20260709 12:25:17.768836  1881 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:17.736253  9251 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.739704  9442 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:17.766507  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.753494  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.771040  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.774235  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.752902  9454 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.775185  1728 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:17.774859  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.772136  1723 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:17.773110  1880 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:17.778769  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.755167  9145 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:17.783595  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:17.781746  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86241.3us
I20260709 12:25:17.782174  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:17.782231  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 88334.3us
I20260709 12:25:17.781260  1806 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:25:17.772560  8961 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:17.786435  9217 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:17.787021  9217 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:17.787524  9217 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:17.787748  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.788084  9395 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.756703  9455 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.774736  9442 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.789469  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.789762  9442 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:17.790817  9179 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:17.791550  9179 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.794988  9362 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.796926  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.797878  1731 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:17.797670  9454 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.798018  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:17.798748  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.799515  1798 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:17.799937  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.800339  9454 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.800522  1725 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:25:17.800674  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:17.801026  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:17.801946  1800 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:17.802345  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:17.787025  9451 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:25:17.803567  8853 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.801630  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:17.804860  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.805580  1724 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
W20260709 12:25:17.807022  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 117713us
I20260709 12:25:17.807188  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:17.797362  9145 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.808415  9459 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:17.808802  9145 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:17.811174  9441 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25.
I20260709 12:25:17.813898  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:17.801272  1798 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:17.816977  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:17.817804  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.799263  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.769553  1732 rpcz_store.cc:276] 0709 12:25:15.237600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.909822 (+1672222us) service_pool.cc:224] Handling call
0709 12:25:16.933445 (+ 23623us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.942519 (+  9074us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.942550 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:16.985295 (+ 42745us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.175611 (+190316us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:17.185679 (+ 10068us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.248525 (+ 62846us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b180034bb80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:25:17.248749 (+   224us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.248828 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.687055 (+438227us) spinlock_profiling.cc:243] Waited 3.39 ms on lock 0x7b180034bb80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:17.746425 (+ 59370us) raft_consensus.cc:1685] finished
0709 12:25:17.746450 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.746621 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":887,"prepare.queue_time_us":10074,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":2170,"raft.queue_time_us":68394,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":30351,"spinlock_wait_cycles":12385280,"thread_start_us":170826,"threads_started":1,"wal-append.queue_time_us":216156}
I20260709 12:25:17.823880  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.824146  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.824983  1729 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:17.825109  1723 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:17.827342  2411 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:17.813781  9458 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:17.833470  9458 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.833879  9458 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:17.834911  1801 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:17.835415  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:17.836191  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:17.836836  1801 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:17.839738  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.840291  9251 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.840504  1725 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:17.840548  2414 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 18. (term mismatch)
I20260709 12:25:17.781471  9077 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:17.799218  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.802659  9313 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.851297  1801 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:17.842588  2414 pending_rounds.cc:85] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 17
I20260709 12:25:17.855122  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.855439  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:17.859273  1803 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: [18.12-20.13]
I20260709 12:25:17.860013  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.859608  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.859524  1802 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:17.864039  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.865216  1806 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:17.855212  2414 pending_rounds.cc:107] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 23.18
I20260709 12:25:17.872447  2414 raft_consensus.cc:2889] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:17.872164  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.873469  1802 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:17.852658  9394 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:17.876062  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:17.864465  9432 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:17.857113  9077 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:17.879673  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.889460  1723 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:17.880352  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:17.786898  8961 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:17.877977  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.899271  1881 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:25:17.883481  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 225832us
I20260709 12:25:17.866403  8989 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:25:17.907114  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.909194  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.801730  9296 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:17.913501  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:17.917423  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.918159  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.918275  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.924316  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:17.925060  1806 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:17.917905  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.933317  9399 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:17.935497  9179 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:17.938104  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.938973  1879 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:17.943154  1875 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:17.942843  9362 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:17.944604  8989 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:17.945097  9315 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:17.881795  9432 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:17.900488  9077 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:17.969132  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:17.949570  1798 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:17.945109  8989 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:17.946717  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.960258  1875 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:25:17.974857  1880 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:25:17.972955  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:17.975873  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:17.980186  1732 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:17.952306  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 124511us
I20260709 12:25:17.934651  9319 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:17.988884  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:17.992204  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 82772.9us
I20260709 12:25:17.993577  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:17.994659  1723 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:17.989872  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:17.999974  1770 leader_election.cc:400] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20.
I20260709 12:25:18.001224  9432 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:18.002593  9432 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:18.027631  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.029345  9442 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:18.029690  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.030313  9300 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:18.040006  9442 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:18.048636  1770 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.041046  9300 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:18.041879  9434 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.601s	user 0.003s	sys 0.000s
W20260709 12:25:18.061829  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 102161us
I20260709 12:25:18.033510  9458 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:18.038259  9464 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:18.074918  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:18.050774  1803 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:18.053323  9457 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:18.051715  9026 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:18.070852  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 128488us
I20260709 12:25:18.077560  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:18.081477  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 1067) took 3564 ms (client timeout 2102 ms). Trace:
W20260709 12:25:18.081990  1804 rpcz_store.cc:269] 0709 12:25:14.509766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.513336 (+  3570us) service_pool.cc:224] Handling call
0709 12:25:18.062635 (+3549299us) spinlock_profiling.cc:243] Waited 2.89 s on lock 0x7b540009fbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:18.071083 (+  8448us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6652072320}
I20260709 12:25:18.083496  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:18.084192  1804 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:18.085086  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:18.085681  1804 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:18.082459  8961 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:18.088824  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:18.090158  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:18.093122  1803 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:18.095495  9158 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:18.026729  9217 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.098902  9455 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:18.100699  9217 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:18.103071  1801 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:25:18.103976  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:18.105240  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:18.105446  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.105898  1726 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:18.106467  9158 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:18.107784  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:18.108242  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.109735  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.078621  9457 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:18.111974  9457 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:18.113168  9457 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:18.108428  1727 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:18.110806  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 312819us
I20260709 12:25:18.116482  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:18.118247  1723 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:18.119074  1732 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:18.122001  1879 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
W20260709 12:25:18.124809  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44552.3us
I20260709 12:25:18.115047  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:18.127357  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3598) took 1976 ms (client timeout 2095 ms). Trace:
I20260709 12:25:18.128116  1726 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:18.127902  1873 rpcz_store.cc:269] 0709 12:25:16.139385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.306145 (+166760us) service_pool.cc:224] Handling call
0709 12:25:18.115349 (+1809204us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b540021b6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:18.115486 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2102167936}
I20260709 12:25:18.135813  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:18.136188  2414 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:18.076187  9464 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:18.138675  9474 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:18.184046  9474 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:18.071313  9394 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:18.185798  9394 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:18.190172  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.192509  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:18.193333  2414 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:18.136626  2416 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:18.147593  1801 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:18.197583  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:18.148630  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:18.151418  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73928us
I20260709 12:25:18.199138  1800 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:18.199568  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:18.152364  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:18.201021  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48793.6us
I20260709 12:25:18.201093  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:18.154404  9464 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:18.153825  9458 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:18.132014  9158 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:18.161445  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 51796.5us
I20260709 12:25:18.163003  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:18.220304  1732 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 16: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:18.221869  2411 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:18.221949  1803 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
W20260709 12:25:18.163450  9445 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.719s	user 0.006s	sys 0.001s
I20260709 12:25:18.164873  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:18.123245  9300 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:18.141927  2412 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:17.706720   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:18.179586  9455 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:25:18.181578  9466 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.250s	user 0.002s	sys 0.002s
I20260709 12:25:18.242750  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:18.243614  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:18.199427  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:18.206916  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:18.161228  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:18.207899  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:18.208033  2414 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:18.218431  1727 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:18.225844  1804 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
W20260709 12:25:18.226908  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 62611us
I20260709 12:25:18.232213  1724 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:25:18.237111  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:18.240897  1723 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:18.242739  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:18.243942  9319 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:25:18.270073  9399 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:18.270746  9319 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:18.271040  9300 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:18.271370  9319 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:18.272480  9300 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:18.272740  9319 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:18.274154  2410 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:18.276429  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 76983.4us
I20260709 12:25:18.278393  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.287946  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.278990  1801 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:18.279151  9463 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:25:18.250092  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:25:18.267079  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64405us
I20260709 12:25:18.295230  1800 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:25:18.282297  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47641.6us
W20260709 12:25:18.270335  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.306177  1725 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:18.280700  1799 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:18.289602  1806 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:18.267401  9313 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:18.213019  9469 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:18.306185  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:18.314150  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 64194us
I20260709 12:25:18.248170  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:18.308243  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.307791  1798 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:25:18.311755  2410 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
W20260709 12:25:18.312291  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 67799.3us
I20260709 12:25:18.267875  9006 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:18.358633  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68436.4us
I20260709 12:25:18.308082  9432 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 12:25:18.362350  9470 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:18.234299  9315 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:18.362888  9399 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:18.413316  1877 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:18.363317  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:18.365375  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.366251  9395 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:18.245042  9460 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.151s
I20260709 12:25:18.369453  9228 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.373634  1802 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
W20260709 12:25:18.375452  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 97131.6us
I20260709 12:25:18.307621  9319 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:18.424180  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 118093us
I20260709 12:25:18.424194  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.315557  1873 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:18.425012  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:18.315975  2414 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
W20260709 12:25:18.409335  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 92353.1us
I20260709 12:25:18.287163  9455 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.128s
I20260709 12:25:18.356709  9311 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.363500  9459 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:18.363375  9379 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:18.414266  1874 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
W20260709 12:25:18.421715  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 136078us
I20260709 12:25:18.432439  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:18.432825  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.404214  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:18.395368  9006 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:18.312986  9077 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.435890  2382 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:18.437027  9006 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:18.437165  9228 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:18.442340  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 65639.2us
I20260709 12:25:18.439725  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:18.430384  9296 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.014s	sys 0.024s
I20260709 12:25:18.450767  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.428619  9395 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:18.434482  9379 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:18.456681  1881 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:18.457636  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.453112  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:18.463871  1725 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
W20260709 12:25:18.448463  9296 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.014s	sys 0.024s
I20260709 12:25:18.441740  9470 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:25:18.451211  9476 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:25:18.443248  9399 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:25:18.474099  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.446552  9228 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:18.475893  9394 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:18.443459  9223 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:25:18.446070  8989 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:18.493255  8989 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:18.465830  9217 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:18.499135  9217 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.499688  9217 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:18.478896  9179 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:18.480342  9296 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:18.480790  9464 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.444954  9442 raft_consensus.cc:2764] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:25:18.450120  9459 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:18.489500  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.455889  9469 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:18.471076  9135 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:18.447003  9311 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.474996  8961 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:18.525897  9464 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:18.530303  9077 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:18.531502  9459 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.533269  1807 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:18.521997  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:18.548677  9217 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.521654  9135 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:18.572367  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 128093us
I20260709 12:25:18.521359  9469 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:18.575906  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86539us
I20260709 12:25:18.579535  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:18.580381  1729 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:18.573858  9394 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:18.584882  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:18.552716  9470 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:18.594157  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 170041us
W20260709 12:25:18.595685  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 142649us
I20260709 12:25:18.595927  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:18.596930  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:18.604585  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.596117  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:18.607252  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 85376.9us
I20260709 12:25:18.607946  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:18.606578  1729 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:18.610580  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:18.613363  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 180949us
I20260709 12:25:18.614993  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:18.616206  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:18.620355  1730 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:18.581722  9135 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:18.581007  9442 raft_consensus.cc:2764] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:25:18.621845  1724 raft_consensus.cc:2368] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25.
I20260709 12:25:18.551574  9395 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:18.582154  9469 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:18.629721  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.614050  9179 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.638048  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.599125  9470 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:18.616603  9449 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:18.641139  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.641999  1877 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:18.646426  1878 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:25:18.650025  1803 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:25:18.651247  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:18.651475  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:18.652271  1801 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:18.653255  1872 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:25:18.654667  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:18.655490  1880 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:18.663350  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:18.628746  9399 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:18.664214  1801 raft_consensus.cc:2368] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24.
I20260709 12:25:18.616127  9494 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:25:18.570290  9379 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.616335  8989 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:25:18.667626  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.671759  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:18.631986  9217 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:18.657459  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:18.692765  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.694111  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.695200  1872 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:18.653260  9470 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:18.704484  1798 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:25:18.706012  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:18.665812  9449 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:18.713032  1806 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:18.712337  1880 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
W20260709 12:25:18.699087  9457 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.585s	user 0.010s	sys 0.003s
I20260709 12:25:18.655640  9179 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:18.726930  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 89530us
I20260709 12:25:18.663600  9319 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:25:18.730108  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.662106  9471 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.223s
W20260709 12:25:18.732084  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 102455us
I20260709 12:25:18.736943  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:18.735525  1880 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:25:18.711899  9217 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:18.743121  9179 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:18.744429  9026 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.629s	user 0.011s	sys 0.001s
I20260709 12:25:18.746655  9319 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:18.747502  9319 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:18.747848  9319 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:18.747948  9490 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:18.749194  9485 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:25:18.748863  9319 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.749778  9453 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:18.750214  9453 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:18.750290  9319 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:18.750713  9453 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:18.767861  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 75223.5us
I20260709 12:25:18.770653  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:18.752847  9493 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.052s	user 0.001s	sys 0.001s
I20260709 12:25:18.782215  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:18.783208  1880 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:18.774451  9311 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:18.751926  9217 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:18.803035  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:18.803732  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67035.7us
I20260709 12:25:18.729923  9473 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
I20260709 12:25:18.767218  9341 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:18.814143  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.707801  9394 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.780148  9490 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:18.793269  9496 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:18.800771  9494 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:18.826462  1880 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:18.807832  9217 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:18.839221  9469 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.839756  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:18.840816  1878 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:18.841975  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.842334  9469 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:18.844791  9503 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:25:18.845376  9503 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:18.846995  9479 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.265s	user 0.000s	sys 0.002s
I20260709 12:25:18.830363  9494 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:18.849208  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.827814  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:18.851084  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.852415  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:18.803004  9489 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:18.848621  9494 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.797005  9458 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.032s
W20260709 12:25:18.832288  9482 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.222s	user 0.000s	sys 0.002s
I20260709 12:25:18.856194  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.846885  9449 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:18.892658  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.893594  9486 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:18.894443  9455 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:18.828446  9496 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:18.895057  9496 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:18.905061  9006 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.907020  9341 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:18.907601  9341 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:18.908051  9341 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:18.906387  9486 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:18.911118  9458 consensus_queue.cc:1243] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 20.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:25:18.916503  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:18.917358  1881 raft_consensus.cc:2376] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24. Re-sending same reply.
I20260709 12:25:18.894018  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.920092  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.921725  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.858467  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.859870  1876 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:18.924840  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:18.925462  1876 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:18.864944  2416 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:25:18.923581  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.928296  1880 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:18.928661  9499 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:18.929853  1880 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:25:18.876994  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:18.931879  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55034.9us
I20260709 12:25:18.932926  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.878379  9311 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:18.846378  9503 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:18.881937  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:18.918532  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:18.935520  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 165061us
I20260709 12:25:18.937724  1730 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:18.941200  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.950058  1881 raft_consensus.cc:2368] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25.
I20260709 12:25:18.950912  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:18.956175  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.972507  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:18.973714  2414 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:18.973970  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:18.990351  1876 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:18.995086  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:18.996582  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.944407  9455 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:19.009222  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:18.944130  9315 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:19.003078  1731 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:19.003757  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:19.007800  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:19.011309  1725 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: [25.11-25.11]
I20260709 12:25:19.024971  1726 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:19.012207  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:18.971585  9457 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:18.968869  9454 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:19.019506  1728 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:25:18.950155  9458 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:18.945822  9488 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:18.993304  9464 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:18.992607  9145 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:18.993803  9432 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:19.042855  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:19.049741  9432 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:19.050321  1726 raft_consensus.cc:2410] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:25:19.050387  9432 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:19.051549  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 61123.5us
W20260709 12:25:19.047582  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 96850us
I20260709 12:25:19.056201  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.058056  9489 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:19.058774  9489 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:19.059840  9489 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.057647  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:19.066114  9492 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:19.066608  9492 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:19.067025  9492 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:19.076426  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.098377  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:19.101558  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.103163  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:19.109709  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:19.027612  9503 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:19.022902  9501 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:19.118945  1725 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:25:19.033536  9158 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:19.141132  1723 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:19.139942  1772 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:19.040370  9311 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:19.144124  1728 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:25:19.156141  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:19.025065  9473 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.100s
I20260709 12:25:19.107264  9494 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:25:19.160881  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 107530us
I20260709 12:25:19.163841  1724 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:19.117362  9395 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:25:19.172269  9502 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:25:19.167975  9006 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:19.176739  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:19.133950  9158 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:19.179838  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:19.181023  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:19.116043  9488 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:19.128367  9501 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:19.185294  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.188557  1803 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:19.190344  1731 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
W20260709 12:25:19.199311  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 124171us
I20260709 12:25:19.200099  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:19.200307  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:19.202469  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:19.203311  1801 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:19.204675  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:19.223253  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:19.223690  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:19.228603  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:19.165169  9449 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.145140  9486 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:25:19.240629  1723 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:19.242313  1725 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:25:19.199779  9300 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:19.256263  1877 raft_consensus.cc:1217] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Deduplicated request from leader. Original: 18.15->[19.16-23.18]   Dedup: 19.16->[21.17-23.18]
I20260709 12:25:19.260607  1877 pending_rounds.cc:85] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 16
I20260709 12:25:19.261086  1877 pending_rounds.cc:107] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 20.17
I20260709 12:25:19.261408  1877 raft_consensus.cc:2889] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:19.200752  9395 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:19.200276  9501 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:19.267778  1801 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:25:19.166851  9311 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.234426  9315 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:19.230173  9499 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:19.222267  9026 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.049s
W20260709 12:25:19.253631  9470 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.522s	user 0.012s	sys 0.000s
W20260709 12:25:19.271602  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48482.9us
I20260709 12:25:19.282119  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:19.293545  1806 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:19.111198  9490 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:19.287993  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 108227us
I20260709 12:25:19.293462  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.294512  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:19.258684  9300 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.301039  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:19.284673  9315 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.260478  9489 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:25:19.312064  1801 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
W20260709 12:25:19.316421  2412 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1485) took 1493 ms (client timeout 1511 ms). Trace:
I20260709 12:25:19.318871  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:19.241148  9453 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.302038  9300 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:19.317143  2412 rpcz_store.cc:269] 0709 12:25:17.816509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.816740 (+   231us) service_pool.cc:224] Handling call
0709 12:25:19.309568 (+1492828us) spinlock_profiling.cc:243] Waited 316 ms on lock 0x7b54004560d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:19.309799 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":727650560}
I20260709 12:25:19.253175  9449 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.272753  9432 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.335523  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:19.337954  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2863) took 1397 ms (client timeout 1743 ms). Trace:
I20260709 12:25:19.340497  1800 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:19.311708  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 88096.9us
I20260709 12:25:19.341614  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.345106  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:19.338675  1875 rpcz_store.cc:269] 0709 12:25:17.940298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.989513 (+ 49215us) service_pool.cc:224] Handling call
0709 12:25:19.337661 (+1348148us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54002dfad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:19.337867 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":722496768}
I20260709 12:25:19.319921  9315 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:19.320674  9474 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:19.379750  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:19.388842  1880 raft_consensus.cc:2368] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24.
I20260709 12:25:19.389065  1801 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:19.380757  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:19.301556  9006 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:25:19.282773  9395 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.330222  9026 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:19.390306  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:19.317354  9447 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.330528  9469 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:19.330757  9145 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:19.298466  9470 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:19.419811  9510 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
I20260709 12:25:19.401177  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:19.421566  9526 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.422945  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:19.423810  1874 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:19.422132  9526 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.425184  1874 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
W20260709 12:25:19.425915  9319 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.663s	user 0.006s	sys 0.003s
I20260709 12:25:19.426890  1732 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
W20260709 12:25:19.405222  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 111857us
W20260709 12:25:19.408504  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114074us
I20260709 12:25:19.412842  2418 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:25:19.411470  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.412881  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:19.419268  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:19.429613  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.426565  1874 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:19.395350  9505 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.387064  9488 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.463764  1875 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
W20260709 12:25:19.468422  9513 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
I20260709 12:25:19.482745  9473 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:19.334883  9492 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.450138  1844 leader_election.cc:400] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25.
W20260709 12:25:19.523279  9476 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
W20260709 12:25:19.465193  9511 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:25:19.382570  9394 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
W20260709 12:25:19.469362  9508 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:25:19.318068  9158 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:19.467581  9507 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
W20260709 12:25:19.469125  9341 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.561s	user 0.012s	sys 0.001s
W20260709 12:25:19.445822  9217 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.611s	user 0.013s	sys 0.000s
W20260709 12:25:19.457270  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45943.1us
I20260709 12:25:19.495527  9519 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.402206  9026 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:19.454620  1880 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:19.414153  9469 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:19.506669  1803 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:19.554214  1872 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:19.500367  9453 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.489776  9395 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.487763  9449 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.410496  9489 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:19.400074  9495 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:19.400169  9515 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:19.448871  9442 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:19.421701  9470 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:19.534720  1844 leader_election.cc:403] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:25:19.568039  9442 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:19.569026  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 121999us
I20260709 12:25:19.421653  9311 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:19.527716  9496 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.632s	user 0.010s	sys 0.003s
I20260709 12:25:19.570590  9442 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.572355  9471 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25.
W20260709 12:25:19.536793  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 107286us
I20260709 12:25:19.572999  9394 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:19.573288  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:19.574333  9470 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:19.574677  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.574863  9470 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:19.575948  9470 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 25.11, Last appended by leader: 11, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.576494  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.577070  9470 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:19.577870  9449 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:19.535194  9145 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:19.550422  1874 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:19.578521  9471 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:19.547394  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:19.579201  9145 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:19.539191  9458 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:19.543443  9464 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:19.580327  9464 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.581851  9458 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.491313  9432 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.416586  9447 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.542580  9473 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:19.564409  9487 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:19.554618  9313 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.572049  9496 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.578974  9453 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:19.578482  9449 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:19.586210  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.549194  1728 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:19.586858  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.559667  1803 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:19.600117  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:19.600296  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.553217  9158 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:19.579551  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:19.557658  9469 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:19.590961  1878 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:19.609751  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2749) took 1471 ms. Trace:
W20260709 12:25:19.583034  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.562469  9517 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:19.554178  9519 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.603338  9533 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:19.530160  9505 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.560788  9489 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:19.549577  9341 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.552973  9217 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.623129  9026 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.641073  9026 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:19.642158  9217 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.596014  9313 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.643090  1806 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:25:19.623221  9494 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:19.612560  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:19.601989  9515 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:19.619062  1872 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:19.616468  1805 rpcz_store.cc:276] 0709 12:25:18.139769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.267190 (+127421us) service_pool.cc:224] Handling call
0709 12:25:18.267697 (+   507us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:18.268232 (+   535us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.268282 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:18.482127 (+213845us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b2c0005c178. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:25:19.248585 (+766458us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b2c0005c178. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:25:19.267681 (+ 19096us) log.cc:844] Serialized 52 byte log entry
0709 12:25:19.448249 (+180568us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:19.448317 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.448487 (+   170us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.609449 (+160962us) raft_consensus.cc:1685] finished
0709 12:25:19.609472 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.609683 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6407,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"spinlock_wait_cycles":42764928,"thread_start_us":138997,"threads_started":1,"wal-append.queue_time_us":250764}
I20260709 12:25:19.622584  1729 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:19.632342  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:19.625459  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:19.608536  9454 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.562s	user 0.009s	sys 0.003s
I20260709 12:25:19.626962  9533 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.644850  1804 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:19.584929  9447 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:19.622972  9503 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:19.650379  1874 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:19.669488  1879 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:25:19.669562  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:19.652998  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:19.653275  1799 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:19.626641  9435 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:19.619705  1696 leader_election.cc:400] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24.
W20260709 12:25:19.645299  9135 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.013s	user 0.015s	sys 0.000s
I20260709 12:25:19.597220  9495 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:25:19.672374  1696 leader_election.cc:403] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:25:19.671069  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:19.672713  1802 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:19.675442  9541 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:25:19.676208  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 90072.9us
I20260709 12:25:19.684170  1872 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:19.672015  9494 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.014s
W20260709 12:25:19.644450  9432 consensus_queue.cc:1175] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 22 status { last_received { term: 19 index: 15 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [21.16-21.16]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:19.631837  9489 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:19.686826  9379 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:19.662734  9319 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.688743  9495 consensus_queue.cc:1243] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 19.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:25:19.691116  9505 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:19.692232  9499 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:19.692678  9499 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:19.692597  9501 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.693053  9315 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.693626  9499 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.694396  1803 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:19.668767  9533 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:19.695154  1799 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:19.695590  1725 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:19.682680  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2712) took 1884 ms. Trace:
W20260709 12:25:19.698071  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 80598.5us
I20260709 12:25:19.698380  1878 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:19.698825  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:19.699357  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.696893  1807 rpcz_store.cc:276] 0709 12:25:17.797873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.042861 (+244988us) service_pool.cc:224] Handling call
0709 12:25:18.043312 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.551065 (+507753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.682366 (+1131301us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b540013a954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:19.682597 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1126269312}
I20260709 12:25:19.699399  1872 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:25:19.704413  1805 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 20 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:25:19.670616  9454 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:25:19.705265  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:19.706136  1801 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: [18.12-20.13]
I20260709 12:25:19.709236  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:19.709921  1873 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:19.704458  9395 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24.
I20260709 12:25:19.714903  9496 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:19.716236  9460 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.716682  9341 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:19.716804  9460 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.717804  9464 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:19.717674  9515 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.718618  9526 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:19.714972  9537 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:25:19.720984  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:19.721589  9459 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:19.721933  9514 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:19.722734  9341 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:19.723147  1728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:19.615139  9449 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.722203  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:19.732517  1628 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 173) took 69454 ms (client timeout 15000 ms). Trace:
I20260709 12:25:19.723999  1728 raft_consensus.cc:606] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 18 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:19.696301  9499 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:19.695855  9135 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.723444  9501 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:19.742901  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:19.744647  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77055.8us
I20260709 12:25:19.743929  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:19.746353  1799 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:19.609972  9473 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:19.688969  9319 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.755657  9145 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:19.758467  9135 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.751538  9541 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:19.753407  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.745981  9526 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.740028  9514 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:19.753466  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:19.747722  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:19.773392  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:19.759637  9474 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:19.732201  9503 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:19.788607  1728 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:19.788434  1873 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:19.790062  1873 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:19.794356  1728 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:19.724354  9505 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:25:19.735204  9395 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:25:19.740446  9464 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:25:19.795477  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:19.767537  9492 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:19.804453  9531 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:25:19.805469  9319 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.806406  1723 pending_rounds.cc:85] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 9
I20260709 12:25:19.807475  9006 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:19.807868  9006 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:19.808692  9006 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.810233  1723 pending_rounds.cc:107] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260709 12:25:19.810775  1723 raft_consensus.cc:2889] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:19.863284  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.766474  9300 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:19.806481  9516 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.352s	user 0.001s	sys 0.000s
I20260709 12:25:19.801396  9514 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:19.911494  1799 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:19.805545  9531 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:19.900967  9135 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.798527  9526 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:19.941430  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.948508  9300 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:19.951328  9460 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:19.933528  9522 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.451s	user 0.000s	sys 0.004s
I20260709 12:25:19.954181  1798 raft_consensus.cc:2368] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23.
I20260709 12:25:19.967754  9135 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:19.968660  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.971971  9526 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.912716  9454 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.928757  9531 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:19.733261  1628 rpcz_store.cc:269] 0709 12:24:10.261502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.305647 (+ 44145us) service_pool.cc:224] Handling call
0709 12:24:11.043084 (+737437us) spinlock_profiling.cc:243] Waited 7.67 ms on lock 0x7b5400002490. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb40b2 00007f16b1c2d759 00007f16b1bc5a76 00007f16b1b96d58 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc
0709 12:25:19.657963 (+68614879us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:15.866472 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:15.903834 (+ 37362us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:15.903915 (+    81us) write_op.cc:435] Acquired schema lock
0709 12:25:15.903944 (+    29us) tablet.cc:662] Decoding operations
0709 12:25:15.905585 (+  1641us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:15.905700 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:15.905741 (+    41us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:25:16.041749 (+136008us) tablet.cc:701] Row locks acquired
0709 12:25:16.041806 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:25:16.042091 (+   285us) write_op.cc:270] Start()
0709 12:25:16.042289 (+   198us) write_op.cc:276] Timestamp: P: 1783599916041969 usec, L: 0
0709 12:25:16.042327 (+    38us) op_driver.cc:348] REPLICATION: starting
0709 12:25:16.043303 (+   976us) log.cc:844] Serialized 4776 byte log entry
0709 12:25:16.220167 (+176864us) op_driver.cc:464] REPLICATION: finished
0709 12:25:16.236119 (+ 15952us) write_op.cc:301] APPLY: starting
0709 12:25:16.236302 (+   183us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:16.822335 (+586033us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:16.863292 (+ 40957us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:17.255645 (+392353us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:17.256158 (+   513us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=24,mrs_lookups=0
0709 12:25:17.256223 (+    65us) write_op.cc:312] APPLY: finished
0709 12:25:17.257152 (+   929us) log.cc:844] Serialized 113 byte log entry
0709 12:25:17.334932 (+ 77780us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:17.361305 (+ 26373us) write_op.cc:454] Released schema lock
0709 12:25:17.361414 (+   109us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:17.362137 (+   723us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:17.362177 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4449940,"reported_tablets":32,"spinlock_wait_cycles":17634560,"child_traces":[["op",{"apply.queue_time_us":15536,"apply.run_cpu_time_us":43734,"apply.run_wall_time_us":1125943,"cfile_cache_hit":50,"cfile_cache_hit_bytes":54335,"num_ops":12,"prepare.queue_time_us":617345,"prepare.run_cpu_time_us":59255,"prepare.run_wall_time_us":347342,"raft.queue_time_us":15622,"raft.run_cpu_time_us":6642,"raft.run_wall_time_us":16074,"replication_time_us":177559,"thread_start_us":574524,"threads_started":5,"wal-append.queue_time_us":290474,"wal-append.run_cpu_time_us":14753,"wal-append.run_wall_time_us":2422274}]]}
I20260709 12:25:19.915071  9145 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.951507  9514 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.957257  9006 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:19.972821  9435 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.974660  9449 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.995474  1874 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:19.978076  9454 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.980630  9145 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:19.989989  9435 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.017568  9435 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:19.999339  9473 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.018643  9473 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.022032  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:20.022981  1872 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
W20260709 12:25:20.026217  9524 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.531s	user 0.002s	sys 0.000s
I20260709 12:25:20.014536  9531 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.997648  9489 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:19.992976  9449 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.040333  9531 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:20.041225  9145 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:20.048310  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 305864us
I20260709 12:25:19.993204  9541 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.049422  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:20.053958  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:20.055437  9526 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:20.056849  9526 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:20.058286  9526 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.060781  9315 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:20.062547  9135 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:20.066560  9473 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:20.067088  9509 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.592s	user 0.002s	sys 0.000s
I20260709 12:25:20.056036  9179 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:20.072702  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:20.073513  1879 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:20.075083  8961 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:20.075517  8961 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:20.076402  8961 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.076887  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2873) took 1102 ms. Trace:
I20260709 12:25:20.077528  1732 rpcz_store.cc:276] 0709 12:25:18.974520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.979649 (+  5129us) service_pool.cc:224] Handling call
0709 12:25:20.076581 (+1096932us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b54002b6bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:20.076798 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":958315264}
I20260709 12:25:20.081919  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:20.082617  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:20.082916  9449 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:20.083063  2414 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 20 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:25:20.084440  9489 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:20.086977  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:20.087477  9471 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:20.087940  9471 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:20.089018  9471 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:20.089650  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:20.091154  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:20.070446  9454 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:20.098430  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:20.099696  1876 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:25:20.107959  2383 leader_election.cc:400] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25.
I20260709 12:25:20.114063  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:20.114158  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:20.115589  1875 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:25:20.128463  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:20.130177  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:20.133818  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:20.138024  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:20.139101  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:20.140558  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:20.113739  9449 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:20.118038  9489 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.141238  1879 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:20.143891  1696 leader_election.cc:400] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24.
I20260709 12:25:20.150821  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:20.153167  1732 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:25:20.153744  1732 pending_rounds.cc:85] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 11
I20260709 12:25:20.154043  1732 pending_rounds.cc:107] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:25:20.154343  1732 raft_consensus.cc:2889] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:20.156024  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.157157  1725 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:20.162959  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:20.138497  9471 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.150537  9503 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.005s
I20260709 12:25:20.108105  9469 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.151489  9026 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:20.142827  9499 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:20.137827  9145 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:20.140990  9540 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:20.142897  9458 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:20.170995  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:20.157917  9319 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:20.157974  9395 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:20.141579  9435 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:25:20.161839  9449 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:20.187533  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:20.157560  9492 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.023s
I20260709 12:25:20.173022  9501 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:25:20.191324  1880 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:20.174661  9470 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:20.162295  9489 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.180214  9515 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:20.179498  9026 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.198313  1728 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:20.191354  9077 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:20.185636  9474 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.203192  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:20.204147  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1189) took 1008 ms. Trace:
W20260709 12:25:20.211161  9554 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:25:20.179144  9145 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.193375  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:20.217818  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:20.233318  2411 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:25:20.240338  2383 leader_election.cc:400] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23.
I20260709 12:25:20.245988  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.200573  1725 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []
I20260709 12:25:20.188009  9395 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:20.260675  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:20.193161  9135 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:20.265393  1731 rpcz_store.cc:276] 0709 12:25:19.196084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.196250 (+   166us) service_pool.cc:224] Handling call
0709 12:25:19.196713 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.197003 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:19.303235 (+106232us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.203454 (+900219us) spinlock_profiling.cc:243] Waited 43.4 ms on lock 0x7b1800389980. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:25:20.203714 (+   260us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.203769 (+    55us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:20.203840 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.203928 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.204073 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":125377,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":3282,"raft.queue_time_us":5775,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"spinlock_wait_cycles":158182912,"thread_start_us":135934,"threads_started":2,"wal-append.queue_time_us":143478}
I20260709 12:25:20.192510  9540 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:20.304059  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:20.261157  1728 pending_rounds.cc:85] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 12
I20260709 12:25:20.303422  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:20.355511  1800 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:20.355538  1728 pending_rounds.cc:107] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:25:20.342031  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:20.356295  1728 raft_consensus.cc:2889] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:20.296330  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:20.342926  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 125230us
I20260709 12:25:20.342993  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.184118  9533 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.265347  2419 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:20.385982  9533 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:20.386420  9520 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.904s	user 0.005s	sys 0.018s
I20260709 12:25:20.350956  9362 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.399123  9546 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:20.399317  9362 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.399856  9546 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.229277  9501 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:20.262652  9432 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.139s
W20260709 12:25:20.405323  9529 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.906s	user 0.002s	sys 0.000s
I20260709 12:25:20.346578  9135 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:20.403306  8989 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:20.400712  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 62962.2us
I20260709 12:25:20.406819  8989 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.405654  9432 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:20.310412  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:20.364575  1724 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:20.364540  1798 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:20.373865  9559 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
I20260709 12:25:20.374087  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.377978  9489 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:20.379607  9540 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.425199  9135 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.380705  9449 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.437523  9501 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:20.381547  9555 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
I20260709 12:25:20.383075  9531 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:20.185508  9458 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.447875  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:20.378684  9561 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:20.448810  1723 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 17: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:20.449319  9561 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:20.449960  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.448809  9458 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:20.388571  9469 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:20.387248  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:20.349478  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 88891.1us
I20260709 12:25:20.392764  9541 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:20.353096  9296 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.352447  9454 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:25:20.395298  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:20.260253  9077 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:20.346414  9514 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:25:20.260995  9026 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:20.235304  9499 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:20.402750  9546 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:20.402774  9519 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:20.354400  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 161107us
I20260709 12:25:20.310359  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:20.401813  9362 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:20.412384  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:20.413741  2415 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:20.196215  9537 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.421149  1770 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:20.492724  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 81911.4us
I20260709 12:25:20.493870  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:20.409145  8989 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:20.421625  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 181400us
I20260709 12:25:20.410043  1801 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:20.422066  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:20.422246  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:20.422420  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.431576  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:20.501919  1805 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:20.344074  9395 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:20.436028  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:20.436591  9505 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:25:20.437484  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.431097  9311 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260709 12:25:20.440378  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:20.450555  9561 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.440378  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:20.509775  1727 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.345264  9145 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:20.462389  1798 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:20.452877  1726 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:20.310972  9474 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.227641  9470 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.471622  2413 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:20.427567  9432 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:20.487980  2419 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.486176  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:20.486066  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 175817us
I20260709 12:25:20.487901  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:20.483240  9490 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.006s	user 0.029s	sys 0.018s
I20260709 12:25:20.479014  9379 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:20.491398  9454 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:20.511696  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:20.476900  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.541312  1723 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:20.502739  9315 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:20.516778  9135 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.519001  9311 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:20.522238  8989 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.478430  9006 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:25:20.497404  9519 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:20.523453  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:20.567454  1724 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:20.526224  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 259058us
I20260709 12:25:20.532833  1807 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:20.535558  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2869) took 1556 ms. Trace:
I20260709 12:25:20.570158  1730 rpcz_store.cc:276] 0709 12:25:18.973357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.973551 (+   194us) service_pool.cc:224] Handling call
0709 12:25:18.973956 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.992541 (+ 18585us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.992578 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:19.005507 (+ 12929us) log.cc:844] Serialized 26 byte log entry
0709 12:25:19.243298 (+237791us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:19.243359 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.243459 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.385744 (+1142285us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:20.504716 (+118972us) raft_consensus.cc:1685] finished
0709 12:25:20.504738 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.530076 (+ 25338us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30802,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":15430,"spinlock_wait_cycles":53920128,"thread_start_us":128552,"threads_started":1,"wal-append.queue_time_us":369904}
I20260709 12:25:20.572427  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.485064  9563 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:20.503235  9541 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:25:20.506708  9487 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 12:25:20.546080  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.540668  9454 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:20.537868  9490 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.066s	user 0.029s	sys 0.018s
I20260709 12:25:20.478188  9296 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.573542  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:20.542563  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56498us
I20260709 12:25:20.538035  9379 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:20.507740  9537 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.555476  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.556530  9362 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:20.512375  9463 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:25:20.583127  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:20.585166  1723 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:20.531417  9077 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.585568  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:20.603603  1726 raft_consensus.cc:606] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:20.604154  1726 raft_consensus.cc:614] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:20.608237  1727 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:20.581995  9454 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.618325  1730 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:20.622153  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:20.581779  9519 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.623553  1807 raft_consensus.cc:2410] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:25:20.623550  1730 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:20.626538  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:20.582067  9546 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.631191  1723 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:20.632171  1727 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:20.566442  8989 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.637336  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:20.605443  9490 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:25:20.643186  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69715us
W20260709 12:25:20.639186  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 165964us
I20260709 12:25:20.716751  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:20.694496  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:20.696277  9574 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.628592  9499 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:20.580029  9563 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.703037  2413 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:20.635293  9077 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:20.704830  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 202363us
I20260709 12:25:20.708230  1727 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:20.719679  9490 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:20.641376  1723 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:25:20.718393  1799 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:20.584339  9432 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:20.731952  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 195110us
I20260709 12:25:20.635066  9540 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:20.734352  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:20.710762  9319 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:20.653606  9435 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.084s
I20260709 12:25:20.635226  9519 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.642093  9546 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.735671  9499 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:20.758311  9584 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:20.755959  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:20.772512  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:20.777586  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:20.778569  1873 raft_consensus.cc:2410] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:25:20.777160  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:20.782058  1877 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:20.699381  9379 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.758937  9463 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:25:20.811223  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:20.804941  1723 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 12:25:20.811945  1723 pending_rounds.cc:85] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 10
I20260709 12:25:20.812366  1723 pending_rounds.cc:107] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:25:20.812785  1723 raft_consensus.cc:2889] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:20.813692  1872 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:20.788115  9470 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:20.788195  9584 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.827020  9296 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:20.770113  9319 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.831214  9135 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:20.832127  9579 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 12:25:20.834403  9546 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:20.835347  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:20.836066  9471 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.662s	user 0.005s	sys 0.007s
I20260709 12:25:20.836190  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2865) took 1491 ms. Trace:
I20260709 12:25:20.768669  9458 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 12:25:20.734017  9574 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:20.836685  1878 rpcz_store.cc:276] 0709 12:25:19.344320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.344457 (+   137us) service_pool.cc:224] Handling call
0709 12:25:20.835986 (+1491529us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54001f7f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:20.836119 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":388449536}
I20260709 12:25:20.840874  1875 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 12:25:20.831434  9562 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.274s	user 0.002s	sys 0.000s
I20260709 12:25:20.836745  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:20.841545  2419 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:20.771315  9487 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:25:20.801505  9474 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:20.856068  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:20.861817  1878 raft_consensus.cc:2410] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:25:20.774446  9573 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.085s
W20260709 12:25:20.849053  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 131499us
I20260709 12:25:20.818960  9379 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:20.918315  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.869472  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:20.919394  1878 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:20.919955  9590 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.920715  9590 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.809499  1799 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:20.882123  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:20.924194  9474 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:20.924605  9474 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.924952  9474 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:20.914155  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48612.9us
I20260709 12:25:20.914877  9473 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:20.916733  9598 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:20.839849  9519 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:20.923550  9590 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:20.927635  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:20.957129  1878 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:25:20.917065  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 314764us
I20260709 12:25:20.837299  9583 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:20.943022  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.836215  9537 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:20.847582  9584 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:20.945564  9587 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.009354  9587 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:20.948203  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:21.011385  9487 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:21.011538  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63438.6us
I20260709 12:25:21.011893  9487 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:21.012538  1772 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:20.927942  9588 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:25:21.014019  1772 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:20.958128  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:21.014937  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.015853  1726 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:21.015762  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.016927  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.017617  1726 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:20.947605  9296 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.836867  9463 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 12:25:20.959973  9598 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.019074  9296 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:20.821774  9490 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.021062  9598 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:21.004341  9563 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.012653  9487 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:21.021540  9296 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:25:21.022312  9296 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:21.023505  9583 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.025972  9583 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:20.837008  9341 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.092s
W20260709 12:25:21.027683  9145 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.001s
I20260709 12:25:21.028131  9145 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.029323  9341 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:21.030318  9145 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:21.030663  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.032023  1724 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:21.032686  1726 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:25:21.033439  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:20.996070  8989 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:20.999631  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:21.036762  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.037304  1724 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:21.040053  1726 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:21.042459  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:21.044252  1878 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:21.045883  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:20.998375  9519 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:21.048271  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48723.3us
I20260709 12:25:20.949548  9533 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.087s
I20260709 12:25:21.020108  9574 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:21.034772  9395 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.509s	user 0.007s	sys 0.004s
I20260709 12:25:21.034318  9473 raft_consensus.cc:2764] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:25:21.045259  9399 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.062161  9587 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:21.070189  1878 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:21.052930  9598 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:21.018460  9454 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:21.031038  9469 consensus_queue.cc:1175] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" responder_term: 17 status { last_received { term: 16 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 16. Current term is 17. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:25:21.070214  9026 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.573s	user 0.012s	sys 0.002s
W20260709 12:25:21.071442  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.053561  9590 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.074486  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:21.070941  9399 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.076997  9587 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:21.077454  9603 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.049325  9296 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:25:20.962416  9499 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.068341  9395 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.089051  9474 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.097738  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.104909  1878 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:21.066473  9519 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.143715  1878 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
W20260709 12:25:21.146669  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49084.1us
I20260709 12:25:21.162014  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.149986  1881 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:21.147317  1723 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:21.109016  9505 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 12:25:21.142805  9489 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:21.108835  9590 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:21.167176  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 121355us
I20260709 12:25:21.107264  9026 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.152242  9296 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:21.164410  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.170698  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.173960  1878 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:21.175599  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.177000  1807 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:25:21.167603  9575 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.463s	user 0.000s	sys 0.002s
I20260709 12:25:21.178987  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.172330  9587 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:21.181205  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.151827  9487 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.182054  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2877) took 1762 ms. Trace:
I20260709 12:25:21.182546  1880 rpcz_store.cc:276] 0709 12:25:19.419840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.556061 (+136221us) service_pool.cc:224] Handling call
0709 12:25:19.556478 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.194261 (+637783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.181725 (+987464us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b5400116f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:21.181943 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1459369856}
I20260709 12:25:21.183288  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:21.183835  9487 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.184059  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.185564  1807 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:21.186013  9584 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.187840  1807 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:21.189225  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:21.191222  8989 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:21.191838  8989 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:21.192812  8989 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.196348  9590 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.134770  9603 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:21.199664  9596 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:25:21.200383  9587 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:21.200872  9515 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:21.201658  9515 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:21.204699  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:21.184855  1880 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:21.205499  1807 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:21.204504  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.207742  1799 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:25:21.210682  9296 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:21.211299  9399 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:21.216823  1798 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:21.217351  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.219311  1805 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:21.220667  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.221401  1801 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:21.223770  9519 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:21.224390  9223 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:21.224852  9223 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.225241  9223 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.170405  9591 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:21.230032  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.231604  9604 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:21.232306  9604 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:21.232931  9536 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.673s	user 0.002s	sys 0.000s
I20260709 12:25:21.233314  9604 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.202242  9587 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.238685  9395 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:21.239518  1807 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:21.240571  9474 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:21.240844  9581 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.426s	user 0.001s	sys 0.000s
I20260709 12:25:21.241336  9591 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.244081  9591 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.246829  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:21.244451  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.248775  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.209440  9590 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.257035  9572 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:21.253780  9223 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:21.211905  9296 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.259145  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.251550  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.267891  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:21.268635  1876 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:21.248545  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.270036  1874 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:21.271400  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.261143  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.261999  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.291003  1880 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:21.261940  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.291530  1807 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:21.292819  1876 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:21.293542  9474 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:21.293780  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.294126  9474 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.294687  2419 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:21.295054  9223 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:25:21.263187  9572 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:21.296682  9474 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.296487  9572 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.282617  9455 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:25:21.297603  9572 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:21.298199  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1496) took 1096 ms. Trace:
I20260709 12:25:21.284876  9394 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 12:25:21.264180  1805 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:21.264739  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.263192  9590 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.299814  1801 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 16.
I20260709 12:25:21.265484  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.300773  1878 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:21.301590  9590 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.250753  9540 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:21.302800  9540 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:21.303791  9540 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.266681  1875 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
W20260709 12:25:21.275866  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 104826us
I20260709 12:25:21.277446  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:21.277774  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:21.307348  1800 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:21.307544  1804 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:21.305253  1880 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:25:21.310600  1880 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:21.317826  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.318523  1876 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:21.306648  1875 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:25:21.323222  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:21.323791  1876 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.280652  8989 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:21.281273  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:21.328951  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47815.1us
W20260709 12:25:21.282164  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 117862us
W20260709 12:25:21.295261  9576 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.598s	user 0.002s	sys 0.001s
I20260709 12:25:21.298625  2414 rpcz_store.cc:276] 0709 12:25:20.201862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.203711 (+  1849us) service_pool.cc:224] Handling call
0709 12:25:20.204002 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.451846 (+247844us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:20.452191 (+   345us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.569869 (+117678us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.569922 (+    53us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:20.569977 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.297965 (+727988us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b540046a354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:21.298010 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.298143 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9118,"prepare.queue_time_us":436,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":9228,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":4743,"spinlock_wait_cycles":569372544,"thread_start_us":108140,"threads_started":1,"wal-append.queue_time_us":108540}
I20260709 12:25:21.305112  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:21.263685  1799 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:21.292641  1807 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:21.340471  2412 raft_consensus.cc:2368] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
I20260709 12:25:21.340411  1807 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:21.325599  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.328087  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1091) took 2143 ms. Trace:
I20260709 12:25:21.336270  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:21.337407  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.344444  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.337214  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:21.338218  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.345757  1804 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:21.345990  2417 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:21.343070  1802 rpcz_store.cc:276] 0709 12:25:19.179831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.190559 (+ 10728us) service_pool.cc:224] Handling call
0709 12:25:19.190918 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:19.684099 (+493181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.323008 (+1638909us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b54002c0954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:21.323267 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1107347200}
I20260709 12:25:21.351454  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:21.354933  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.355556  1876 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:21.352415  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.305109  9540 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:21.356977  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.361610  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.361759  9616 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 12:25:21.363279  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.363950  9615 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.012s
I20260709 12:25:21.364452  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.365127  1875 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:21.322692  9319 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.364239  9313 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.370756  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.374804  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:21.353807  1807 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.368865  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.376204  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:21.370240  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:21.377384  1876 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.376547  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.383827  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:21.354804  9590 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:21.391757  1772 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.389201  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.394832  1876 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:21.396178  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.359987  9394 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:21.396823  1876 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:21.391119  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.398438  9605 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:21.399175  9605 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:21.399616  9605 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:21.402956  9487 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:21.403503  1876 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:21.403736  9514 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:21.404145  9514 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.404490  9514 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:21.410351  1875 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:21.411181  9614 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:25:21.403467  9487 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.409459  1623 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 10 to 17, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 17 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:21.419355  9584 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:21.393919  9313 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:25:21.367324  9223 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:21.325917  9515 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:21.422115  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.392720  9616 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:21.396827  9474 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:25:21.386873  9618 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:25:21.422936  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.370729  9591 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:21.437804  1875 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:25:21.423310  1878 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:21.421604  9399 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:21.466629  9399 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.467258  9399 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:21.468806  9026 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:21.472417  1878 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:25:21.472476  9547 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:21.473431  9547 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:21.433202  9487 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:21.396106  9505 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:21.396613  9537 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:21.481197  9505 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:21.481828  9505 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:21.392976  9590 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:21.483493  9590 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:21.439841  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:21.441107  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45613.6us
I20260709 12:25:21.432749  9587 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:25:21.491464  9587 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:25:21.491881  9587 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:21.443177  9223 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:21.492816  9587 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.494359  9587 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:21.448179  9474 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.495523  9474 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:21.443819  9313 consensus_queue.cc:1243] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:25:21.421849  9394 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.496990  9394 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:21.454427  9591 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.461292  9515 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:21.463464  9569 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:25:21.436787  9616 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:21.499425  9616 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:21.494918  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:21.474444  1878 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:21.501541  1799 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:21.503130  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63433.9us
W20260709 12:25:21.505566  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.506929  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:21.453228  9459 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:21.526328  1878 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:21.518244  9006 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:21.519361  9379 raft_consensus.cc:2764] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:25:21.555019  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.558035  1878 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:21.524188  9455 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:25:21.570523  1799 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:21.525908  9573 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
I20260709 12:25:21.580801  2416 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:25:21.577914  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.494359  9223 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.535611  9459 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:21.531194  9608 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:25:21.529369  9296 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:25:21.592664  1878 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:21.574887  9591 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.474462  9547 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:21.598989  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 596749us
I20260709 12:25:21.600127  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1191) took 2403 ms. Trace:
I20260709 12:25:21.600972  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:21.602355  9531 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:25:21.604704  9621 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:25:21.606200  1729 rpcz_store.cc:276] 0709 12:25:19.196774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.258453 (+ 61679us) service_pool.cc:224] Handling call
0709 12:25:19.260129 (+  1676us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.662169 (+402040us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.599893 (+1937724us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54002d14d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:21.600064 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":826294656}
I20260709 12:25:21.585386  9315 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:21.667932  1729 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:21.669986  1732 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:25:21.674610  1732 pending_rounds.cc:85] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 15
I20260709 12:25:21.670809  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:21.674921  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.675333  1730 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.606333  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.669345  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:21.677081  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.677032  1723 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:21.619019  9470 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:21.674949  1732 pending_rounds.cc:107] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:25:21.691243  1732 raft_consensus.cc:2889] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:21.693543  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.694355  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:21.695116  1725 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.695842  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.696790  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.699378  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:21.700924  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.701736  1732 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:21.703450  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 352038us
I20260709 12:25:21.704276  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:21.686206  1730 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:21.707201  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:21.708706  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:21.709102  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 102832us
I20260709 12:25:21.709524  1730 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:21.684489  9594 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.687752  9315 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.713860  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:21.663084  9604 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.082s
I20260709 12:25:21.607292  9514 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.699357  9503 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.701498  9533 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.722018  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.699419  9179 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.724228  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:21.725127  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.722967  1730 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:21.743073  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:21.730495  1723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:21.743964  1723 raft_consensus.cc:606] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:21.744379  1723 raft_consensus.cc:614] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:21.744444  9614 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:21.733969  1725 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:21.745398  9614 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:21.745565  9584 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:21.746146  9584 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.746527  9614 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.747509  9179 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.737231  2382 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:21.749944  9179 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:21.746546  9584 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:21.751605  9627 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:25:21.753024  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae12891834034d968900107cc58fa821"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:21.753741  1726 raft_consensus.cc:606] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:21.753309  9533 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.756166  9533 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:21.738929  9399 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.761554  9399 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:21.717027  9315 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:21.764334  9458 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:21.665177  1623 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 15 to 17, leader changed from <none> to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 17 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:21.768042  9584 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.768247  1623 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 12 to 19, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 19 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:21.729674  1696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.125s (SENT) [suppressed 18 similar messages]
I20260709 12:25:21.771400  9584 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:21.675565  9595 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.077s
I20260709 12:25:21.690531  9006 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.031s
I20260709 12:25:21.774271  9145 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:21.776271  9503 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.742774  9598 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:21.780140  9514 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:21.720912  9459 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:21.781136  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.739529  9610 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 21.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.042s
I20260709 12:25:21.739475  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:21.762266  9399 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:25:21.798594  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:21.721174  9501 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:21.800386  1804 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:25:21.801694  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:21.805579  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54758.2us
W20260709 12:25:21.807322  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 88891.5us
W20260709 12:25:21.815953  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:21.817278  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 87617.3us
I20260709 12:25:21.818938  1732 raft_consensus.cc:2410] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 25 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:21.820912  1880 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:25:21.762279  9540 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:21.825887  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:21.826640  1875 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:21.802923  9501 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.830204  9501 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:21.796818  9598 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:21.835532  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:21.799868  9470 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:21.845294  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:21.784055  9590 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.796742  9505 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.776993  9605 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.720687  9594 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:21.855445  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:21.825418  9540 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:21.870952  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62707.4us
I20260709 12:25:21.847947  9498 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:21.851671  9598 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:21.839192  9607 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:21.878531  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.878548  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:21.882963  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:21.867619  9574 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:21.879547  9540 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:21.857488  9006 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:25:21.860739  9610 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
I20260709 12:25:21.906229  9594 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:21.931947  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 89938.5us
I20260709 12:25:21.876258  9584 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:25:21.901839  9574 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:21.931450  9547 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:21.949296  9632 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:21.975709  9632 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:21.977053  9315 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.895931  9607 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:21.929379  1623 catalog_manager.cc:5697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 11 to 19, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 19 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:21.953827  9487 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:21.956341  9640 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:25:21.959925  9505 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:22.097608  1875 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:25:22.098361  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3662) took 1138 ms. Trace:
I20260709 12:25:22.098767  1881 rpcz_store.cc:276] 0709 12:25:20.959677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.015466 (+ 55789us) service_pool.cc:224] Handling call
0709 12:25:21.981070 (+965604us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5400205dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:22.098292 (+117222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":301150592}
I20260709 12:25:22.101044  2419 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:22.101986  1873 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:22.107270  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:22.108142  9590 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:22.110020  9632 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.111527  9632 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:22.101926  9395 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:22.112432  9395 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:22.113606  9587 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.601s	user 0.013s	sys 0.005s
I20260709 12:25:22.113452  9395 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.115008  9395 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:22.115087  9605 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:22.115657  1807 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:22.110531  1623 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 9 to 17, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 17 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:22.118654  1730 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:22.083212  9616 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.583s	user 0.007s	sys 0.005s
W20260709 12:25:22.119724  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 219581us
I20260709 12:25:22.120498  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:22.103288  9594 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:22.118418  9394 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.621s	user 0.005s	sys 0.016s
I20260709 12:25:22.129607  1806 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:25:22.132334  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:22.122020  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 251381us
I20260709 12:25:22.149350  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:22.150271  1804 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:22.156344  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:22.181352  1807 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:22.183840  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:22.191287  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:22.191787  1804 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:22.192219  1729 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:22.169401  9505 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:22.200852  1723 pending_rounds.cc:85] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
I20260709 12:25:22.201252  1723 pending_rounds.cc:107] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:25:22.201519  1723 raft_consensus.cc:2889] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:22.209568  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:22.210561  1728 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:22.143415  9590 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
W20260709 12:25:22.196168  9311 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 20 current term: 20 is higher.
I20260709 12:25:22.160367  9394 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.139282  9616 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.196251  9583 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:22.234308  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.238276  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.152892  9574 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.203619  9561 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:22.241413  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.234328  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58468.1us
I20260709 12:25:22.200989  9505 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:22.178954  9594 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.214963  9644 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:22.194176  9634 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 21.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.043s
I20260709 12:25:22.240535  9541 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.306393  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:22.307011  1799 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:25:22.308187  1803 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 12:25:22.308266  1799 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:25:22.309497  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:22.309706  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:22.309962  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:22.310818  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:22.309515  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:22.310878  1801 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:22.310871  9512 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.966s	user 0.030s	sys 0.023s
I20260709 12:25:22.310714  1800 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:22.327205  1729 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:22.327911  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:22.328919  1729 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:22.257570  9561 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.336506  1728 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:25:22.243688  9583 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.333675  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.261744  9505 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:22.262041  9540 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.340520  1803 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:22.341099  1801 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:22.339685  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:22.343559  1800 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260709 12:25:22.332283  9512 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.128s	user 0.031s	sys 0.024s
I20260709 12:25:22.323786  9315 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:22.324970  9487 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:22.275488  9541 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.269111  9501 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.362609  9540 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:22.352316  1729 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.337682  9610 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:22.408535  1729 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:22.354970  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.370358  9590 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:22.407241  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:22.428237  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:22.431891  1728 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:22.367795  9474 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.872s	user 0.011s	sys 0.003s
W20260709 12:25:22.408663  9649 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:25:22.364375  9594 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:22.349622  9642 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:25:22.406625  9646 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:25:22.365029  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:22.427287  9636 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.267s	user 0.001s	sys 0.001s
I20260709 12:25:22.363236  9541 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:22.364038  9561 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:22.441278  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:22.460392  1800 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:22.365257  9394 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:22.467068  1728 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:22.366202  9643 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.261s
I20260709 12:25:22.427436  9610 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.501863  1772 leader_election.cc:400] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
I20260709 12:25:22.503332  1772 leader_election.cc:403] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:25:22.439152  9474 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.459870  9458 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.437511  9590 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:22.459955  9594 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:22.510545  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.517809  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.511075  9607 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:22.529726  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.549147  1730 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:22.509590  9313 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
W20260709 12:25:22.555508  9655 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.118s	user 0.000s	sys 0.001s
W20260709 12:25:22.560119  9650 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 12:25:22.512228  9458 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.575240  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.579653  1725 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:22.583523  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.584817  1730 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 27: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:22.586040  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:22.587502  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.592109  9603 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:22.601924  9498 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.718s	user 0.006s	sys 0.005s
I20260709 12:25:22.605770  9503 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:22.542093  9594 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.610172  1730 raft_consensus.cc:2368] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
W20260709 12:25:22.613217  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.622200  1729 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:25:22.569018  9583 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:22.663823  9659 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.664387  9659 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.673437  9659 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:22.631862  9592 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 23 current term: 23 is higher.
I20260709 12:25:22.636628  1770 leader_election.cc:400] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
W20260709 12:25:22.683537  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47065.4us
I20260709 12:25:22.684556  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.637760  9006 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:22.638237  9615 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.639964  9435 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.611565  9591 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.611379  9587 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.648061  9474 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:22.649834  9458 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:22.614751  9590 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 25.11, Last appended by leader: 11, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.677601  9447 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:22.677186  9601 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:22.653680  9026 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:22.655381  9610 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:22.655484  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:22.612298  9598 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.719s	user 0.010s	sys 0.003s
I20260709 12:25:22.660696  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:22.663182  9632 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.551s	user 0.004s	sys 0.008s
I20260709 12:25:22.492870  1623 catalog_manager.cc:5697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 9 to 15, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 15 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:22.621229  9501 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:22.627313  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.670377  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.680509  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:22.777765  1726 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:22.685609  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.623209  9498 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:22.743237  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:22.686506  9006 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:22.778828  2414 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:22.676195  9594 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:22.778959  1732 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:22.785836  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:22.909354  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:22.908147  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:22.910389  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:22.890853  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:22.802546  1726 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:22.914614  1880 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:22.821779  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.674680  8961 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:22.797278  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:22.921517  2412 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:22.892719  1807 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:22.890094  9568 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:25:22.750174  9601 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:22.924579  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:22.924705  9601 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.925844  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:22.926484  9601 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:22.818950  9660 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.108s
I20260709 12:25:22.891347  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:22.780179  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:22.928733  2411 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:22.929024  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:22.929621  1799 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:22.915602  1805 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:22.930186  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:22.809751  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:22.931046  1732 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 27: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:22.888819  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.749619  2419 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:22.794896  9505 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.918001  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:22.955937  1805 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:25:22.918301  1803 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:25:22.956465  1726 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:22.919106  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:22.957078  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:22.919396  9006 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:22.796583  9395 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.668s	user 0.009s	sys 0.004s
I20260709 12:25:22.748894  9447 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:22.744719  9435 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.796765  2417 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:22.923369  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:22.973735  1807 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:22.974093  1732 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
W20260709 12:25:22.859107  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.797883  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:22.810824  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:22.820942  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:22.823051  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:22.963428  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:22.931320  2415 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:22.975740  1804 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:22.964082  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 275653us
I20260709 12:25:22.966090  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:22.778451  9494 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.975453  1729 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:22.977267  1878 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:22.778020  9026 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:22.981779  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 160912us
I20260709 12:25:22.987703  1730 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:22.989962  1726 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:22.992789  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:23.003415  1726 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:25:23.003868  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:23.004614  1803 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:22.747035  9587 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.998207  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:23.007428  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:22.997125  1880 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:23.005928  1726 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:25:22.972702  9519 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:23.011792  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:22.887303  9512 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.108s
I20260709 12:25:23.019862  2414 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:23.013350  1881 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:22.744580  9615 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.910657  9634 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.087s
I20260709 12:25:23.022967  2411 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:23.003567  9494 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.036772  9587 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:22.995995  9296 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:25:23.037961  9494 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.002732  9574 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:23.020562  1801 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:22.983563  9583 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.747242  9591 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:22.790401  9598 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:22.991716  9470 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.032968  2419 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.021023  9006 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.021056  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:23.043582  1803 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:23.045030  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:23.046310  1881 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:23.002481  9077 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:23.006896  9642 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.018316  1799 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:23.029594  9630 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:23.006460  9568 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:25:23.067282  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:23.033095  9651 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:23.071471  1881 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:25:23.012588  9561 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.053s
I20260709 12:25:23.071945  1878 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:23.047564  9572 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:23.040548  9634 consensus_queue.cc:1243] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:25:23.080730  1880 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:25:23.068058  2411 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:23.081681  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:23.090988  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:23.069132  9574 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:23.093529  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:23.096194  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 406050us
I20260709 12:25:23.099794  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:22.812044  1623 catalog_manager.cc:5697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 13 to 16, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 16 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:23.079188  9541 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.115533  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:23.089624  9498 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:23.139281  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:23.065668  9583 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.073784  9642 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.094820  9449 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:23.094442  9582 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:23.052815  9447 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 21.18, Last appended by leader: 18, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.143810  1726 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:23.073709  9077 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.139415  9395 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:23.106830  9458 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:25:23.146235  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46529.4us
W20260709 12:25:23.169610  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 195942us
I20260709 12:25:23.069046  9470 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:23.094517  9394 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.061s
I20260709 12:25:23.154731  9603 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:23.076536  9630 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.114511  2419 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:23.154083  9594 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:23.161620  9582 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.156466  9296 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.109916  9568 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:25:23.175873  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:23.190811  9594 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.197324  1726 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:23.197919  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:23.195731  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:23.205427  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:23.177883  9447 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:23.227836  9662 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:25:23.230013  9077 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:23.231231  1723 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:23.177336  9449 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:25:23.233866  1723 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:23.234993  1623 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 15 to 17. New cstate: current_term: 17 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:23.241201  9598 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:23.169389  9489 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:25:23.187650  9458 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:23.242568  9533 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:23.243114  9533 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:23.243688  9458 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.244040  9533 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.245550  9533 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:23.246044  9490 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:23.184427  9505 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:23.247208  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71434us
I20260709 12:25:23.198925  9574 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.248950  9642 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:23.249431  9675 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:23.249972  9675 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:23.179315  9395 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.252697  9395 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:23.250864  9675 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.253552  9591 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:23.201081  9582 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:23.254429  9470 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:23.254976  1874 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 16.
I20260709 12:25:23.147295  9541 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.207022  9594 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:23.199628  9603 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:23.258476  9603 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.260993  9026 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.262408  9026 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.262646  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1214) took 1420 ms. Trace:
I20260709 12:25:23.263126  1731 rpcz_store.cc:276] 0709 12:25:21.842588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.842978 (+   390us) service_pool.cc:224] Handling call
0709 12:25:22.669862 (+826884us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:22.670201 (+   339us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:22.670227 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:22.670731 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.054375 (+383644us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:23.054434 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.054677 (+   243us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.260706 (+206029us) raft_consensus.cc:1685] finished
0709 12:25:23.260728 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.262379 (+  1651us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54000d2ad0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:23.262577 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1202,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":88,"spinlock_wait_cycles":1900459392,"thread_start_us":383198,"threads_started":1,"wal-append.queue_time_us":588258}
I20260709 12:25:23.213366  9498 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:23.214385  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:23.265057  9641 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.125s	user 0.000s	sys 0.003s
I20260709 12:25:23.209156  9296 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:23.265499  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 51201us
I20260709 12:25:23.266235  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:23.267452  9296 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.268502  8961 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:25:23.244009  9665 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:25:23.269052  8961 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:23.217626  9615 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:23.270566  1732 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
W20260709 12:25:23.223413  9593 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 2.089s	user 0.011s	sys 0.002s
I20260709 12:25:23.224217  9435 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:23.256618  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3669) took 2171 ms (client timeout 2829 ms). Trace:
W20260709 12:25:23.213032  9661 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
I20260709 12:25:23.273069  1803 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:23.281138  9223 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:23.283541  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.287009  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.302721  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1116) took 1554 ms. Trace:
I20260709 12:25:23.305253  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:23.289640  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:23.277274  1874 rpcz_store.cc:269] 0709 12:25:21.084944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.271021 (+186077us) service_pool.cc:224] Handling call
0709 12:25:23.256439 (+1985418us) spinlock_profiling.cc:243] Waited 1.98 s on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:23.256560 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4559968000}
I20260709 12:25:23.303545  1798 rpcz_store.cc:276] 0709 12:25:21.748564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.784699 (+ 36135us) service_pool.cc:224] Handling call
0709 12:25:21.785102 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.796720 (+ 11618us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.796753 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:21.803434 (+  6681us) log.cc:844] Serialized 26 byte log entry
0709 12:25:21.881396 (+ 77962us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:21.881456 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.881686 (+   230us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.264432 (+1382746us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:23.270457 (+  6025us) raft_consensus.cc:1685] finished
0709 12:25:23.270479 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.302617 (+ 32138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11428,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":13042,"spinlock_wait_cycles":27718400,"thread_start_us":30172,"threads_started":1,"wal-append.queue_time_us":250146}
I20260709 12:25:23.316871  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:23.318197  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:23.319752  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:23.320585  1724 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:23.264940  9498 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:23.250703  9630 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:23.333323  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.336664  1803 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:23.276696  9590 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:23.193765  9561 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.120s
I20260709 12:25:23.310014  9642 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:23.269982  8961 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.341445  2419 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:23.237283  1623 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:23.348446  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:23.247594  9677 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.088s
I20260709 12:25:23.350864  9677 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:23.351289  9677 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:23.352332  9677 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.353453  2411 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:23.353935  9677 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:23.354415  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.312559  9300 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:23.339656  1798 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:25:23.361470  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.321771  9659 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:23.365095  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:23.339480  9590 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.316602  9223 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.368868  9642 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:23.417251  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.386595  9341 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:23.397225  9505 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:23.427136  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.429560  1801 raft_consensus.cc:2410] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:25:23.401026  9590 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:23.405236  9519 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.379619  9300 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.402711  9591 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:23.386087  9659 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.432155  9505 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.433035  9470 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.456339  1799 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:23.425217  9341 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.453477  9505 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.474817  1725 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:23.475306  1725 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:23.481089  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:23.481909  1731 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
W20260709 12:25:23.482432  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.472323  9583 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.447387  9591 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.485988  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:23.348497  1623 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 13 to 16, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 16 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:23.488152  9223 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.489607  1623 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:25:23.452605  9498 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.502310  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:23.492950  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:23.476107  1725 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:23.458727  9470 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:23.494030  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.486140  9582 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.510924  1725 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:23.513943  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:23.502403  1799 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:23.442663  9519 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.516592  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2751) took 2013 ms. Trace:
W20260709 12:25:23.492884  9601 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.566s	user 0.014s	sys 0.004s
I20260709 12:25:23.520409  1802 rpcz_store.cc:276] 0709 12:25:21.503257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.503387 (+   130us) service_pool.cc:224] Handling call
0709 12:25:21.518836 (+ 15449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.523970 (+  5134us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:21.568700 (+ 44730us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.242641 (+1673941us) spinlock_profiling.cc:243] Waited 3.37 ms on lock 0x7b180035d4c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:25:23.390125 (+147484us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.401553 (+ 11428us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.401732 (+   179us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:23.401778 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.432798 (+ 31020us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.516219 (+ 83421us) raft_consensus.cc:1685] finished
0709 12:25:23.516312 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.516518 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4576,"prepare.queue_time_us":30636,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":2879,"raft.queue_time_us":14863,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":5615,"spinlock_wait_cycles":7747328,"thread_start_us":147152,"threads_started":1,"wal-append.queue_time_us":1872695}
I20260709 12:25:23.517938  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:23.529983  9470 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.535543  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:23.536659  1874 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
W20260709 12:25:23.546196  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44007.5us
I20260709 12:25:23.547284  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:23.548254  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:23.513242  9680 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.509871  9341 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:23.555061  1725 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260709 12:25:23.555814  1725 pending_rounds.cc:85] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 12
I20260709 12:25:23.556085  1725 pending_rounds.cc:107] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260709 12:25:23.556365  1725 raft_consensus.cc:2889] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:23.553735  9514 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:25:23.557986  9683 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:25:23.561800  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:23.468310  9642 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.592165  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:23.578670  9223 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:25:23.584316  9676 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.215s	user 0.001s	sys 0.001s
I20260709 12:25:23.611255  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:23.593637  9487 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:23.586516  9296 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:25:23.576272  9680 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.605325  9684 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:25:23.645218  9679 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:25:23.629604  9514 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:25:23.637718  9697 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:23.635874  9223 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.026s
W20260709 12:25:23.656261  9651 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.574s	user 0.020s	sys 0.000s
I20260709 12:25:23.649915  9659 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.654767  9300 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:23.667443  9684 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:23.695375  9519 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.700176  9680 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:23.708846  9694 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
W20260709 12:25:23.710721  9696 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:25:23.729068  9594 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.730713  9582 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:23.754730  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.755470  1805 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:23.731909  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:23.758042  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3751) took 1178 ms. Trace:
I20260709 12:25:23.758312  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.759590  1803 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:25:23.761987  9497 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:23.762450  9497 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:23.764700  9505 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.733562  9077 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:23.735837  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:23.769013  1725 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:23.770040  1696 connection.cc:256] RPC call timeout handler was delayed by -1.7099229729997205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1574s for a response
I20260709 12:25:23.673503  9697 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.683518  9341 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:23.758472  1732 rpcz_store.cc:276] 0709 12:25:22.579618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.975897 (+396279us) service_pool.cc:224] Handling call
0709 12:25:23.270386 (+294489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.275736 (+  5350us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.757827 (+482091us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b5400435dd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:23.757977 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":676615552}
I20260709 12:25:23.739172  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:23.739948  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:23.741567  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:23.742249  1802 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 11. Preceding OpId from leader: term: 27 index: 12. (index mismatch)
W20260709 12:25:23.744861  9648 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.433s	user 0.003s	sys 0.000s
I20260709 12:25:23.748083  1804 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:25:23.723317  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3680) took 2116 ms. Trace:
I20260709 12:25:23.752722  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.747572  9684 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.774430  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:23.756377  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:23.775162  1731 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:23.770542  1696 connection.cc:256] RPC call timeout handler was delayed by -1.6946229729996958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15910000000000002s for a response
I20260709 12:25:23.776191  1728 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:23.778453  1798 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 17: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:23.688547  9651 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:23.777923  1872 rpcz_store.cc:276] 0709 12:25:21.606972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.607364 (+   392us) service_pool.cc:224] Handling call
0709 12:25:21.607759 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.608297 (+   538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:21.608329 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.268711 (+1660382us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b180038ba20. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a
0709 12:25:23.269142 (+   431us) spinlock_profiling.cc:243] Waited 7.94 ms on lock 0x7b2c00009c38. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:25:23.269986 (+   844us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.450698 (+180712us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:23.450902 (+   204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.450987 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.723025 (+272038us) raft_consensus.cc:1685] finished
0709 12:25:23.723052 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.723234 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7212,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":48597,"spinlock_wait_cycles":22583936,"thread_start_us":180236,"threads_started":1,"wal-append.queue_time_us":342001}
I20260709 12:25:23.773837  9077 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.790385  9077 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.791705  9594 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.795774  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:23.795779  1725 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:23.798676  9505 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:23.773808  9341 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:23.800361  9573 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:23.800067  9341 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:23.717944  9590 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.808100  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:23.804605  1879 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:23.918947  1725 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:25:23.841275  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:23.878662  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.810161  9454 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:23.826598  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:23.923506  1724 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:23.773918  9697 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:23.782086  1696 connection.cc:256] RPC call timeout handler was delayed by -1.6946229729996958s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15910000000000002s for a response
I20260709 12:25:23.832800  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:23.926934  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.927327  1803 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:23.927513  1696 connection.cc:256] RPC call timeout handler was delayed by -1.6343229729995983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1658s for a response
W20260709 12:25:23.928083  1696 connection.cc:256] RPC call timeout handler was delayed by -1.6073229729995546s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1688s for a response
W20260709 12:25:23.928587  1696 connection.cc:256] RPC call timeout handler was delayed by -1.5668229729999439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1733s for a response
W20260709 12:25:23.929024  1696 connection.cc:256] RPC call timeout handler was delayed by -1.5668229729999439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1733s for a response
W20260709 12:25:23.929687  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.929818  1725 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:23.800774  9573 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:23.939998  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4957229729998289s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
W20260709 12:25:23.942086  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4957229729998289s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18120000000000003s for a response
W20260709 12:25:23.943602  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4066229729996849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19110000000000002s for a response
W20260709 12:25:23.945425  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2527229729998908s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
I20260709 12:25:23.907511  9077 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.868839  9487 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:23.833552  9497 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:23.805718  9341 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:24.046254  1803 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:24.020628  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:24.052873  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2527229729998908s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2082s for a response
W20260709 12:25:24.022300  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:24.040572  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.919019  9158 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:24.069491  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0718229729995983s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2283s for a response
I20260709 12:25:24.073246  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:24.023747  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:24.074316  1801 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:24.070209  1728 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 19
W20260709 12:25:23.992122  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 304009us
I20260709 12:25:24.078673  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:24.034026  9432 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:24.082050  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:24.087088  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1504) took 2802 ms. Trace:
I20260709 12:25:24.019714  9590 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:24.071046  9594 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.089067  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.088442  1881 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:24.093016  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.097378  1804 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:24.097409  1801 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:24.094154  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.099063  1878 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 15.
I20260709 12:25:24.087638  2416 rpcz_store.cc:276] 0709 12:25:21.284871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.285041 (+   170us) service_pool.cc:224] Handling call
0709 12:25:21.285417 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:21.584105 (+298688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.086769 (+2502664us) spinlock_profiling.cc:243] Waited 293 ms on lock 0x7b54003fc854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:24.087026 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":674925952}
I20260709 12:25:24.086017  9158 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.085075  9608 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:24.110808  9501 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:24.084874  9395 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:24.110671  9608 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.015342  9573 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.114110  9573 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:24.114774  9541 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.115970  1728 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:24.118585  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2901) took 2276 ms. Trace:
I20260709 12:25:24.066449  9077 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.119041  1875 rpcz_store.cc:276] 0709 12:25:21.842088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.884999 (+ 42911us) service_pool.cc:224] Handling call
0709 12:25:22.097289 (+212290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:22.100591 (+  3302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.118362 (+2017771us) spinlock_profiling.cc:243] Waited 212 ms on lock 0x7b5400205dd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:24.118517 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":487398656}
I20260709 12:25:24.107122  1880 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:24.088569  9659 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:24.132895  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:24.134066  1875 raft_consensus.cc:2368] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20.
I20260709 12:25:24.134757  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.135838  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.141396  1803 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:24.139933  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.144764  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.148056  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.155941  9713 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:25:24.156350  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68961.9us
I20260709 12:25:24.176266  9713 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:24.167655  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 3752) took 1550 ms (client timeout 2008 ms). Trace:
I20260709 12:25:24.157569  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:24.177702  1730 rpcz_store.cc:269] 0709 12:25:22.613024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.618332 (+  5308us) service_pool.cc:224] Handling call
0709 12:25:24.163550 (+1545218us) spinlock_profiling.cc:243] Waited 302 ms on lock 0x7b54002a9c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:24.163732 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":695417088}
I20260709 12:25:24.178885  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:24.120931  9077 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:24.183326  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.180311  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.130601  9582 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:24.182181  9684 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.158470  9394 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.181677  9145 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.132599  9490 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:24.106498  9594 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.199795  1875 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:25:24.222982  1875 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:25:24.145213  9546 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.147526  9616 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:24.187790  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:24.161173  9498 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:24.180436  9590 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.223587  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:24.237291  1730 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 26 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:25:24.189170  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.238409  1877 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:24.239833  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:24.233569  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56585.7us
I20260709 12:25:24.237030  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:24.241545  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:24.245016  1880 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:24.252396  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.253768  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:24.264844  1723 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bb48ef23a7434834929994987a36f360"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:24.265599  1723 raft_consensus.cc:606] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:24.265965  1723 raft_consensus.cc:614] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:24.213790  9582 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.269704  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.224367  9659 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.049s
W20260709 12:25:24.306380  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.307683  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:24.309049  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.288508  9594 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:24.311484  9594 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:25:24.289491  9710 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:25:24.290023  9315 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:24.313925  9315 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.314404  9315 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:24.224602  9638 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 12:25:24.316184  9608 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.226629  9394 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.320564  9394 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.231501  9145 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.322364   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:25:24.322530  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.323341  9145 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:24.324019  1878 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:24.324544   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:24.325397  9582 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:24.233310  9490 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.327788  9490 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.328665  9697 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.274962  9587 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:24.329581  9587 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.329939  9587 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:24.332267  1877 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:24.335234  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.336964  1880 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:24.338379  1880 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:25:24.339808  1880 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:24.341034  1880 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:24.342144  1880 raft_consensus.cc:1217] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Deduplicated request from leader. Original: 14.10->[16.11-21.13]   Dedup: 16.11->[19.12-21.13]
I20260709 12:25:24.236943  9546 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.218881  9684 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.268287  9713 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.294072  9592 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:24.260309  9590 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.243822  9498 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.295724  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1118) took 2508 ms. Trace:
I20260709 12:25:24.300947  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.379141  1876 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:24.381238  1876 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:24.301903  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2769) took 1014 ms. Trace:
I20260709 12:25:24.383597  1730 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:24.302525  1803 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:24.303578  1801 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:25:24.305941  9659 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:25:24.310847  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2889) took 3561 ms (client timeout 1812 ms). Trace:
I20260709 12:25:24.321276  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:24.270293  9395 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:24.292407  9651 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:24.356395  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.342631  1880 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [16.11-21.13]
I20260709 12:25:24.290570  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.377889  1806 rpcz_store.cc:276] 0709 12:25:21.787677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.787865 (+   188us) service_pool.cc:224] Handling call
0709 12:25:21.788286 (+   421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:22.153676 (+365390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.295447 (+2141771us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54001d5954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:24.295658 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":775683456}
W20260709 12:25:24.385938  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64734.9us
I20260709 12:25:24.385303  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.382244  1800 rpcz_store.cc:276] 0709 12:25:23.287978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.288138 (+   160us) service_pool.cc:224] Handling call
0709 12:25:23.288547 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.289031 (+   484us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:23.338923 (+ 49892us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.457179 (+118256us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.461015 (+  3836us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.461232 (+   217us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:23.461285 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.461392 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.281187 (+819795us) spinlock_profiling.cc:243] Waited 32.3 ms on lock 0x7b24002b7888. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:25:24.301520 (+ 20333us) raft_consensus.cc:1685] finished
0709 12:25:24.301643 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.301826 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":83894,"prepare.queue_time_us":56865,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":42,"raft.queue_time_us":5469,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":10449,"spinlock_wait_cycles":74898432,"thread_start_us":116036,"threads_started":1,"wal-append.queue_time_us":223027}
W20260709 12:25:24.385474  1879 rpcz_store.cc:269] 0709 12:25:20.749500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.749616 (+   116us) service_pool.cc:224] Handling call
0709 12:25:24.310598 (+3560982us) spinlock_profiling.cc:243] Waited 2.94 s on lock 0x7b540001efd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:24.310729 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6769846144}
I20260709 12:25:24.396405  1805 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
W20260709 12:25:24.401331  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 110846us
I20260709 12:25:24.407460  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:24.410102  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 207178us
I20260709 12:25:24.377247  9722 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:24.392153  9713 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:24.395187  9026 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:24.422391  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:24.412149  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.388675  9582 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:24.377142  9573 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:25:24.385766  9494 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:24.409242  9645 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:25:24.411350  9616 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:25:24.385301  9315 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.389319  9677 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:25:24.398025  9463 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:25:24.391227  9341 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 12:25:24.380882  9608 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:25:24.432013  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.476881  1803 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:24.421809  9722 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:24.447527  1807 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:24.426645  9379 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:24.451895  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:24.463198  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.448928  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.422797  9026 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.453127  9616 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:24.479243  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75659.5us
I20260709 12:25:24.451501  9582 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.426857  9498 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.528975  9724 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:24.532502  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:24.529621  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77796.2us
I20260709 12:25:24.405565  9592 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:24.485702  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.544636  1803 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:25:24.546307  1801 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:24.534309  9724 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:24.552672  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:24.464815  9687 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.870s	user 0.007s	sys 0.010s
I20260709 12:25:24.463356  9490 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:24.556916  9700 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.920s	user 0.000s	sys 0.008s
I20260709 12:25:24.464044  9463 consensus_queue.cc:1243] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:25:24.483572  9590 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.513511  9645 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:24.536041  9616 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:24.578157  1799 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:25:24.462857  9158 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:24.531595  9026 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:24.445953  9684 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.482928  9315 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.557301  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:24.547636  9697 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:24.531356  9379 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:24.549467  9592 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:24.552309  9607 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.593380  9546 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:24.587486  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:24.551337  9724 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:24.554702  9223 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 12:25:24.561869  9432 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:24.452569  9494 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.555353  9498 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:25:24.558034  9541 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:24.607470  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:24.638377  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.555224  9608 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:25:24.597290  9158 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:24.597349  9490 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.482347  9395 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:24.634577  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47241.2us
I20260709 12:25:24.600564  9379 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:24.640004  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.655966  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.658598  9490 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:24.644135  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 237517us
I20260709 12:25:24.609747  9501 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:24.581800  9645 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.601292  9697 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:24.614226  9432 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.661691  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.638933  9546 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:24.666282  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 400942us
I20260709 12:25:24.637482  9684 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.639844  9158 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.687022  9158 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:24.667001  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.688784   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:24.687059  9697 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.691592  9582 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.692160  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3755) took 2078 ms. Trace:
I20260709 12:25:24.692397  9645 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:24.692668  1729 rpcz_store.cc:276] 0709 12:25:22.613827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.984007 (+370180us) service_pool.cc:224] Handling call
0709 12:25:23.011498 (+ 27491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.011872 (+   374us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.011907 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.012376 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.246492 (+1234116us) spinlock_profiling.cc:243] Waited 7.17 ms on lock 0x7b18002281c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:25:24.549755 (+303263us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:24.549823 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.549927 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.691830 (+141903us) raft_consensus.cc:1685] finished
0709 12:25:24.691853 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.692088 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13914,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":6842,"spinlock_wait_cycles":16498432,"thread_start_us":302275,"threads_started":1,"wal-append.queue_time_us":1613171}
I20260709 12:25:24.665259  8961 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:24.670187  9498 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:24.672477  9313 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:24.673107  9684 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.655243  9459 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.606726  9607 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.655468  9663 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:24.668337  9432 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:24.679661  9546 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:24.682689  1798 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 14. (term mismatch)
I20260709 12:25:24.597710  9722 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.724792  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.685398  9490 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:24.685601  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:24.694947  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:24.696194  1727 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:24.725582  2416 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:24.692806  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:24.702894  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:24.701907  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 293594us
I20260709 12:25:24.712128  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:24.699877  9006 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.716526  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.733403  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.718170  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:24.739835  1803 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:24.701119  9561 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:24.683254  9463 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.668017  9501 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.692230  9582 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:24.722610  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:24.697997  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:24.698853  8961 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:24.728960  1728 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:24.713212  1730 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:24.729885  1731 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:24.731657  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.729169  1727 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:24.730772  1724 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:24.760246  2410 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:25:24.701259  9313 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:24.743788  1729 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:24.761070  1799 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:24.761467  1732 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:24.768189  8961 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:24.775671  9313 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.776892  9313 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:24.741580  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:24.778214  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:24.747251  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:24.781488  1881 raft_consensus.cc:1217] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Deduplicated request from leader. Original: 14.10->[16.11-22.13]   Dedup: 16.11->[19.12-22.13]
I20260709 12:25:24.781594  9607 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:24.782019  1881 pending_rounds.cc:85] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 11
I20260709 12:25:24.782282  1881 pending_rounds.cc:107] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:25:24.718230  1798 pending_rounds.cc:85] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 13
I20260709 12:25:24.782974  1798 pending_rounds.cc:107] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 16.14
I20260709 12:25:24.765585  9487 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.783299  1798 raft_consensus.cc:2889] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:24.783692  9487 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.749789  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.748052  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:24.784394  9546 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.785133  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2797) took 1032 ms. Trace:
I20260709 12:25:24.785234  1880 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:24.785555  1807 rpcz_store.cc:276] 0709 12:25:23.752287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.757961 (+  5674us) service_pool.cc:224] Handling call
0709 12:25:24.784968 (+1027007us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b5400167bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:24.785073 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1582308608}
I20260709 12:25:24.786108  9487 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.724716  9459 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.740571  9722 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:24.793432  9459 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:24.758000  9721 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.048s
I20260709 12:25:24.754930  9006 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.795432  9721 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:24.797284  9006 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.728776  9736 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.078s
I20260709 12:25:24.782545  1881 raft_consensus.cc:2889] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:24.798674  8961 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.712656  9498 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:24.767920  2416 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:24.772388  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.801484  1873 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:24.757603  9730 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:25:24.782980  9659 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:24.800729  9498 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:24.809674  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.810168  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:24.811933  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 270293us
I20260709 12:25:24.813848  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:24.814917  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:24.741822  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.805898  1807 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:24.808207  1880 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:24.817188  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.769266  1727 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.820134  1727 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:24.818383  2416 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:24.818773  1728 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 10. Preceding OpId from leader: term: 27 index: 12. (index mismatch)
I20260709 12:25:24.813342  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.822475  9494 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:24.822938  9582 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:24.823374  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:24.814352  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:24.824890  9501 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.827512  1873 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
W20260709 12:25:24.828126  9727 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.158s	user 0.000s	sys 0.001s
I20260709 12:25:24.766872  9645 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:25:24.759552  9697 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:24.665139  9592 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.769192  9728 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.073s
W20260709 12:25:24.833354  9587 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.503s	user 0.010s	sys 0.003s
I20260709 12:25:24.835505  2415 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:24.819343  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.765295  9463 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.837255  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.841123  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.840690  9724 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.897653  2419 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:24.897753  2418 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:24.866223  9497 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:24.899977  1724 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:25:24.847039  1880 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:24.847479  1807 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:24.842638  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1347) took 2048 ms. Trace:
I20260709 12:25:24.905066  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.844024  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.905880  2419 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
W20260709 12:25:24.853204  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.906986  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.762408  9561 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:24.904932  2410 rpcz_store.cc:276] 0709 12:25:22.794508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.963507 (+168999us) service_pool.cc:224] Handling call
0709 12:25:22.963815 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.770528 (+1806713us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.842304 (+ 71776us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b5400525154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:24.842554 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4104278784}
I20260709 12:25:24.856442  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:24.848731  1878 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:25:24.870103  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.906616  2416 raft_consensus.cc:2410] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 24 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
W20260709 12:25:24.909128  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52818us
W20260709 12:25:24.909680  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.875074  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.876291  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:24.830852  9684 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:24.880591  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 65794.1us
I20260709 12:25:24.731473  9663 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.848902  1727 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
I20260709 12:25:24.898018  2411 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:25:24.910827  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 91565.5us
I20260709 12:25:24.834573  9616 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.851282  9659 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:24.916887  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:24.909883  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.854390  2417 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:24.885680  9607 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:25:24.921384  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:24.923759  1724 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 14. (term mismatch)
I20260709 12:25:25.096215  1724 pending_rounds.cc:85] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
I20260709 12:25:25.096611  1724 pending_rounds.cc:107] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 19.14
I20260709 12:25:25.096964  1724 raft_consensus.cc:2889] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:24.924279  9501 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:25.099311  9501 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:24.884191  9713 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:24.914086  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.963892  1727 pending_rounds.cc:85] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
I20260709 12:25:25.103091  9713 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:25.103404  1727 pending_rounds.cc:107] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 12:25:25.103631  9713 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:25.103737  1727 raft_consensus.cc:2889] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:25.105310  1727 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:25.105652  1727 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:24.965821  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.971179  9582 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.107234  9607 consensus_queue.cc:1243] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:25:25.109222  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:24.973237  9697 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:24.883898  9611 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:25.037074  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:24.848057  9736 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:25.115091  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:25.116376  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:25.117203  2419 raft_consensus.cc:2376] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20. Re-sending same reply.
I20260709 12:25:25.118201  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:25.115135  9736 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.119226  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:25.120610  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:25.121104  9724 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:24.957516  9591 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:25.043330  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:24.978407  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 58704.9us
W20260709 12:25:25.060799  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.884160  9744 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:25.138039  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 221288us
I20260709 12:25:24.960558  9463 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:24.914469  9537 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:24.951088  9490 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:25.111282  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:25.144773  9490 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:25.106449  1727 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:25.041195  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.123052  1729 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:25.145143  9026 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.000s
W20260709 12:25:25.131639  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 217640us
I20260709 12:25:25.146430  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:25.146925  1729 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 12:25:25.147330  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:24.972725  9077 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 12:25:25.123265  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 213436us
I20260709 12:25:25.118078  9432 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:25.150162  9632 raft_consensus.cc:2764] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:25:25.150241  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:24.920656  9587 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.150945  9632 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:25.151160  1727 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:25.151396  9632 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.151752  9632 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:25.151888  9432 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:25.152671  9587 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:25.140054  9744 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:25.153653  9744 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.140455  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:25.156033  9537 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:25.139652  9742 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:25.145291  9490 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:25.155509  1798 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:25.156193  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:25.165256  1799 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
W20260709 12:25:25.157833  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50466.5us
I20260709 12:25:25.141515  9663 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:25.140295  9659 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.158607  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:25.113231  9501 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:25.167733  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:24.938694  9158 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:25.171180  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:25.146085  9026 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:25.161778  1805 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 12:25:25.120934  9736 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:25.186677  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:25.168874  9638 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.019s
W20260709 12:25:25.173199  9734 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.008s	sys 0.029s
I20260709 12:25:25.172217  9724 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:25.198734  2417 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:25.157739  9394 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:25.160431  9541 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:25.112087  9487 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:25:25.138646  9697 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:25.159546  9741 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:25.112974  9379 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.188544  1727 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:25.176117  9006 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.037s
W20260709 12:25:25.205328  9734 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.008s	sys 0.031s
I20260709 12:25:25.208415  1872 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:25:25.198086  9721 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.209038  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:25.198302  9742 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:25.168256  9590 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:25:25.216142  2412 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:25.204066  9455 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.233750  8989 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:25.205698  9591 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:25.203547  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.213162  9684 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:25.243041  9546 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.239668  9741 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.201670  9463 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:25.212615  9541 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.291087  9541 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.290714  9455 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.213346  9394 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:25.292958  9394 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.294762  9455 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:25.282024  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.274994  1874 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
W20260709 12:25:25.200778  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 422620us
I20260709 12:25:25.254729  2415 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:25.290154  9463 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.303346  9077 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.304075  9077 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.305570  9463 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:25.307235  9735 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:25.309228  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:25.303347  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 489021us
W20260709 12:25:25.286646  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 77722.7us
W20260709 12:25:25.312688  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.316493  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 181942us
I20260709 12:25:25.324826  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:25.305615  9546 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.317102  9591 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.353432  9610 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:25.353940  9610 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:25.285957  9684 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.354077  9541 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.354625  9684 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:25.355587  9394 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:25.356094  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.358163  9394 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:25.260998  9721 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.254441  9561 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:25.372291  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:25.381356  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:25.359481  9747 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:25:25.342192  9395 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:25:25.433058  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.437657  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.441526  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:25.442494  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.443297  1727 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:25.442553  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 255564us
I20260709 12:25:25.449756  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:25.450665  1727 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:25.368710  9616 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:25.472117  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.478222  1731 raft_consensus.cc:2368] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22.
I20260709 12:25:25.425647  9341 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:25.479728  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:25.439869  9455 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:25.485498  2415 raft_consensus.cc:2368] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
I20260709 12:25:25.487058  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:25.488299  1727 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:25.489676  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.479866  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:25.496423  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:25.496841  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55391.1us
I20260709 12:25:25.498692  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:25.499794  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:25.496388  9313 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:25.502875  9741 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:25.503156  9313 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.503561  9313 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.505481  9735 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:25.505908  9735 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:25.374433  9745 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.141s
I20260709 12:25:25.506219  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.507431  9379 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:25.473565  9736 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:25.509893  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 353144us
I20260709 12:25:25.508337  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.477473  9546 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:25.523420  1772 leader_election.cc:400] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
I20260709 12:25:25.523815  1772 leader_election.cc:403] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:25.524870  9616 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:25.525434  9616 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:25.528584  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.529481  1729 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:25.530238  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:25.533871  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.534336  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.368121  9697 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.536837  9313 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.538280  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.539615  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.539896  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.540231  1729 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:25.541020  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.542264  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.546823  9158 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:25.371686  9610 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:25.547883  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:25.548369  9610 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:25.541087  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.549631  1729 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 12:25:25.550479  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.551121  1729 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:25:25.552280  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:25.552765  9026 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:25.553962  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:25.554386  9541 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:25.554579  1877 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:25.556084  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:25.556695  1877 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:25:25.534659  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.523729  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.493798  9455 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.567402  1878 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:25.570423  9158 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:25.573308  9077 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:25.575057  9642 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:25.575490  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.576172  1725 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:25.575501  9642 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:25.576771  9642 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:25.576995  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:25.577534  1725 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:25.583606  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:25.583865  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:25.584581  9494 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:25.584995  9494 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:25.585378  9494 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:25.587441  9722 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.795s	user 0.011s	sys 0.000s
I20260709 12:25:25.587239  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.588316  1723 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:25:25.588367  1731 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:25.588940  1723 pending_rounds.cc:85] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 11
I20260709 12:25:25.589293  1723 pending_rounds.cc:107] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 21.12
I20260709 12:25:25.589524  1723 raft_consensus.cc:2889] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:25.592864  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.593137  1877 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:25.593998  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:25.595393  9379 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:25.595993  9721 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:25.596336  9379 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:25.555501  9432 consensus_queue.cc:1175] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "075704438992440697732181e49ccd09" responder_term: 22 status { last_received { term: 21 index: 16 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:25.597380  9642 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.598003  9341 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.599169  9632 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.600987  9494 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.590276  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:25.604355  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.605326  1729 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:25.506251  9735 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:25.605999  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 107223us
I20260709 12:25:25.605247  1731 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.606750  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:25.615459  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:25.514927  9395 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:25.632025  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:25.632723  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:25.602393  9632 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:25.637928  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:25.640343  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:25.640883  1728 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:25.543016  9313 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:25.616441  1727 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:25.616842  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:25.619180  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.628297  1725 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
W20260709 12:25:25.634253  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 312994us
I20260709 12:25:25.484699  9501 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.083s
I20260709 12:25:25.641976  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:25.614506  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:25.633327  2418 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:25.671260  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:25.677398  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:25.605024  9455 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:25.678126  1875 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:25.681113  1727 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
W20260709 12:25:25.657476  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 332717us
I20260709 12:25:25.683386  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:25.647022  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 329001us
I20260709 12:25:25.648824  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:25.685859  1731 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:25.686144  1877 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:25.526331  9616 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:25.695490  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53643.6us
I20260709 12:25:25.697516  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:25.704432  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:25.708527  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:25.514449  9750 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.173s
I20260709 12:25:25.711953  1875 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:25.668318  9724 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.709705  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:25.716439  1728 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:25.667927  9145 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:25.637141  9541 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.081s
I20260709 12:25:25.720177  1877 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:25.746152  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:25.567667  9736 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:25.668443  9611 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:25.711874  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:25.735867  1878 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
W20260709 12:25:25.745940  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.744561  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.690237  9753 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:25:25.637868  9590 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.089s
I20260709 12:25:25.645032  9515 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
I20260709 12:25:25.751260  1728 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:25.737530  9772 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:25.755043  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45481.7us
I20260709 12:25:25.747574  9494 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:25.755285  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50937.1us
I20260709 12:25:25.788300  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:25.706607  9741 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:25.706421  9722 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:25.778726  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:25.701918  9077 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:25.765882  9395 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:25.701318  9313 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:25.775784  9379 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:25.789153  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3708) took 1044 ms. Trace:
I20260709 12:25:25.793229  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:25.795337  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:25.797931  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.707039  9745 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:25:25.799654  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:25.791976  9713 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.681s	user 0.009s	sys 0.006s
I20260709 12:25:25.834000  1728 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:25.748039  9561 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:25.778009  9145 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.761121  9026 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.045s
I20260709 12:25:25.764284  9638 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.047s
I20260709 12:25:25.753659  9724 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:25.772384  9611 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:25.839644  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.900156  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2941) took 1209 ms. Trace:
I20260709 12:25:25.858538  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3712) took 1112 ms. Trace:
W20260709 12:25:25.907276  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.853112  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53522.3us
I20260709 12:25:25.863749  2417 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
W20260709 12:25:25.842849  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 124792us
I20260709 12:25:25.945833  1872 rpcz_store.cc:276] 0709 12:25:24.744880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.746871 (+  1991us) service_pool.cc:224] Handling call
0709 12:25:24.772286 (+ 25415us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:25.243177 (+470891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.788752 (+545575us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b5400116a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:25.789060 (+   308us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002378368}
I20260709 12:25:25.946614  1723 rpcz_store.cc:276] 0709 12:25:24.690805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.695252 (+  4447us) service_pool.cc:224] Handling call
0709 12:25:24.695592 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.589964 (+894372us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.899868 (+309904us) spinlock_profiling.cc:243] Waited 892 ms on lock 0x7b54002a9754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:25.900081 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2052246784}
I20260709 12:25:25.802165  9594 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:25.803275  8989 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.512s	user 0.010s	sys 0.001s
I20260709 12:25:25.954353  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:25.947798  1876 rpcz_store.cc:276] 0709 12:25:24.746452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.748654 (+  2202us) service_pool.cc:224] Handling call
0709 12:25:24.749083 (+   429us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:24.769701 (+ 20618us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.769737 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:24.770356 (+   619us) log.cc:844] Serialized 52 byte log entry
0709 12:25:25.269623 (+499267us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:25.269689 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.270724 (+  1035us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.858295 (+587571us) raft_consensus.cc:1685] finished
0709 12:25:25.858319 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.858476 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52391,"raft.queue_time_us":11663,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":6350,"thread_start_us":430032,"threads_started":1,"wal-append.queue_time_us":625809}
W20260709 12:25:26.011169  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53288.1us
I20260709 12:25:25.957609  1723 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:25:25.793249  9771 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:25.957980  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:26.014420  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:26.016397  1723 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
W20260709 12:25:26.017387  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.909097  9752 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.618s	user 0.000s	sys 0.002s
W20260709 12:25:25.882335  9767 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
W20260709 12:25:25.906708  9751 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.616s	user 0.001s	sys 0.002s
W20260709 12:25:25.881680  9760 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:25:26.022017  9659 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:25.835219  9313 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:25.878226  9777 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:25.852082  9722 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.840536  9561 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.840435  9776 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:25.876677  9573 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:25.908785  9736 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.061695  1723 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:26.028644  9501 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.089s
I20260709 12:25:26.137498  1723 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:25.888681  9762 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:25.804325  9515 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:25.956212  9594 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:26.199285  9594 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:26.014045  8989 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:25.880244  9764 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:25:26.108024  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:26.222376  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.223680  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.024565  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 313097us
I20260709 12:25:26.111578  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:25.814302  9642 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:25.802527  9772 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.060972  9546 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.954556  9713 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.257978  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:26.259415  1798 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:26.015080  9395 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.095s
W20260709 12:25:26.121939  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 450741us
I20260709 12:25:26.265553  9515 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:26.265977  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:26.266417  9651 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:25:26.125528  9341 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.527s	user 0.005s	sys 0.005s
I20260709 12:25:25.958988  9145 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:26.135438  9490 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.951s	user 0.008s	sys 0.005s
I20260709 12:25:26.125629  9561 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:26.117801  9659 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.147313  9501 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:25:26.149811  9611 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:26.153247  9610 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.604s	user 0.012s	sys 0.000s
I20260709 12:25:26.021684  9697 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.097s
I20260709 12:25:26.030107  9771 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.156338  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3710) took 1410 ms. Trace:
I20260709 12:25:26.158993  2410 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:25:26.159437  9744 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.005s	user 0.013s	sys 0.001s
I20260709 12:25:26.160310  9786 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:26.160753  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1238) took 1250 ms. Trace:
I20260709 12:25:26.164852  8961 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.141672  9762 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:26.160949  9785 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:26.167891  9724 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.168133  9313 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.296743  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:26.168277  9459 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:26.170023  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:26.188004  9684 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.833s	user 0.013s	sys 0.000s
I20260709 12:25:26.130522  9722 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:26.190621  9760 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:25:26.191143  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:26.192979  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3699) took 1942 ms (client timeout 1702 ms). Trace:
W20260709 12:25:26.212682  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.193903  9741 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:26.212829  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:26.214490  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:26.215943  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:26.303047  1878 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:26.303162  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:26.304944  1803 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:26.305763  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 114776us
I20260709 12:25:26.219524  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:26.230172  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:26.218426  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:26.307827  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89483us
I20260709 12:25:26.246289  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:26.247030  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:26.309536  2414 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:26.247296  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:26.309203  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:26.311347  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:26.198190  9455 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.212085  9487 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.244469  9736 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.277591  1723 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
W20260709 12:25:26.289279  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 274947us
I20260709 12:25:26.284152  1879 rpcz_store.cc:276] 0709 12:25:24.745705 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.745880 (+   175us) service_pool.cc:224] Handling call
0709 12:25:24.765622 (+ 19742us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:24.882566 (+116944us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:25.312823 (+430257us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.636186 (+323363us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:25.646417 (+ 10231us) log.cc:844] Serialized 52 byte log entry
0709 12:25:25.646643 (+   226us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:25.646702 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.155893 (+509191us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b540021bbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:26.155949 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:26.156000 (+    51us) raft_consensus.cc:1685] finished
0709 12:25:26.156096 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.156270 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113300,"prepare.queue_time_us":553987,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":101433,"raft.run_cpu_time_us":368,"raft.run_wall_time_us":43043,"spinlock_wait_cycles":86367488,"thread_start_us":658313,"threads_started":2,"wal-append.queue_time_us":542057}
I20260709 12:25:26.286558  1724 rpcz_store.cc:276] 0709 12:25:24.910019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.125827 (+215808us) service_pool.cc:224] Handling call
0709 12:25:25.165783 (+ 39956us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:25.166235 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.166280 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:25.167224 (+   944us) log.cc:844] Serialized 52 byte log entry
0709 12:25:25.443121 (+275897us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:25.443190 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.443290 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:26.160525 (+717235us) raft_consensus.cc:1685] finished
0709 12:25:26.160546 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.160695 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":21743,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"thread_start_us":235906,"threads_started":1,"wal-append.queue_time_us":450343}
I20260709 12:25:26.284413  2410 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:26.273963  9315 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.295723  2411 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:26.278498  9341 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
W20260709 12:25:26.300720  1874 rpcz_store.cc:269] 0709 12:25:24.250840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.308783 (+ 57943us) service_pool.cc:224] Handling call
0709 12:25:26.192713 (+1883930us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b54000f0854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:26.192876 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2219824384}
I20260709 12:25:26.349862  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:26.350358  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:26.301757  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:26.304112  1875 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:26.216918  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:26.332244  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:26.346928  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 392617us
I20260709 12:25:26.352516  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:26.352859  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:26.282087  9501 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:26.283368  9748 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:26.356204  1876 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:26.354650  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 82691.2us
I20260709 12:25:26.334353  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:26.264015  9546 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.338546  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:26.363770  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:26.364259  1803 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 22
W20260709 12:25:26.339452  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.332334  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:26.346928  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 116851us
I20260709 12:25:26.366645  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:26.351115  1874 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:26.353839  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:26.282603  9490 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.332789  8961 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:26.367687  1879 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:26.336740  9498 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:26.371615  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:26.397097  1807 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:26.346254  9611 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:26.376231  9541 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:26.351168  9459 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:26.377643  9638 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:26.379968  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:26.380695  9501 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.337285  9494 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:26.384840  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:26.385826  1798 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:26.336748  9724 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:26.400153  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:26.476509  1731 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:26.347265  9487 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.354909  9315 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.344775  9730 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.337095  9798 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.362282  9748 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:26.293448  9744 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.294296  9792 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.095s
I20260709 12:25:26.425890  1878 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:25:26.431665  1803 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:26.354831  9495 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:26.339546  8989 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.079s
I20260709 12:25:26.520932  9495 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.302032  9785 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.525249  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:26.526436  9785 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.382656  9145 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:26.376250  9659 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.332127  9684 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.462936  1804 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:25:26.523412  9495 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.530534  9684 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.397210  9776 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:26.537853  1804 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:26.473289  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94559.6us
W20260709 12:25:26.423291  9735 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.776s	user 0.006s	sys 0.009s
I20260709 12:25:26.540634  9735 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.542758  1806 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:26.483402  8961 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:26.555269  2413 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:26.492231  1874 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 16.
I20260709 12:25:26.351696  9736 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.495738  9501 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:25:26.421053  9611 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:26.423192  9541 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.567021  9541 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:26.515293  9798 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.449911  9490 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:26.423045  9498 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:26.570864  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.572256  9487 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.507148  2410 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.573148  9722 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.573400  2410 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:26.566262  9611 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.574880  9455 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.575135  9611 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:26.557984  8961 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:26.576611  8961 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:26.547086  9771 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.498366  9494 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:26.579919  9494 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.580715  9736 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.513262  9748 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:26.585153  9591 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.586723  9594 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.564456  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:26.587678  9772 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.566907  9315 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.450546  9379 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:26.395459  9762 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.589051  9744 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.589818  9762 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:26.509649  9724 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:26.591452  9713 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.592247  9724 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.569502  9459 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.591390  9541 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.592895  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:26.596589  9592 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:26.597028  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.597867  9802 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.151s	user 0.002s	sys 0.000s
I20260709 12:25:26.296775  9786 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:26.600350  9541 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.601799  9786 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.513517  9730 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:26.605275  9730 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.611774  9594 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.549456  9735 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.616861  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1534) took 1500 ms. Trace:
I20260709 12:25:26.617311  2413 rpcz_store.cc:276] 0709 12:25:25.116786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.121587 (+  4801us) service_pool.cc:224] Handling call
0709 12:25:25.121945 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.559423 (+1437478us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.616572 (+ 57149us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b5400561b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:26.616780 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3295607168}
I20260709 12:25:26.618189  9546 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:26.571799  9498 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:26.646615  9798 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:26.410262  1696 leader_election.cc:400] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20.
W20260709 12:25:26.685573  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.690506  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2960) took 1145 ms. Trace:
I20260709 12:25:26.691365  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2967) took 1144 ms. Trace:
I20260709 12:25:26.692243  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3771) took 1249 ms. Trace:
I20260709 12:25:26.692760  1729 rpcz_store.cc:276] 0709 12:25:25.442684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.606076 (+163392us) service_pool.cc:224] Handling call
0709 12:25:25.606364 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.606555 (+   191us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:26.139772 (+533217us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.691438 (+551666us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.691588 (+   150us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:26.691719 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.691863 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.692046 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":197985,"prepare.queue_time_us":677277,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"thread_start_us":845565,"threads_started":2,"wal-append.queue_time_us":432757}
I20260709 12:25:26.422376  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:26.450119  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 83553.7us
I20260709 12:25:26.707981  1875 raft_consensus.cc:1217] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Deduplicated request from leader. Original: 14.10->[16.11-21.13]   Dedup: 19.12->[21.13-21.13]
I20260709 12:25:26.764189  1875 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [16.11-21.13]
I20260709 12:25:26.765054  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:26.401000  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53044.9us
I20260709 12:25:26.772825  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2958) took 1228 ms. Trace:
I20260709 12:25:26.481269  1879 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:26.551357  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3713) took 1804 ms. Trace:
I20260709 12:25:26.692144  1730 rpcz_store.cc:276] 0709 12:25:25.546762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.196444 (+649682us) service_pool.cc:224] Handling call
0709 12:25:26.196900 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.222514 (+ 25614us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:26.299159 (+ 76645us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.690274 (+391115us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.690449 (+   175us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:26.690604 (+   155us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.690765 (+   161us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.691127 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":265463,"prepare.queue_time_us":154434,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":77064,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":6307,"thread_start_us":208199,"threads_started":2,"wal-append.queue_time_us":335623}
I20260709 12:25:26.782683  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:26.773659  1723 rpcz_store.cc:276] 0709 12:25:25.544323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.161091 (+616768us) service_pool.cc:224] Handling call
0709 12:25:26.161350 (+   259us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.346246 (+184896us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.771907 (+425661us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b540030af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:26.772472 (+   565us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266827520}
I20260709 12:25:26.795358  2410 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:25:26.799768  1804 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:26.777127  9454 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.801474  2413 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:26.803282  9611 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:26.804914  1724 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:26.806001  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:25:26.806699  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.695822  1726 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:26.836362  1806 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:26.843679  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:26.695401  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 304980us
I20260709 12:25:26.691255  1732 rpcz_store.cc:276] 0709 12:25:25.544821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.166048 (+621227us) service_pool.cc:224] Handling call
0709 12:25:26.166449 (+   401us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:26.202676 (+ 36227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.202704 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:26.203283 (+   579us) log.cc:844] Serialized 52 byte log entry
0709 12:25:26.419239 (+215956us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:26.438250 (+ 19011us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.439614 (+  1364us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.439873 (+   259us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:26.689535 (+249662us) raft_consensus.cc:1685] finished
0709 12:25:26.689635 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.690175 (+   540us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":18341,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":36016,"raft.run_cpu_time_us":804,"raft.run_wall_time_us":27319,"thread_start_us":144349,"threads_started":1,"wal-append.queue_time_us":286903}
I20260709 12:25:26.807849  9755 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:26.854368  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:26.866988  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:26.908764  1729 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:26.900502  2410 pending_rounds.cc:85] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 12
I20260709 12:25:26.922652   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:26.923252  9454 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:26.927788  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 120103us
I20260709 12:25:26.922775  2410 pending_rounds.cc:107] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:25:26.934705  2410 raft_consensus.cc:2889] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:26.934960  9546 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.959309  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1526) took 2034 ms. Trace:
I20260709 12:25:26.928642  1881 rpcz_store.cc:276] 0709 12:25:24.763634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.780895 (+ 17261us) service_pool.cc:224] Handling call
0709 12:25:24.781266 (+   371us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:24.867790 (+ 86524us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:24.897046 (+ 29256us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.436271 (+539225us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b18004328e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:25.439921 (+  3650us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:25.440668 (+   747us) log.cc:844] Serialized 52 byte log entry
0709 12:25:25.440815 (+   147us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:25.441033 (+   218us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.447191 (+  6158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.447312 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:26.374470 (+927158us) spinlock_profiling.cc:243] Waited 14.7 ms on lock 0x7b18004328e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f
0709 12:25:26.375946 (+  1476us) spinlock_profiling.cc:243] Waited 3.14 ms on lock 0x7b48000ce7d8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:25:26.549244 (+173298us) raft_consensus.cc:1685] finished
0709 12:25:26.549282 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.551198 (+  1916us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82984,"prepare.queue_time_us":38702,"prepare.run_cpu_time_us":135,"prepare.run_wall_time_us":9330,"raft.queue_time_us":158327,"raft.run_cpu_time_us":5379,"raft.run_wall_time_us":952249,"spinlock_wait_cycles":47197952,"thread_start_us":409377,"threads_started":1,"wal-append.queue_time_us":739467}
I20260709 12:25:26.967404  2410 rpcz_store.cc:276] 0709 12:25:24.915955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.917836 (+  1881us) service_pool.cc:224] Handling call
0709 12:25:24.918281 (+   445us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.158866 (+1240585us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54005922a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:25:26.506566 (+347700us) spinlock_profiling.cc:243] Waited 8.34 ms on lock 0x7b4c002706c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa0a4a20 00007f16aa0b5a29 00007f16aa07b644 00007f16aa0a2f2d 00007f16aa0a2b93 00007f16abdd318b 00007f16abdd3cb5 00007f16abdcef30 00007f16abdd07de 00007f16abddab49 00007f16abddb416
0709 12:25:26.935647 (+429081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.949907 (+ 14260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":456536704}
I20260709 12:25:26.993315  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:26.999229  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:27.030332  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82918.4us
I20260709 12:25:27.044471  1731 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:27.047248  9611 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:27.047755  9454 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:27.048035  9611 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:27.048741  9611 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:27.050947  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.051357  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.051646  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47291.4us
I20260709 12:25:27.052618  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:27.051393  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.054343  9313 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:25:27.031615  9459 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:27.058885  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 97329.1us
I20260709 12:25:27.059514  9459 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:27.059966  8961 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:27.061302  9459 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:27.063199  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:27.063797  1804 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:25:27.064893  9798 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:27.065477  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:27.031828  9394 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:27.066299  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:27.068320  9798 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:27.068674  9632 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:27.071610  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 265979us
I20260709 12:25:27.038551  9546 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:27.073647  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:27.074707  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:27.075214  9798 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:27.043040  1730 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
W20260709 12:25:27.076848  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.066583  1804 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:27.077536  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:27.080076  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:27.084311  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3796) took 1390 ms. Trace:
I20260709 12:25:27.051239  9796 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:25:27.046633  1731 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:27.089700  1723 rpcz_store.cc:276] 0709 12:25:25.693654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.082502 (+1388848us) service_pool.cc:224] Handling call
0709 12:25:27.083198 (+   696us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.083653 (+   455us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.083683 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.083813 (+   130us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:27.083955 (+   142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.084025 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.084146 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:27.039171  1727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.009s	sys 0.016s
I20260709 12:25:27.085731  2413 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:27.069790  9632 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:27.051476  9645 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260709 12:25:27.100502  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3793) took 1418 ms. Trace:
I20260709 12:25:27.101167  1731 rpcz_store.cc:276] 0709 12:25:25.677870 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.045829 (+1367959us) service_pool.cc:224] Handling call
0709 12:25:27.046405 (+   576us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.090634 (+ 44229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.096593 (+  5959us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:27.096132  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43818.8us
I20260709 12:25:27.111135  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:27.111785  9592 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:27.100113  9772 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:27.103505  9546 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:27.114923  9394 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.009s
W20260709 12:25:27.140800  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53028.3us
I20260709 12:25:27.152319  9797 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:27.155875  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 81346.4us
I20260709 12:25:27.144611  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:27.159101  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:27.162423  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 308742us
I20260709 12:25:27.161362  9454 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:27.169073  2410 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:27.169351  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:27.176366  9541 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.177922  2411 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:27.179018  9735 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 12:25:27.183640  9494 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:27.184794  9494 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:27.185575  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 2943) took 2494 ms (client timeout 2621 ms). Trace:
I20260709 12:25:27.187083  9494 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:27.183737  9632 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:27.192775  9573 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:27.193634  9573 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:27.196723  9762 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:27.157294  8961 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:27.177755  9541 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:27.199391  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:27.205416  2411 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:27.186305  1726 rpcz_store.cc:269] 0709 12:25:24.691430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.692477 (+  1047us) service_pool.cc:224] Handling call
0709 12:25:27.185199 (+2492722us) spinlock_profiling.cc:243] Waited 1.97 s on lock 0x7b54002d0ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:27.185421 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4520318208}
I20260709 12:25:27.216533  9632 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.217623  9632 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:27.221776  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:27.224131  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52583.6us
I20260709 12:25:27.224606  1804 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:27.227001  9796 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.007s
I20260709 12:25:27.239817  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:27.250600  9454 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:27.256376  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1248) took 1030 ms. Trace:
I20260709 12:25:27.257098  1731 rpcz_store.cc:276] 0709 12:25:26.225663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.107630 (+881967us) service_pool.cc:224] Handling call
0709 12:25:27.149676 (+ 42046us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.216412 (+ 66736us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.226759 (+ 10347us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.248865 (+ 22106us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:27.255962 (+  7097us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.256094 (+   132us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.256293 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:27.254873  9561 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.854s	user 0.010s	sys 0.014s
I20260709 12:25:27.254287  9313 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:27.262583  9822 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.063s	user 0.001s	sys 0.001s
I20260709 12:25:27.274981  9561 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:27.276541  9379 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:27.283488  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2796) took 1070 ms. Trace:
I20260709 12:25:27.286330  9541 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:27.287680  9632 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:27.253676  9454 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:27.301072  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1250) took 1074 ms. Trace:
W20260709 12:25:27.309034  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 135407us
W20260709 12:25:27.310978  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 231260us
I20260709 12:25:27.308508  1723 rpcz_store.cc:276] 0709 12:25:26.226301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.138171 (+911870us) service_pool.cc:224] Handling call
0709 12:25:27.201072 (+ 62901us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.269861 (+ 68789us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.269891 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.291532 (+ 21641us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:27.300627 (+  9095us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.300752 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.300962 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:27.293871  9379 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:27.343969  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3720) took 1733 ms. Trace:
I20260709 12:25:27.344662  1880 rpcz_store.cc:276] 0709 12:25:25.494960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.495070 (+   110us) service_pool.cc:224] Handling call
0709 12:25:25.495444 (+   374us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:25:25.508457 (+ 13013us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.508488 (+    31us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:25:25.509370 (+   882us) log.cc:844] Serialized 104 byte log entry
0709 12:25:26.847756 (+1338386us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x7b18002f3e00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:25:27.132442 (+284686us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:27.145866 (+ 13424us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.172134 (+ 26268us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.199174 (+ 27040us) raft_consensus.cc:1685] finished
0709 12:25:27.199186 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.227890 (+ 28704us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17461,"raft.queue_time_us":12937,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":8249,"spinlock_wait_cycles":98049280,"thread_start_us":184178,"threads_started":1,"wal-append.queue_time_us":1666108}
W20260709 12:25:27.310745  9821 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.228s	user 0.001s	sys 0.001s
I20260709 12:25:27.344764  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1245) took 1003 ms. Trace:
I20260709 12:25:27.251170  9573 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:27.357765  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:27.355945  1725 rpcz_store.cc:276] 0709 12:25:26.224630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.098524 (+873894us) service_pool.cc:224] Handling call
0709 12:25:27.140333 (+ 41809us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.215676 (+ 75343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.215718 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.225636 (+  9918us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:27.225676 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.225730 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.228084 (+  2354us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:27.357223  9394 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:27.363682  9394 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.364080  9394 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:27.366882  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 156311us
I20260709 12:25:27.369115  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:27.370285  9494 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:25:27.371816  9501 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.802s	user 0.006s	sys 0.017s
I20260709 12:25:27.372584  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:27.373752  9796 raft_consensus.cc:2764] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:25:27.293949  1805 rpcz_store.cc:276] 0709 12:25:26.213416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.213589 (+   173us) service_pool.cc:224] Handling call
0709 12:25:26.214039 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.703144 (+489105us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:26.703872 (+   728us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.963169 (+259297us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.963251 (+    82us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:26.963321 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.217448 (+254127us) spinlock_profiling.cc:243] Waited 3.72 ms on lock 0x7b1800342340. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:27.283011 (+ 65563us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54001671d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:27.283076 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.283294 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16053,"prepare.queue_time_us":26071,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":93,"raft.queue_time_us":31035,"raft.run_cpu_time_us":511,"raft.run_wall_time_us":35072,"spinlock_wait_cycles":1131350016,"thread_start_us":244666,"threads_started":1,"wal-append.queue_time_us":329418}
I20260709 12:25:27.359275  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:27.387169  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.399000  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:27.404379  2413 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:27.399999  1798 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:27.410023  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2962) took 1864 ms. Trace:
W20260709 12:25:27.411625  9748 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.828s	user 0.009s	sys 0.017s
I20260709 12:25:27.418105  9748 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:27.412539  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:27.420739  9748 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:27.421752  2416 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:27.427712  1804 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:27.431730  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 115134us
I20260709 12:25:27.431998  8961 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:27.432976  8961 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:27.435422  9454 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:27.436792  9454 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:27.444339  9501 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:27.452235  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:27.459389  9379 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:27.460023  9796 raft_consensus.cc:2764] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:25:27.460897  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:27.461490  1728 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:25:27.462322  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2973) took 1191 ms. Trace:
I20260709 12:25:27.406697  9394 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:27.465344  1807 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:25:27.465996  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2813) took 1209 ms. Trace:
I20260709 12:25:27.462679  1728 rpcz_store.cc:276] 0709 12:25:26.270654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.460439 (+1189785us) service_pool.cc:224] Handling call
0709 12:25:27.460974 (+   535us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.461940 (+   966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.462242 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:27.471233  1807 rpcz_store.cc:276] 0709 12:25:26.256630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.705614 (+448984us) service_pool.cc:224] Handling call
0709 12:25:27.464523 (+758909us) spinlock_profiling.cc:243] Waited 8.34 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:25:27.464800 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.465709 (+   909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.465936 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19184512}
I20260709 12:25:27.479904  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:27.434451  8961 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:27.487589  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 364504us
I20260709 12:25:27.416148  1727 rpcz_store.cc:276] 0709 12:25:25.545332 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.179823 (+634491us) service_pool.cc:224] Handling call
0709 12:25:26.180263 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:26.222250 (+ 41987us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:26.428615 (+206365us) log.cc:844] Serialized 14 byte log entry
0709 12:25:27.137962 (+709347us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.229346 (+ 91384us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.229579 (+   233us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:27.229633 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.233091 (+  3458us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.408840 (+175749us) raft_consensus.cc:1685] finished
0709 12:25:27.408864 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.409936 (+  1072us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53919,"prepare.queue_time_us":118282,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":7623,"raft.queue_time_us":85780,"raft.run_cpu_time_us":553,"raft.run_wall_time_us":39361,"thread_start_us":842051,"threads_started":2,"wal-append.queue_time_us":634947}
W20260709 12:25:27.515575  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73649.8us
I20260709 12:25:27.488380  9394 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:27.520169  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:27.510741  9379 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:27.528019  1807 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:27.532066  9792 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:27.532900  9792 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:27.548813  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:27.550328  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62910.7us
I20260709 12:25:27.551179  9394 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:27.552784  9796 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:27.553364  9796 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.553763  9796 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:27.560511  9755 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:27.561615  9755 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:27.560708  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2803) took 1341 ms. Trace:
I20260709 12:25:27.567579  1806 rpcz_store.cc:276] 0709 12:25:26.218941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.219118 (+   177us) service_pool.cc:224] Handling call
0709 12:25:27.560406 (+1341288us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b5400133b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:27.560606 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":542512512}
I20260709 12:25:27.566885  1798 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
W20260709 12:25:27.595103  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.598706  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.485960  9772 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:27.605842  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 60930.1us
W20260709 12:25:27.622968  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.644914  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50690us
I20260709 12:25:27.650812  1724 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:25:27.597977  9541 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.661573  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:27.657589  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2811) took 1349 ms. Trace:
I20260709 12:25:27.660758  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3790) took 2011 ms. Trace:
W20260709 12:25:27.661572  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 298535us
I20260709 12:25:27.674671  1724 rpcz_store.cc:276] 0709 12:25:25.647024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.564105 (+1917081us) service_pool.cc:224] Handling call
0709 12:25:27.595867 (+ 31762us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.657647 (+ 61780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.657926 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:27.676659  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:27.680143  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2975) took 1400 ms. Trace:
I20260709 12:25:27.687857  1727 rpcz_store.cc:276] 0709 12:25:26.280080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.515339 (+1235259us) service_pool.cc:224] Handling call
0709 12:25:27.600342 (+ 85003us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.662387 (+ 62045us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.671240 (+  8853us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.678691 (+  7451us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:27.679113 (+   422us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.679328 (+   215us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.679982 (+   654us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:27.676618  9772 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:27.695182  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 107735us
I20260709 12:25:27.696979  9541 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:27.699648  9541 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:27.700896  9838 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:25:27.702099  9838 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:27.702479  9838 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:27.703649  9838 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:27.705188  9838 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:27.719120  9561 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:27.719537  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:27.720525  9561 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:27.722909  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:27.724164  9829 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:25:27.730244  9755 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:27.733318  9837 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:27.658399  9501 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:27.731531  9663 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:25:27.674482  1803 rpcz_store.cc:276] 0709 12:25:26.222389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.518557 (+296168us) service_pool.cc:224] Handling call
0709 12:25:26.518995 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:26.519410 (+   415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.519456 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:26.520009 (+   553us) log.cc:844] Serialized 26 byte log entry
0709 12:25:26.708521 (+188512us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:26.708665 (+   144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.708851 (+   186us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.571455 (+862604us) raft_consensus.cc:1685] finished
0709 12:25:27.571487 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.571700 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7722,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":186021,"threads_started":1,"wal-append.queue_time_us":415123}
I20260709 12:25:27.781083  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:27.783715  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:27.789644  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 51950.3us
W20260709 12:25:27.801211  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:27.814426  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 74964.7us
W20260709 12:25:27.824967  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 415996us
I20260709 12:25:27.849020  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1251) took 1622 ms. Trace:
I20260709 12:25:27.861513  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:27.862978  9840 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.863684  9840 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:27.857481  9748 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.865244  9837 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:27.866003  9840 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:27.864022  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:27.788208  9573 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 12:25:27.857939  1730 rpcz_store.cc:276] 0709 12:25:26.226612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.235209 (+1008597us) service_pool.cc:224] Handling call
0709 12:25:27.267366 (+ 32157us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.315905 (+ 48539us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:27.598136 (+282231us) log.cc:844] Serialized 14 byte log entry
0709 12:25:27.837926 (+239790us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.838003 (+    77us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:27.848605 (+ 10602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.848732 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.848932 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":410932,"threads_started":2}
I20260709 12:25:27.822463  9755 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.052s
I20260709 12:25:27.887518  2411 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:27.879200  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2976) took 1598 ms. Trace:
I20260709 12:25:27.826370  9501 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:27.834937  9735 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:25:27.866662  9748 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:27.890427  1727 rpcz_store.cc:276] 0709 12:25:26.280627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.689780 (+1409153us) service_pool.cc:224] Handling call
0709 12:25:27.827801 (+138021us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.878520 (+ 50719us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.878637 (+   117us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:27.878716 (+    79us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:27.878805 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.878908 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.879104 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:27.931520  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 219336us
I20260709 12:25:27.933063  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1361) took 1343 ms. Trace:
W20260709 12:25:27.934114  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 109010us
I20260709 12:25:27.933612  2416 rpcz_store.cc:276] 0709 12:25:26.589798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.599632 (+  9834us) service_pool.cc:224] Handling call
0709 12:25:26.599972 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.423582 (+823610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.900857 (+477275us) spinlock_profiling.cc:243] Waited 821 ms on lock 0x7b5400525154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:27.932978 (+ 32121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1888431872}
I20260709 12:25:27.944682  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:27.946759  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82875.4us
I20260709 12:25:27.941638  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:27.953191  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 256228us
I20260709 12:25:27.926276  9645 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:25:27.898293  9796 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:28.005131  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 109852us
W20260709 12:25:28.017249  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.027146  1724 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:25:28.028173  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:28.028966  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 110080us
I20260709 12:25:28.031383  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2978) took 1686 ms. Trace:
I20260709 12:25:28.035889  1724 rpcz_store.cc:276] 0709 12:25:26.349951 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.888349 (+1538398us) service_pool.cc:224] Handling call
0709 12:25:27.954956 (+ 66607us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:28.027620 (+ 72664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.031321 (+  3701us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:28.040109  9839 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.251s	user 0.000s	sys 0.001s
W20260709 12:25:28.049064  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.050833  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1147) took 1280 ms. Trace:
I20260709 12:25:28.051373  1800 rpcz_store.cc:276] 0709 12:25:26.770364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.150537 (+380173us) service_pool.cc:224] Handling call
0709 12:25:27.151071 (+   534us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.151661 (+   590us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.151692 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.152476 (+   784us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.453578 (+301102us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b48001c0158. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:25:27.582834 (+129256us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:27.890214 (+307380us) log.cc:844] Serialized 14 byte log entry
0709 12:25:27.890413 (+   199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.890486 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.042495 (+152009us) raft_consensus.cc:1685] finished
0709 12:25:28.050471 (+  7976us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.050631 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7493,"prepare.queue_time_us":415568,"raft.queue_time_us":93367,"raft.run_cpu_time_us":3588,"raft.run_wall_time_us":311443,"spinlock_wait_cycles":25414656,"thread_start_us":705024,"threads_started":2,"wal-append.queue_time_us":585840}
I20260709 12:25:28.053679  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:28.055033  9866 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.056013  9870 raft_consensus.cc:2764] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 12:25:28.057195  9501 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:28.058868  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:28.059980  8961 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:28.060577  8961 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:28.061015  8961 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:28.061721  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:28.055727  9866 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:28.070225  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3792) took 2398 ms. Trace:
W20260709 12:25:28.040186  9847 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:25:28.079284  1729 rpcz_store.cc:276] 0709 12:25:25.672289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.198496 (+1526207us) service_pool.cc:224] Handling call
0709 12:25:27.198964 (+   468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.234207 (+ 35243us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:27.650155 (+415948us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.069012 (+418857us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:28.069225 (+   213us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:28.069373 (+   148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.069666 (+   293us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.070160 (+   494us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":640793,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":4566,"raft.queue_time_us":120421,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"thread_start_us":657955,"threads_started":3}
W20260709 12:25:28.095463  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 147043us
W20260709 12:25:28.075181  9313 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.769s	user 0.010s	sys 0.006s
I20260709 12:25:28.103889  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:27.996027  9837 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.092s
I20260709 12:25:28.088841  9772 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:28.145303  1801 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:28.121455  9313 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:28.155011  9846 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 12:25:28.156284  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:25:28.160472  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 132548us
I20260709 12:25:28.187505  9796 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:28.189639  9880 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:28.190344  9880 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.190802  9880 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:28.199252  9748 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:28.200835  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.179303  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1255) took 1916 ms. Trace:
W20260709 12:25:28.165750  9611 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.116s	user 0.013s	sys 0.005s
W20260709 12:25:28.169656  9855 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:25:28.171383  9879 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.174104  9494 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:28.176170  9866 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:28.177470  9853 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.182s	user 0.002s	sys 0.000s
W20260709 12:25:28.178268  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.208953  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2974) took 1932 ms. Trace:
I20260709 12:25:28.254582  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1254) took 2008 ms. Trace:
I20260709 12:25:28.201368  1725 rpcz_store.cc:276] 0709 12:25:26.263370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.361587 (+1098217us) service_pool.cc:224] Handling call
0709 12:25:27.413603 (+ 52016us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.471930 (+ 58327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.471976 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.472689 (+   713us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.902579 (+429890us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:27.902642 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.902741 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.179003 (+276262us) raft_consensus.cc:1685] finished
0709 12:25:28.179030 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.179217 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6770,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":429322,"threads_started":1,"wal-append.queue_time_us":589831}
W20260709 12:25:28.258551  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 236579us
I20260709 12:25:28.253494  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1258) took 1831 ms. Trace:
I20260709 12:25:28.260156  1724 rpcz_store.cc:276] 0709 12:25:26.422529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.062945 (+1640416us) service_pool.cc:224] Handling call
0709 12:25:28.136458 (+ 73513us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:28.253017 (+116559us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:28.253062 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:28.253126 (+    64us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:28.253167 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.253252 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.253415 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:28.279721  9879 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:28.281705  9792 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:28.284925  9838 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.579s	user 0.015s	sys 0.000s
I20260709 12:25:28.282524  9879 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:28.285755  2417 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:28.303925  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1252) took 2063 ms. Trace:
I20260709 12:25:28.235798  9645 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:28.258397  9379 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.253192  1728 rpcz_store.cc:276] 0709 12:25:26.276518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.472183 (+1195665us) service_pool.cc:224] Handling call
0709 12:25:27.472857 (+   674us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.473382 (+   525us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:27.900045 (+426663us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.208573 (+308528us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:28.208626 (+    53us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:28.208667 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.208740 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.208886 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9400,"prepare.queue_time_us":476226,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":92696,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":698900,"threads_started":3}
I20260709 12:25:28.255146  1723 rpcz_store.cc:276] 0709 12:25:26.246095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.325284 (+1079189us) service_pool.cc:224] Handling call
0709 12:25:27.386546 (+ 61262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.418389 (+ 31843us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:27.845811 (+427422us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.217368 (+371557us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:28.217440 (+    72us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:28.254296 (+ 36856us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.254380 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.254524 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4515,"prepare.queue_time_us":509982,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":72625,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"thread_start_us":740824,"threads_started":3}
I20260709 12:25:28.318043  9882 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:28.320027  9877 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.322881  9866 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.316075  9573 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.316567  9851 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:28.345696  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 308790us
I20260709 12:25:28.316475  9755 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:28.176283  9772 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.217625  9611 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:28.331123  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 391994us
I20260709 12:25:28.333202  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:28.312995  1726 rpcz_store.cc:276] 0709 12:25:26.240389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.253073 (+1012684us) service_pool.cc:224] Handling call
0709 12:25:27.345744 (+ 92671us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.416915 (+ 71171us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.416967 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.475699 (+ 58732us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.900545 (+424846us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:27.900609 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.900712 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.295063 (+394351us) raft_consensus.cc:1685] finished
0709 12:25:28.295103 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.303453 (+  8350us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5386,"raft.queue_time_us":30631,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":8138,"thread_start_us":383695,"threads_started":1,"wal-append.queue_time_us":521016}
W20260709 12:25:28.356781  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.346539  9846 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.325155  9501 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 12:25:28.379469  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2818) took 1311 ms. Trace:
I20260709 12:25:28.387370  9865 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:28.387184  9645 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.388139  9865 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.386291  1799 rpcz_store.cc:276] 0709 12:25:27.086263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.212687 (+126424us) service_pool.cc:224] Handling call
0709 12:25:27.247667 (+ 34980us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.324934 (+ 77267us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:27.691938 (+367004us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.365164 (+673226us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.365501 (+   337us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:28.365587 (+    86us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:28.365661 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.365765 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.365954 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35301,"prepare.queue_time_us":536215,"prepare.run_cpu_time_us":136,"prepare.run_wall_time_us":141,"raft.queue_time_us":94403,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":14117,"thread_start_us":759713,"threads_started":3,"wal-append.queue_time_us":446309}
I20260709 12:25:28.383538  9379 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.385725  9573 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:28.528174  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 466553us
W20260709 12:25:28.529764  9849 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 12:25:28.590893  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1253) took 2348 ms. Trace:
I20260709 12:25:28.591295  1731 rpcz_store.cc:276] 0709 12:25:26.242544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.258494 (+1015950us) service_pool.cc:224] Handling call
0709 12:25:27.265767 (+  7273us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.314947 (+ 49180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.314993 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.405935 (+ 90942us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.792627 (+386692us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:27.792667 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.792741 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.590643 (+797902us) raft_consensus.cc:1685] finished
0709 12:25:28.590663 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.590832 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9184,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":55009,"thread_start_us":361437,"threads_started":1,"wal-append.queue_time_us":827539}
I20260709 12:25:28.534680  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1256) took 2140 ms. Trace:
I20260709 12:25:28.541147  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:28.543417  1803 raft_consensus.cc:1217] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Deduplicated request from leader. Original: 20.11->[21.12-21.12]   Dedup: 21.12->[]
I20260709 12:25:28.552211  9379 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.553465  9611 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:28.557377  1880 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 15. Current term is 16. Ops: []
I20260709 12:25:28.557377  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:28.381893  9882 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:28.377951  9877 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:28.536471  9869 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:25:28.544258  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:28.568912  9872 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:28.491420  9866 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:28.570887  9796 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:28.534891  9755 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:28.576568  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.395605  9851 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:28.399089  9846 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:28.579628  9875 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:25:28.580319  9892 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:28.580499  9573 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.580994  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:28.614732  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 92250.5us
I20260709 12:25:28.614984  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:28.615957  1801 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:28.537994  9865 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.588260  9772 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:28.595492  1727 rpcz_store.cc:276] 0709 12:25:26.394531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.927411 (+1532880us) service_pool.cc:224] Handling call
0709 12:25:27.976506 (+ 49095us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.992377 (+ 15871us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:28.030379 (+ 38002us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.350151 (+319772us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.533933 (+183782us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b1800368680. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:25:28.534215 (+   282us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:28.534279 (+    64us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:28.534354 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.534433 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.534604 (+   171us) inbound_call.cc:177] Queueing success response
0709 12:25:28.561897 (+ 27293us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b1800368680. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
Metrics: {"mutex_wait_us":153112,"prepare.queue_time_us":46201,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":120,"raft.queue_time_us":187209,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":25895,"spinlock_wait_cycles":7207424,"thread_start_us":252650,"threads_started":1,"wal-append.queue_time_us":449617}
W20260709 12:25:28.598220  1805 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.572s	user 0.012s	sys 0.000s
I20260709 12:25:28.599016  1803 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:25:28.604318  9872 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:28.616784  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:28.573832  9880 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.606458  9796 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:28.625551  1801 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
W20260709 12:25:28.591316  8961 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.530s	user 0.006s	sys 0.010s
I20260709 12:25:28.623487  1799 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:28.620414  1805 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:28.628916  9893 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:28.629122  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:28.629784  1801 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:28.630200  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:28.631235  9882 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:28.633148  9866 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:28.632431  9796 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:28.631304  1801 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:25:28.636085  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1148) took 1408 ms. Trace:
I20260709 12:25:28.618096  9895 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.639885  9893 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:28.637527  1807 rpcz_store.cc:276] 0709 12:25:27.227638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.539222 (+311584us) service_pool.cc:224] Handling call
0709 12:25:27.539652 (+   430us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.540139 (+   487us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.540175 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.540795 (+   620us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.918888 (+378093us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:27.989664 (+ 70776us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.634045 (+644381us) spinlock_profiling.cc:243] Waited 3.39 ms on lock 0x7b180042f460. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287
0709 12:25:28.634169 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.634246 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.634292 (+    46us) raft_consensus.cc:1685] finished
0709 12:25:28.635855 (+  1563us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.636020 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82426,"prepare.queue_time_us":99383,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":4748,"raft.queue_time_us":135288,"raft.run_cpu_time_us":530,"raft.run_wall_time_us":15460,"spinlock_wait_cycles":7789440,"thread_start_us":512669,"threads_started":3,"wal-append.queue_time_us":514176}
I20260709 12:25:28.639462  9840 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:28.664985  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.663019  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:28.634908  8961 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.655792  9882 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:28.665557  9840 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.629841  9892 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:28.690294  1732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.005s	sys 0.021s
W20260709 12:25:28.691213  1732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.005s	sys 0.021s
I20260709 12:25:28.693756  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:28.694717  1807 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
W20260709 12:25:28.695853  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 117623us
I20260709 12:25:28.672562  9796 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:28.659693  9772 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.025s
I20260709 12:25:28.697507  1801 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:28.698133  9755 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.700136  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:28.684772  9838 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:28.690033  9840 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:28.707032  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.781s (SENT) [suppressed 8 similar messages]
I20260709 12:25:28.717120  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:28.717865  1801 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
W20260709 12:25:28.723302  9883 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.146s	user 0.001s	sys 0.000s
I20260709 12:25:28.723955  9892 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.724318  9877 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.693190  9882 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.685762  9872 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:28.724864  9841 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.039s	user 0.005s	sys 0.002s
I20260709 12:25:28.690167  9895 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.733493  9893 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.743409  9872 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:28.744930  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1545) took 1574 ms. Trace:
I20260709 12:25:28.745434  2415 rpcz_store.cc:276] 0709 12:25:27.170102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.209880 (+ 39778us) service_pool.cc:224] Handling call
0709 12:25:27.213626 (+  3746us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.217422 (+  3796us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:25:27.217479 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.218598 (+  1119us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.515271 (+296673us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:25:27.515344 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.515474 (+   130us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.655125 (+1139651us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:28.744466 (+ 89341us) raft_consensus.cc:1685] finished
0709 12:25:28.744496 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.744739 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":205841,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":96,"spinlock_wait_cycles":25021824,"thread_start_us":291026,"threads_started":1,"wal-append.queue_time_us":405184}
I20260709 12:25:28.752206  9851 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:28.756203  9880 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:28.757264  9895 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.739506  9882 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.767334  9866 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:28.767557  9884 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.768222  9866 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.768229  9884 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.770920  9884 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.771200  9866 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.774839  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 2977) took 2493 ms. Trace:
I20260709 12:25:28.775300  1732 rpcz_store.cc:276] 0709 12:25:26.280999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.814515 (+1533516us) service_pool.cc:224] Handling call
0709 12:25:27.874458 (+ 59943us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:27.979538 (+105080us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.979573 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:28.040553 (+ 60980us) log.cc:844] Serialized 52 byte log entry
0709 12:25:28.691663 (+651110us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:28.709927 (+ 18264us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.710317 (+   390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.710415 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.774545 (+ 64130us) raft_consensus.cc:1685] finished
0709 12:25:28.774575 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.774739 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":6297,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":1027,"raft.queue_time_us":177507,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":4061,"thread_start_us":743063,"threads_started":2,"wal-append.queue_time_us":690013}
I20260709 12:25:28.774362  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.779285  1727 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
W20260709 12:25:28.777410  1696 connection.cc:256] RPC call timeout handler was delayed by -1.8786382500002219s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32010000000000005s for a response
I20260709 12:25:28.758404  8961 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.783452  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:28.788769  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1154) took 1520 ms. Trace:
I20260709 12:25:28.789374  1806 rpcz_store.cc:276] 0709 12:25:27.268517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.569160 (+300643us) service_pool.cc:224] Handling call
0709 12:25:27.569759 (+   599us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:27.570552 (+   793us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:27.903107 (+332555us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.013064 (+109957us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:28.013803 (+   739us) log.cc:844] Serialized 52 byte log entry
0709 12:25:28.013972 (+   169us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:28.018759 (+  4787us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.363717 (+344958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.364179 (+   462us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.577461 (+213282us) spinlock_profiling.cc:243] Waited 5.79 ms on lock 0x7b18002f38c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16b0d01328 00007f16b0cf8901 00007f16b0cf88a7 00007f16b0df0288 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508
0709 12:25:28.788475 (+211014us) raft_consensus.cc:1685] finished
0709 12:25:28.788501 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.788679 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":457406,"prepare.queue_time_us":467649,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":78,"raft.queue_time_us":217635,"raft.run_cpu_time_us":3956,"raft.run_wall_time_us":915233,"spinlock_wait_cycles":13404672,"thread_start_us":682216,"threads_started":3,"wal-append.queue_time_us":438733}
I20260709 12:25:28.801540  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.803064  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:28.780258  9866 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.805924  1729 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:28.808506  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.811002  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.807009  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.809545  1727 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:28.812075  1731 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
W20260709 12:25:28.813009  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 106013us
W20260709 12:25:28.814244  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 92806.1us
W20260709 12:25:28.816192  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.817296  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.811753  9884 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:28.827353  1616 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet 8bb813e4fab14a61b810a24b56899586 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 123095127f334445889c21a42ba1d41e is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 20 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:28.831233  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:28.832151  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:28.833734  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:28.853426  1804 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:28.827358  9846 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:28.841200  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57889.5us
I20260709 12:25:28.842617  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:28.845412  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:28.842479  9836 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.956s	user 0.008s	sys 0.008s
I20260709 12:25:28.855798  2417 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:28.851744  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:28.857260  1731 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:28.858829  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:28.859448  2415 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:28.853752  9884 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.870002  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:28.817214  9866 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:28.871364  1800 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:25:28.876016  9880 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:28.876724  9880 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.878011  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:28.878628  2410 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:28.865222  9561 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:25:28.879824  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:28.880743  2410 raft_consensus.cc:2410] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
W20260709 12:25:28.881088  9867 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.362s	user 0.002s	sys 0.000s
I20260709 12:25:28.881850  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:28.890987  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:28.891502  1804 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:28.891782  1732 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:28.870738  9884 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:28.888382  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:28.898870  9908 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:28.899796  9908 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.898962  9645 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.002s
I20260709 12:25:28.903319  1725 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:25:28.903457  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:28.904623  9866 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:28.905517  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44843.1us
I20260709 12:25:28.906641  9772 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:28.906756  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:28.907476  9772 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.908010  9772 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:28.908504  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:28.909492  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:28.909559  9501 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:28.910646  9501 consensus_queue.cc:1243] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:25:28.911043  9541 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:28.911723  9541 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.912569  9879 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:28.914062  9541 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.912019  9887 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:28.914954  9904 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:28.915982  9755 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:25:28.917053  9501 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:25:28.922524  9876 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.584s	user 0.001s	sys 0.003s
I20260709 12:25:28.927253  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.927088  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.927663  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.928188  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.931561  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.932359  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.949033  1727 raft_consensus.cc:2410] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
W20260709 12:25:28.950726  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 135059us
W20260709 12:25:28.947731  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 101761us
I20260709 12:25:28.902696  9908 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.973453  1732 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:28.937379  9887 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:28.966681  1728 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:28.976480  1798 raft_consensus.cc:1217] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Deduplicated request from leader. Original: 17.11->[18.12-21.14]   Dedup: 20.13->[21.14-21.14]
I20260709 12:25:28.982286  1798 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [18.12-21.14]
I20260709 12:25:28.939633  9755 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:28.984944  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.985754  1729 raft_consensus.cc:2368] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22.
I20260709 12:25:28.986076  1732 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:28.987306  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:28.988754  9907 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.989611  9907 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:28.992298  1731 raft_consensus.cc:2410] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:25:28.992506  9907 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:28.993834  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.994131  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.943046  9541 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.995232  1728 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:29.002998  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:28.984107  9887 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:29.021287  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:29.025352  1731 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
W20260709 12:25:29.025904  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.028216  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:29.029096  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:29.027045  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.013747  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:29.035770  1726 raft_consensus.cc:2368] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
W20260709 12:25:29.036723  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 142318us
W20260709 12:25:29.040113  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 114935us
I20260709 12:25:29.042098  2383 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:28.975746  9904 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:29.054132  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 114891us
I20260709 12:25:29.052626  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:29.046639  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.055943  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.053530  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:29.011710  9907 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
I20260709 12:25:29.010881  9541 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.080663  1873 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:29.041719  9888 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.087285  1730 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:29.014307  9772 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.079469  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.090931  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.091400  1880 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:29.090560  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.087702  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:29.095616  1801 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 28 because replica has last-logged OpId of term: 27 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 10.
I20260709 12:25:29.096858  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:29.098160  1807 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 24 because replica has last-logged OpId of term: 23 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:25:29.105901  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:29.107909  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 1153) took 1835 ms (client timeout 1588 ms). Trace:
W20260709 12:25:29.108417  1805 rpcz_store.cc:269] 0709 12:25:27.268166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.460552 (+192386us) service_pool.cc:224] Handling call
0709 12:25:29.103263 (+1642711us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b5400166cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:29.103363 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1298586880}
I20260709 12:25:29.116221  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2815) took 1922 ms. Trace:
I20260709 12:25:29.120994  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:29.122038  1726 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:29.122815  1875 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:29.129199  1803 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:29.135032  1873 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
W20260709 12:25:29.135699  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 122058us
I20260709 12:25:29.141126  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:29.153574  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:29.147467  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.168287  2418 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:29.137938  9846 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:29.140213  9840 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.176704  2411 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:29.150346  9762 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:29.186146  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 22 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.188683  1730 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:29.103920  9888 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.116984  1802 rpcz_store.cc:276] 0709 12:25:27.053815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.192394 (+138579us) service_pool.cc:224] Handling call
0709 12:25:27.193179 (+   785us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:27.193921 (+   742us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.193963 (+    42us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:27.195565 (+  1602us) log.cc:844] Serialized 78 byte log entry
0709 12:25:27.490143 (+294578us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:27.903344 (+413201us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.019329 (+115985us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.396405 (+377076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.396517 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.573755 (+177238us) spinlock_profiling.cc:243] Waited 9.82 ms on lock 0x7b18002a74e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:28.787509 (+213754us) spinlock_profiling.cc:243] Waited 4.46 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:28.975772 (+188263us) raft_consensus.cc:1685] finished
0709 12:25:28.975806 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.976070 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":452097,"prepare.queue_time_us":522382,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":79,"raft.queue_time_us":258304,"raft.run_cpu_time_us":492,"raft.run_wall_time_us":427308,"spinlock_wait_cycles":32827904,"thread_start_us":1142046,"threads_started":4,"wal-append.queue_time_us":620629}
I20260709 12:25:29.095752  9755 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:29.191116  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.208503  1872 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:25:29.210405  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.213071  9904 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:29.214221  9882 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:29.215075  9882 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:29.216377  9924 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:25:29.217702  9882 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:29.218333  9919 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:29.219098  9919 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:29.219203  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.220309  9919 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:29.222949  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63157.1us
I20260709 12:25:29.223112  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.224006  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.224990  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.225521  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.219211  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:29.226512  1881 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:25:29.227375  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.233799  1877 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 21 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:25:29.228307  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.228916  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:29.211323  1873 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
W20260709 12:25:29.248775  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.248994  1726 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:29.232527  1879 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:29.251253  1730 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:29.203969  9762 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:29.252362  1875 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:29.188081  9846 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.291503  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:29.292738  1723 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:29.255123  9755 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:29.311245  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.315999  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.317482  1880 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 16.
I20260709 12:25:29.322670  1723 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:25:29.315624  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.332191  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:29.337453  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 283355us
I20260709 12:25:29.333792  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:29.347414  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 268373us
I20260709 12:25:29.328011  9772 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:29.352077  1874 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:29.326470  9541 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:29.356151  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 325359us
I20260709 12:25:29.361951  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:29.360103  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:29.381915  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.382519  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.384545  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:29.384850  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.385699  1803 raft_consensus.cc:2368] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22.
I20260709 12:25:29.387604  1806 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:25:29.388182  9888 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:29.388847  9888 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:29.389240  9888 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:29.389626  9888 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:29.391245  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:29.398721  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 313301us
I20260709 12:25:29.398973  1723 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 20 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:25:29.406661  9928 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:25:29.407155  9865 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:29.407644  9865 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:29.408074  9865 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:29.410446  1732 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:25:29.421558  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.361433  9866 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:29.422430  1874 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:29.426749  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:29.428162  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:29.431217  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.433820  1730 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:25:29.437240  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.445210  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.445533  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:29.446480  1804 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:29.448320  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 161515us
I20260709 12:25:29.452193  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.465828  1874 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:29.472047  1803 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:29.474038  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.474999  1799 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:29.491899  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.356418  9762 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 21.18, Last appended by leader: 18, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.501075  1730 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:29.422619  9884 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.480695  9846 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:29.512049  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:29.516752  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 157938us
I20260709 12:25:29.449414  9866 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.517534  1875 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:29.528045  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.545377  1731 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:29.551596  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 107257us
I20260709 12:25:29.513495  9840 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:29.567087  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.564808  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:29.565788  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.572714  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.576792  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:29.578289  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 133599us
I20260709 12:25:29.578624  1803 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:29.600258  1730 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:29.600951  1730 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:29.604200  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.607702  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:29.608392  1802 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:29.608510  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:29.609553  1803 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
W20260709 12:25:29.617581  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114299us
I20260709 12:25:29.619580  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:29.620514  1874 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:29.620976  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.643675  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:29.644255  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:29.645900  9931 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
I20260709 12:25:29.646183  9953 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 12:25:29.647553  1730 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:29.639508  9935 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:29.648905  1730 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:29.649235  9935 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:29.637830  9938 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.651422  9938 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.663024  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.663877  1880 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:29.644526  1799 raft_consensus.cc:2376] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24. Re-sending same reply.
I20260709 12:25:29.677366  9946 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:29.683513  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:29.684027  9934 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:25:29.684840  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:29.651448  9884 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:29.697329  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53158.2us
I20260709 12:25:29.637645  9940 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.179s
I20260709 12:25:29.731006  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:29.733026  9947 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:29.733394  9945 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:25:29.733626  9947 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.740267  9946 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.740854  9946 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.743660  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:29.744417  9943 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:29.745121  9943 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:29.745610  9943 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:29.744781  1724 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:29.748684  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.749888  1876 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:25:29.755643  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:29.756394  1873 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:29.751734  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.759966  1875 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:29.762339  1802 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:29.763692  1728 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:25:29.764133  1728 pending_rounds.cc:85] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 11
I20260709 12:25:29.764379  1728 pending_rounds.cc:107] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:25:29.764626  1728 raft_consensus.cc:2889] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:29.784408  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.716646  9953 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:29.718849  9954 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:29.799685  1877 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:25:29.803712  1730 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 12. (term mismatch)
I20260709 12:25:29.786916  9944 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:29.808282  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.808553  1874 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:29.709296  9865 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.806317  1881 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:29.804339  1730 pending_rounds.cc:85] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 11
W20260709 12:25:29.813485  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 117498us
I20260709 12:25:29.818529  1730 pending_rounds.cc:107] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:25:29.818944  1730 raft_consensus.cc:2889] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:29.809026  1872 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
W20260709 12:25:29.822489  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.823500  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:29.824755  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 92305.3us
I20260709 12:25:29.798305  9948 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:29.826196  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 133234us
I20260709 12:25:29.830916  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.814393  9866 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:29.835983  1881 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:25:29.836545  1877 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:29.831583  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:29.846861  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:29.826463  9952 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:25:29.807922  9944 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:29.844923  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:29.852837  1724 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:29.813851  9953 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.862295  1872 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:25:29.865334  9953 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:29.865883  1881 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:25:29.869668  1723 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:29.871776  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2803) took 1328 ms. Trace:
I20260709 12:25:29.872191  1800 rpcz_store.cc:276] 0709 12:25:28.543100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.545156 (+  2056us) service_pool.cc:224] Handling call
0709 12:25:28.545531 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:28.873448 (+327917us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.871520 (+998072us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b54001d4a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:29.871712 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":744658176}
W20260709 12:25:29.877605  9959 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:25:29.879359  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:29.879998  9865 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:29.882203  9944 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:29.831781  9957 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:25:29.887749  9494 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:29.808032  9954 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:29.890753  9954 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:29.852572  9948 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:29.891598  9948 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.893220  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.893537  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.895140  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.895356  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:29.887749  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56221.6us
I20260709 12:25:29.896879  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:29.898463  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:29.899138  1802 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:29.901590  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:29.902062  1873 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:29.902496  1731 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:29.903102  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.894500  9948 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:29.909746  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:29.914367  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.914326  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:29.923331  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.924140  1873 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:29.928516  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.930102  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.931519  1873 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:29.932677  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.933444  2417 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:25:29.936523  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:29.940946  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:29.950186  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.955344  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.956727  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.957664  1873 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:29.954070  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:29.959811  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.962356  1873 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:29.963853  1802 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:29.973625  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.951401  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:29.974766  1878 raft_consensus.cc:2410] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 23 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:25:29.975047  1729 raft_consensus.cc:606] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:29.975416  1729 raft_consensus.cc:614] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:29.975709  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:29.976799  2418 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:29.978670  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:29.980052  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:29.980888  2414 raft_consensus.cc:2368] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23.
I20260709 12:25:29.985713  9947 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:29.955717  9964 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:29.986888  9964 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:29.987133  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.987427  9964 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:29.987656  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.926187  9394 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:29.991132  9394 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:29.980080  9888 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.590s	user 0.009s	sys 0.004s
I20260709 12:25:29.991683  9394 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:29.991856  9888 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.994424  9888 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:29.996201  9944 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.000242  9944 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:30.001392  9960 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:25:30.003144  9944 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:30.002588  9964 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:29.998555  9501 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.997882  9866 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:30.005414  1872 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:25:30.005937  9943 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:29.962767  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:30.008788  1724 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:29.999065  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:30.014443  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 60472.5us
I20260709 12:25:30.022223  2416 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:30.023417  1729 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
W20260709 12:25:30.025233  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62604.5us
W20260709 12:25:30.030488  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 192972us
W20260709 12:25:30.008317  9887 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.966s	user 0.014s	sys 0.003s
I20260709 12:25:30.034598  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:30.037583  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 197805us
I20260709 12:25:30.038017  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:30.054373  9964 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:30.045401  9904 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:30.055725  9904 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.056205  9904 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:30.049330  1873 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:30.049813  1846 leader_election.cc:400] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23.
W20260709 12:25:30.044654  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 103799us
I20260709 12:25:30.059623  1846 leader_election.cc:403] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:25:30.061565  9865 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:30.069600  9494 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:30.071254  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.072562  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.073325  1730 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:30.076678  2416 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:30.079082  9947 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:30.080212  9865 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23.
I20260709 12:25:30.087285  2414 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:30.089174  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.090199  1732 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:30.091971  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:30.093848  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.095537  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:30.096081  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49065.6us
W20260709 12:25:30.096868  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45498us
W20260709 12:25:30.097349  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98380.9us
I20260709 12:25:30.098031  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:30.040153  9872 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:30.046726  9906 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:25:30.114586  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 194781us
I20260709 12:25:30.109074  9944 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:25:30.119727  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.108368  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:30.116490  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.116797  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:30.117383  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.106146  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:30.029837  9501 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:30.116922  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:30.121624  1732 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:25:30.030483  9866 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:30.125887  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.115562  9906 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.047456  9887 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:30.200311  1729 raft_consensus.cc:606] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:30.202930  1729 raft_consensus.cc:614] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:30.207474  9949 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.208233  9949 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:30.208727  9937 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:30.209208  9937 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.209538  9937 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:30.212733  9962 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:30.213383  9962 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:30.214367  9962 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:30.215291  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 107059us
I20260709 12:25:30.216599  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:30.218679  9884 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:30.201073  9893 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:30.220943  9893 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:30.221958  9893 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:30.216543  9944 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:30.225543  9501 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:30.225708  9944 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:30.226240  9944 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:30.110196  9954 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.025s
I20260709 12:25:30.227301  9944 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:30.229156  9944 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:30.223212  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:30.232543  1725 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:30.244506  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:30.199194  9872 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:30.250576  9872 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:30.251114  9872 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:30.251534  1872 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:30.252238  9866 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:30.198915  9888 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:30.252786  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:30.253187  9888 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:30.253715  1872 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:30.254146  9888 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:30.240439  9963 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:30.256333  9846 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:25:30.254707  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:30.245465  1880 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:30.201686  9906 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:30.263932  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:30.264226  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:30.265904  1879 raft_consensus.cc:2376] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24. Re-sending same reply.
I20260709 12:25:30.210721  9949 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:30.279312  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 141978us
I20260709 12:25:30.283083  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:30.278532  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:30.301307  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:30.305375  1879 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:25:30.280182  9961 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:30.319422  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:30.320398  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:30.308717  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 199753us
I20260709 12:25:30.318073  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:30.328083  1805 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:25:30.322918  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:30.323185  9938 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:30.323724  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:30.335279  1800 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:30.336277  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2815) took 1210 ms. Trace:
I20260709 12:25:30.336690  1805 rpcz_store.cc:276] 0709 12:25:29.125976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.317610 (+1191634us) service_pool.cc:224] Handling call
0709 12:25:30.336209 (+ 18599us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:30.330232  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:30.342682  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:30.331190  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:30.333398  1804 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:30.289921  9866 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:30.345255  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:30.332757  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:30.308627  9884 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:25:30.432312  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101202us
I20260709 12:25:30.433712  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:30.437237  1800 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
W20260709 12:25:30.333953  9941 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.734s	user 0.000s	sys 0.004s
I20260709 12:25:30.441599  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:30.441844  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:30.442390  1879 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:30.328648  9943 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:30.444736  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:30.445428  1802 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:30.449728  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:30.450773  1805 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:30.427273  9954 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:30.471480  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 138813us
I20260709 12:25:30.438283  9501 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:25:30.490224  9974 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.490919  9977 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:30.494059  9950 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.495703  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:30.511705  9541 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:30.513331  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:30.518476  9975 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.210s	user 0.000s	sys 0.000s
I20260709 12:25:30.519837  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3811) took 1589 ms. Trace:
I20260709 12:25:30.622260  1728 rpcz_store.cc:276] 0709 12:25:28.930634 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.007334 (+ 76700us) service_pool.cc:224] Handling call
0709 12:25:29.024038 (+ 16704us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.778703 (+754665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.519461 (+740758us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b540030b454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:30.519754 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1681827584}
I20260709 12:25:30.427991  9866 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:30.436386  9884 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:30.462747  9983 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.618131  1726 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:30.608009  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 135502us
I20260709 12:25:30.624075  1876 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:30.566478  9954 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:30.595991  9941 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:25:30.672399  9866 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:30.561070  9977 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:30.697695  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 264031us
I20260709 12:25:30.673105  9884 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:30.619715  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:30.700838  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:30.723349  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 103738us
I20260709 12:25:30.725975  1845 leader_election.cc:400] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22.
I20260709 12:25:30.618644  9541 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:30.727178  9541 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:30.723590  9394 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.731s	user 0.015s	sys 0.007s
I20260709 12:25:30.723284  9938 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:30.726305  1845 leader_election.cc:403] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Cancelling election due to peer responding with higher term
I20260709 12:25:30.728544  9541 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:30.731094  9907 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 21. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22.
I20260709 12:25:30.733227  9938 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:30.730062  9394 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:30.739504  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:30.688565  9985 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:30.673280  9980 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:30.742187  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:30.524194  9501 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:30.745251  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.568447  9974 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:30.759635  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:30.742036  9975 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:25:30.715139  9887 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:30.771360  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:30.753801  9938 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:30.763980  9953 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:30.774204  9907 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:25:30.783861  9973 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.345s	user 0.001s	sys 0.000s
W20260709 12:25:30.785924  9978 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260709 12:25:30.786337  9981 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.787364  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3812) took 1843 ms. Trace:
I20260709 12:25:30.831727  1872 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
W20260709 12:25:30.789269  9982 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:25:30.790045  9937 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.580s	user 0.010s	sys 0.001s
W20260709 12:25:30.790299  9986 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 12:25:30.791293  9979 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.445s	user 0.001s	sys 0.000s
I20260709 12:25:30.793205  9880 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:30.795826  9994 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.796046  9927 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:25:30.796976  2411 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1448) took 1955 ms (client timeout 2199 ms). Trace:
I20260709 12:25:30.797029  9887 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.798686  1801 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:30.848412  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2829) took 1433 ms. Trace:
W20260709 12:25:30.806105  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 84191.4us
I20260709 12:25:30.809981  1879 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:30.762945  9501 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:30.624228  9950 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:30.824621  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.824467  1726 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:30.826275  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.828472  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:30.832067  1727 rpcz_store.cc:276] 0709 12:25:28.944305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.008792 (+ 64487us) service_pool.cc:224] Handling call
0709 12:25:29.041293 (+ 32501us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:29.070161 (+ 28868us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:29.246062 (+175901us) log.cc:844] Serialized 14 byte log entry
0709 12:25:29.330740 (+ 84678us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:29.344063 (+ 13323us) log.cc:844] Serialized 26 byte log entry
0709 12:25:29.344227 (+   164us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:29.344440 (+   213us) log.cc:844] Serialized 14 byte log entry
0709 12:25:29.358945 (+ 14505us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.359034 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:29.869042 (+510008us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b1800310ba0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:30.008324 (+139282us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b1800310ba0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:30.323664 (+315340us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:30.787036 (+463372us) raft_consensus.cc:1685] finished
0709 12:25:30.787060 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.787287 (+   227us) inbound_call.cc:177] Queueing success response
0709 12:25:30.788440 (+  1153us) spinlock_profiling.cc:243] Waited 7.78 ms on lock 0x7b4800041c58. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
Metrics: {"mutex_wait_us":29248,"prepare.queue_time_us":257921,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":2614,"raft.queue_time_us":26636,"raft.run_cpu_time_us":13638,"raft.run_wall_time_us":1741713,"spinlock_wait_cycles":103645952,"thread_start_us":178765,"threads_started":2,"wal-append.queue_time_us":237243}
W20260709 12:25:30.835983  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 123278us
W20260709 12:25:30.802729  9872 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.551s	user 0.008s	sys 0.000s
W20260709 12:25:30.838162  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56152.4us
I20260709 12:25:30.716832  9983 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:30.790243  9935 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:30.774735  9991 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.848873  1801 rpcz_store.cc:276] 0709 12:25:29.414822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.414978 (+   156us) service_pool.cc:224] Handling call
0709 12:25:30.798340 (+1383362us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3eee 000000000036eb39 00007f16b0d271e0 00007f16b0df00c4 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:25:30.798496 (+   156us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:30.848040 (+ 49544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.848335 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36399872}
I20260709 12:25:30.759920  9985 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:30.859213  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:30.847054  2411 rpcz_store.cc:269] 0709 12:25:28.841671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.842334 (+   663us) service_pool.cc:224] Handling call
0709 12:25:30.796748 (+1954414us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54008c0354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:30.796896 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":739713280}
I20260709 12:25:30.775619  9980 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:30.918094  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.913120  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:30.915449  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.921136  1726 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:25:30.921491  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:30.840415  9937 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:30.857117  9977 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:30.968147  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:30.975322  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:30.961102  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 43677.2us
I20260709 12:25:30.989782  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1179) took 1144 ms. Trace:
I20260709 12:25:30.979142  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1268) took 1898 ms. Trace:
I20260709 12:25:30.996043  1728 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:30.968868  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47475.4us
I20260709 12:25:30.997274  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3742) took 1365 ms. Trace:
I20260709 12:25:31.036116  1802 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 12:25:30.967748  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:30.960753  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.039383  1800 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:31.038862  1805 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:30.959452  1727 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:25:30.965952  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 194684us
I20260709 12:25:30.923763  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:30.978329  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89020.5us
I20260709 12:25:31.040614  1723 rpcz_store.cc:276] 0709 12:25:29.081226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.402127 (+320901us) service_pool.cc:224] Handling call
0709 12:25:29.869283 (+467156us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.870115 (+   832us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.978874 (+1108759us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54000d1bd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:30.979065 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1073363328}
I20260709 12:25:30.912765 10004 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.042835  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:31.011516  9906 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.740s	user 0.010s	sys 0.002s
W20260709 12:25:31.055474  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 136849us
I20260709 12:25:30.884057  9888 raft_consensus.cc:2764] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
W20260709 12:25:30.985284  9990 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:25:31.039947  1874 rpcz_store.cc:276] 0709 12:25:29.632333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.632532 (+   199us) service_pool.cc:224] Handling call
0709 12:25:29.637872 (+  5340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.814081 (+176209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.996919 (+1182838us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:30.997179 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":385599744}
W20260709 12:25:30.969576  9982 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:25:31.095788  1880 raft_consensus.cc:2368] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25.
I20260709 12:25:31.036981  1881 rpcz_store.cc:276] 0709 12:25:29.845864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.866846 (+ 20982us) service_pool.cc:224] Handling call
0709 12:25:29.867162 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:29.896786 (+ 29624us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:29.909810 (+ 13024us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.198700 (+288890us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.198960 (+   260us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.208555 (+  9595us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.208726 (+   171us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:30.208764 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.208852 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.989495 (+780643us) raft_consensus.cc:1685] finished
0709 12:25:30.989552 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.989696 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62137,"prepare.queue_time_us":12843,"prepare.run_wall_time_us":56,"raft.queue_time_us":32958,"raft.run_cpu_time_us":685,"raft.run_wall_time_us":63035,"spinlock_wait_cycles":6790272,"thread_start_us":162381,"threads_started":1,"wal-append.queue_time_us":340536}
I20260709 12:25:30.889987  9992 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:31.082357 10004 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.099466  9992 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.099704  9923 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:31.100821  1801 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:25:30.994832  9394 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:30.993196  9980 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:30.994968  9954 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:30.993511  9946 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.055722  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:30.923697  9880 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:30.837265  9981 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:30.975597  9997 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.994135  9974 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:30.991832  9983 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:30.990134  9985 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:30.975394  9993 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:30.881752  9887 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.100035  1803 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:31.111228  1728 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 18. Preceding OpId from leader: term: 23 index: 19. (index mismatch)
I20260709 12:25:30.975505  9998 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:30.889010  9872 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:30.975991 10005 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:31.105115  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 49535.2us
W20260709 12:25:31.105428  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.017s	sys 0.023s
W20260709 12:25:31.119982  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72841.9us
W20260709 12:25:31.122843  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.018s	sys 0.023s
I20260709 12:25:31.107663  1727 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:30.856556  9994 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:31.098431  9904 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.037s	user 0.008s	sys 0.007s
I20260709 12:25:30.964685  9996 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.151s
I20260709 12:25:31.101550  9992 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.124771  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:31.097108  9977 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:31.127705  1770 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.132850  9974 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:31.119855  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 80966.3us
I20260709 12:25:31.133371  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:31.134063  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.061220  9906 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.135967  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:31.137051  9974 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:31.137255  9906 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.140269  1805 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:31.140874  9946 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:31.141379  9946 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:31.141444  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.141999  9946 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:31.142375  1805 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:31.135140  9992 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:31.135301  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.143473  9541 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:31.150480  9992 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:31.101579 10004 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.150700  1803 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:31.150920  9541 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:31.151933  9541 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:31.114028  9880 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.151188  9992 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:31.152935  9541 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.103150 10000 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:31.153337  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.154134  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1181) took 1292 ms. Trace:
I20260709 12:25:31.109855  9954 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:31.108229  9394 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:31.154529  1805 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:25:31.151551 10003 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:25:31.102958  9950 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:31.163290  2383 leader_election.cc:336] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.588s (SENT)
I20260709 12:25:31.123224  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:31.166043  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:30.926003  9991 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:31.166692  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:31.167186  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:31.168690  1872 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:31.175287  1879 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:31.172194  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:31.177242  1874 raft_consensus.cc:2376] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22. Re-sending same reply.
I20260709 12:25:31.155643  9981 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.155296  9541 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:31.157209  9887 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.181846  9937 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.184571  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:31.185304  2411 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:31.186383  9880 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.186829  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2833) took 1745 ms. Trace:
I20260709 12:25:31.187575  1728 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:31.187942  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3008) took 1108 ms. Trace:
I20260709 12:25:31.188274  1725 rpcz_store.cc:276] 0709 12:25:30.079015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.890541 (+811526us) service_pool.cc:224] Handling call
0709 12:25:30.957267 (+ 66726us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.983448 (+ 26181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.983513 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.984311 (+   798us) log.cc:844] Serialized 26 byte log entry
0709 12:25:31.105210 (+120899us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:31.105259 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.105332 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.187701 (+ 82369us) raft_consensus.cc:1685] finished
0709 12:25:31.187723 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.187871 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":167,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":108,"thread_start_us":120402,"threads_started":1,"wal-append.queue_time_us":198114}
I20260709 12:25:31.189216  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:31.189945  2411 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:31.191138  9977 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:31.191569  9977 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.192001  9977 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:31.169842  9906 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.193903  9906 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.157632 10000 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:31.203907  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 80771.7us
I20260709 12:25:31.207530  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:31.196976  9906 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:31.197173  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:31.160066  1878 rpcz_store.cc:276] 0709 12:25:29.861589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.977680 (+116091us) service_pool.cc:224] Handling call
0709 12:25:29.977984 (+   304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.039087 (+ 61103us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.039135 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.095804 (+ 56669us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.803428 (+707624us) spinlock_profiling.cc:243] Waited 7.19 ms on lock 0x7b18001c6780. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:25:31.035762 (+232334us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:31.035810 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.035899 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.153656 (+117757us) raft_consensus.cc:1685] finished
0709 12:25:31.153683 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.153940 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71262,"raft.queue_time_us":65836,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":52432,"spinlock_wait_cycles":16536832,"thread_start_us":232116,"threads_started":1,"wal-append.queue_time_us":1002317}
I20260709 12:25:31.200953  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:31.201129  9950 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:31.201749  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.187214  1807 rpcz_store.cc:276] 0709 12:25:29.441490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.441769 (+   279us) service_pool.cc:224] Handling call
0709 12:25:30.505767 (+1063998us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:25:30.506048 (+   281us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:30.699566 (+193518us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.699604 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:30.729271 (+ 29667us) log.cc:844] Serialized 52 byte log entry
0709 12:25:30.956502 (+227231us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:30.956575 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.957985 (+  1410us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.186447 (+228462us) raft_consensus.cc:1685] finished
0709 12:25:31.186478 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.186736 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":338499,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"spinlock_wait_cycles":3221888,"thread_start_us":418333,"threads_started":2,"wal-append.queue_time_us":291942}
I20260709 12:25:31.196743  9991 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:25:31.223137  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3014) took 1093 ms. Trace:
I20260709 12:25:31.229558  1730 rpcz_store.cc:276] 0709 12:25:30.129535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.998020 (+868485us) service_pool.cc:224] Handling call
0709 12:25:30.998448 (+   428us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.008188 (+  9740us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.008220 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.008742 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:25:31.130856 (+122114us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:31.130895 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.130956 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.222866 (+ 91910us) raft_consensus.cc:1685] finished
0709 12:25:31.222890 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.223063 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2633,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":17462,"thread_start_us":121664,"threads_started":1,"wal-append.queue_time_us":211954}
I20260709 12:25:31.164633  9394 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.226594  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:31.169186  9993 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:31.217278  1876 raft_consensus.cc:2376] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23. Re-sending same reply.
W20260709 12:25:31.228276  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62333.3us
I20260709 12:25:31.242012  1872 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:31.219087  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:31.131642  9904 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.225241  9946 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.234925  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1375) took 1006 ms. Trace:
I20260709 12:25:31.113720  9997 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.236140  9755 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:31.236384  9994 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.228570 10002 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.009s
W20260709 12:25:31.237372  9944 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.981s	user 0.007s	sys 0.007s
I20260709 12:25:31.237812  9501 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:31.238555  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 85767.3us
I20260709 12:25:31.219641  9981 raft_consensus.cc:2764] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:25:31.239084  9948 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:31.227126  9996 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.012s
I20260709 12:25:31.283309  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.239486  9394 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:31.244031  2382 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:31.214422 10000 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:31.246434 10007 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:31.246591  9872 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.247730  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.248450  9954 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:31.249589  9950 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:31.244812  2414 rpcz_store.cc:276] 0709 12:25:30.229020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.229711 (+   691us) service_pool.cc:224] Handling call
0709 12:25:30.250106 (+ 20395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.250619 (+   513us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.250658 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.258675 (+  8017us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.700063 (+441388us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:30.700113 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.700188 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.234615 (+534427us) raft_consensus.cc:1685] finished
0709 12:25:31.234643 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.234850 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4603,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":95,"thread_start_us":326147,"threads_started":1,"wal-append.queue_time_us":574233}
I20260709 12:25:31.263798  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:31.265559  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.293443  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:31.232842 10004 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
W20260709 12:25:31.271018  9884 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.506s	user 0.011s	sys 0.002s
I20260709 12:25:31.298131  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:31.272248  9904 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:31.252709  9991 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:31.272488  9946 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:31.276182  1695 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.018s	sys 0.049s
W20260709 12:25:31.301170  1695 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.019s	sys 0.049s
I20260709 12:25:31.279093  9933 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
W20260709 12:25:31.282099  9866 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.535s	user 0.003s	sys 0.005s
I20260709 12:25:31.278455  9998 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:31.290390 10007 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:31.291404  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.294587  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.270553  9997 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.285373  9974 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:31.294034  9954 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.295440  9994 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:31.253372  9755 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:31.297132  9772 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.007s
I20260709 12:25:31.314337  1802 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:31.304159  9993 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.304276  9981 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:31.305768  9980 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:31.327646  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.329941  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:31.300730  9884 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.310030 10007 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:31.317883  9997 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:31.318965  9954 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:31.312413  9887 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:31.306244  9866 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.319187  9994 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.345329  9925 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.050s
I20260709 12:25:31.346032  9951 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:31.332506  9980 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:31.372947  9951 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:31.374046  9951 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.374812  9981 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.375567  9981 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.366092  9994 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:31.380312  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40200 (request call id 68) took 1370 ms. Trace:
I20260709 12:25:31.380826  1729 rpcz_store.cc:276] 0709 12:25:30.009801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.023195 (+ 13394us) service_pool.cc:224] Handling call
0709 12:25:31.380041 (+1356846us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b540044aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd2853 00007f16b0da060a 00007f16aa3267d4 00007f16aa326762 00007f16aa326691 00007f16aa32660c 00007f16aa3258c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:31.380203 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":400555904}
I20260709 12:25:31.382848 10005 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:31.383360 10005 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:31.384485  9866 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.385205  9866 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:31.331933  9993 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.384262 10005 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.387252 10005 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:31.388027  9993 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.339546  9755 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 9, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.395833  9755 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:31.396245  9884 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.398898  1723 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:31.403716  9772 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:31.404667  9838 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:31.405118  9838 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:31.405371  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3760) took 1524 ms. Trace:
I20260709 12:25:31.407168  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:31.405956  9838 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.408551  9838 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:31.412712  1724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
W20260709 12:25:31.413123  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 165824us
I20260709 12:25:31.413376  1724 raft_consensus.cc:606] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.413906  1724 raft_consensus.cc:614] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:31.405418  9501 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.419423  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2801) took 2877 ms. Trace:
I20260709 12:25:31.419901  1798 rpcz_store.cc:276] 0709 12:25:28.542098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.542332 (+   234us) service_pool.cc:224] Handling call
0709 12:25:28.976232 (+433900us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:28.986565 (+ 10333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.419163 (+2432598us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b540035fa50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:31.419329 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":996612992}
I20260709 12:25:31.420630  9932 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:31.421896  9932 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:31.405864  1873 rpcz_store.cc:276] 0709 12:25:29.880735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.982688 (+101953us) service_pool.cc:224] Handling call
0709 12:25:29.983104 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:30.059194 (+ 76090us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.405121 (+1345927us) spinlock_profiling.cc:243] Waited 66 ms on lock 0x7b5400260a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:31.405305 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151720192}
I20260709 12:25:31.422398  9932 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:31.413296  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:31.338109 10002 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:31.338881  9974 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:31.351420  9997 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.318336  9992 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.430524  1724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:31.431268  1724 raft_consensus.cc:606] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.431651  1724 raft_consensus.cc:614] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:31.432916  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3003) took 1357 ms. Trace:
I20260709 12:25:31.365512 10004 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 12:25:31.378665  9981 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.433313  1732 rpcz_store.cc:276] 0709 12:25:30.075409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.135680 (+ 60271us) service_pool.cc:224] Handling call
0709 12:25:30.136116 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.138822 (+  2706us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:30.139341 (+   519us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.309693 (+170352us) log.cc:844] Serialized 14 byte log entry
0709 12:25:30.309957 (+   264us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.310704 (+   747us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.310899 (+   195us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:31.426694 (+1115795us) spinlock_profiling.cc:243] Waited 63.7 ms on lock 0x7b1800320920. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf 00007f16abdf2038
0709 12:25:31.427988 (+  1294us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.428209 (+   221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.432662 (+  4453us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:31.432708 (+    46us) raft_consensus.cc:1685] finished
0709 12:25:31.432721 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.432860 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23228,"prepare.queue_time_us":18636,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":103,"raft.queue_time_us":13617,"raft.run_cpu_time_us":409,"raft.run_wall_time_us":6080,"spinlock_wait_cycles":165489920,"thread_start_us":165244,"threads_started":1,"wal-append.queue_time_us":330967}
W20260709 12:25:31.437196  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 107309us
I20260709 12:25:31.423879  9980 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:31.439411  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.442679  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.460558  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:31.497357  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54797.6us
I20260709 12:25:31.428989  9997 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:31.567850  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 276474us
W20260709 12:25:31.578027 10019 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:25:31.579895  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.581575  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:31.583266  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.585026  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.585830  1798 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:31.585994  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.586578  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:31.587042  1804 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:31.587422  1873 raft_consensus.cc:2376] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23. Re-sending same reply.
I20260709 12:25:31.583333  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.606765  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:31.607865  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:31.433990  9991 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:31.499125  9957 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:31.597399  9982 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:25:31.623818  9872 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:31.624092  9866 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:31.624809  9872 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.507726  9925 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:31.507427  9501 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:31.640120  9991 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.593997  9884 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.646206 10024 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:31.646938 10024 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.649258 10024 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.640792  9992 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:31.688944  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.690675  1770 leader_election.cc:400] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
I20260709 12:25:31.695135  1770 leader_election.cc:403] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:25:31.650568  9872 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.707973  2415 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:25:31.713809  1876 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:31.729807  1798 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:25:31.730669  1798 raft_consensus.cc:606] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.731138  1798 raft_consensus.cc:614] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:31.735028  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:31.660598  9887 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.734807  9993 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:31.714820  9932 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.739771  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.609805 10002 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.098s
I20260709 12:25:31.646572  9884 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.742164  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:31.624888  9866 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.696707  9501 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:31.731987  9977 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.539s	user 0.007s	sys 0.006s
I20260709 12:25:31.747957  1807 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:31.729861  9872 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.728999 10024 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.755189  1876 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:31.755376  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 184991us
I20260709 12:25:31.745187 10021 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
I20260709 12:25:31.745532  9887 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.747181  9993 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.775518  9932 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:31.756881  9501 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.772251  9951 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:31.757464  9977 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:31.859292  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:31.869716  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 113319us
I20260709 12:25:31.871834  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.875783  1770 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.882155  9993 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.882946  9993 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:31.883471  9993 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.887082  1876 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:31.887643  9887 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.888792  9993 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.892695  9884 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.897241  9866 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:31.898087 10000 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.604s	user 0.008s	sys 0.000s
I20260709 12:25:31.898587  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:31.788995  9948 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:31.763996 10024 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.761740  9872 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.901156  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.925004  1724 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:31.878338  9951 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.934928  1770 leader_election.cc:400] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22.
I20260709 12:25:31.935110  1724 raft_consensus.cc:606] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:31.935758  1724 raft_consensus.cc:614] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:31.935787  2417 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:31.927752  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:31.937443  9951 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.923828 10022 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:31.908715 10020 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
I20260709 12:25:31.935382  1770 leader_election.cc:403] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:31.930099  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:31.935874  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2824) took 1015 ms. Trace:
I20260709 12:25:31.947366  1801 rpcz_store.cc:276] 0709 12:25:30.920723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.951508 (+ 30785us) service_pool.cc:224] Handling call
0709 12:25:31.935668 (+984160us) spinlock_profiling.cc:243] Waited 65 ms on lock 0x7b54002f3354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:31.935792 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149504256}
I20260709 12:25:31.943169  2410 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:31.938344  9992 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 12:25:31.918399  9954 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.549s	user 0.007s	sys 0.002s
I20260709 12:25:31.947541  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3011) took 1867 ms. Trace:
I20260709 12:25:31.956818  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:31.947501  2418 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
W20260709 12:25:31.912943  9394 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.616s	user 0.005s	sys 0.005s
W20260709 12:25:31.933418  9755 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.537s	user 0.008s	sys 0.003s
I20260709 12:25:31.910655  9501 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.923475  9872 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.900873 10002 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:31.914033 10007 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.550s	user 0.005s	sys 0.005s
I20260709 12:25:31.957602  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:31.956990  1726 rpcz_store.cc:276] 0709 12:25:30.079928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.922232 (+842304us) service_pool.cc:224] Handling call
0709 12:25:30.922535 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:30.952075 (+ 29540us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:31.713978 (+761903us) log.cc:844] Serialized 14 byte log entry
0709 12:25:31.947030 (+233052us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:31.947109 (+    79us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:31.947191 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.947287 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:31.947461 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":476377,"prepare.queue_time_us":801469,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":2984,"raft.queue_time_us":1735,"thread_start_us":508136,"threads_started":2}
W20260709 12:25:32.013978  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 79254.2us
I20260709 12:25:32.015293  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:31.953591 10005 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22.
I20260709 12:25:31.908483  9977 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:31.898576 10000 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:31.921625 10024 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:31.958552 10022 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:31.900483  9994 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:31.902984 10021 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:32.038256  9961 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:32.045163  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:32.046460  9923 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260709 12:25:32.061549  9948 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:32.062018  9948 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.062397  9948 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:32.064497  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:32.067869  2414 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:32.073463  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:31.975797  9954 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.048825 10004 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:32.076215  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:32.009917  9394 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.081462  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:32.086418  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.087028  1798 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:32.087287  1801 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:32.091655  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.066980  9961 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:32.048544  9996 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.056s
I20260709 12:25:32.019115 10007 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.102245  1879 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:32.096043  1873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.022s	sys 0.037s
I20260709 12:25:32.099082  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.105062  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:32.105939  1873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.022s	sys 0.037s
I20260709 12:25:32.105124  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:32.107148  1801 raft_consensus.cc:2368] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23.
I20260709 12:25:32.108799  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.075459  9951 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.016s
I20260709 12:25:32.122880  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:32.122907  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.127226  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.127977  1801 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:32.128176  1798 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:32.129200  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.152453  1803 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 13. (term mismatch)
I20260709 12:25:32.153255  1803 pending_rounds.cc:85] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 12
I20260709 12:25:32.153549  1803 pending_rounds.cc:107] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 24.13
I20260709 12:25:32.153771  1803 raft_consensus.cc:2889] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:32.155797  1807 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:25:32.157312  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.176700  9994 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:32.179256  9954 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:32.182245  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:32.185043 10007 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:32.193503  9394 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:32.195990 10000 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:32.199204 10022 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:32.202821  9980 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.683s	user 0.012s	sys 0.001s
I20260709 12:25:32.203521  9980 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.117518 10024 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:32.210969 10024 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.131229  9872 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.215678  9872 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:32.237363  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.242898  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:32.250892  2415 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:32.130896  9755 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.257889  1807 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:32.206481  9980 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:32.228879  9962 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.265653  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.267011  1806 raft_consensus.cc:2410] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:25:32.260814  9755 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:32.273862  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.275281  2415 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:32.275722  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:32.276621  1726 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:32.277571  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:32.286527  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.287429  1803 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:32.289443  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:32.290122  1729 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:32.292522  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.293046  1731 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:32.294662  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:32.295181  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:32.295648  1723 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:32.288252  9755 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:32.304080  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 285874us
I20260709 12:25:32.306635  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:32.334126  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:32.336338  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:32.341378  1807 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: [23.12-23.12]
I20260709 12:25:32.348812  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:32.349831  1803 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:32.351890  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:32.352381  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:32.350653  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:32.353443  1726 raft_consensus.cc:2410] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 22 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:25:32.354691  1730 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:32.287845  9962 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.358349  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:32.355409  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:32.339125  9980 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:32.357753  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:32.362905  1723 raft_consensus.cc:2410] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:25:32.359599  1726 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:32.356832  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1285) took 1862 ms. Trace:
I20260709 12:25:32.364902  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:32.359052  1730 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:32.360121  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:32.342562  9882 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:32.340335  9932 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:32.365914  1724 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:32.381749  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:32.384016  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.385041  1807 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
W20260709 12:25:32.362401  9997 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.734s	user 0.015s	sys 0.000s
I20260709 12:25:32.357967  9879 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:32.388899  9932 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.359050 10024 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:32.411022  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 287763us
W20260709 12:25:32.413686  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.438761  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:32.439653  1798 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:32.439872  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:32.441133  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.442138  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.443203  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.370309 10009 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:32.486241  1730 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:25:32.468593  1726 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:32.442683  1806 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
W20260709 12:25:32.500903  9957 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.881s	user 0.014s	sys 0.003s
I20260709 12:25:32.487969 10024 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.402395  9997 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.522713  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.533095  9948 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.534905  2418 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:32.537241 10043 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:32.538008 10043 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.510432  9872 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:32.539682 10032 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:25:32.540860  9932 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:32.541479  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1206) took 1036 ms. Trace:
I20260709 12:25:32.541635  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:32.541931  1873 rpcz_store.cc:276] 0709 12:25:31.505201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.588282 (+ 83081us) service_pool.cc:224] Handling call
0709 12:25:31.588693 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.589097 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.589127 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.592565 (+  3438us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.106806 (+514241us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:32.135282 (+ 28476us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.414417 (+279135us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.414562 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:32.541240 (+126678us) raft_consensus.cc:1685] finished
0709 12:25:32.541264 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.541411 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":23635,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":61,"raft.queue_time_us":383715,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":32936,"thread_start_us":792276,"threads_started":2,"wal-append.queue_time_us":676587}
I20260709 12:25:32.463336 10009 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:32.545424  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1182) took 1507 ms. Trace:
I20260709 12:25:32.546097 10009 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:32.545863  1802 rpcz_store.cc:276] 0709 12:25:31.037802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.048046 (+ 10244us) service_pool.cc:224] Handling call
0709 12:25:31.048361 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.325452 (+277091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.545164 (+1219712us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b5400132254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:32.545352 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":611160832}
I20260709 12:25:32.547724  9962 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:32.548882 10035 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:25:32.552035  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.552933  1875 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:32.553514  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1200) took 1383 ms. Trace:
I20260709 12:25:32.553830  1875 rpcz_store.cc:276] 0709 12:25:31.169953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.170084 (+   131us) service_pool.cc:224] Handling call
0709 12:25:32.551826 (+1381742us) spinlock_profiling.cc:243] Waited 2.34 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:25:32.553411 (+  1585us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5370624}
I20260709 12:25:32.519136 10031 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.555588 10031 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.559019  9947 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:25:32.560381 10036 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.058s	user 0.003s	sys 0.000s
I20260709 12:25:32.561277  9947 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:32.562026   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:25:32.568116  1801 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:32.524078 10024 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:32.570075  9997 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:32.522429  9957 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:32.571909  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2833) took 1067 ms. Trace:
I20260709 12:25:32.572355  1805 rpcz_store.cc:276] 0709 12:25:31.504550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.510915 (+  6365us) service_pool.cc:224] Handling call
0709 12:25:32.571675 (+1060760us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b54001e3a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:32.571840 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1187809152}
I20260709 12:25:32.572898  1845 leader_election.cc:400] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23.
I20260709 12:25:32.573614  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.573969  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:32.574877  9919 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:32.572096  2415 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
W20260709 12:25:32.577359  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.578202  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.570467  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.579782  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:32.580660  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1198) took 1411 ms (client timeout 1632 ms). Trace:
I20260709 12:25:32.581156  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.583439  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:32.543622 10043 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:32.570585  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:32.589570  1798 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:32.581799  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:32.590343 10040 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.592895  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1379) took 1188 ms. Trace:
I20260709 12:25:32.620585  2410 rpcz_store.cc:276] 0709 12:25:31.404682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.406899 (+  2217us) service_pool.cc:224] Handling call
0709 12:25:32.592694 (+1185795us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54005230d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:32.592814 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":430542720}
I20260709 12:25:32.594369  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.558521 10031 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:32.598479  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.590480 10049 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.607674  1800 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:32.581055  1877 rpcz_store.cc:269] 0709 12:25:31.169243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.169441 (+   198us) service_pool.cc:224] Handling call
0709 12:25:32.579625 (+1410184us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:25:32.580595 (+   970us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6376192}
I20260709 12:25:32.608326  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:32.618958  1729 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:32.624322  1881 raft_consensus.cc:2368] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26.
I20260709 12:25:32.625002  1730 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:25:32.619561  1803 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:25:32.629130  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:32.630393  1881 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:32.642341  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:32.645126  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:32.641306  1806 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:32.648809  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:32.651553  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:32.652433  1874 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:32.656116  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:32.657866  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.659377  1881 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:32.649528  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:32.667443  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:32.365370  1728 rpcz_store.cc:276] 0709 12:25:30.494679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.188401 (+693722us) service_pool.cc:224] Handling call
0709 12:25:31.188725 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.189071 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:31.947794 (+758723us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.356494 (+408700us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.356544 (+    50us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:32.356588 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.356658 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.356781 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":259737,"prepare.queue_time_us":588911,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":10801,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":3951,"thread_start_us":830780,"threads_started":2}
I20260709 12:25:32.669615  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:32.670492  1876 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:32.671176  1798 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:32.662614  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:32.672089  2410 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:32.641407  9961 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.526s	user 0.012s	sys 0.006s
I20260709 12:25:32.667444  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.665341  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.664605  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:32.641010 10009 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:32.675256  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:32.663022  9872 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:32.693007  1877 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:32.694185  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.699131  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:32.711196  1803 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:32.648406 10031 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:32.690754  9923 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:32.627715  9944 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:32.705377  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:32.722679  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.647869  9919 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:25:32.672667 10007 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:32.660156 10043 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:25:32.660754  9964 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:25:32.673868 10034 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:32.709417  9872 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.728866  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.645892 10040 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:32.726735  1873 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:32.734562 10048 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.143s	user 0.002s	sys 0.000s
I20260709 12:25:32.647398 10049 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:32.725538  2410 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:32.742225  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:32.751273  9872 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:32.762331  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:32.735190 10007 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.824780  1881 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
I20260709 12:25:32.701918  9961 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:32.731451  9944 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:32.748967 10034 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:32.837718  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.840112  1873 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:32.763393 10031 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:32.741510  9947 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.095s
I20260709 12:25:32.829068 10007 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:32.837786  9944 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:32.838133 10034 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:32.882742  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.886608  1873 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:32.868175 10031 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:32.874882  9919 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:32.973548  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:32.978626  1873 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
W20260709 12:25:32.994975  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.884334 10055 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:33.039414 10055 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:33.039964 10055 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:33.041152 10055 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:32.903163  9948 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:33.043872  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:33.046698  1877 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:33.100890  1800 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:32.949693 10031 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:32.949155  9919 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:33.114364  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 453727us
I20260709 12:25:33.042763 10055 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:33.050030 10040 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:33.119863  1874 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:25:33.116117  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 412741us
I20260709 12:25:33.045378  9885 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:33.048592 10049 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:33.080787  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1201) took 1874 ms. Trace:
I20260709 12:25:33.123849  1872 rpcz_store.cc:276] 0709 12:25:31.206465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.206638 (+   173us) service_pool.cc:224] Handling call
0709 12:25:31.241586 (+ 34948us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.242483 (+   897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.080506 (+1838023us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b54002df5d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:33.080711 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41031936}
W20260709 12:25:33.124878  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 463753us
I20260709 12:25:33.135394  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:33.136400  1729 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:33.113080  9919 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:33.120388  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:33.139710  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.143487  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.144290  1873 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:33.147229  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:33.068014 10052 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.194s	user 0.000s	sys 0.000s
I20260709 12:25:33.154192  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:33.124513 10009 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:33.071944 10050 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.187s	user 0.000s	sys 0.000s
I20260709 12:25:33.152669  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:33.153633  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:33.196630  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.197388  1726 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:33.153203  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:33.197633  1872 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
W20260709 12:25:33.198369  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45268.7us
I20260709 12:25:33.188822  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1281) took 2722 ms. Trace:
I20260709 12:25:33.199503  1727 rpcz_store.cc:276] 0709 12:25:30.427564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.060078 (+632514us) service_pool.cc:224] Handling call
0709 12:25:31.096562 (+ 36484us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.130274 (+ 33712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.149854 (+2019580us) spinlock_profiling.cc:243] Waited 8.78 ms on lock 0x7b54003cc3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:33.150096 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20189440}
I20260709 12:25:33.190960  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:33.191913  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:33.193084  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:33.207917  1723 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:33.123603  9885 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:33.209795  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:33.210721  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.207931  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.211987  1874 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:33.212450  1873 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:33.215898  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.015s	sys 0.023s
W20260709 12:25:33.216303  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.015s	sys 0.023s
W20260709 12:25:33.216429  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 454158us
W20260709 12:25:33.216619  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 541400us
I20260709 12:25:33.219291  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:33.219741  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:33.220391  2382 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:33.220650  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:33.221207  2410 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:33.221385  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:33.221549  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:33.223569  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:33.224347  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:33.225168  2415 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:33.227180  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.228097  1872 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:33.188858 10049 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:33.232808  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:33.234544  1728 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:33.053972  9944 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.252947  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:33.245492  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:33.145838 10040 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.077s
W20260709 12:25:33.257056  1876 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.582s	user 0.013s	sys 0.000s
I20260709 12:25:33.257427  2410 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:33.257556  1876 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:33.260053  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.228478  9893 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.036s
I20260709 12:25:33.258744  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:33.269928  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.270815  1876 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:33.273375  1877 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:33.232812 10049 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.273674  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.273048  9893 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:33.253484  9645 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:33.278263  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.279091  9893 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.267253  9961 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:33.282501  1874 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:33.291733  1873 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
W20260709 12:25:33.316586  1802 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.006s	sys 0.018s
W20260709 12:25:33.317256  1802 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.006s	sys 0.018s
W20260709 12:25:33.318136  1805 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.006s	sys 0.016s
W20260709 12:25:33.318565  1805 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.006s	sys 0.016s
I20260709 12:25:33.278400 10049 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:33.274416 10058 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:33.280272 10043 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:33.292567  9893 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:33.352028  1725 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:33.353018 10057 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.213s	user 0.001s	sys 0.000s
I20260709 12:25:33.377990  1731 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:33.358556 10067 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.302909 10040 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:33.282240  9645 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:33.282020 10060 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:33.279753  9885 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.288431 10009 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:33.404224  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:33.329145 10051 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.426s	user 0.000s	sys 0.001s
W20260709 12:25:33.375725  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 152237us
I20260709 12:25:33.407696  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:33.411330  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 165896us
I20260709 12:25:33.379869  1731 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: []
I20260709 12:25:33.382110  2416 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
W20260709 12:25:33.380069  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 158752us
I20260709 12:25:33.338586 10063 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:33.412619  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:33.420186  2410 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:33.422484  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:33.424397  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:33.424993  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.425217  1731 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:33.426649  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:33.434713  1725 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:33.429468  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:33.396667  9645 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:33.439734 10078 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:33.419281 10062 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.450312  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:33.450798 10062 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.444128  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3842) took 1085 ms. Trace:
I20260709 12:25:33.419917 10063 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:33.437230 10043 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:33.398072 10067 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.449309 10076 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.453325  1732 rpcz_store.cc:276] 0709 12:25:32.358501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.556010 (+197509us) service_pool.cc:224] Handling call
0709 12:25:32.556466 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:32.556869 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.556907 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:32.557438 (+   531us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.760365 (+202927us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:32.760423 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.760512 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.443830 (+683318us) raft_consensus.cc:1685] finished
0709 12:25:33.443863 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:33.444049 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33650,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":113,"thread_start_us":202402,"threads_started":1,"wal-append.queue_time_us":455535}
I20260709 12:25:33.427663 10061 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:33.424062 10068 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:33.415674 10034 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.449391 10040 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:25:33.469321  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:33.469533  1806 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:33.470759  1802 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:33.470098  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47697.9us
I20260709 12:25:33.477347  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:33.459106 10043 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:33.464798 10061 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:33.475678 10007 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.581s	user 0.012s	sys 0.001s
W20260709 12:25:33.494728  1846 connection.cc:256] RPC call timeout handler was delayed by -2.0353444190000118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21989999999999998s for a response
I20260709 12:25:33.465485 10074 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:33.452471 10078 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.467437 10076 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.507192  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.508122  1877 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:33.508955  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.509152 10076 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:33.510151  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.510233  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.511047  1876 raft_consensus.cc:2376] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22. Re-sending same reply.
I20260709 12:25:33.511785  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.512036  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.512434  1872 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:33.513145  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.514812  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:33.519543  9995 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 2.715s	user 0.016s	sys 0.049s
W20260709 12:25:33.522502  2418 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1387) took 1341 ms (client timeout 1717 ms). Trace:
W20260709 12:25:33.523030  2418 rpcz_store.cc:269] 0709 12:25:32.181537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.181835 (+   298us) service_pool.cc:224] Handling call
0709 12:25:33.522276 (+1340441us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b5400525154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:33.522440 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":807964928}
I20260709 12:25:33.470624  9947 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:25:33.527130 10078 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:33.499480  9944 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:33.490701 10064 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.047s
I20260709 12:25:33.529237 10064 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:25:33.475600 10075 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:33.531992  1802 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
W20260709 12:25:33.534108  2383 leader_election.cc:336] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.604s (SENT)
I20260709 12:25:33.505548  1731 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:33.535578  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:33.504329  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:33.537513  1874 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:33.503782  1697 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:33.541338  1697 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:33.542303  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:33.545166  1806 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:33.545843  1802 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:33.470438 10060 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:25:33.550218  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1205) took 2045 ms. Trace:
I20260709 12:25:33.555630  1697 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:33.529626 10064 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:33.479753 10071 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.084s
I20260709 12:25:33.496179 10082 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:33.498127 10068 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.556411  1879 rpcz_store.cc:276] 0709 12:25:31.504795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.504933 (+   138us) service_pool.cc:224] Handling call
0709 12:25:31.585351 (+ 80418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.102870 (+517519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.549971 (+1447101us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b5400206cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:33.550149 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1188139264}
I20260709 12:25:33.482465 10073 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
I20260709 12:25:33.560894  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:33.492249 10061 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:33.567046  1725 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:33.568795  1876 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:25:33.544723  9944 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:33.546165  9879 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:33.562206  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:33.550750 10078 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:33.574935  9872 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.813s	user 0.011s	sys 0.004s
I20260709 12:25:33.575269 10009 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:33.575706 10043 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.577443  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:33.578140  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.539777 10074 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:33.580526 10067 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:33.581017  9919 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.584182  9879 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.495137 10058 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:33.503655 10007 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.564340  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:33.591985  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 140276us
I20260709 12:25:33.562724 10075 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:33.582461 10082 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:33.570231  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:33.601897 10009 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:33.504997 10063 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.597576 10077 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:33.618608  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 54462.6us
I20260709 12:25:33.619658  1874 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
W20260709 12:25:33.617919  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 148843us
I20260709 12:25:33.622454 10007 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:33.634238 10034 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:33.619413  9919 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:33.622920 10063 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:33.619036 10055 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:33.614127  9879 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:33.621035 10031 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.000s
I20260709 12:25:33.570215 10068 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.593050  9872 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.617404 10049 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.626616 10077 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:33.678131  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 140805us
I20260709 12:25:33.622419 10062 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:33.665107 10043 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:33.689123  2383 leader_election.cc:336] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.576s (SENT)
W20260709 12:25:33.683395  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 202438us
I20260709 12:25:33.700579  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:33.702666  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:33.704273  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:33.706923  1725 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:33.686204 10077 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:33.660588 10083 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:25:33.622040 10009 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.702245  9846 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:33.701706  9944 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:33.728739  9944 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.683838 10031 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.733435 10031 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:33.704260 10065 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.734498 10065 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.735977  9872 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:33.705401  9919 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:33.716243 10079 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:25:33.612566 10064 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:33.726043  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.703882 10074 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.718968 10058 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.620317 10082 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:33.744431  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45438.2us
I20260709 12:25:33.704048  9645 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.738922  9919 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:33.740795  9846 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.739120  9980 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:33.782130  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:33.784171  9992 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:33.791018  9919 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:33.790601 10049 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:33.795209  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 233059us
I20260709 12:25:33.804092  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:33.804948  2418 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:33.808084  2418 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:25:33.814694  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:33.815979  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:33.821880  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:33.823488  2416 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:33.827150  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 141907us
I20260709 12:25:33.828399  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:33.830358  1728 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:33.792752  9949 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:33.842124  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:33.832176  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:33.836701  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:33.821852  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:33.855492  2410 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:33.795197 10084 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:33.857864  1724 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:33.875393  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:33.883890  1804 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:33.884064  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:33.886134  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:33.888581  1724 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:33.838217 10062 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:33.842469 10049 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:33.890184  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:33.801448 10081 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:33.794831  9980 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.798101  9992 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.901949  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:33.902705  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:33.909672  1800 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:33.908327  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:33.918900  1724 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:33.921921  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:33.923383  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:33.926760  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.895592 10062 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:33.907887  9893 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.510s	user 0.014s	sys 0.000s
I20260709 12:25:33.941659   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:33.943351   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:33.943562 10061 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:33.879057 10091 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.076s
I20260709 12:25:33.945115 10091 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:33.945470 10091 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:33.945947 10061 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:33.966610  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:33.910840 10084 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:33.971465  1724 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:33.972844  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1381) took 2458 ms. Trace:
I20260709 12:25:33.973476  2414 rpcz_store.cc:276] 0709 12:25:31.514808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.514994 (+   186us) service_pool.cc:224] Handling call
0709 12:25:31.616937 (+101943us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.068631 (+451694us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.972387 (+1903756us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b5400487454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:33.972719 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1036431232}
I20260709 12:25:33.976415  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:33.977495  1732 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:33.921206 10068 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:33.933454 10062 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:33.939239 10002 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.996912  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:33.997722  1804 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:34.000142  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 111167us
I20260709 12:25:33.965236 10097 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:33.907277 10107 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.100s
I20260709 12:25:34.012508  2410 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:34.017452 10084 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.003870  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:33.972934  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:34.064035  1732 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:33.954998  9893 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:33.995345  9933 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 12:25:34.070662  1724 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:34.098452  1732 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:25:34.099655  1804 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:34.098871  1724 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:25:34.001410 10002 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:34.069166 10110 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:34.113833  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 321292us
W20260709 12:25:34.107946  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 141390us
I20260709 12:25:34.094448  9645 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:34.091924 10097 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:34.133244  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:34.123562  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:34.099560 10061 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:34.037771 10009 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.074s
I20260709 12:25:34.109254  9846 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:34.111140 10118 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:34.161058 10118 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:34.163420 10118 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:34.120153 10105 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:34.164568  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.193852 10111 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:34.196358 10111 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:34.128096 10110 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.201846 10110 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:34.204221 10002 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:34.204949 10091 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:34.206152  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.206488 10091 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:34.190097 10113 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:25:34.192072 10112 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:34.203837 10097 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.211263  9992 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:34.211905  9980 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:34.214196  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1293) took 1081 ms. Trace:
I20260709 12:25:34.214656  9893 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:34.219699 10121 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.155364 10125 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.155759 10061 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:34.223855 10009 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:34.112311 10077 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.253896  9872 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:34.278913  9872 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:34.288373  9879 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.611s	user 0.004s	sys 0.008s
I20260709 12:25:34.257091 10111 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.288933  9879 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.277643 10121 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.253153 10112 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.277118 10098 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:34.310386 10113 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:34.273610  9992 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:34.310966 10113 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:34.311578  9992 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.312192 10113 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.287804  9872 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:34.251629  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 118476us
I20260709 12:25:34.256493  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:34.253907  1727 rpcz_store.cc:276] 0709 12:25:33.133145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.275341 (+142196us) service_pool.cc:224] Handling call
0709 12:25:33.284243 (+  8902us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:33.452927 (+168684us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:33.453039 (+   112us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:33.581037 (+127998us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b2c0005a338. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:25:33.581512 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 12:25:33.801140 (+219628us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:33.801206 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.801327 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:34.213733 (+412406us) raft_consensus.cc:1685] finished
0709 12:25:34.213767 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.214120 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13863,"raft.queue_time_us":154681,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":38292736,"thread_start_us":373098,"threads_started":2,"wal-append.queue_time_us":535785}
W20260709 12:25:34.267901  1771 connection.cc:256] RPC call timeout handler was delayed by -1.2554493829998137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
W20260709 12:25:34.319643  1771 connection.cc:256] RPC call timeout handler was delayed by -1.1029957509995256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1643s for a response
W20260709 12:25:34.320178  1771 connection.cc:256] RPC call timeout handler was delayed by -1.0772493829999803s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
W20260709 12:25:34.320544  1771 connection.cc:256] RPC call timeout handler was delayed by -1.0367493829999148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1795s for a response
I20260709 12:25:34.269099  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:34.212392 10043 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.000s
W20260709 12:25:34.231863  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 108395us
I20260709 12:25:34.298569  9645 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:34.329954  1879 raft_consensus.cc:2410] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 23 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
W20260709 12:25:34.339900  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.319310 10113 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:34.381670  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 62647.7us
I20260709 12:25:34.322815  9949 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:34.297717 10009 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:34.335649  9645 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:34.379931  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:34.397908  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:34.276700 10125 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:34.400331  2383 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.896s (SENT) [suppressed 19 similar messages]
I20260709 12:25:34.378283 10077 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:34.383769 10062 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.401670  9879 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:34.439324 10037 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:34.429567 10126 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.075s
W20260709 12:25:34.486496  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2946) took 1901 ms (client timeout 1533 ms). Trace:
W20260709 12:25:34.487083  1875 rpcz_store.cc:269] 0709 12:25:32.584856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.486256 (+1901400us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:34.486399 (+   143us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:34.438832 10002 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.036s
W20260709 12:25:34.491202  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 93362.7us
I20260709 12:25:34.494314  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:34.504411  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 160627us
W20260709 12:25:34.505753  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 105452us
W20260709 12:25:34.506570  2383 leader_election.cc:336] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.929s (SENT)
I20260709 12:25:34.497670 10087 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:34.510567  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:34.511057 10087 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:34.512248 10022 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:34.512925 10022 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:34.515372 10097 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.514508 10022 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.516004 10097 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.516352  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:34.521847  9957 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:34.499078 10061 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:34.477623  9949 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:34.566882  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51252us
I20260709 12:25:34.571018  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:34.574822  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:34.577184  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 61793us
I20260709 12:25:34.537776 10089 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.032s
W20260709 12:25:34.578263  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 90282.5us
I20260709 12:25:34.578624  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:34.568401 10127 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:34.586934  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:34.581969 10061 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:34.611009 10061 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:25:34.628652  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.634042  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:34.634951  1729 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:34.570475 10002 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:34.640213  9846 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:34.640655 10002 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:34.644657  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 57794.6us
I20260709 12:25:34.584513 10009 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:34.580163 10098 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:34.641119  9846 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:34.614882  9893 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
I20260709 12:25:34.669687  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3840) took 2311 ms. Trace:
I20260709 12:25:34.675108  1730 rpcz_store.cc:276] 0709 12:25:32.357987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.488315 (+130328us) service_pool.cc:224] Handling call
0709 12:25:32.624779 (+136464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:32.625470 (+   691us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.669320 (+2043850us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54001bf3d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:34.669584 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":311027968}
I20260709 12:25:34.584794  9923 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.079s
I20260709 12:25:34.642515 10063 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:34.642729 10134 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:34.689148  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 110612us
W20260709 12:25:34.694128  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2952) took 1991 ms (client timeout 1726 ms). Trace:
W20260709 12:25:34.694639  1881 rpcz_store.cc:269] 0709 12:25:32.702617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.728482 (+ 25865us) service_pool.cc:224] Handling call
0709 12:25:34.693890 (+1965408us) spinlock_profiling.cc:243] Waited 75.8 ms on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:34.694047 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":174366080}
I20260709 12:25:34.658723 10087 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.700217 10087 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:34.641095 10002 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:34.647002 10062 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:34.684849 10125 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:34.711622  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.709514 10061 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:34.779500 10024 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:34.795966  9919 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.994s	user 0.011s	sys 0.004s
I20260709 12:25:34.796619  9919 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.797107 10112 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:34.711158  9846 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 12:25:34.766449 10125 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 12:25:34.775137 10111 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:34.814422  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 243512us
I20260709 12:25:34.819353  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:34.809631  9919 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:34.827688  1801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.001s	sys 0.014s
W20260709 12:25:34.828285  1801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.501s	user 0.001s	sys 0.014s
I20260709 12:25:34.822454  9980 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:34.836824 10135 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
W20260709 12:25:34.837054 10140 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260709 12:25:34.838531  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1307) took 1126 ms. Trace:
I20260709 12:25:34.839119  1731 rpcz_store.cc:276] 0709 12:25:33.711719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.711852 (+   133us) service_pool.cc:224] Handling call
0709 12:25:33.712259 (+   407us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:33.716754 (+  4495us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:33.716788 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:33.721040 (+  4252us) log.cc:844] Serialized 52 byte log entry
0709 12:25:34.211597 (+490557us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b180046ace0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:25:34.404553 (+192956us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:34.404615 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.404711 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:34.838197 (+433486us) raft_consensus.cc:1685] finished
0709 12:25:34.838218 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.838466 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4174,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":3879,"spinlock_wait_cycles":28681088,"thread_start_us":192691,"threads_started":1,"wal-append.queue_time_us":919806}
I20260709 12:25:34.836043  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:34.831985 10129 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
W20260709 12:25:34.838467 10139 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.226s	user 0.000s	sys 0.001s
W20260709 12:25:34.831915 10138 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
W20260709 12:25:34.831633 10132 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
I20260709 12:25:34.822721 10111 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.842249  9846 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:34.848548  9980 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.849099 10127 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:34.904408 10154 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.081s	user 0.001s	sys 0.001s
I20260709 12:25:34.918370 10064 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:25:34.921396  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2866) took 1075 ms. Trace:
I20260709 12:25:34.923899  1803 rpcz_store.cc:276] 0709 12:25:33.846226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.846467 (+   241us) service_pool.cc:224] Handling call
0709 12:25:33.846859 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:33.937818 (+ 90959us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:33.947734 (+  9916us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:33.967373 (+ 19639us) log.cc:844] Serialized 26 byte log entry
0709 12:25:34.437556 (+470183us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:34.439260 (+  1704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.439339 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:34.921107 (+481768us) raft_consensus.cc:1685] finished
0709 12:25:34.921147 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.921321 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":117752,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":27645,"thread_start_us":520810,"threads_started":2,"wal-append.queue_time_us":505468}
I20260709 12:25:34.927771 10111 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:34.932899 10112 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:34.933491 10112 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:34.934423 10112 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:34.935789 10112 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:34.940840 10113 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.560s	user 0.005s	sys 0.005s
I20260709 12:25:34.942468  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3780) took 1389 ms. Trace:
I20260709 12:25:34.942921  1879 rpcz_store.cc:276] 0709 12:25:33.552597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.560040 (+  7443us) service_pool.cc:224] Handling call
0709 12:25:34.942263 (+1382223us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b54001f8454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:34.942407 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1692497152}
I20260709 12:25:34.944763 10121 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:34.945490  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 109524us
I20260709 12:25:34.970599 10105 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:34.952080 10097 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:34.957232  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:34.961388 10100 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:34.981405 10100 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:34.982115  1696 leader_election.cc:400] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26.
I20260709 12:25:34.982333 10100 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.983254  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:34.983592 10100 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:34.984721  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:34.988008 10105 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:34.992020  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.994854  1803 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:34.995968  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:34.966336  9645 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:34.968902  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:34.962296 10037 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:34.918851 10095 log.cc:927] Time spent T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.096s	user 0.010s	sys 0.005s
I20260709 12:25:35.015554 10077 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:34.915735 10125 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:35.019284  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:35.019974  2414 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:35.001047  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:35.014596  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:35.023979  2416 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:35.024547  1807 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:35.025314  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.002218  2415 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:25:35.003225  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:35.003638  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.004041  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.005699  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.006523  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.028643  1728 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:35.011096  1723 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:35.012446  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:35.015097  9879 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:35.037808  2414 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
W20260709 12:25:34.979256 10164 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:25:35.050604  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47456.8us
I20260709 12:25:35.051752  2418 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:25:35.008839 10091 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.764s	user 0.000s	sys 0.011s
I20260709 12:25:35.054646  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.059310 10077 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.016s
I20260709 12:25:35.076144 10131 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:35.076607 10131 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:35.077378 10131 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.054772 10002 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.080637 10065 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.082929  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.083437 10083 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:35.084028 10083 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:35.059815  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:35.086869  9885 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:35.087208  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:35.089223 10091 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:35.088606  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:35.090534 10091 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:35.091096 10091 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:35.092384 10160 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.176s	user 0.002s	sys 0.000s
I20260709 12:25:35.092043 10091 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.094365  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:35.095037  2415 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:35.095877 10055 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:35.098012 10055 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.023841 10061 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:35.023638  9755 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:35.099850 10096 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:35.101225 10107 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.101755 10107 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.103507  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3779) took 1551 ms. Trace:
I20260709 12:25:35.103941  1872 rpcz_store.cc:276] 0709 12:25:33.552237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.620738 (+ 68501us) service_pool.cc:224] Handling call
0709 12:25:33.621130 (+   392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.621354 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:33.672540 (+ 51186us) log.cc:844] Serialized 14 byte log entry
0709 12:25:33.831439 (+158899us) log.cc:844] Serialized 14 byte log entry
0709 12:25:33.831797 (+   358us) log.cc:844] Serialized 14 byte log entry
0709 12:25:34.163387 (+331590us) spinlock_profiling.cc:243] Waited 9.3 ms on lock 0x7b18000962e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287
0709 12:25:34.163800 (+   413us) log.cc:844] Serialized 14 byte log entry
0709 12:25:34.819779 (+655979us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b18000962e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:34.819972 (+   193us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b18000962e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb557 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:35.051120 (+231148us) spinlock_profiling.cc:243] Waited 4.07 ms on lock 0x7b18000962e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240
0709 12:25:35.083199 (+ 32079us) spinlock_profiling.cc:243] Waited 7.33 ms on lock 0x7b18000962e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16afcac58f 00007f16a8ceb571 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd
0709 12:25:35.096018 (+ 12819us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b18000962e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240
0709 12:25:35.096598 (+   580us) spinlock_profiling.cc:243] Waited 4.56 ms on lock 0x7b18000962e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240
0709 12:25:35.097621 (+  1023us) spinlock_profiling.cc:243] Waited 15.4 ms on lock 0x7b18000962e0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f
0709 12:25:35.098750 (+  1129us) spinlock_profiling.cc:243] Waited 4.05 ms on lock 0x7b48000ce7d8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:25:35.103112 (+  4362us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:35.103173 (+    61us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:35.103234 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.103306 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.103440 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1060925,"prepare.queue_time_us":61384,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":87,"raft.queue_time_us":217848,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":38122,"spinlock_wait_cycles":196159232,"thread_start_us":169684,"threads_started":3,"wal-append.queue_time_us":190264}
W20260709 12:25:35.106418 10148 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.362s	user 0.001s	sys 0.000s
I20260709 12:25:35.106565  9645 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:35.107267 10123 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:35.107681 10123 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:35.108731 10123 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.110095 10123 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:35.059592  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.112128  1731 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 17.
I20260709 12:25:35.065640  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.113108  1729 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:35.100311  9762 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:35.059082 10024 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:35.069303 10005 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:35.134467 10005 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:35.134709  2417 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:35.135332 10005 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.136906 10005 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:35.065800 10162 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
I20260709 12:25:35.116395  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:35.139133  1881 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:35.140457 10107 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.084899 10083 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.116887  1875 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:35.118212  1877 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:35.083017  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.065933  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:35.072180  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.144381  2414 raft_consensus.cc:2410] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 21 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:25:35.121116  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:35.056715  1697 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:25:35.132220  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 59800.5us
I20260709 12:25:35.144496  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:35.145318  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 90151.1us
I20260709 12:25:35.146569  2411 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:35.131659  9762 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:35.152266  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:35.149724  1807 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:35.153461  2416 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:35.144526  2415 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:35.155227  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:35.157868  1730 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:25:35.158618  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.154745  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:35.162317  1726 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:35.098489 10055 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.163213  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:35.183027  1874 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:35.164934  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:35.094617  9992 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.184124  1879 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:35.172595  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.175220  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.182754  1873 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:35.185369  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:35.068176  9980 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.203358  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:35.154883  9755 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:35.165545  9962 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.217957  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 245374us
I20260709 12:25:35.064348 10113 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.140s
I20260709 12:25:35.062353  9879 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:35.238977  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 1.33852e+06us
W20260709 12:25:35.221608  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59685.8us
I20260709 12:25:35.239892  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:35.161672 10061 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:25:35.231911  1879 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:35.190989 10062 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:35.191764 10100 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:35.190763 10002 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:35.243770  1723 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:25:35.262205  1725 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
W20260709 12:25:35.254765  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.252146  1881 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:35.226329  9888 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:35.273787  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 3031) took 1396 ms. Trace:
I20260709 12:25:35.277963  1723 rpcz_store.cc:276] 0709 12:25:33.865324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.964299 (+ 98975us) service_pool.cc:224] Handling call
0709 12:25:34.968757 (+1004458us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:25:35.262020 (+293263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12787584}
I20260709 12:25:35.238615 10077 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:35.277026  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.277324  1727 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:35.281054  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:35.319598  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 134354us
I20260709 12:25:35.251273 10065 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:35.331880  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:35.239061 10126 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.074s
I20260709 12:25:35.239245 10022 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 12:25:35.246990  9962 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.432600  2415 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:35.432768  1800 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260709 12:25:35.284994  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 300761us
I20260709 12:25:35.282392  9888 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:35.502921  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 171123us
I20260709 12:25:35.505836  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:35.439934  9755 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:35.381350 10073 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.479517 10062 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.507503  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:35.520524  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 296978us
I20260709 12:25:35.439404  9949 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:35.517223  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:35.454932  9992 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:35.523185  1723 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:35.479665 10061 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:35.528025  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:35.541139  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:35.495949 10100 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:35.545359  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:35.549888  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.550135  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44376.2us
I20260709 12:25:35.551081  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:35.528271  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2863) took 1686 ms. Trace:
I20260709 12:25:35.554088  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:35.501816 10112 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
W20260709 12:25:35.530009 10166 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
I20260709 12:25:35.590154 10024 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:35.274955  9762 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.507439  9888 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:35.552779  1801 rpcz_store.cc:276] 0709 12:25:33.841894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.842045 (+   151us) service_pool.cc:224] Handling call
0709 12:25:33.842337 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.930722 (+ 88385us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:34.316027 (+385305us) log.cc:844] Serialized 14 byte log entry
0709 12:25:34.854008 (+537981us) log.cc:844] Serialized 14 byte log entry
0709 12:25:34.935581 (+ 81573us) spinlock_profiling.cc:243] Waited 6.46 ms on lock 0x7b1800316420. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240
0709 12:25:34.936572 (+   991us) spinlock_profiling.cc:243] Waited 2.31 ms on lock 0x7b1800316420. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1
0709 12:25:34.937687 (+  1115us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b1800316420. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:35.140772 (+203085us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b1800316420. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:25:35.141011 (+   239us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:35.141068 (+    57us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:35.141125 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.527994 (+386869us) spinlock_profiling.cc:243] Waited 88.1 ms on lock 0x7b54003b83d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:35.528051 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.528203 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":389328,"prepare.queue_time_us":453338,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":129,"raft.queue_time_us":322236,"raft.run_cpu_time_us":427,"raft.run_wall_time_us":78076,"spinlock_wait_cycles":232094336,"thread_start_us":848491,"threads_started":2,"wal-append.queue_time_us":536152}
W20260709 12:25:35.567833 10175 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.156s	user 0.000s	sys 0.001s
W20260709 12:25:35.568363 10165 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
I20260709 12:25:35.552054  9980 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:35.573629 10174 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
W20260709 12:25:35.574770 10171 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:25:35.552481  9992 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.529434  9879 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.583631  1621 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 13 to 16. New cstate: current_term: 16 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:35.601878 10074 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.588603 10131 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:35.559273 10176 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:35.504142 10002 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.048s
I20260709 12:25:35.508926 10055 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.523001 10183 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:35.547477 10110 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:35.610924  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 56897.5us
I20260709 12:25:35.489017 10065 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:35.619160  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 379324us
I20260709 12:25:35.559695 10091 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:35.657966 10058 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.587148  2410 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
W20260709 12:25:35.631129  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.540995 10073 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.664158  9919 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.664762  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.663172  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:35.694571  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.615612 10024 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:35.668349  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:35.624150  9980 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:35.799005  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:35.799624  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:35.800307  1875 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:35.742944 10176 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:35.659619 10170 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.180s	user 0.000s	sys 0.000s
I20260709 12:25:35.671541 10163 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.659307 10068 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.530370 10062 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.657085 10076 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:35.759999  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:35.763319  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.641311 10168 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:25:35.664505 10072 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.657624 10040 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.689499 10123 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.579s	user 0.011s	sys 0.005s
I20260709 12:25:35.781147  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:35.852849  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 71791.9us
I20260709 12:25:35.853710 10062 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.659775 10108 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.660431 10042 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:25:35.785425  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 240120us
I20260709 12:25:35.629657 10111 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:25:35.624049 10105 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.631s	user 0.009s	sys 0.004s
W20260709 12:25:35.660666  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51399.9us
I20260709 12:25:35.856527 10108 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.860831 10108 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.856711 10042 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:35.861799 10189 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:35.818720 10176 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:35.862629 10042 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.863842 10042 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:35.663316 10080 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:35.865192 10080 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.857287  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:35.791549  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.643246 10064 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.666s	user 0.007s	sys 0.005s
I20260709 12:25:35.676066  9947 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:35.868448  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:35.869057  9947 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:35.869952 10169 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.502s	user 0.002s	sys 0.002s
I20260709 12:25:35.802827  9980 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:35.871970  9947 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.643134  9992 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:35.857584 10111 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:35.872997  9992 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:35.873318 10111 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:35.873704  9992 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.874239  9992 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:35.857961 10076 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:25:35.876121 10179 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.231s	user 0.005s	sys 0.002s
I20260709 12:25:35.858212 10123 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:25:35.698958  9846 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.628505 10005 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.171s
I20260709 12:25:35.661073 10066 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.125s
I20260709 12:25:35.797887  2415 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:35.798005  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:35.891278  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:35.623829 10184 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.890012 10076 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:35.904392 10076 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:35.903811 10184 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.558059  9949 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.905355 10076 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.906762  9949 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:35.801479  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:35.759569 10122 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:35.907764  1880 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:35.802604  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:35.908051 10122 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.760043 10107 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.802977  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:35.911067 10107 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:35.911549  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 108712us
I20260709 12:25:35.912636 10118 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:35.815902  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:35.915930  1879 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:35.916833 10123 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:35.917418 10123 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:35.893239  9846 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.919209  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:35.919586  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3778) took 2367 ms. Trace:
I20260709 12:25:35.919884  1800 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:35.920166  9846 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:35.827376 10068 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.921672  1723 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:25:35.918329 10123 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.923498 10068 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:35.924445  1875 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:35.925407 10073 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:35.817016  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:35.779827 10121 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:35.923377 10188 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:35.781455 10183 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.784663 10114 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.783213 10058 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:35.823961  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 273505us
I20260709 12:25:35.919974  1876 rpcz_store.cc:276] 0709 12:25:33.551781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.569658 (+ 17877us) service_pool.cc:224] Handling call
0709 12:25:34.701674 (+1132016us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.141141 (+439467us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.141173 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:35.141644 (+   471us) log.cc:844] Serialized 26 byte log entry
0709 12:25:35.216647 (+ 75003us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:35.216712 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.216832 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:35.672195 (+455363us) raft_consensus.cc:1685] finished
0709 12:25:35.672226 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.919306 (+247080us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400223150. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:35.919497 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":438989,"raft.queue_time_us":884131,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"spinlock_wait_cycles":2557650688,"thread_start_us":74577,"threads_started":1,"wal-append.queue_time_us":132888}
I20260709 12:25:35.829172  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:35.832229  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72304.6us
I20260709 12:25:35.824579  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:35.951803  2416 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:25:35.851338 10072 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:35.788064  1731 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.001s	sys 0.010s
W20260709 12:25:35.957865  1731 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.002s	sys 0.011s
I20260709 12:25:35.858388  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:35.881547  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.959703  1724 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:35.743942 10074 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.883049  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:35.781255 10061 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.886000  9951 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.825802 10163 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.901155  9762 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:35.851402 10040 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.948271 10188 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:35.946342 10121 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:35.957048 10114 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.902252 10066 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:36.040407  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:36.046837  2415 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:36.048343  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:35.857806 10105 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:35.903048 10002 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:35.876122 10112 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:35.903136 10005 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:35.640635  9755 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:35.947670  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:36.085268  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:36.086074  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 138525us
I20260709 12:25:36.085183 10005 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:36.087762 10005 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:35.842051  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.088992 10005 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:36.089818  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.091017 10159 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:36.091477 10159 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:36.092424 10125 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:36.092275 10159 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.093623 10159 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:36.033823 10058 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:36.094201  9846 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:36.039403 10188 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:35.970280  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 111982us
I20260709 12:25:35.971800  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.096352  1725 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:36.096457  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:36.097296  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.097530  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:36.097930  1725 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:35.855930  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:36.098960  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 243105us
I20260709 12:25:35.797866  9919 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.101765 10097 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:36.103027 10096 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:36.103943  9919 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:36.104592  9919 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:36.104977  9919 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.105332  9919 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:36.106433  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:35.995318  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:35.979379  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:36.107996  2418 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:25:35.825279 10055 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.123s
I20260709 12:25:36.109100 10055 consensus_queue.cc:1243] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 23.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:25:35.838636 10077 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:36.109664 10077 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:36.109907 10055 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:36.110425 10055 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:36.111392 10055 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.112931 10055 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:36.002862  2416 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:36.009235 10083 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:36.121896  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:36.011126  9645 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.904s	user 0.013s	sys 0.000s
I20260709 12:25:36.132203  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:36.016897 10163 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.018553 10074 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:35.999698 10172 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:36.019284  9951 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.020237  1873 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:36.022876  1621 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 13 to 23. New cstate: current_term: 23 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:36.023788  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1568) took 1796 ms. Trace:
I20260709 12:25:36.024266 10184 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:36.025709 10080 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.801074 10100 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.022073 10111 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.001394  9962 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.026705  2417 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:36.125931 10113 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:36.076838  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.058717  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.038961 10186 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.451s	user 0.002s	sys 0.000s
I20260709 12:25:36.106570 10084 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:36.085338 10037 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
I20260709 12:25:36.005541 10065 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:35.953400 10183 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:36.118703  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.176640  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:36.041147 10180 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.458s	user 0.000s	sys 0.002s
I20260709 12:25:36.046319 10066 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:36.098492  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:36.121031  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:36.040014 10182 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.411s	user 0.002s	sys 0.000s
I20260709 12:25:36.088809  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:36.128299  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:36.191835  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 63637.3us
I20260709 12:25:36.063261  1875 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:36.129845  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:36.195310  2416 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:36.139286  2410 rpcz_store.cc:276] 0709 12:25:34.227616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.244694 (+ 17078us) service_pool.cc:224] Handling call
0709 12:25:36.023544 (+1778850us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54004560d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:36.023706 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2910035840}
I20260709 12:25:36.027617 10122 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.140594  2419 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:36.027405 10107 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:36.142568  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1398) took 1141 ms. Trace:
I20260709 12:25:36.035921 10040 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:36.041486  1877 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:36.156172  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.165436 10072 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:36.168313 10176 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.211896 10040 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:36.214003 10176 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:36.214879  1732 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:36.125560 10002 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:36.211928  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 113510us
I20260709 12:25:36.217083 10002 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.141974  9846 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:36.217730  9846 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.218129  9846 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:36.144261  9933 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:25:36.175735 10094 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.214488  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:36.219936 10094 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.220629  2416 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 26 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:36.222409 10094 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.188023  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1314) took 1120 ms. Trace:
W20260709 12:25:36.200549  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 111807us
I20260709 12:25:36.201627  1731 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:36.201107  2411 rpcz_store.cc:276] 0709 12:25:35.001189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.001333 (+   144us) service_pool.cc:224] Handling call
0709 12:25:35.050649 (+ 49316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.147060 (+ 96411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.142229 (+995169us) spinlock_profiling.cc:243] Waited 95.7 ms on lock 0x7b5400487454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:36.142480 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":220132096}
I20260709 12:25:36.210757  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:36.232853  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:36.040467 10121 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.210624 10080 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:36.233421 10172 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:36.233879 10172 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:36.008548 10185 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.420s	user 0.001s	sys 0.002s
I20260709 12:25:36.234071 10121 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:36.234113 10072 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:36.235080 10072 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:36.235599  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1399) took 1083 ms. Trace:
I20260709 12:25:36.223532 10084 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:36.235963 10072 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.235996  2414 rpcz_store.cc:276] 0709 12:25:35.152449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.153078 (+   629us) service_pool.cc:224] Handling call
0709 12:25:35.203640 (+ 50562us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.226591 (+ 22951us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.226621 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:35.256069 (+ 29448us) log.cc:844] Serialized 26 byte log entry
0709 12:25:35.436978 (+180909us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:35.437444 (+   466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.437980 (+   536us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.007652 (+569672us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:36.235313 (+227661us) raft_consensus.cc:1685] finished
0709 12:25:36.235334 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.235539 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22526,"raft.queue_time_us":22985,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":14667,"spinlock_wait_cycles":4493184,"thread_start_us":103344,"threads_started":1,"wal-append.queue_time_us":262311}
I20260709 12:25:36.223872 10074 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:36.238428 10074 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:36.239281 10074 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:36.239928  9962 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.240191 10074 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.240548  9962 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.231187  1728 rpcz_store.cc:276] 0709 12:25:35.067967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.150318 (+ 82351us) service_pool.cc:224] Handling call
0709 12:25:35.857462 (+707144us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.857675 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:35.857972 (+   297us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.124161 (+266189us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:36.124212 (+    51us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:36.124264 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.124351 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.187761 (+ 63410us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54002d14d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:36.187942 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":685,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":2211,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":1531077376,"thread_start_us":265792,"threads_started":1}
I20260709 12:25:36.232508 10066 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.243007 10066 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:36.199783  9939 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.243968  9939 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.246179  9939 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:36.224557  9394 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:36.248301  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:36.249459  2414 raft_consensus.cc:2410] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:25:36.252609  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:36.253187  1725 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:36.254828  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.247242 10040 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:36.265408  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:36.179312 10113 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.262219  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:36.261013  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 302925us
W20260709 12:25:36.262972  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.269276  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.262812  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:36.264246  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.237548 10072 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:36.237484 10084 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:36.270393  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:36.234246 10172 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:36.273150  1724 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:36.274009  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:36.272571  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:36.282642  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:36.259222 10065 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:36.190380  9992 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.226544 10077 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.265456 10074 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:36.288743  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.288882  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.289438  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.294091  1731 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:36.271306  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.294703  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.296178  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.289942  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:36.290851  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.297931 10062 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:36.298421 10062 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:36.298857 10062 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:36.291862  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.302047  2413 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:36.271895 10113 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:36.303996  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1574) took 1045 ms. Trace:
I20260709 12:25:36.304397  2413 rpcz_store.cc:276] 0709 12:25:35.258328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.258527 (+   199us) service_pool.cc:224] Handling call
0709 12:25:36.303802 (+1045275us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b54004745d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:36.303928 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2262327680}
I20260709 12:25:36.271215 10111 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.307341  9962 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:36.296985  1724 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:36.299001  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.312932  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:36.274186 10187 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:36.314458  1728 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:36.277938 10084 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:36.314419  9980 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.327472  1803 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:36.248003 10125 consensus_queue.cc:1175] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 24 status { last_received { term: 23 index: 17 } last_committed_idx: 17 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:36.291404 10112 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:36.287132 10163 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:36.331827 10163 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.332273 10163 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:36.341799  9919 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.283711 10161 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:36.343034 10161 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.343525 10161 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:36.324283  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:36.349466  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 247498us
I20260709 12:25:36.352487  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:36.344871  9888 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.720s	user 0.012s	sys 0.000s
I20260709 12:25:36.364163  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:36.286576 10110 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
I20260709 12:25:36.377403  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.378942  1876 raft_consensus.cc:2410] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 25 because replica has last-logged OpId of term: 24 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 18.
I20260709 12:25:36.338233  9846 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.366010  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.380784  1873 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:36.374713  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:36.326030 10077 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.384944  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:36.376579  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.394668  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:36.386336  1872 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:36.394043 10107 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:36.414652  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:36.415380 10068 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:36.415112 10107 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.415927 10068 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.417680 10107 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:36.418186 10068 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:36.424679 10105 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.395207 10058 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.431231  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:36.431804 10058 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.432809  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:36.434351 10058 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:36.435104 10076 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:36.437965  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:36.439314  9762 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:36.439782  9762 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.440177  9762 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:36.442301  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.444234 10114 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.467675 10084 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.468350  9919 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:36.469033 10084 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:36.469717  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:36.438187 10192 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.088s	user 0.001s	sys 0.001s
I20260709 12:25:36.472548  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:36.473057  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.473801  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:36.474429  9919 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:36.454555  1880 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:36.456102  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:36.460429  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3861) took 1452 ms. Trace:
I20260709 12:25:36.464883  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:36.464530  1621 catalog_manager.cc:5697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 20. New cstate: current_term: 20 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:36.474716  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.489270  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:36.490638  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.504968  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.486152  1729 rpcz_store.cc:276] 0709 12:25:35.008011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.113745 (+105734us) service_pool.cc:224] Handling call
0709 12:25:35.164304 (+ 50559us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:35.185899 (+ 21595us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.185930 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:35.212221 (+ 26291us) log.cc:844] Serialized 52 byte log entry
0709 12:25:35.445911 (+233690us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:35.478728 (+ 32817us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.485029 (+  6301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.485195 (+   166us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.040303 (+555108us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:36.459970 (+419667us) raft_consensus.cc:1685] finished
0709 12:25:36.460129 (+   159us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.460355 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49000,"prepare.queue_time_us":31944,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":29943,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":19046,"spinlock_wait_cycles":5421952,"thread_start_us":166528,"threads_started":1,"wal-append.queue_time_us":271224}
I20260709 12:25:36.511715  1874 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:36.486425  1723 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 27: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:36.486626  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.487780  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.517536  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.518159  1880 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:36.519102  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.475562  2412 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:25:36.519227  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:36.519722  1880 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
W20260709 12:25:36.522514  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4767466510002123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
I20260709 12:25:36.523102  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.523922  1880 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:36.525430  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:36.526036  1880 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:36.523679  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:36.531260  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.531466  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:36.534154  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.529596  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:36.455229 10111 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:36.540248  1798 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:25:36.483035 10091 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.700s	user 0.008s	sys 0.004s
I20260709 12:25:36.476614  9846 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.538688  1803 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:36.540079  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:36.384265 10096 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:36.537580  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:36.475046  9919 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.527196 10176 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.537091 10058 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:36.571336  1874 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:36.574141  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.493772  1876 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:36.575879  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:36.579586  1873 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:36.579972  1880 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:36.582106  1727 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:36.540172  9879 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:36.588979  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.557150 10040 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:36.575883 10123 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:36.564421 10121 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.378509  9888 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.559263 10055 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.484874  9762 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.558969 10111 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.572944 10091 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:36.610947  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:36.615140  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.617460  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.619562  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:36.597822  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.573575  9394 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.622862  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:36.602659  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.624101  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.624897  1804 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:36.626137  1802 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 12:25:36.576182  9919 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:36.536044 10113 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.621109  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:36.626586  1802 pending_rounds.cc:85] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 10
I20260709 12:25:36.648257  1802 pending_rounds.cc:107] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:25:36.648685  1802 raft_consensus.cc:2889] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:36.578994 10058 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.624850  1803 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:36.666290  2414 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:36.667477  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:36.631707  9951 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:36.668778  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.640550  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 331305us
I20260709 12:25:36.647430  1873 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:25:36.601233 10123 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.636461  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:36.559185 10172 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.654522  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.655712  1728 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:36.664572  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:36.684538  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63535us
I20260709 12:25:36.679236  1724 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:36.676499  2383 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.036s (SENT)
I20260709 12:25:36.608835 10111 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:36.684015  1799 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:36.682569  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.676254  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.683401  1874 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
W20260709 12:25:36.623353 10108 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.034s	sys 0.057s
I20260709 12:25:36.686093  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:36.687150  1872 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:25:36.604029  9879 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:36.685575  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:36.584980 10176 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.663262 10100 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:25:36.707762  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.758544  9962 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.711024 10064 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:36.766511  1875 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:36.713994  1802 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:39.258387  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2893) took 2545 ms. Trace:
I20260709 12:25:39.258847  1802 rpcz_store.cc:276] 0709 12:25:36.712739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.713351 (+   612us) service_pool.cc:224] Handling call
0709 12:25:36.713753 (+   402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:39.257720 (+2543967us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.258189 (+   469us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:39.259730  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.940s	user 0.025s	sys 0.037s
W20260709 12:25:39.260486  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.941s	user 0.025s	sys 0.037s
I20260709 12:25:39.261435  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:39.263108  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 2.97294e+06us
I20260709 12:25:39.264675  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:36.717754  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:36.710443  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:36.726495  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.718353  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:39.774816  1873 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
W20260709 12:25:39.775436  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2993) took 3112 ms (client timeout 1533 ms). Trace:
W20260709 12:25:39.775828  1873 rpcz_store.cc:269] 0709 12:25:36.667402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.717944 (+ 50542us) service_pool.cc:224] Handling call
0709 12:25:39.775328 (+3057384us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:36.762555  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:38.449754 10219 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.780277 10219 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:39.781533  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2983) took 3122 ms (client timeout 1715 ms). Trace:
W20260709 12:25:39.782315  1873 rpcz_store.cc:269] 0709 12:25:36.659618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.781269 (+3121651us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:39.781460 (+   191us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:39.104238  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.721s	user 0.052s	sys 0.086s
I20260709 12:25:39.119848  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:38.671402  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 221222us
W20260709 12:25:36.759658  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:36.917924  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:39.128475  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1224) took 3011 ms. Trace:
W20260709 12:25:37.127086  2410 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.044s	sys 0.134s
W20260709 12:25:36.759851  2418 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.030s	sys 0.071s
W20260709 12:25:39.163587  1799 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.446s	user 0.021s	sys 0.045s
I20260709 12:25:37.081379  9762 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:36.977654  9879 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:39.822520  2410 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.222s	user 0.048s	sys 0.134s
I20260709 12:25:39.823961  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1580) took 3308 ms. Trace:
I20260709 12:25:39.824378  2410 rpcz_store.cc:276] 0709 12:25:36.515855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.516057 (+   202us) service_pool.cc:224] Handling call
0709 12:25:36.540581 (+ 24524us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:36.560288 (+ 19707us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:36.560332 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:36.573780 (+ 13448us) log.cc:844] Serialized 26 byte log entry
0709 12:25:39.823295 (+3249515us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:39.823388 (+    93us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.823522 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.823606 (+    84us) raft_consensus.cc:1685] finished
0709 12:25:39.823617 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.823850 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16536,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":3120,"thread_start_us":3222483,"threads_started":1,"wal-append.queue_time_us":903646,"wal-append.run_cpu_time_us":2623,"wal-append.run_wall_time_us":1002501}
W20260709 12:25:37.569895  9919 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.897s	user 0.006s	sys 0.009s
I20260709 12:25:36.665930  9933 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.102s
I20260709 12:25:36.657696  9394 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.774199  9962 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.937328  9980 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:38.641775  9992 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.627007 10055 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:36.672250 10058 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:36.749101 10061 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.134s
W20260709 12:25:36.911712 10062 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.605s	user 0.000s	sys 0.019s
I20260709 12:25:39.857395 10062 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:39.858486 10235 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:39.858980 10235 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:39.860136 10062 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.663990 10065 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:39.850142 10055 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:39.863087 10065 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:37.555995 10066 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.313s	user 0.015s	sys 0.001s
I20260709 12:25:36.733878 10068 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:39.864244  9394 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:37.303133 10074 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.975s	user 0.007s	sys 0.006s
I20260709 12:25:39.873821  1876 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:37.091454 10076 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.651s	user 0.010s	sys 0.000s
I20260709 12:25:36.697962 10080 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:37.602533 10084 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.133s	user 0.012s	sys 0.004s
I20260709 12:25:36.705741 10091 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.759971 10096 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.758896 10100 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:39.903978 10100 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:36.702382 10108 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.035s	sys 0.057s
W20260709 12:25:37.573758 10111 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.863s	user 0.009s	sys 0.005s
I20260709 12:25:37.115458 10113 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:36.616974 10121 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:36.680335 10123 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:37.148999 10157 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.097s	sys 0.097s
W20260709 12:25:39.949455 10157 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.484s	user 0.099s	sys 0.099s
W20260709 12:25:37.538594 10161 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.195s	user 0.015s	sys 0.000s
I20260709 12:25:39.954376 10161 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:37.359036 10163 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.026s	user 0.015s	sys 0.000s
I20260709 12:25:39.959309 10163 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:39.961018  1805 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:39.962255  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2881) took 3420 ms (client timeout 3062 ms). Trace:
I20260709 12:25:37.132397 10172 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:36.712123 10176 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:36.787629 10183 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.561s	user 0.010s	sys 0.000s
I20260709 12:25:36.691114 10184 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:37.070359 10185 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
W20260709 12:25:36.773725 10188 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.678s	user 0.011s	sys 0.002s
W20260709 12:25:36.735330 10194 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
W20260709 12:25:37.079119 10199 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.397s	user 0.001s	sys 0.000s
I20260709 12:25:39.983075  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1323) took 4098 ms. Trace:
I20260709 12:25:39.983634  1730 rpcz_store.cc:276] 0709 12:25:35.884184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.884422 (+   238us) service_pool.cc:224] Handling call
0709 12:25:35.884795 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.888873 (+  4078us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.888925 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:35.928224 (+ 39299us) log.cc:844] Serialized 26 byte log entry
0709 12:25:36.356046 (+427822us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:36.356097 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.356170 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.982510 (+3626340us) raft_consensus.cc:1685] finished
0709 12:25:39.982554 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.982818 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3573,"raft.queue_time_us":12874,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":28498,"thread_start_us":427369,"threads_started":1,"wal-append.queue_time_us":647749}
W20260709 12:25:37.464382 10200 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:25:37.570163 10203 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.290s	user 0.000s	sys 0.001s
W20260709 12:25:37.355341 10204 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
W20260709 12:25:37.461865 10210 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.347s	user 0.000s	sys 0.000s
I20260709 12:25:39.195623 10077 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:39.198565 10223 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.994413  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1336) took 3566 ms. Trace:
I20260709 12:25:39.203480 10224 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:39.208289 10226 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.995936 10226 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:39.266274 10159 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:39.995086  1731 rpcz_store.cc:276] 0709 12:25:36.427511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.433936 (+  6425us) service_pool.cc:224] Handling call
0709 12:25:36.434309 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:36.434697 (+   388us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:36.434730 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:36.436629 (+  1899us) log.cc:844] Serialized 26 byte log entry
0709 12:25:36.626983 (+190354us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:36.627033 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.627116 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.993891 (+3366775us) raft_consensus.cc:1685] finished
0709 12:25:39.993932 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.994194 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":445,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":189959,"threads_started":1,"wal-append.queue_time_us":620851}
I20260709 12:25:39.267781  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:39.253463  1803 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:39.999076  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 731573us
I20260709 12:25:39.999337  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2896) took 3285 ms. Trace:
I20260709 12:25:39.999765  1803 rpcz_store.cc:276] 0709 12:25:36.714113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.714908 (+   795us) service_pool.cc:224] Handling call
0709 12:25:39.252322 (+2537414us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:25:39.252879 (+   557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:39.999030 (+746151us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.999263 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106816768}
I20260709 12:25:40.001063  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.001958  1803 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:39.309309  1621 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 19 to 22, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 22 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:40.003227 10002 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:40.003815 10002 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:40.003641  1621 catalog_manager.cc:5697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 10 to 19, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 19 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:25:40.005167 10002 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:39.369908  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1239) took 2692 ms (client timeout 1562 ms). Trace:
I20260709 12:25:39.994835 10161 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:40.006996  1872 rpcz_store.cc:269] 0709 12:25:36.677236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.369338 (+2692102us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:39.369574 (+   236us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:40.008327  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2988) took 3347 ms (client timeout 2021 ms). Trace:
W20260709 12:25:40.008670  1872 rpcz_store.cc:269] 0709 12:25:36.661245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.008098 (+3346853us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:40.008245 (+   147us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:40.009522  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2987) took 3348 ms (client timeout 2101 ms). Trace:
W20260709 12:25:40.009889  1872 rpcz_store.cc:269] 0709 12:25:36.660920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.009344 (+3348424us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:40.009465 (+   121us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:39.388199  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2989) took 2726 ms (client timeout 1635 ms). Trace:
W20260709 12:25:39.389786  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.688s	user 0.003s	sys 0.025s
W20260709 12:25:39.391762  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:40.010953  1874 rpcz_store.cc:269] 0709 12:25:36.661576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.387785 (+2726209us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:39.387968 (+   183us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:39.385809   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:25:39.472559  1806 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.756s	user 0.027s	sys 0.034s
W20260709 12:25:40.012069  1845 connection.cc:256] RPC call timeout handler was delayed by -1.6629876279998825s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1511s for a response
W20260709 12:25:40.012809  1806 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.297s	user 0.029s	sys 0.035s
W20260709 12:25:40.012873  1845 connection.cc:256] RPC call timeout handler was delayed by -1.6521876279998651s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1523s for a response
W20260709 12:25:40.013592  1845 connection.cc:256] RPC call timeout handler was delayed by -1.5495876279996992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1637s for a response
I20260709 12:25:39.479674 10107 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.438076 10196 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.482254  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.780s	user 0.007s	sys 0.024s
I20260709 12:25:40.014600 10107 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.014745 10196 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:39.994948 10223 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.017061 10107 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.017648 10107 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.501842  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.018038 10223 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.018445 10084 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.012158  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2981) took 3353 ms (client timeout 2157 ms). Trace:
I20260709 12:25:39.489777 10221 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:40.012127 10096 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:40.019351  1874 rpcz_store.cc:269] 0709 12:25:36.658831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.011929 (+3353098us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:40.012091 (+   162us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:39.694422  1724 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 3.009s	user 0.014s	sys 0.041s
W20260709 12:25:39.726774  1623 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 254) took 56079 ms (client timeout 15000 ms). Trace:
I20260709 12:25:39.738107 10230 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:40.020491  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2992) took 3358 ms (client timeout 2164 ms). Trace:
I20260709 12:25:39.753759 10042 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:39.756270  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 3.0455e+06us
W20260709 12:25:39.766678  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.019939 10221 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:40.023329  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:40.023968  1846 connection.cc:256] RPC call timeout handler was delayed by -2.7737316669999927s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
W20260709 12:25:39.784449  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.401s	user 0.053s	sys 0.087s
I20260709 12:25:40.025547 10221 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.021178 10230 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:39.998462 10226 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:39.785933 10219 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:39.796231  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.797950  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1238) took 3126 ms (client timeout 1917 ms). Trace:
I20260709 12:25:40.029768 10230 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:39.795945  1800 rpcz_store.cc:276] 0709 12:25:36.117135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.118876 (+  1741us) service_pool.cc:224] Handling call
0709 12:25:36.123324 (+  4448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:36.135424 (+ 12100us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:36.135465 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:36.136006 (+   541us) log.cc:844] Serialized 26 byte log entry
0709 12:25:36.521615 (+385609us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:36.521659 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.521728 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:39.127618 (+2605890us) raft_consensus.cc:1685] finished
0709 12:25:39.127683 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:39.128138 (+   455us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6838,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":5603,"thread_start_us":338764,"threads_started":1,"wal-append.queue_time_us":825143,"wal-append.run_cpu_time_us":2930,"wal-append.run_wall_time_us":1398083}
I20260709 12:25:40.031088 10230 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.026999 10224 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.031594  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.032420 10113 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:40.032850  1800 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:25:40.033273 10113 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.018069 10196 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:40.030474  1873 rpcz_store.cc:269] 0709 12:25:36.671575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.797706 (+3126131us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:39.797865 (+   159us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:40.040060  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2986) took 3379 ms (client timeout 2808 ms). Trace:
W20260709 12:25:40.020967  1874 rpcz_store.cc:269] 0709 12:25:36.662526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.020312 (+3357786us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:40.020438 (+   126us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:40.038996  1724 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:40.034818 10112 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.041810  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2985) took 3381 ms (client timeout 2822 ms). Trace:
I20260709 12:25:40.041986 10112 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.042948 10229 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.043561 10229 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.044487 10112 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:37.283344  9888 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:39.828900  9919 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.058014  9919 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:39.829934  1799 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.113s	user 0.021s	sys 0.045s
I20260709 12:25:39.831878  1732 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:39.835301  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 715933us
I20260709 12:25:40.060855  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:40.061200  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3051) took 3447 ms. Trace:
I20260709 12:25:40.061787  1732 rpcz_store.cc:276] 0709 12:25:36.613703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.618598 (+  4895us) service_pool.cc:224] Handling call
0709 12:25:36.663690 (+ 45092us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.060136 (+3396446us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.060817 (+   681us) spinlock_profiling.cc:243] Waited 3.17 s on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:40.061108 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7285108480}
I20260709 12:25:39.836400  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:40.063370  1880 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 22. Ops: []
I20260709 12:25:40.064277  9933 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:39.841266  9962 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:39.850103  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.029852 10065 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.860324 10235 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.066514 10065 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.067952 10235 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:40.069168 10065 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.069938 10065 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:40.070353 10065 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:39.861464  9980 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.071382  9980 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.073836  1732 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260709 12:25:40.074294  1732 pending_rounds.cc:85] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 18
I20260709 12:25:39.872272  1878 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:25:40.075179  1731 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:25:40.075309 10040 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.076020  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2976) took 3688 ms (client timeout 1979 ms). Trace:
W20260709 12:25:40.076547  1878 rpcz_store.cc:269] 0709 12:25:36.387172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.388009 (+   837us) service_pool.cc:224] Handling call
0709 12:25:40.075702 (+3687693us) spinlock_profiling.cc:243] Waited 3.46 s on lock 0x7b5400052154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:40.075923 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7947255680}
W20260709 12:25:40.078042  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2990) took 3416 ms (client timeout 3229 ms). Trace:
I20260709 12:25:39.908492 10058 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:39.879006 10227 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.079381 10227 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.080511 10058 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:39.886250 10091 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:39.899439  1876 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:39.919046  1877 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:39.927297  1879 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
W20260709 12:25:39.935401  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2975) took 3561 ms (client timeout 1886 ms). Trace:
I20260709 12:25:39.968348 10184 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.976470 10188 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:39.940325 10121 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:39.962772  1805 rpcz_store.cc:269] 0709 12:25:36.541360 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.541489 (+   129us) service_pool.cc:224] Handling call
0709 12:25:39.961960 (+3420471us) spinlock_profiling.cc:243] Waited 3.38 s on lock 0x7b5400132254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:39.962163 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7776304896}
I20260709 12:25:39.994166 10163 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:39.994504 10077 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:39.997088 10159 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:39.269446  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2897) took 2554 ms (client timeout 1801 ms). Trace:
I20260709 12:25:39.278074  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:40.011308  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.310s	user 0.004s	sys 0.025s
W20260709 12:25:40.014016  1845 connection.cc:256] RPC call timeout handler was delayed by -1.4163876279999386s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
W20260709 12:25:40.015134  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 3.313s	user 0.008s	sys 0.024s
I20260709 12:25:40.018599  1728 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:39.529034  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1227) took 3131 ms (client timeout 2076 ms). Trace:
W20260709 12:25:39.689036  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.726702  1801 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 3.192s	user 0.037s	sys 0.041s
I20260709 12:25:40.020107 10096 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.020912 10107 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:39.811220  2382 connection.cc:256] RPC call timeout handler was delayed by -1.0787528040000325s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
I20260709 12:25:40.031898 10230 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:39.812587  2418 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 3.649s	user 0.031s	sys 0.072s
I20260709 12:25:40.033995  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:39.812057  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.034258 10219 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.034394 10157 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.010712  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2984) took 3350 ms (client timeout 2106 ms). Trace:
I20260709 12:25:40.005841  1621 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 reported cstate change: term changed from 14 to 18, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 18 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:39.991366 10100 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.082670 10227 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:40.112835 10234 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:40.153231 10234 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.153695 10077 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.040553 10113 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.104087  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:39.814220  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:40.042359  1724 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 3056) took 3422 ms (client timeout 2102 ms). Trace:
I20260709 12:25:39.828890 10232 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.040568  1873 rpcz_store.cc:269] 0709 12:25:36.660612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.039815 (+3379203us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:40.039975 (+   160us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:40.036335  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:40.042227  1874 rpcz_store.cc:269] 0709 12:25:36.660308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.041572 (+3381264us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:40.041725 (+   153us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:40.038844 10084 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.035290  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.039044  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.034720 10224 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.048929 10076 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:40.047533 10055 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.050560  9394 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.051119 10068 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.051574 10229 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:40.020733  1623 rpcz_store.cc:269] 0709 12:24:43.646830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.647734 (+   904us) service_pool.cc:224] Handling call
0709 12:25:39.726429 (+56078695us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:24.086495 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:24.109609 (+ 23114us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:24.109663 (+    54us) write_op.cc:435] Acquired schema lock
0709 12:25:24.109684 (+    21us) tablet.cc:662] Decoding operations
0709 12:25:24.110743 (+  1059us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:24.110886 (+   143us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:24.110917 (+    31us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:25:24.113767 (+  2850us) tablet.cc:701] Row locks acquired
0709 12:25:24.113824 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:25:24.114012 (+   188us) write_op.cc:270] Start()
0709 12:25:24.114175 (+   163us) write_op.cc:276] Timestamp: P: 1783599924113945 usec, L: 0
0709 12:25:24.114196 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:25:24.271384 (+157188us) log.cc:844] Serialized 4421 byte log entry
0709 12:25:25.522097 (+1250713us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd3e28 00007f16af80e581 00007f16af80dd58 00007f16af8129f1 00007f16af8129a9 00007f16af812939 00007f16af812901 00007f16af811b2d 0000000000388634 0000000000388479
0709 12:25:25.603349 (+ 81252us) op_driver.cc:464] REPLICATION: finished
0709 12:25:25.893260 (+289911us) write_op.cc:301] APPLY: starting
0709 12:25:25.893369 (+   109us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:26.607205 (+713836us) spinlock_profiling.cc:243] Waited 9.8 ms on lock 0x7b18002a5140. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16abb170ec 00007f16abb19af7 00007f16abb1afb0 00007f16af82f1bb 00007f16af82f532 00007f16af876b93 00007f16af723de3 00007f16af7236ab 00007f16af724e77
0709 12:25:26.613901 (+  6696us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:26.613939 (+    38us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:26.770685 (+156746us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:26.835278 (+ 64593us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=22,key_file_lookups=22,delta_file_lookups=22,mrs_lookups=0
0709 12:25:26.835378 (+   100us) write_op.cc:312] APPLY: finished
0709 12:25:26.836474 (+  1096us) log.cc:844] Serialized 105 byte log entry
0709 12:25:27.087131 (+250657us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:27.171812 (+ 84681us) write_op.cc:454] Released schema lock
0709 12:25:27.210149 (+ 38337us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:27.219641 (+  9492us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:27.219723 (+    82us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":660200,"reported_tablets":49,"spinlock_wait_cycles":332672,"child_traces":[["op",{"apply.queue_time_us":289613,"apply.run_cpu_time_us":53188,"apply.run_wall_time_us":1326164,"cfile_cache_hit":46,"cfile_cache_hit_bytes":50151,"num_ops":11,"prepare.queue_time_us":355403,"prepare.run_cpu_time_us":42297,"prepare.run_wall_time_us":1527367,"raft.queue_time_us":594583,"raft.run_cpu_time_us":13191,"raft.run_wall_time_us":509742,"replication_time_us":1441974,"spinlock_wait_cycles":45973632,"thread_start_us":1608701,"threads_started":5,"wal-append.queue_time_us":612972,"wal-append.run_cpu_time_us":23159,"wal-append.run_wall_time_us":2705640}]]}
I20260709 12:25:40.051576 10074 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:40.063299  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 227261us
I20260709 12:25:39.839758 10233 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:40.065136  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1221) took 3669 ms. Trace:
I20260709 12:25:40.066938 10108 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:40.076020 10040 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:39.935914 10111 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:39.966624 10183 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:40.155462  1845 connection.cc:256] RPC call timeout handler was delayed by -1.3137522239999271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20260709 12:25:40.078506  1878 rpcz_store.cc:269] 0709 12:25:36.661894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.077743 (+3415849us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:40.077967 (+   224us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:40.151149 10231 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:40.153971  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.156301  1800 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:40.151221  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2894) took 3438 ms. Trace:
I20260709 12:25:40.125561  1731 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
W20260709 12:25:40.155205  1872 rpcz_store.cc:269] 0709 12:25:36.659963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.010501 (+3350538us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:40.010650 (+   149us) inbound_call.cc:177] Queueing failure response
Metrics: {}
W20260709 12:25:40.170964  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2968) took 3858 ms (client timeout 1526 ms). Trace:
W20260709 12:25:40.251380  1724 rpcz_store.cc:269] 0709 12:25:36.619496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.622591 (+  3095us) service_pool.cc:224] Handling call
0709 12:25:40.042110 (+3419519us) spinlock_profiling.cc:243] Waited 55.6 ms on lock 0x7b54002b70d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:40.042264 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":127963392}
W20260709 12:25:40.252051  1845 connection.cc:256] RPC call timeout handler was delayed by -1.0581876279998141s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2183s for a response
I20260709 12:25:40.256939  1802 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:40.258033  1845 connection.cc:256] RPC call timeout handler was delayed by -1.0133769899998697s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
W20260709 12:25:40.155475  1804 rpcz_store.cc:269] 0709 12:25:36.714479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.717446 (+  2967us) service_pool.cc:224] Handling call
0709 12:25:39.269160 (+2551714us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.197844  1807 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:40.160980  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2991) took 3498 ms (client timeout 1577 ms). Trace:
I20260709 12:25:40.162986  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:40.256357  1877 rpcz_store.cc:269] 0709 12:25:36.312400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.312654 (+   254us) service_pool.cc:224] Handling call
0709 12:25:40.170581 (+3857927us) spinlock_profiling.cc:243] Waited 3.57 s on lock 0x7b54001f8e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:40.170853 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8200724864}
I20260709 12:25:40.203702  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1484) took 4072 ms. Trace:
W20260709 12:25:40.155215  1875 rpcz_store.cc:269] 0709 12:25:36.397953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.504197 (+106244us) service_pool.cc:224] Handling call
0709 12:25:39.528547 (+3024350us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b5400223b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:39.528809 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":597947264}
W20260709 12:25:40.168304  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 2973) took 3810 ms (client timeout 2008 ms). Trace:
W20260709 12:25:40.160660  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 3.46316e+06us
W20260709 12:25:40.180778  1902 debug-util.cc:398] Leaking SignalData structure 0x7b0800364d20 after lost signal to thread 564
I20260709 12:25:40.070722 10065 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:40.223558 10159 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:40.194520 10184 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:40.155022  1881 rpcz_store.cc:269] 0709 12:25:36.374119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.374342 (+   223us) service_pool.cc:224] Handling call
0709 12:25:39.934921 (+3560579us) spinlock_profiling.cc:243] Waited 3.55 s on lock 0x7b540021acd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:39.935204 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8160883456}
I20260709 12:25:40.253074  1880 rpcz_store.cc:276] 0709 12:25:36.396150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.623433 (+227283us) service_pool.cc:224] Handling call
0709 12:25:36.623759 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.064090 (+3440331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.064749 (+   659us) spinlock_profiling.cc:243] Waited 3.44 s on lock 0x7b54001f7f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:40.065023 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7909333248}
W20260709 12:25:40.276029  1902 debug-util.cc:398] Leaking SignalData structure 0x7b08001503c0 after lost signal to thread 1664
I20260709 12:25:40.158918 10072 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.169776  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.083144 10091 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.287052  1798 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.272181  1801 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:40.277031  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 1236) took 1166 ms. Trace:
I20260709 12:25:40.074481  9980 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:40.271477  1876 rpcz_store.cc:269] 0709 12:25:36.662217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.762234 (+100017us) service_pool.cc:224] Handling call
0709 12:25:40.160714 (+3398480us) spinlock_profiling.cc:243] Waited 92 ms on lock 0x7b54003525d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:40.160885 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211687168}
I20260709 12:25:40.074568  1732 pending_rounds.cc:107] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260709 12:25:40.160629 10100 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.203622 10064 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:40.104998 10161 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:40.274210  1879 rpcz_store.cc:269] 0709 12:25:36.358118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.358355 (+   237us) service_pool.cc:224] Handling call
0709 12:25:40.167931 (+3809576us) spinlock_profiling.cc:243] Waited 3.49 s on lock 0x7b540001ead4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:40.168200 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8026003200}
I20260709 12:25:40.106359 10221 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:40.285279  1726 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:40.289323  1732 raft_consensus.cc:2889] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:40.291574  1902 debug-util.cc:398] Leaking SignalData structure 0x7b080015d060 after lost signal to thread 1756
I20260709 12:25:40.277508 10230 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.102317 10123 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.290351  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:40.289031  1802 rpcz_store.cc:276] 0709 12:25:39.106854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.277591 (+170737us) service_pool.cc:224] Handling call
0709 12:25:40.273093 (+995502us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.294395  2410 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:40.199404 10188 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.296818  1805 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 18: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.287089  1727 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:40.252066 10108 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:40.306466  1902 debug-util.cc:398] Leaking SignalData structure 0x7b08003f1e60 after lost signal to thread 1831
I20260709 12:25:40.288700  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.298591  9919 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.299716 10049 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.301327 10080 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:40.311738 10080 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.312625  9962 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:40.293545 10072 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.314219  9933 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:40.314453 10107 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:40.302295  1844 connection.cc:256] RPC call timeout handler was delayed by -1.6205232090001118s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
I20260709 12:25:40.315650  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1228) took 3917 ms. Trace:
I20260709 12:25:40.316084  1879 rpcz_store.cc:276] 0709 12:25:36.398230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.296202 (+3897972us) service_pool.cc:224] Handling call
0709 12:25:40.315029 (+ 18827us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.315301 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:40.315324 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:40.315365 (+    41us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:40.315412 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.315465 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.315559 (+    94us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:40.317298  1845 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.511s (SENT) [suppressed 6 similar messages]
I20260709 12:25:40.297657 10058 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.318465 10058 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.320472  1880 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:40.321224  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1230) took 3922 ms. Trace:
I20260709 12:25:40.321332 10058 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.321593  1880 rpcz_store.cc:276] 0709 12:25:36.398764 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.319806 (+3921042us) service_pool.cc:224] Handling call
0709 12:25:40.320209 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.320933 (+   724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.321145 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.304662 10221 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.161296 10219 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.317206  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:40.154287 10157 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.308542  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.252606  1806 rpcz_store.cc:276] 0709 12:25:36.713225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.714908 (+  1683us) service_pool.cc:224] Handling call
0709 12:25:36.715327 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.715591 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:40.013607 (+3298016us) log.cc:844] Serialized 14 byte log entry
0709 12:25:40.150544 (+136937us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:40.150657 (+   113us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:40.150797 (+   140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.150943 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.151161 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":722425,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":8089,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":153,"thread_start_us":3433781,"threads_started":2,"wal-append.queue_time_us":136897}
I20260709 12:25:40.337740  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.338229  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.293974 10113 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.339129  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.310166 10064 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.311079 10183 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.314847  9933 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:40.118204  9992 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.205s
I20260709 12:25:40.304875 10084 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:40.322724  1801 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2879) took 3791 ms (client timeout 1511 ms). Trace:
I20260709 12:25:40.273222  2418 rpcz_store.cc:276] 0709 12:25:36.130885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.131010 (+   125us) service_pool.cc:224] Handling call
0709 12:25:36.134359 (+  3349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:36.135862 (+  1503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:36.135893 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:36.138095 (+  2202us) log.cc:844] Serialized 26 byte log entry
0709 12:25:40.154857 (+4016762us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:40.154934 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.203309 (+ 48375us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.203386 (+    77us) raft_consensus.cc:1685] finished
0709 12:25:40.203408 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.203615 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28125,"raft.queue_time_us":2741,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":9528,"thread_start_us":3990831,"threads_started":1,"wal-append.queue_time_us":1021175}
I20260709 12:25:40.324005  1803 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.323820  1879 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: [24.15-24.15]
W20260709 12:25:40.326633  1902 debug-util.cc:398] Leaking SignalData structure 0x7b08001be320 after lost signal to thread 1832
I20260709 12:25:40.322669  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.308439 10055 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.369906 10188 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:40.370434  1880 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:40.370831 10188 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.371989 10055 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.373164  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.330149  2419 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:25:40.331166 10111 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.296615 10231 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:40.379660  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:40.332458  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.380963  1802 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.381999  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.332537 10221 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.287781 10232 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.335011  9919 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.384609  9919 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.278908 10074 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.379660  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.386536  2414 raft_consensus.cc:2410] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 24 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260709 12:25:40.386981 10074 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:40.336278  1800 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:40.388412  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.389392 10096 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:40.389772  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.390458 10227 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:40.336125 10108 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.391806 10108 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:40.392654 10227 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:40.393117 10227 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:40.342285 10161 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.396507 10123 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:40.396812 10161 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.397570 10064 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.342682  1876 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:25:40.318585 10049 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.256556 10068 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:40.347325  1801 rpcz_store.cc:269] 0709 12:25:36.530886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.531105 (+   219us) service_pool.cc:224] Handling call
0709 12:25:40.306775 (+3775670us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.343891 10113 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.431423  2411 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:40.350236 10058 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:40.432821 10113 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:40.350533  1902 debug-util.cc:398] Leaking SignalData structure 0x7b08002c7880 after lost signal to thread 1905
W20260709 12:25:40.433614  1902 debug-util.cc:398] Leaking SignalData structure 0x7b080022fe80 after lost signal to thread 1906
I20260709 12:25:40.350605 10230 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:40.435532  1902 debug-util.cc:398] Leaking SignalData structure 0x7b0800049260 after lost signal to thread 2443
I20260709 12:25:40.351332  9933 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.355295  9992 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:40.356278  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1229) took 4222 ms. Trace:
I20260709 12:25:40.359313  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.361078  1872 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:40.362449 10219 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.290657 10076 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.363394  1877 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:25:40.364180 10080 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:40.453822  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 136333us
I20260709 12:25:40.454501  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1232) took 4030 ms. Trace:
I20260709 12:25:40.454952  1872 rpcz_store.cc:276] 0709 12:25:36.424382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.360435 (+3936053us) service_pool.cc:224] Handling call
0709 12:25:40.360854 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.454199 (+ 93345us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.454436 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.366547 10112 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:40.367141  9980 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.367914  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3864) took 4487 ms. Trace:
I20260709 12:25:40.372512  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1231) took 3959 ms. Trace:
I20260709 12:25:40.349511  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1229) took 3951 ms. Trace:
I20260709 12:25:40.373934  2413 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:40.379503  2418 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:25:40.384300 10107 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:25:40.457969  1726 rpcz_store.cc:276] 0709 12:25:35.880554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.881151 (+   597us) service_pool.cc:224] Handling call
0709 12:25:35.899991 (+ 18840us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.303037 (+4403046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.367603 (+ 64566us) spinlock_profiling.cc:243] Waited 4.38 s on lock 0x7b54002d19d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:40.367835 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10081283328}
I20260709 12:25:40.387331 10157 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.400784 10040 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:40.405463 10072 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:40.475620 10065 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:40.406242  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.407975  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.479192  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.409157  1873 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:25:40.410202  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2887) took 3802 ms (client timeout 1785 ms). Trace:
I20260709 12:25:40.479858  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.308939  9394 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.384609 10232 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.414215  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2899) took 1311 ms. Trace:
I20260709 12:25:40.414930 10231 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:25:40.339087  1806 raft_consensus.cc:2368] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 27.
I20260709 12:25:40.416224 10183 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:40.417152 10062 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:40.417680 10096 consensus_queue.cc:1175] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" responder_term: 27 status { last_received { term: 26 index: 14 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:40.418299  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.418648  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.372722  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.420940  1800 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:40.349522  1875 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:40.422891  1728 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:25:40.424312  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1226) took 4023 ms. Trace:
I20260709 12:25:40.426981  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.160445  1621 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 21, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 21 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:40.451076  9992 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:40.507489  1876 rpcz_store.cc:276] 0709 12:25:36.397674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.293847 (+3896173us) service_pool.cc:224] Handling call
0709 12:25:40.340398 (+ 46551us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.420392 (+ 79994us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.420621 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.452661  2410 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:25:40.453425  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 2982) took 3791 ms. Trace:
I20260709 12:25:40.449338  1807 rpcz_store.cc:276] 0709 12:25:36.133537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.137113 (+  3576us) service_pool.cc:224] Handling call
0709 12:25:36.154343 (+ 17230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.283900 (+4129557us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.355881 (+ 71981us) spinlock_profiling.cc:243] Waited 4.04 s on lock 0x7b5400167bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:40.356194 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9297642368}
I20260709 12:25:40.508952  2419 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:40.509773  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.509967  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.509995  1727 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:40.511265  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.512360  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:40.512679  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.513319  2413 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:40.513456  1807 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:40.460512  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.458850  1880 rpcz_store.cc:276] 0709 12:25:36.413533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.342065 (+3928532us) service_pool.cc:224] Handling call
0709 12:25:40.350143 (+  8078us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.371300 (+ 21157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.372217 (+   917us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b5400073854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:40.372422 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45840128}
I20260709 12:25:40.459453  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.508776  1877 rpcz_store.cc:276] 0709 12:25:36.659253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.362563 (+3703310us) service_pool.cc:224] Handling call
0709 12:25:40.362976 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.450302 (+ 87326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.450600 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.515846  1725 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:40.516146  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.516330  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.517040 10234 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.517848  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:40.462939  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 145933us
I20260709 12:25:40.517792  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.466051  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.459888  2418 pending_rounds.cc:85] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 15
I20260709 12:25:40.462950 10091 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:40.508486  1801 raft_consensus.cc:2410] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:25:40.458981  1879 rpcz_store.cc:276] 0709 12:25:36.398498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.317446 (+3918948us) service_pool.cc:224] Handling call
0709 12:25:40.323634 (+  6188us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:40.349052 (+ 25418us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.349424 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.472748  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.478926 10112 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.484243  1803 rpcz_store.cc:276] 0709 12:25:39.103815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.002664 (+898849us) service_pool.cc:224] Handling call
0709 12:25:40.414112 (+411448us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:40.480134  1798 rpcz_store.cc:269] 0709 12:25:36.607367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.607600 (+   233us) service_pool.cc:224] Handling call
0709 12:25:40.409925 (+3802325us) spinlock_profiling.cc:243] Waited 3.6 s on lock 0x7b5400066154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:40.410121 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8272578432}
I20260709 12:25:40.518735  1880 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:40.471901  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.514575  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.524060  2418 pending_rounds.cc:107] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:25:40.524777  2418 raft_consensus.cc:2889] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:40.509773  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.518954  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2895) took 3805 ms. Trace:
I20260709 12:25:40.528543  1726 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:40.530431  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.531996  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.532735  1875 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:40.533442 10091 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.533788  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.535848  1872 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:40.536089 10091 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.536935  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.537199 10049 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:40.537753  1872 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:40.521181  1723 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:40.506687  9980 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.539727  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.541112  9980 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.520744 10230 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.542951 10183 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.543408  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.544344  1878 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 18: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:40.544394 10230 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.545356 10183 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.550663  2415 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:25:40.479038 10065 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.552310 10064 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:40.554308  9394 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.554641 10112 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:40.555354  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.555395  1802 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:40.556173 10061 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:40.557112  1805 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:40.556945 10061 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.557677  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:40.558188  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.559127  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:40.559618  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.560354  2412 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:40.561383  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.562114  1729 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:40.559165 10188 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.563199  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.545560  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.565141  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1343) took 1021 ms. Trace:
I20260709 12:25:40.565487  1878 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:40.534067  2414 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:40.550315  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:40.533057  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:40.568099  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.537642  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:40.486480  1696 connection.cc:256] RPC call timeout handler was delayed by -2.6625644000000648s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1506s for a response
W20260709 12:25:40.570482  1696 connection.cc:256] RPC call timeout handler was delayed by -2.6445644000000357s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1526s for a response
W20260709 12:25:40.570978  1696 connection.cc:256] RPC call timeout handler was delayed by -2.5986643999999615s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
W20260709 12:25:40.571501  1696 connection.cc:256] RPC call timeout handler was delayed by -2.5464643999998771s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1635s for a response
I20260709 12:25:40.563442  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.563927  1730 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:40.456718 10100 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:40.565609  1728 rpcz_store.cc:276] 0709 12:25:39.500312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.501158 (+   846us) service_pool.cc:224] Handling call
0709 12:25:40.521288 (+1020130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.576122  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:40.571924  1696 connection.cc:256] RPC call timeout handler was delayed by -2.477164399999765s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17120000000000002s for a response
W20260709 12:25:40.577271  1696 connection.cc:256] RPC call timeout handler was delayed by -2.4744643999997606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1715s for a response
W20260709 12:25:40.577746  1696 connection.cc:256] RPC call timeout handler was delayed by -2.4744643999997606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1715s for a response
I20260709 12:25:40.528151  1799 rpcz_store.cc:276] 0709 12:25:36.713607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.716273 (+  2666us) service_pool.cc:224] Handling call
0709 12:25:36.716616 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:36.716875 (+   259us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:40.059326 (+3342451us) log.cc:844] Serialized 14 byte log entry
0709 12:25:40.421579 (+362253us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:40.452401 (+ 30822us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:40.457229 (+  4828us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.482581 (+ 25352us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.518884 (+ 36303us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5190,"prepare.queue_time_us":721462,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":38347,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":154,"thread_start_us":3577585,"threads_started":2,"wal-append.queue_time_us":398983}
I20260709 12:25:40.580448  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.581712  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.582003  1730 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:40.530418  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:40.583271  2411 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:40.583266  1873 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:40.552302  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:40.584528  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:40.578423  1696 connection.cc:256] RPC call timeout handler was delayed by -2.4141387680001571s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1533s for a response
I20260709 12:25:40.573139  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:40.585510  1696 connection.cc:256] RPC call timeout handler was delayed by -2.3898644000000786s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1809s for a response
I20260709 12:25:40.586488  2410 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:40.586179  1696 connection.cc:256] RPC call timeout handler was delayed by -2.3448644000000058s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
W20260709 12:25:40.588419  1696 connection.cc:256] RPC call timeout handler was delayed by -2.3205643999999666s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1886s for a response
W20260709 12:25:40.588810  1696 connection.cc:256] RPC call timeout handler was delayed by -2.3205643999999666s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1886s for a response
W20260709 12:25:40.589291  1696 connection.cc:256] RPC call timeout handler was delayed by -2.2368643999998312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260709 12:25:40.589643  1696 connection.cc:256] RPC call timeout handler was delayed by -2.2368643999998312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19790000000000002s for a response
W20260709 12:25:40.589555  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:40.590262  1696 connection.cc:256] RPC call timeout handler was delayed by -2.210764399999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
W20260709 12:25:40.591078  1696 connection.cc:256] RPC call timeout handler was delayed by -2.210764399999789s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2008s for a response
I20260709 12:25:40.585294  1725 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:40.592221  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.592614  1732 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:40.593746  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.575362  1804 raft_consensus.cc:2376] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24. Re-sending same reply.
I20260709 12:25:40.594983  1771 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:40.595155  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.595822  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.596336  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.596609  1804 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 17.
I20260709 12:25:40.597108  1806 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:40.597725  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.596383  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.598960  1801 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.543408  9992 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.587234  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.600999  9992 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:40.602648  9933 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:40.535887 10068 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.603356 10061 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.604161  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.604842  1877 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:40.605345  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.606130  2415 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:40.607196  1807 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:40.608062  2415 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:25:40.608345  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:40.608862  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:40.609306  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.609715  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.610404  1807 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:40.610572  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.610947  9394 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:40.561386 10076 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:40.614456  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:40.612694 10064 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.615114  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.617411  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.544582  1621 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 21, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 21 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:40.618657 10234 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:40.618878  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:40.618949  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.620260  9980 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.620139 10230 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:40.620102  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.658258  1876 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:40.621910  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:40.622135  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.659806  2418 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:40.550302 10184 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.599771  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.593791  1729 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:40.612201  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.628175  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.627643  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.629269  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.629918  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:40.674640  1876 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:40.644009  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.631475  1799 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:40.654187  1880 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
W20260709 12:25:40.638569  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50918.3us
I20260709 12:25:40.650481  1879 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:25:40.620589  1723 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:25:40.628258  9394 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.631053 10064 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:40.631719 10112 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:40.666499  9980 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:40.667641  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45920.5us
I20260709 12:25:40.668362  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.669935  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.659919  1878 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:40.671651  2413 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:40.674453  1877 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
W20260709 12:25:40.591563  1696 connection.cc:256] RPC call timeout handler was delayed by -2.19906439999977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2021s for a response
I20260709 12:25:40.712143  1805 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:40.532948 10062 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.680677  1872 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:40.718513  1803 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:25:40.681823  1800 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:40.714214  2411 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:40.677731  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.719794  1724 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:40.720938  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.706346  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.721732  1876 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:40.715704  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1224) took 4292 ms. Trace:
I20260709 12:25:40.687883  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:40.713099  1696 connection.cc:256] RPC call timeout handler was delayed by -2.1864644000002045s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20350000000000001s for a response
W20260709 12:25:40.725473  1696 connection.cc:256] RPC call timeout handler was delayed by -2.1270644000001084s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
W20260709 12:25:40.725980  1696 connection.cc:256] RPC call timeout handler was delayed by -2.1270644000001084s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2101s for a response
W20260709 12:25:40.726405  1696 connection.cc:256] RPC call timeout handler was delayed by -2.1225644000001012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
W20260709 12:25:40.726856  1696 connection.cc:256] RPC call timeout handler was delayed by -2.0766644000000269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2157s for a response
I20260709 12:25:40.506934 10227 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.730541  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.731441  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.727937  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.728084  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.705288  9992 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:40.732673  1877 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:40.727265  1696 connection.cc:256] RPC call timeout handler was delayed by -2.0766644000000269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2157s for a response
I20260709 12:25:40.731247  1799 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
W20260709 12:25:40.734869  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47044.1us
I20260709 12:25:40.732059  1807 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
W20260709 12:25:40.733162  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 359526us
W20260709 12:25:40.736891  1696 connection.cc:256] RPC call timeout handler was delayed by -2.015464399999928s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
I20260709 12:25:40.680423  9394 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:40.735510  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.737825  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.738517  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:40.738656  1878 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:40.739189  1876 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:40.739799 10184 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:40.739888  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.740759 10245 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:40.741223 10245 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.741566 10245 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:40.743013  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 223072us
I20260709 12:25:40.743372  2416 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:25:40.746752  1801 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.684700 10076 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260709 12:25:40.747805  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.748579  1801 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:25:40.714502 10112 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:40.749339  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.749387 10112 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:40.749986  1801 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:40.735507  2412 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:40.751293  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.752254  1723 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:40.742584 10159 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.657948 10049 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.067s
I20260709 12:25:40.753126  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.722229  1881 rpcz_store.cc:276] 0709 12:25:36.397047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.290974 (+3893927us) service_pool.cc:224] Handling call
0709 12:25:40.311617 (+ 20643us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:40.312159 (+   542us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:40.312194 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:40.312864 (+   670us) log.cc:844] Serialized 26 byte log entry
0709 12:25:40.436278 (+123414us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:40.681568 (+245290us) log.cc:844] Serialized 14 byte log entry
0709 12:25:40.688713 (+  7145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.688802 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.688843 (+    41us) raft_consensus.cc:1685] finished
0709 12:25:40.688857 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.689014 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6560,"raft.queue_time_us":8522,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":11981,"thread_start_us":340839,"threads_started":2,"wal-append.queue_time_us":265961}
I20260709 12:25:40.801862  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:40.737912  1696 connection.cc:256] RPC call timeout handler was delayed by -1.8462387680001484s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21640000000000004s for a response
W20260709 12:25:40.756515  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.754262  1806 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.780611  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.781351  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:40.807056  1696 connection.cc:256] RPC call timeout handler was delayed by -1.8462387680001484s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21640000000000004s for a response
I20260709 12:25:40.807297  1803 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:40.807483  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4907643999999891s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2808s for a response
I20260709 12:25:40.780766  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.796352  2410 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:40.755169  1880 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:40.807850  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4907643999999891s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2808s for a response
I20260709 12:25:40.756285  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.689276 10100 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.808322  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:40.818869  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.819628  1732 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:40.737882  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:40.808736  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4781643999999687s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2822s for a response
I20260709 12:25:40.809583  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.735872 10040 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:40.818161  1876 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:40.821869  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:40.822396  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 203666us
I20260709 12:25:40.828167  1872 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
W20260709 12:25:40.821481  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4781643999999687s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2822s for a response
W20260709 12:25:40.829162  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1649643999999171s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.317s for a response
W20260709 12:25:40.829535  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1118643999998312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3229s for a response
I20260709 12:25:40.829344  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:40.829944  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1118643999998312s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3229s for a response
I20260709 12:25:40.736240  9992 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:40.822463  1726 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:40.838049  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 91987.9us
I20260709 12:25:40.831981  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.833340  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.834507  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.836352  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.830313  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.839704  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.841741  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.844584  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.845573  1805 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
W20260709 12:25:40.840619  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 102817us
I20260709 12:25:40.840297  1723 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:40.851287  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.851754  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.852608  1801 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:40.853653  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.854492  1881 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:40.857339  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.858630  1807 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.863132  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:40.784227  9962 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 23.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.051s
I20260709 12:25:40.826262 10065 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:40.866858  1806 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.867127  1799 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:40.870428  1877 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:40.833462 10040 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:40.871304  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 158159us
I20260709 12:25:40.878746  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.877599  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.846238  9992 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:40.878311  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.904063  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.904740  1723 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:40.874208 10076 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:40.880992  1798 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:40.907404  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2898) took 4192 ms. Trace:
I20260709 12:25:40.907775  1798 rpcz_store.cc:276] 0709 12:25:36.763804 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.870097 (+4106293us) service_pool.cc:224] Handling call
0709 12:25:40.877115 (+  7018us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.907101 (+ 29986us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.907320 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.910252  1798 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:25:40.881551  1805 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:25:40.910986  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.911842  1730 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 18: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:40.912819  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.914283 10062 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.914501  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.915807  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.916575  1881 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:40.917250 10061 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:40.917591  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:40.918381  1881 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:40.918241 10183 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:40.918866 10061 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:40.919795 10183 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:40.920348 10061 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:40.886991  9962 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:40.921067  9962 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:40.871958 10230 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.050s
I20260709 12:25:40.922122  2416 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:25:40.921986  9962 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.923377  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.886991  1725 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:40.924150  1732 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:25:40.924633  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.925141  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:40.879810  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:40.906606 10076 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:40.927582  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:40.927655 10076 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:40.929980  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:40.930333 10065 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:40.884537  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.931916  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.931448 10065 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.932940  1724 raft_consensus.cc:2410] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
W20260709 12:25:40.933892  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54146.2us
I20260709 12:25:40.920651 10183 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:40.934907 10230 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:25:40.935683  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69061.8us
I20260709 12:25:40.923511  9962 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:40.925837  2417 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:40.936561  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:40.948163  1728 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:40.891094  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.937367 10161 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.003s
I20260709 12:25:40.940662 10183 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:40.937646 10040 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:40.934623  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:40.941969  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66937.9us
I20260709 12:25:40.944052  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1232) took 1839 ms. Trace:
I20260709 12:25:40.957818  1804 rpcz_store.cc:276] 0709 12:25:39.104975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.943127 (+1838152us) service_pool.cc:224] Handling call
0709 12:25:40.943497 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.943728 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:40.943753 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:40.943793 (+    40us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:40.943821 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.943868 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.943976 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:40.956854  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:40.926256 10234 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:40.935001 10246 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:40.879578  1727 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:40.936745  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:40.916369  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1223) took 4519 ms. Trace:
I20260709 12:25:40.925876  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.841502  1621 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 22, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 22 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:40.949486  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.973635  1730 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:40.890342  1806 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:25:40.976521  1729 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:25:40.965651  1803 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:40.977922  1726 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:40.926466  1732 raft_consensus.cc:2376] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21. Re-sending same reply.
I20260709 12:25:40.896436  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:40.979030  1725 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:40.967749  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.980676  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.980738  1801 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:40.948191  1873 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:40.971702  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.984894  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.985677  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.986176  1872 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 19. (term mismatch)
I20260709 12:25:40.937091 10231 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.050s
I20260709 12:25:40.967422 10080 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:40.987237 10080 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.987669 10080 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:40.986374  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:40.988112  1876 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:40.989539  1876 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:25:40.989676 10227 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:40.991096 10184 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:25:40.991480  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55053.4us
I20260709 12:25:40.992313  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:40.993821 10183 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.994602 10061 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:40.994930 10062 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:40.995430 10062 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:40.996377 10062 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.997680 10062 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:41.000782  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:40.972036  1874 rpcz_store.cc:276] 0709 12:25:36.396729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.291175 (+3894446us) service_pool.cc:224] Handling call
0709 12:25:40.420267 (+129092us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:40.587286 (+167019us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:40.587322 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:40.613289 (+ 25967us) log.cc:844] Serialized 26 byte log entry
0709 12:25:40.752099 (+138810us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:40.752144 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.824639 (+ 72495us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:40.824691 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.916064 (+ 91373us) raft_consensus.cc:1685] finished
0709 12:25:40.916083 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.916287 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11319,"raft.queue_time_us":11918,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":332487040,"thread_start_us":138384,"threads_started":1,"wal-append.queue_time_us":224045}
I20260709 12:25:41.002918 10096 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:41.001022  1727 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:25:41.003793  1727 pending_rounds.cc:85] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 11
I20260709 12:25:40.985137  1879 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:41.004142  1727 pending_rounds.cc:107] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:25:41.004520  1727 raft_consensus.cc:2889] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:41.004877  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48098.2us
I20260709 12:25:41.005357  1879 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:41.005864  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:40.935024 10058 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:25:41.013399  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 185096us
I20260709 12:25:41.020910  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:41.012297  1872 pending_rounds.cc:85] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 18
I20260709 12:25:41.021711  1872 pending_rounds.cc:107] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 24.19
I20260709 12:25:41.022032  1872 raft_consensus.cc:2889] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:41.015865  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.017978  1876 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:41.022908  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:41.025478  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 129144us
I20260709 12:25:40.986339  1730 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:40.938007 10100 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:41.014302  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:40.909663 10064 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:40.935702 10230 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:41.028821  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:40.993963  1621 catalog_manager.cc:5697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 19 to 22. New cstate: current_term: 22 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:25:41.034838  1872 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:25:41.036880  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:41.052991  1880 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:41.039040  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:41.031581  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.043792 10068 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:25:41.044155  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 168857us
I20260709 12:25:40.987717  9980 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:40.995272 10183 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:41.047065  1873 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:41.029588  1731 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:41.049978  1727 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:41.021450 10058 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.034421 10159 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:41.023672 10234 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:41.012543 10246 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:41.083688 10246 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:41.086934  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.034904 10074 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:41.059288  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:41.037834 10111 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:25:41.091674  2416 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
W20260709 12:25:41.092233  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 100026us
I20260709 12:25:41.093398  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:41.094982 10061 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:41.068662  9980 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:41.099440 10080 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.053928 10049 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:41.045675 10100 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:41.021312 10091 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:25:41.160279  1874 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:25:41.074249  1621 catalog_manager.cc:5697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 16 to 19, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 19 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:41.075356  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 52507us
I20260709 12:25:41.077549 10245 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:41.104619  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.065253  1844 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.102299 10184 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:41.136102  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.069074 10058 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:41.138985 10230 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.140199 10234 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.140964 10100 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:41.145972  1723 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
W20260709 12:25:41.237671  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.147179 10112 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.149277 10064 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:41.099630 10161 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:41.155349  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:41.240715  2411 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:25:41.156317  1877 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:25:41.182731  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 117601us
W20260709 12:25:41.202767  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 272868us
I20260709 12:25:41.199456  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.102536 10080 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:41.242424  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.164274  9980 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:41.245914  9980 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.249084  9980 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:41.239651 10100 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:41.253515  9394 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.255353 10080 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:41.255839  9394 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:41.256026 10080 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.256666 10080 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:41.259472  1621 catalog_manager.cc:5697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:25:41.163059  9992 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.263609 10245 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:41.263098  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.264467 10254 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:41.264894  9992 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:41.264999 10254 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:41.265462 10254 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:41.250614  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 182135us
I20260709 12:25:41.266873  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:41.267313  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 134874us
I20260709 12:25:41.268086  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.240732 10161 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.269474 10161 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:41.261686  1621 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 17 to 24, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 24 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:41.275767  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:41.277361 10184 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:41.278551 10249 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:41.278678 10219 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:41.279120 10249 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:41.279325 10219 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:41.279526 10249 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:41.280378 10219 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.284920  1621 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 19 to 22. New cstate: current_term: 22 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:41.293859 10066 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:41.295208 10066 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:41.295908 10066 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:41.296517 10112 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:41.297187 10066 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.198585 10096 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:41.300468  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:41.302425 10072 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:41.268160  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:41.303511  1621 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 reported cstate change: term changed from 14 to 20. New cstate: current_term: 20 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:25:41.334357  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67575.5us
W20260709 12:25:41.336670  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68618.8us
I20260709 12:25:41.353499  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3800) took 1074 ms. Trace:
I20260709 12:25:41.354272  1881 rpcz_store.cc:276] 0709 12:25:40.279149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.920111 (+640962us) service_pool.cc:224] Handling call
0709 12:25:40.920377 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:40.920629 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:40.920964 (+   335us) log.cc:844] Serialized 14 byte log entry
0709 12:25:41.353046 (+432082us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:41.353117 (+    71us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:41.353183 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.353265 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:41.353421 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27226,"prepare.queue_time_us":788,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"thread_start_us":404255,"threads_started":1}
I20260709 12:25:41.358147  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:41.386232  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:41.413512  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:41.414386  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:41.396759  2410 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:41.421654  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:41.435618  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 77770.1us
I20260709 12:25:41.391258  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.344861 10068 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:41.439579  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.345664 10061 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:41.441325  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:41.442178  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:41.447677  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.432585  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:41.440508  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:41.458793 10072 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:41.439375  9992 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:41.462288 10074 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:41.464125  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:41.512792  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 491888us
I20260709 12:25:41.513119  2416 raft_consensus.cc:2410] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:25:41.463135  1621 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 15, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 15 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:41.464994  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:41.475056 10061 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:41.475382  2412 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:41.478657  1727 raft_consensus.cc:2410] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 22 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:25:41.489780 10080 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.479396  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:41.500366 10040 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.520s	user 0.009s	sys 0.001s
I20260709 12:25:41.433436  9962 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:41.502548 10227 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.512s	user 0.010s	sys 0.003s
W20260709 12:25:41.502933  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 338310us
I20260709 12:25:41.504045 10232 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:25:41.508985  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.513721  2414 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 27: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:41.345947  9394 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:41.520635 10061 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:41.529904  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:41.537050  2415 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.000s	sys 0.014s
W20260709 12:25:41.539848  2415 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.000s	sys 0.015s
I20260709 12:25:41.512493  9992 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:41.562585  1807 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.005s	sys 0.010s
W20260709 12:25:41.563099  1807 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.006s	sys 0.010s
I20260709 12:25:41.566188  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3806) took 1158 ms. Trace:
I20260709 12:25:41.566895  1874 rpcz_store.cc:276] 0709 12:25:40.407990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.002969 (+594979us) service_pool.cc:224] Handling call
0709 12:25:41.053840 (+ 50871us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.164483 (+110643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.370830 (+206347us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b5400117e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:41.562007 (+191177us) spinlock_profiling.cc:243] Waited 50.4 ms on lock 0x7b5400117e50. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:41.566075 (+  4068us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":354839168}
W20260709 12:25:41.570497  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.074s	sys 0.113s
W20260709 12:25:41.571281  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.075s	sys 0.113s
I20260709 12:25:41.578593  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:41.579264  2410 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:41.579690  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.014s	sys 0.040s
W20260709 12:25:41.580291  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.014s	sys 0.040s
I20260709 12:25:41.530642 10232 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:41.581782  1696 leader_election.cc:400] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 27.
I20260709 12:25:41.602944  1696 leader_election.cc:403] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
W20260709 12:25:41.606405  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 85627.5us
I20260709 12:25:41.608456  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.608544 10100 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.609476  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:41.610879 10232 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.612355 10232 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:41.613071  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:41.530462 10040 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.532974  9962 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:41.615688  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53479.4us
I20260709 12:25:41.615207  1621 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 22, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 22 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:41.616601 10040 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:41.617000  9962 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:41.617255  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.617579  1621 catalog_manager.cc:5697] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 13 to 16, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 16 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:25:41.619406 10062 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 27.
I20260709 12:25:41.538265  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6388b1a5243d463589750794e8ac5511"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:41.631353  1727 raft_consensus.cc:606] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:41.631505 10062 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:41.632450 10062 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:41.632871 10062 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:41.620078 10058 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.620287  9992 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.633267 10062 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:41.634881 10091 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:25:41.635674  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.623871  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2928) took 1011 ms. Trace:
I20260709 12:25:41.635535  1621 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 13 to 17, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 17 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:25:41.636562  1802 rpcz_store.cc:276] 0709 12:25:40.612373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.928941 (+316568us) service_pool.cc:224] Handling call
0709 12:25:40.929311 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:40.974555 (+ 45244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:40.974592 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:41.034054 (+ 59462us) log.cc:844] Serialized 26 byte log entry
0709 12:25:41.345559 (+311505us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:41.345602 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.345693 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:41.623534 (+277841us) raft_consensus.cc:1685] finished
0709 12:25:41.623566 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:41.623737 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4273,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":149,"thread_start_us":310949,"threads_started":1,"wal-append.queue_time_us":586891}
I20260709 12:25:41.639511  1878 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:41.639645  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:41.640379  1727 raft_consensus.cc:606] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.640785  1727 raft_consensus.cc:614] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:41.643461  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:41.527860 10161 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.644377 10058 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:41.644698 10188 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:41.645198 10188 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:41.645913 10161 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:41.646802 10068 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:41.647105 10227 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:41.627118  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3812) took 1217 ms. Trace:
I20260709 12:25:41.647790 10230 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:41.647993  1879 rpcz_store.cc:276] 0709 12:25:40.410114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.006205 (+596091us) service_pool.cc:224] Handling call
0709 12:25:41.006549 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.010965 (+  4416us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:41.011003 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:41.011747 (+   744us) log.cc:844] Serialized 26 byte log entry
0709 12:25:41.340002 (+328255us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:41.340072 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.340194 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:41.626875 (+286681us) raft_consensus.cc:1685] finished
0709 12:25:41.626903 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:41.627047 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":327728,"threads_started":1,"wal-append.queue_time_us":608072}
I20260709 12:25:41.628212  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2917) took 1153 ms. Trace:
I20260709 12:25:41.649520  1805 rpcz_store.cc:276] 0709 12:25:40.474476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.912267 (+437791us) service_pool.cc:224] Handling call
0709 12:25:40.912587 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:40.948233 (+ 35646us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:40.948270 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:41.002277 (+ 54007us) log.cc:844] Serialized 26 byte log entry
0709 12:25:41.355886 (+353609us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:41.355936 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.356012 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:41.627968 (+271956us) raft_consensus.cc:1685] finished
0709 12:25:41.627992 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:41.628147 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35204,"raft.queue_time_us":36494,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":4402,"thread_start_us":353139,"threads_started":1,"wal-append.queue_time_us":623522}
W20260709 12:25:41.651198  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:41.652478  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 138816us
I20260709 12:25:41.653316  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3001) took 1246 ms. Trace:
I20260709 12:25:41.653674  1878 rpcz_store.cc:276] 0709 12:25:40.420591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.739591 (+319000us) service_pool.cc:224] Handling call
0709 12:25:41.653110 (+913519us) spinlock_profiling.cc:243] Waited 899 ms on lock 0x7b5400117454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:41.653228 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2066739456}
I20260709 12:25:41.654102  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:41.646008  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:41.655453 10076 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:41.656287  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:41.656996  1727 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:41.631652 10249 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.659093  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:41.659796  2418 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:41.660720  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.660739  1875 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:41.663012  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
W20260709 12:25:41.576829  9933 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.006s	sys 0.010s
I20260709 12:25:41.629221 10080 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:41.573654 10074 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:41.629961 10254 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.663640  1727 raft_consensus.cc:606] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:41.669160  1727 raft_consensus.cc:614] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:25:41.628835  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 692112us
I20260709 12:25:41.668149  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:41.673666  2412 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:41.674954  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3808) took 1260 ms. Trace:
I20260709 12:25:41.675377  1875 rpcz_store.cc:276] 0709 12:25:40.408695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.618619 (+209924us) service_pool.cc:224] Handling call
0709 12:25:41.669334 (+1050715us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b5400074254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:41.669443 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2028273920}
I20260709 12:25:41.684188 10249 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:41.670451 10270 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:41.661943 10100 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:41.636267 10091 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:41.688246 10091 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:41.689466 10091 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.687227 10245 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:41.673863 10231 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:25:41.691219 10231 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:41.690510 10245 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.666818 10072 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:41.691471 10091 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:41.692859  9394 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:41.693090  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:41.692253  1727 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "908089442ca44109875257bfbc9d06c2"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:41.698271  1727 raft_consensus.cc:606] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:41.698771  1727 raft_consensus.cc:614] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:41.695530  2411 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
W20260709 12:25:41.669510  9933 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.007s	sys 0.010s
I20260709 12:25:41.701107  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:25:41.665306  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 381533us
W20260709 12:25:41.703608  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 312452us
W20260709 12:25:41.711474  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52491us
I20260709 12:25:41.712355  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:41.713080  1730 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:41.717809  2410 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
W20260709 12:25:41.737543  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 57358.6us
I20260709 12:25:41.738942  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:41.741151  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 87163.3us
I20260709 12:25:41.691720 10245 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:41.746287  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:41.691581 10231 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:41.753400  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.751212  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:41.637754  1621 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 11 to 18, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 18 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:41.760340  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:41.761581  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:41.762423  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.711160 10076 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:41.721972 10100 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:41.769688  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.646140 10188 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.721626  9394 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:41.775415  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:41.781819  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:41.770514 10072 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.044s
I20260709 12:25:41.767140 10270 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
I20260709 12:25:41.773406 10066 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:41.791267  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.756531 10080 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:25:41.825834  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 208626us
I20260709 12:25:41.770056 10234 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:41.828567  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:41.782294 10074 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:25:41.832010  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:41.832260  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47290.1us
I20260709 12:25:41.838219  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:41.841614  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 57718.4us
I20260709 12:25:41.789963 10100 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:41.784606 10096 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:41.831192  1726 raft_consensus.cc:606] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:41.845381  1726 raft_consensus.cc:614] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:41.748401 10161 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.062s
I20260709 12:25:41.773618 10108 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:41.842002  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 232575us
I20260709 12:25:41.825872 10068 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:41.853645  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:41.855041 10112 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:41.855939 10112 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:41.856375 10112 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.856783 10112 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:41.895473  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46850us
I20260709 12:25:41.876784  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:41.836874  9992 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:41.904119 10068 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:41.904647 10074 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:41.905364 10074 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.905547 10068 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.907879  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:41.908414  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 136724us
I20260709 12:25:41.920459 10263 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:41.920815 10264 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:41.921942 10264 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:25:41.922669 10278 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:25:41.923346 10266 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:41.924140 10266 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:41.925732  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:41.927057  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:41.928862  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:41.929801  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:41.933175  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:41.841248 10279 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:41.879254 10188 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:41.828320 10265 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.119s
W20260709 12:25:41.947170  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.901700 10076 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:41.985220  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 194009us
I20260709 12:25:41.909600  9980 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:41.848847 10111 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.092s
I20260709 12:25:41.986127  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:41.988875  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:41.959790 10183 raft_consensus.cc:2764] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:25:41.968132  9992 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:42.001857  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86696.4us
W20260709 12:25:42.007193  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 196287us
W20260709 12:25:42.011555  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 164955us
I20260709 12:25:41.997145  9980 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:41.843416 10231 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:41.983770 10266 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:41.988137  9394 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:42.032339  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.010361 10264 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:42.034442  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:42.039561  1726 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:41.984561 10080 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:42.050938  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 121224us
I20260709 12:25:42.051815  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:42.051970 10080 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.053272  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:42.056303  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:42.056936  2411 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:42.056785  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:42.057729  1727 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.019541 10270 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:42.065681  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.028266  9980 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:42.077181  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.078148  1804 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:42.032604 10183 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:42.082098  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.085778  1799 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 23
W20260709 12:25:42.088546  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.089102  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:42.089766  1732 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:42.056182 10096 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:42.047227 10264 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:42.045372 10254 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:42.095173  1798 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:42.026675 10049 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260709 12:25:42.064637 10074 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:42.097797  1805 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:42.098186  1726 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:42.070503 10270 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.111660  2416 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:25:41.909349  1621 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 reported cstate change: term changed from 8 to 20, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 20 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:42.119102  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:42.085007 10183 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:42.019264 10100 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.058879 10062 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.113345 10184 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:42.107008 10280 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:42.096756 10295 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:42.125170 10270 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:42.130030 10183 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:42.133289 10282 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.216s	user 0.000s	sys 0.001s
I20260709 12:25:42.153645  1723 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:42.156229  2411 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:42.151749  1728 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:42.149410  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 96201.9us
I20260709 12:25:42.135564 10184 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.139061 10280 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:42.164297  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:42.155238 10274 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.260s	user 0.002s	sys 0.000s
W20260709 12:25:42.179163  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.135362 10298 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.167452  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:42.191654  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.192482  1805 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:42.180722 10184 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:42.224107  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 238067us
I20260709 12:25:42.227399  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:42.185142 10280 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:42.208997  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:42.209688 10293 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:25:42.210930 10303 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:42.198439 10298 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.212019 10289 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:25:42.213615 10294 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:25:42.214958 10295 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:42.216426 10288 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:42.218741  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:42.219274 10285 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:25:42.205176  1730 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:42.235468  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:42.257982  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 119088us
W20260709 12:25:42.262733  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.197290 10301 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.293897  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:42.223055 10277 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
W20260709 12:25:42.209013 10286 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
W20260709 12:25:42.233398 10272 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.244s	user 0.002s	sys 0.000s
W20260709 12:25:42.287264  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68536.6us
I20260709 12:25:42.143357  9992 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.248739 10302 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:42.291618 10283 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.269s	user 0.000s	sys 0.001s
I20260709 12:25:42.250020 10303 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:42.334122 10310 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:42.248430 10112 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.189826 10264 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.339279 10266 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:42.339252 10066 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:42.329546 10303 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.374325  1800 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:42.377116 10309 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.377764 10309 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.379113 10062 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:42.379418 10297 raft_consensus.cc:2764] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
W20260709 12:25:42.382082 10232 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.769s	user 0.007s	sys 0.003s
I20260709 12:25:42.381763 10295 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:42.366937 10277 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:25:42.294314 10301 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.410174  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1418) took 1401 ms. Trace:
I20260709 12:25:42.410069  1732 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:42.306392  9933 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.062s
I20260709 12:25:42.340211 10263 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.046s
I20260709 12:25:42.295699 10288 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.418851  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:42.410710  2419 rpcz_store.cc:276] 0709 12:25:40.982855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.983547 (+   692us) service_pool.cc:224] Handling call
0709 12:25:41.028842 (+ 45295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.565728 (+536886us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:41.565775 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:41.581371 (+ 15596us) log.cc:844] Serialized 26 byte log entry
0709 12:25:41.789042 (+207671us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:41.789120 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.789302 (+   182us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.383792 (+594490us) raft_consensus.cc:1685] finished
0709 12:25:42.383815 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.384026 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1131,"raft.queue_time_us":38163,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":25728,"thread_start_us":617889,"threads_started":2,"wal-append.queue_time_us":338202}
I20260709 12:25:42.362342 10310 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:25:42.218962  1621 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 16 to 17, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 17 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:42.417726 10111 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:42.442224  1727 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:42.355176 10302 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.430606 10058 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.428753 10066 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:42.436968 10111 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.452116  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1421) took 1186 ms. Trace:
W20260709 12:25:42.455428  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.046s	sys 0.052s
I20260709 12:25:42.454954  1799 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
W20260709 12:25:42.460345  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 225016us
W20260709 12:25:42.465327  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.047s	sys 0.052s
I20260709 12:25:42.465960  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:42.465965  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 558143us
I20260709 12:25:42.444941  9933 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:42.467968  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:42.462127  2417 rpcz_store.cc:276] 0709 12:25:41.266244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.274992 (+  8748us) service_pool.cc:224] Handling call
0709 12:25:41.275418 (+   426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.275848 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:41.275878 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:41.276557 (+   679us) log.cc:844] Serialized 26 byte log entry
0709 12:25:41.697990 (+421433us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:41.698049 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.698134 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.451747 (+753613us) raft_consensus.cc:1685] finished
0709 12:25:42.451780 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.452022 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2673,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":96,"thread_start_us":419855,"threads_started":1,"wal-append.queue_time_us":563725}
W20260709 12:25:42.459636 10230 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.809s	user 0.011s	sys 0.003s
I20260709 12:25:42.469628  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:42.471874  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:42.464329 10111 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.470834 10112 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:42.511245  1621 catalog_manager.cc:5697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 reported cstate change: term changed from 12 to 18. New cstate: current_term: 18 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:42.470340  9933 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:42.500835 10315 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:42.520176  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:42.514925  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49042.9us
I20260709 12:25:42.512779 10066 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:42.524137  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 162521us
I20260709 12:25:42.529353  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1235) took 3408 ms. Trace:
I20260709 12:25:42.464488 10058 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:42.537572 10188 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.002s
I20260709 12:25:42.538642  1807 rpcz_store.cc:276] 0709 12:25:39.106455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.951890 (+1845435us) service_pool.cc:224] Handling call
0709 12:25:40.952424 (+   534us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:41.563524 (+611100us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:41.563560 (+    36us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:41.670960 (+107400us) log.cc:844] Serialized 78 byte log entry
0709 12:25:41.833154 (+162194us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:42.006670 (+173516us) log.cc:844] Serialized 14 byte log entry
0709 12:25:42.013623 (+  6953us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.013717 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:42.500433 (+486716us) raft_consensus.cc:1685] finished
0709 12:25:42.500455 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:42.514538 (+ 14083us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48489,"prepare.queue_time_us":324029,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":3408,"raft.queue_time_us":84804,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":16997,"thread_start_us":726730,"threads_started":3,"wal-append.queue_time_us":293829}
W20260709 12:25:42.529968  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55581.1us
W20260709 12:25:42.553850 10271 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:25:42.556288 10314 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:25:42.556591 10301 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:42.558403  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:42.540979  1805 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:42.490881 10230 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:42.539273 10061 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.966s	user 0.016s	sys 0.001s
I20260709 12:25:42.573691 10302 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:42.578459 10076 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.587s	user 0.008s	sys 0.000s
I20260709 12:25:42.588485 10062 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:42.587121  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.656351  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:42.657141  1799 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 27 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:25:42.588944  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:42.583918 10184 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.596541 10061 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.591609 10288 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:42.579162  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:42.590858  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.579967  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:42.672358  1877 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:42.544155 10112 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:42.587989  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:42.584462  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:42.583734  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:42.581444  9980 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:42.587414 10108 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.710s	user 0.011s	sys 0.002s
I20260709 12:25:42.582968 10298 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:42.626366  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53457.7us
I20260709 12:25:42.584805 10297 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.633329  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.546263 10312 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:42.690203 10108 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:42.713027 10280 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.734952  1806 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:42.734287  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:42.719012 10058 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:42.661027 10183 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:42.528332 10318 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.092s
I20260709 12:25:42.663182  1723 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:42.581450 10096 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:42.763243  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.677599  1879 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:42.678534  1881 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:42.678107  1874 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:42.649039 10076 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:42.652076 10062 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.684563  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:42.685972  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:42.687602  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 218046us
I20260709 12:25:42.548473 10315 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.663686 10230 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:42.676172 10091 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.974s	user 0.012s	sys 0.003s
I20260709 12:25:42.690359  1803 raft_consensus.cc:2410] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 27 because replica has last-logged OpId of term: 26 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
W20260709 12:25:42.742923  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:42.783725  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:42.654219  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 95933.8us
W20260709 12:25:42.746604  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 306112us
I20260709 12:25:42.787317  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:42.749619  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:42.750003  2410 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:25:42.751511  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:42.751925 10265 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:42.753026  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:42.792073  2418 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:42.754503  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:42.760254 10307 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.385s	user 0.004s	sys 0.000s
I20260709 12:25:42.663482  9933 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:42.803409  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68379.3us
I20260709 12:25:42.780210  2417 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:25:42.762004 10270 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.586s	user 0.014s	sys 0.000s
I20260709 12:25:42.782817 10230 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:42.789290  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:42.793649  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:42.785449 10159 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:42.821035  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:42.791397 10265 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:42.827255 10076 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:42.827798 10279 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:42.828251  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:42.825222 10062 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:42.830101  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:42.795143 10317 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.052s
I20260709 12:25:42.796165 10091 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.843282 10091 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.844794 10280 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:42.846184 10279 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:42.784291 10184 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:42.847309 10245 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.078s	user 0.007s	sys 0.008s
I20260709 12:25:42.847898 10245 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.856637  2414 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:25:42.799862 10329 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:25:42.800199  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.802031 10315 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:42.774600  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:42.719020 10297 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.812028  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:42.819976  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:42.821636  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:42.877175  1881 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
W20260709 12:25:42.877260  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 65321.7us
I20260709 12:25:42.823558  1875 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:25:42.819314 10230 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:42.824476 10335 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.880293  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.881004  1799 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:42.825330 10188 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:42.826223 10336 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:42.825239  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:42.827211  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:42.840991 10331 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:42.842409 10334 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:42.846889 10279 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:42.789913 10111 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.871717  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:42.886204  1879 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:42.871922  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 67436.7us
W20260709 12:25:42.870651  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 181677us
I20260709 12:25:42.878352  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:42.876581  1874 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:42.820879 10159 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:42.879334  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:42.883716  1801 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:42.890547  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 97455.1us
I20260709 12:25:42.892663  1881 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:42.815738 10270 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.898281  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:42.881961 10318 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:42.903249  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:42.906512  1874 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:42.907326  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:42.907979  1881 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:42.890355 10245 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:42.938071  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:42.896302 10184 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:42.895514 10334 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:42.897791 10336 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:42.953292  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:42.953810  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:42.956143 10338 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:42.959030 10325 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:42.959725 10336 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.963197 10159 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.964593 10159 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:42.965858 10279 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:42.967123 10062 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:42.968024 10062 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:42.968634  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:42.977463  1799 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:42.979404  1725 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:42.980013  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:42.980890  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:42.981568  1726 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 19: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:42.983800  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:42.969033 10062 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.016s
I20260709 12:25:42.985630  1877 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:42.987483  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:42.988293  1878 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:42.991739  1725 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:42.945209 10188 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:42.992473  1725 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:42.888989 10335 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:42.994773  1801 raft_consensus.cc:2410] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 26 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:25:42.904083 10112 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:25:42.943907 10280 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:43.058410  1879 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:43.060981  1881 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:43.062887  1802 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:25:42.994295  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:43.068015  1732 raft_consensus.cc:2368] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26.
I20260709 12:25:43.067365  1801 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:43.070102  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 101644us
I20260709 12:25:42.990473 10184 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:43.074627  1728 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:42.970013  9992 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:43.075970  1729 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:43.076757  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:43.101603  1724 raft_consensus.cc:2410] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 21 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:25:43.080904  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:43.103063  1731 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:42.995587 10318 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:42.969784 10317 consensus_queue.cc:1175] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 23 status { last_received { term: 23 index: 12 } last_committed_idx: 12 last_received_current_leader { term: 23 index: 12 } }
I20260709 12:25:43.079181  1725 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.112098  1725 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:43.082333 10234 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.164254  1878 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:43.065563 10183 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:43.102095 10188 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:43.107667 10288 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.107868 10318 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:43.160398 10341 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:25:43.108191  9992 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:43.105048 10303 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.686s	user 0.006s	sys 0.007s
I20260709 12:25:43.125655 10310 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:43.177898  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 106687us
I20260709 12:25:43.102595 10280 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.191449 10325 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:43.198495  1725 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
W20260709 12:25:43.206622  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.207857  1725 pending_rounds.cc:85] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 14
I20260709 12:25:43.208217  1725 pending_rounds.cc:107] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:25:43.179126 10227 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.208498  1725 raft_consensus.cc:2889] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:43.195263 10318 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:43.178771 10234 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.183758 10270 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:43.215615  1696 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Remote error: Timed out: Call waited in the queue past client deadline
I20260709 12:25:43.223044 10343 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:43.223431 10330 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.224073  2412 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:25:43.224455 10299 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:43.224926 10299 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:43.225132 10280 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:43.225867 10299 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.214334 10111 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:43.220394 10349 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.231128  9992 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:43.231809 10345 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:25:43.175411 10334 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:43.270036 10344 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:25:43.209442 10310 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.216279 10321 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:43.282951  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 79236.7us
I20260709 12:25:43.273293 10335 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:43.233736 10297 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:43.224134 10330 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.210544 10184 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.351150 10338 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:43.284072 10349 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.309724 10188 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.303282 10325 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.205331 10303 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:43.325101  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.441404 10338 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:43.442095 10338 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.442075 10265 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.443511 10265 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:43.444772 10338 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:43.445365 10338 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.442556 10349 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:43.445791 10338 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.448115 10338 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:43.449577  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.450555 10303 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:43.455523 10234 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:43.322407  2412 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 12:25:43.328977 10316 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.952s	user 0.003s	sys 0.000s
I20260709 12:25:43.195745 10288 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.223110 10227 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.326774 10040 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:43.377140 10321 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.385970 10330 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:43.392645 10231 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:43.404235 10299 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:43.406924 10100 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:43.417335 10281 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.048s	user 0.009s	sys 0.000s
I20260709 12:25:43.422817 10318 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.430989  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:43.493770  1732 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:25:43.496109  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3100) took 1329 ms. Trace:
I20260709 12:25:43.496173  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:43.470161  2411 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:43.470512  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:43.473624  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:43.474949  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3102) took 1092 ms. Trace:
I20260709 12:25:43.500186  1725 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:43.501796  1724 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:43.495601 10061 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:43.477180 10040 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:43.461438 10323 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.727s	user 0.000s	sys 0.007s
I20260709 12:25:43.480499  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:43.517551  1728 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:43.482517  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:43.469116  1770 leader_election.cc:400] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26.
I20260709 12:25:43.496775  1723 rpcz_store.cc:276] 0709 12:25:42.166475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.166632 (+   157us) service_pool.cc:224] Handling call
0709 12:25:42.166978 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.746811 (+579833us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.495851 (+749040us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b54000d16d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:43.496028 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1140627840}
I20260709 12:25:43.499919  1730 rpcz_store.cc:276] 0709 12:25:42.366255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.366458 (+   203us) service_pool.cc:224] Handling call
0709 12:25:42.383499 (+ 17041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.383721 (+   222us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:42.620193 (+236472us) log.cc:844] Serialized 14 byte log entry
0709 12:25:43.405993 (+785800us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b1800415f00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287
0709 12:25:43.406201 (+   208us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:43.406287 (+    86us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:43.406416 (+   129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.458435 (+ 52019us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.458606 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41148,"prepare.queue_time_us":78207,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":2347,"raft.queue_time_us":39342,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":4887296,"thread_start_us":413375,"threads_started":3,"wal-append.queue_time_us":274686}
I20260709 12:25:43.518800  1770 leader_election.cc:403] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:25:43.519024  1731 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:43.475205  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 505413us
I20260709 12:25:43.499091  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:43.481540 10303 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:43.526569  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68418.1us
I20260709 12:25:43.509801 10288 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:43.514257 10061 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:43.522356  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:43.509716 10230 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.619s	user 0.008s	sys 0.006s
W20260709 12:25:43.527343  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 96539.2us
I20260709 12:25:43.527997  1732 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:43.521911  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:43.529414  1799 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:43.477547 10072 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:25:43.538697 10326 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:43.538864 10040 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.540373 10040 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:43.541740 10318 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:43.507791 10219 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:43.533643 10230 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.546042  1724 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:43.546160  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2947) took 1207 ms. Trace:
I20260709 12:25:43.549301  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:43.533419 10232 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26.
W20260709 12:25:43.547114 10068 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.171s	user 0.003s	sys 0.013s
I20260709 12:25:43.516692 10108 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.562453  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:43.526357 10304 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.548655  1804 rpcz_store.cc:276] 0709 12:25:42.338398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.338577 (+   179us) service_pool.cc:224] Handling call
0709 12:25:42.338960 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:42.425957 (+ 86997us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:42.425993 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:42.437201 (+ 11208us) log.cc:844] Serialized 26 byte log entry
0709 12:25:42.493112 (+ 55911us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:42.493270 (+   158us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:42.588220 (+ 94950us) spinlock_profiling.cc:243] Waited 71.9 ms on lock 0x7b54001eb4d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:42.588277 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:43.421858 (+833581us) spinlock_profiling.cc:243] Waited 6.16 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:43.545890 (+124032us) raft_consensus.cc:1685] finished
0709 12:25:43.545915 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.546088 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3441,"raft.queue_time_us":9106,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":4208,"spinlock_wait_cycles":179605248,"thread_start_us":28875,"threads_started":1,"wal-append.queue_time_us":182429}
I20260709 12:25:43.597393  1724 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:25:43.573069  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:43.559993 10363 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:25:43.563577 10068 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.193s	user 0.003s	sys 0.014s
I20260709 12:25:43.596452  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:43.603089  1728 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:43.573685  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:43.599359  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 77603.6us
I20260709 12:25:43.563012 10230 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:43.531786 10361 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:43.594766 10219 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:43.595679 10232 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:43.560482 10091 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:43.607301  1723 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:43.536353 10303 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.563792 10325 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:25:43.620414  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1419) took 2637 ms. Trace:
I20260709 12:25:43.572599 10108 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.613889 10232 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.612768 10363 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:43.622160  2415 rpcz_store.cc:276] 0709 12:25:40.983144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:41.010879 (+ 27735us) service_pool.cc:224] Handling call
0709 12:25:41.011259 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:41.573223 (+561964us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:41.573258 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:41.600332 (+ 27074us) log.cc:844] Serialized 26 byte log entry
0709 12:25:41.767769 (+167437us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:41.768158 (+   389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:41.768393 (+   235us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:43.416809 (+1648416us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b2400305318. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d2a09d 00007f16ab95c10e 00007f16ab97388c 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221
0709 12:25:43.620108 (+203299us) raft_consensus.cc:1685] finished
0709 12:25:43.620128 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.620354 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8136,"raft.queue_time_us":36950,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":9229,"spinlock_wait_cycles":83618944,"thread_start_us":652161,"threads_started":2,"wal-append.queue_time_us":662469}
I20260709 12:25:43.630137  1723 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:43.632498  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:43.633597 10184 raft_consensus.cc:2764] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election decision vote started in defunct term 25: won
I20260709 12:25:43.642625  1728 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:43.642968  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:43.645479  1730 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:43.646425 10299 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:43.648365  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:43.649719  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:43.643206  1804 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ae23688e76624153b5a7089586669bb0"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:25:43.650996  1804 raft_consensus.cc:606] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:43.653327 10184 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:25:43.653954 10184 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:43.654927 10184 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.657823 10299 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:43.658567 10299 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 26 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:43.658934  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 120635us
I20260709 12:25:43.659993  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:43.659716 10299 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.661489  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:43.664994  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.664721  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:43.621969 10227 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:43.668450  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:43.611613 10361 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:43.674415  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:43.664165 10232 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:43.678225 10108 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:43.678709 10309 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:43.678936 10310 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:43.679366 10249 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:43.673022 10246 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:43.682889  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:43.686208  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.692325 10066 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:43.692762 10326 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:25:43.693743   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:25:43.624953 10230 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:43.684900 10219 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:43.682554 10361 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:43.714365  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:43.595351 10304 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.717363  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:43.740469  1728 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:43.720450  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:43.740337  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:43.741104  1801 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:43.644289 10303 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:43.686412 10227 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:43.725108  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:43.713155  1730 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:43.715570 10249 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:43.742403  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59593.3us
I20260709 12:25:43.751111  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:43.731107   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:43.721637 10246 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:43.726963 10326 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:43.751981  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:43.727799 10219 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:43.755865  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1237) took 1211 ms. Trace:
I20260709 12:25:43.759286  1798 rpcz_store.cc:276] 0709 12:25:42.544621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.557671 (+ 13050us) service_pool.cc:224] Handling call
0709 12:25:42.580935 (+ 23264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:43.425115 (+844180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.425154 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:43.425201 (+    47us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:43.425242 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.755651 (+330409us) spinlock_profiling.cc:243] Waited 844 ms on lock 0x7b5400133654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:43.755695 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:43.755811 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1940757760}
I20260709 12:25:43.719976 10091 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:25:43.743096 10339 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:43.746677 10061 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:43.759778  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:43.776475  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:43.757472 10246 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:43.754905 10249 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:43.766189  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51957.1us
I20260709 12:25:43.789578  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:43.760425 10326 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:43.782523  1730 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
I20260709 12:25:43.783129  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:43.796757  1730 pending_rounds.cc:85] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 18
I20260709 12:25:43.782720  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:43.780117  1798 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:43.789909  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:43.747004 10230 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.767122 10219 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:43.797075  1730 pending_rounds.cc:107] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 27.19
W20260709 12:25:43.776366 10263 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.065s	user 0.008s	sys 0.025s
I20260709 12:25:43.798043  1728 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:43.758970 10254 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:25:43.799958  1732 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:43.801234  1730 raft_consensus.cc:2889] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:43.804536 10263 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.093s	user 0.008s	sys 0.025s
W20260709 12:25:43.819315  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.819936   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:43.764567 10245 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:25:43.749536 10363 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.826434 10263 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:43.827144 10263 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:43.828612  1732 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:43.828253 10263 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.829563 10263 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:43.831568  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:43.832612 10368 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:25:43.783723 10061 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:43.797979 10339 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:43.763926 10300 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.047s	user 0.008s	sys 0.013s
I20260709 12:25:43.798444  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:43.808122  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:43.833626  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.833670  1725 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:43.838491 10347 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:43.802942  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:43.808054 10335 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:25:43.840374 10367 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.188s	user 0.002s	sys 0.000s
I20260709 12:25:43.840246  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:43.868573  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:43.869504 10066 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:43.802760 10184 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.039s
I20260709 12:25:43.873162  1725 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:25:43.855334  1801 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:25:43.854537  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 56205.4us
W20260709 12:25:43.862658  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59857.2us
I20260709 12:25:43.865520  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:43.866895  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2941) took 1796 ms (client timeout 2209 ms). Trace:
I20260709 12:25:43.875425  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:43.875046  1800 rpcz_store.cc:269] 0709 12:25:42.065282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.065432 (+   150us) service_pool.cc:224] Handling call
0709 12:25:43.860922 (+1795490us) spinlock_profiling.cc:243] Waited 308 ms on lock 0x7b540026fa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:43.861162 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":707852672}
I20260709 12:25:43.879381  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:43.859560 10300 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.143s	user 0.009s	sys 0.013s
I20260709 12:25:43.883241  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:43.885425  1730 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:25:43.882961  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:43.886898  1878 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:25:43.888604  1873 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:43.891295  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:43.892601  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:43.894192  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:43.894057  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:43.895876  1874 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:43.872870 10339 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:43.872596 10303 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:43.899231  1879 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: [22.15-22.15]
I20260709 12:25:43.865007 10378 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:43.899874  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:43.877575 10227 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:43.901446  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:43.901096  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:43.904757  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:43.881631 10335 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:43.911634  1801 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:43.887079 10347 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:43.893486 10245 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:43.921526  1877 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:43.923339  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:43.966676  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2946) took 1745 ms. Trace:
I20260709 12:25:43.967311  1806 rpcz_store.cc:276] 0709 12:25:42.221506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.221637 (+   131us) service_pool.cc:224] Handling call
0709 12:25:42.247593 (+ 25956us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:42.737206 (+489613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:43.966337 (+1229131us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:43.966596 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1084765056}
I20260709 12:25:43.905714 10227 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:43.883560 10219 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.971841  1726 raft_consensus.cc:2410] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 24 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:43.972757 10227 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:43.973138 10219 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:43.933625  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.939205 10304 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:43.976740  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:43.940394 10230 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:43.941308 10335 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:43.942140  1873 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:43.954180  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:43.955583 10339 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:43.956902  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:43.956755 10061 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:43.977711  1801 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:43.991747  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:43.915136 10378 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:44.001428  1805 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:44.006466  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.008155  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.013303  1806 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:25:44.016278  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:44.017541  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.019093  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.020566  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.021176  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.022239  1872 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 28 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:25:44.024375  1806 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:44.024533  9980 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.020625  1873 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:44.028507  1877 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:43.996958 10245 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:43.972958 10232 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.040951  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:43.998574 10349 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:44.038749  1800 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:44.045156  1801 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:25:44.051023  1800 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:44.046237  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.009517 10303 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:25:44.019071 10297 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:44.054415  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.105767  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.059580  9933 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:44.062422  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.052729 10340 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:44.063860 10245 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.069375 10326 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.051915  1800 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.066689 10361 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.073580  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.082657  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:44.083210  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.118503  1872 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 26: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:44.084754 10322 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:44.086448 10066 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:44.086946  1724 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:25:44.087322  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:44.087193 10310 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:44.087713 10076 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:25:44.066759 10232 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.088704  1725 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260709 12:25:44.090056 10040 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:44.090734  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:44.108536 10072 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.051999  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:44.117626  1800 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:44.139497  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 87709us
I20260709 12:25:44.122278  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.141536  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.143095  9394 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:44.128542  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.130667  1725 pending_rounds.cc:85] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 14
I20260709 12:25:44.143574  9394 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:44.143850  1725 pending_rounds.cc:107] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260709 12:25:44.144568  1725 raft_consensus.cc:2889] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:44.144845 10184 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:44.145471 10184 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:44.092121  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:44.146360  1804 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:44.146872 10299 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:44.147511  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:44.148119  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 61410.2us
I20260709 12:25:44.149111  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.151201 10306 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:44.151741 10306 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.131239  1729 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 24 index: 20. (index mismatch)
I20260709 12:25:44.131212 10355 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.146530 10184 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.153286 10355 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.156354 10355 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:44.143978  1802 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:44.157955  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.159709  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:44.160326  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:44.152130 10306 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:44.161940  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.163035 10091 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:44.163793 10091 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:44.164976 10091 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.093019 10246 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.168624  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.158730  1802 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:44.172444  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:44.173404  1802 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:44.150492  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:44.175066  1804 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:44.056738 10349 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.207901  1800 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 19 index: 15. (term mismatch)
I20260709 12:25:44.208652  1800 pending_rounds.cc:85] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 14
I20260709 12:25:44.209125  1800 pending_rounds.cc:107] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 18.15
I20260709 12:25:44.209515  1800 raft_consensus.cc:2889] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:44.209532  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:44.211901  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 320666us
I20260709 12:25:44.195490  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:44.212832  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:44.214582  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.216189  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.215350  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.131275 10340 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:44.186602 10301 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:44.221585  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.228657  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:44.053514 10347 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.138135 10227 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.196146 10112 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:44.242844  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.248519  1800 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:25:44.249722  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:44.251291  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.254026  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.254978  1801 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:44.256525  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:44.227708 10301 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:44.187717 10072 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.130931 10378 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.234309 10263 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.270692 10100 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:44.271298 10100 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:44.271716  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:44.273176  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.279482 10264 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:44.231878 10322 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:25:44.247350 10246 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.243903 10219 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:44.262712 10299 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:44.302455  1880 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: []
I20260709 12:25:44.264408 10300 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:44.265389 10269 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.283689 10303 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:44.316939  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:44.247187 10112 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.301211 10383 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:44.307089 10254 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:44.295313 10263 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.325251 10309 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:44.335688 10249 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.535s	user 0.012s	sys 0.002s
I20260709 12:25:44.253773  9394 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.307408 10306 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.310055 10219 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.372637 10263 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:44.375473  9394 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:44.347460 10326 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.349726 10254 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.309516 10340 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.320735 10269 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:44.392689  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.349256 10383 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:44.351493 10386 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.383540 10361 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.363961 10249 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:44.420759  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 164340us
I20260709 12:25:44.421720  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.423429  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.424938  1874 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:44.401392 10254 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.427436 10299 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:44.429294 10306 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:44.433106  1729 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:25:44.442943  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.408109 10386 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.453940  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:44.432461 10227 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:44.443725 10390 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
I20260709 12:25:44.459034 10061 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:25:44.462332 10346 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.256s	user 0.005s	sys 0.000s
I20260709 12:25:44.488862 10249 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.490206 10219 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.491477 10040 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.492327 10040 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.493386 10304 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:44.493976 10304 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.494388 10321 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:44.495613  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.496150 10304 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:44.496834  1805 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 1247) took 1825 ms (client timeout 1598 ms). Trace:
W20260709 12:25:44.497356  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.500870  1881 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:25:44.456166 10335 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:44.463341 10383 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.464838  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.509205  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:44.510535  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45832.4us
I20260709 12:25:44.465288 10301 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:44.469333  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.475414 10386 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.479152  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.481118  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1248) took 1806 ms. Trace:
I20260709 12:25:44.518211  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.518563  1799 rpcz_store.cc:276] 0709 12:25:42.674947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.101351 (+426404us) service_pool.cc:224] Handling call
0709 12:25:43.101751 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:43.529938 (+428187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.480774 (+950836us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b540026f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:44.481031 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":983024384}
I20260709 12:25:44.520375  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.440894 10299 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:44.479279  1800 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:44.532306  1874 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:44.488380  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:44.534523  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:44.497290  1805 rpcz_store.cc:269] 0709 12:25:42.671176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:42.762929 (+ 91753us) service_pool.cc:224] Handling call
0709 12:25:44.496590 (+1733661us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540009fbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:44.496734 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2718014848}
I20260709 12:25:44.534689  1798 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:44.536963  1805 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:44.507464  1881 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:25:44.511305  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.512580  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.529110  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:44.530038  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:44.539482  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.539826  1874 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:44.530984  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.540781  1724 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
W20260709 12:25:44.533047  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 259933us
I20260709 12:25:44.532232  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:44.505566  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.541576  1806 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:44.540659  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:44.482846 10072 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:44.549157  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.550797  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.545581  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.551733  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.493057 10349 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.548498  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.507791 10068 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:44.565904  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.552628  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.566367  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:44.556357  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.556754  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.568286  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.568934  1724 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:25:44.557389  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:44.569521  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:44.551757  1799 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:44.563128  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.507850 10335 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:44.564308  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.508074 10331 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.570482  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.563890  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.573340  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.576642  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.580062  2418 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:25:44.583752  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.584993  1874 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:44.586826  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:44.587639  2418 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:44.530244 10061 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:44.593851  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.543186 10401 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:44.609277  2410 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:25:44.600168  2419 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:44.606204  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.614331  1728 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:44.616346  1874 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:44.601262  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:44.613332  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:44.615818  1876 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:44.617745  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.593700 10340 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:44.544608 10300 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:44.618641  1876 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 27. Ops: []
I20260709 12:25:44.619290  1874 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:44.621719  1802 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 18.
I20260709 12:25:44.618155  2418 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:44.624006  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:44.599555  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:44.627382  1876 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:44.627890  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:44.629078  1876 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:25:44.631438  1880 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:25:44.626056  2412 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:44.565518 10386 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.509831 10100 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.598412 10301 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:44.595152 10335 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:44.661321  1881 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
W20260709 12:25:44.669843  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47098.9us
I20260709 12:25:44.671823  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:44.640246  9933 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.521s	user 0.011s	sys 0.003s
I20260709 12:25:44.673440  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:44.684391  2410 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:44.613946 10324 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:44.598505 10040 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:44.688014  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:44.697607  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:44.678472  2418 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:44.678807  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49962.1us
W20260709 12:25:44.682096  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 82632.4us
I20260709 12:25:44.626454 10363 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.763078  1876 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:44.594604 10068 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.763101  1881 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:25:44.765543  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:44.648104 10269 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:44.770452  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 93729.9us
I20260709 12:25:44.751720  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:44.624650 10299 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:25:44.792201  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.793190  1876 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:44.788843  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 117109us
I20260709 12:25:44.766269  1881 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:44.687314 10184 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:44.614944 10331 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.809686  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.804329  1805 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:44.685284 10254 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.753129 10249 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.780546 10309 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.759608 10232 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:44.835137  2412 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 24 index: 20. (index mismatch)
I20260709 12:25:44.846050  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.674391 10386 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.683137 10304 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 27.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.133s
I20260709 12:25:44.803272 10040 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:44.865504  9394 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.002s
I20260709 12:25:44.880944 10076 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:44.894687  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:44.898458  9394 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:44.899040  9394 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.899421  9394 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:44.865415  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:44.876317  1800 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:44.784665 10363 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.828224 10184 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.802973 10049 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:44.820619  2418 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:44.890601  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:44.904862  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:44.796806 10264 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.915756  1881 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:44.916016 10264 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.789455 10324 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:44.922281  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 121690us
W20260709 12:25:44.928658  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 127892us
I20260709 12:25:44.930146  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:44.940189  9394 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:44.904053 10040 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.920574 10184 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:45.243389  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 373350us
I20260709 12:25:44.857132 10249 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:45.062335  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 132263us
I20260709 12:25:44.871281 10309 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:45.248996 10415 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.376s	user 0.000s	sys 0.009s
I20260709 12:25:44.906010 10232 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:44.935066 10264 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:44.972451 10061 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.120s
I20260709 12:25:44.942871 10403 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.161s
I20260709 12:25:44.834730 10072 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
I20260709 12:25:44.904696 10321 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.126s
I20260709 12:25:44.802644 10404 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.116s
W20260709 12:25:45.127281  2411 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.000s	sys 0.012s
W20260709 12:25:45.286033  2411 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.660s	user 0.002s	sys 0.014s
I20260709 12:25:45.231561  2410 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:44.930464  2417 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:45.235402  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:45.236291  1798 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.686s	user 0.012s	sys 0.005s
I20260709 12:25:45.236706  9394 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:45.291558  1798 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
W20260709 12:25:45.293136  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 364756us
W20260709 12:25:45.272401 10335 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.502s	user 0.011s	sys 0.003s
W20260709 12:25:45.238675  1806 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.697s	user 0.008s	sys 0.011s
I20260709 12:25:45.300678  1806 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:25:45.302083  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:45.239850  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:45.229728 10397 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.461s	user 0.001s	sys 0.008s
W20260709 12:25:45.250104 10406 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.467s	user 0.009s	sys 0.000s
I20260709 12:25:45.310017  1806 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:25:45.310353  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:45.248890 10313 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:45.254144 10309 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:45.070895 10396 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.456s	user 0.002s	sys 0.000s
I20260709 12:25:45.256927  1876 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:45.340737  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.340978  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:45.258214  1879 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:45.265060  2418 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:45.284101 10264 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:45.284747 10419 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.011s
I20260709 12:25:45.288415  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:45.296870  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:45.238314  1881 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:45.301576  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:45.311924  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83037.6us
I20260709 12:25:45.320022  1806 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:25:45.282176 10420 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:45.342497  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:45.346031  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 426237us
I20260709 12:25:45.287145 10232 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:45.370604 10420 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:45.372412  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:45.375128  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2974) took 1119 ms. Trace:
I20260709 12:25:45.375329  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3033) took 1303 ms. Trace:
W20260709 12:25:45.376924  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80141.7us
I20260709 12:25:45.377022  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.375602  1807 rpcz_store.cc:276] 0709 12:25:44.255559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.255736 (+   177us) service_pool.cc:224] Handling call
0709 12:25:44.266936 (+ 11200us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:44.267481 (+   545us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:44.302957 (+ 35476us) log.cc:844] Serialized 14 byte log entry
0709 12:25:44.510419 (+207462us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:44.550373 (+ 39954us) log.cc:844] Serialized 52 byte log entry
0709 12:25:44.550528 (+   155us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:44.550561 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.550635 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:45.374862 (+824227us) raft_consensus.cc:1685] finished
0709 12:25:45.374890 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.375063 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":31196,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":204,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":55401,"thread_start_us":173453,"threads_started":1,"wal-append.queue_time_us":591650}
I20260709 12:25:45.377709  2418 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:45.295754 10304 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:45.378538  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.378585 10304 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:45.380014 10245 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:45.380502 10245 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:45.380815  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.380924 10245 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:45.382476 10100 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:45.369766 10356 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:45.370184  1805 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:45.383453 10100 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:45.383769  1805 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:45.383941 10100 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:45.384502 10100 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:45.385051  1805 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:45.312491 10335 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:45.316635 10313 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:45.341514  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.388188  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:45.388589  2410 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:25:45.375838 10426 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:45.377322  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.381971 10304 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:45.389854  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:45.390177 10264 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:45.316295 10091 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 12:25:45.368840  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:45.374410  1844 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:45.387952 10335 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:45.409300  2419 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:45.311324 10299 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.111s
I20260709 12:25:45.411928  1804 pending_rounds.cc:85] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 16
I20260709 12:25:45.386613  1805 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:45.351119  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 70076.5us
I20260709 12:25:45.412492  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:45.412912  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:45.393358  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3923) took 1754 ms. Trace:
I20260709 12:25:45.441097  1806 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:45.390661  1881 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
W20260709 12:25:45.392119  1771 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.654s (SENT) [suppressed 25 similar messages]
I20260709 12:25:45.427734  1875 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:45.398558  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:45.410544  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:45.445423  1875 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 15. (term mismatch)
W20260709 12:25:45.438925  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64609us
W20260709 12:25:45.444387  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52324.4us
I20260709 12:25:45.445312  1807 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:45.396605  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:45.449657  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:45.441569  1723 rpcz_store.cc:276] 0709 12:25:43.639240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:43.670416 (+ 31176us) service_pool.cc:224] Handling call
0709 12:25:43.670801 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:43.685328 (+ 14527us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:43.685359 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:43.685785 (+   426us) log.cc:844] Serialized 26 byte log entry
0709 12:25:44.503345 (+817560us) spinlock_profiling.cc:243] Waited 9.7 ms on lock 0x7b1800134260. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:25:44.765163 (+261818us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:44.765237 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.765332 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:45.392983 (+627651us) raft_consensus.cc:1685] finished
0709 12:25:45.393010 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.393283 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12306,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"spinlock_wait_cycles":22316800,"thread_start_us":261539,"threads_started":1,"wal-append.queue_time_us":1659665}
I20260709 12:25:45.450139  1876 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:45.446079  1875 pending_rounds.cc:85] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 14
I20260709 12:25:45.451524  1875 pending_rounds.cc:107] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 26.15
I20260709 12:25:45.451861  1875 raft_consensus.cc:2889] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:45.452915  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:45.453166  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:45.454689  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:45.375692  1873 rpcz_store.cc:276] 0709 12:25:44.072031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.105804 (+ 33773us) service_pool.cc:224] Handling call
0709 12:25:44.106397 (+   593us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:44.110881 (+  4484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:44.110929 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:44.123682 (+ 12753us) log.cc:844] Serialized 26 byte log entry
0709 12:25:44.381779 (+258097us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:44.651364 (+269585us) log.cc:844] Serialized 14 byte log entry
0709 12:25:44.670477 (+ 19113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.670555 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:45.375064 (+704509us) raft_consensus.cc:1685] finished
0709 12:25:45.375087 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.375231 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29579,"prepare.queue_time_us":63975,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":53,"raft.queue_time_us":26905,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":12898,"spinlock_wait_cycles":53257600,"thread_start_us":446794,"threads_started":2,"wal-append.queue_time_us":383789}
I20260709 12:25:45.377799  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:45.439222  1804 pending_rounds.cc:107] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:25:45.343530 10386 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:45.370512 10408 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:25:45.460345  1804 raft_consensus.cc:2889] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:45.461114  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 83401.4us
W20260709 12:25:45.450537  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54091.6us
I20260709 12:25:45.300361 10061 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:45.464332  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.464432  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:45.410938 10393 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.958s	user 0.003s	sys 0.012s
I20260709 12:25:45.460310  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.507536  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.465013  2410 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:45.509140  2418 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:45.390589 10232 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:45.416244 10301 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.726s	user 0.009s	sys 0.003s
I20260709 12:25:45.360389 10400 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:45.465860  1876 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 12:25:45.513171  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 244887us
I20260709 12:25:45.474165  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:45.515775  1875 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:25:45.543331  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:45.480947  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:45.515650  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:45.510303  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:45.545163  1881 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:45.419258 10112 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:45.546437  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 65602.7us
I20260709 12:25:45.546999  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:45.513082  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:45.459169 10304 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:45.554607  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:45.509658 10061 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:45.523367  1805 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:25:45.421784 10419 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:45.346171 10254 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.174s
I20260709 12:25:45.448859 10420 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:45.531509 10360 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:45.534734  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:45.585374  1873 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
W20260709 12:25:45.558441  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45462.2us
I20260709 12:25:45.587432  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:45.466187 10339 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:45.468235  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:45.590983  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 122953us
I20260709 12:25:45.591193  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:45.391258 10264 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:45.623281  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.539992 10072 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:45.556653 10301 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:45.550706 10340 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:45.640349  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.557400 10361 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:45.556905 10383 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:45.600204 10304 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:45.589073  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 343722us
I20260709 12:25:45.578534 10404 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:25:45.650796  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58082.3us
I20260709 12:25:45.519155 10356 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:45.556908 10068 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:45.578397 10112 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.077s
I20260709 12:25:45.465623 10049 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.118s
I20260709 12:25:45.669198  1880 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:45.551744 10315 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:45.551237 10400 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:45.657325 10360 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:25:45.623090 10420 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.705742  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:45.595307 10419 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:45.684116 10040 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:45.697908 10412 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.152s	user 0.001s	sys 0.001s
I20260709 12:25:45.550217 10232 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:45.475978  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:45.671370 10393 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:25:45.646282 10361 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:45.656412 10324 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:45.685972 10339 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:45.677618  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:45.675014 10072 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:45.704685 10418 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.393s	user 0.001s	sys 0.001s
I20260709 12:25:45.695030 10404 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:45.739851  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 263982us
I20260709 12:25:45.657953 10340 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:45.691802 10386 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:45.760409  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82906.4us
I20260709 12:25:45.801026  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:45.644568 10061 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:45.724572 10428 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:45.743865 10414 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
W20260709 12:25:45.799083  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 325037us
I20260709 12:25:45.765401 10400 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:45.866698  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 60740.6us
I20260709 12:25:45.733626 10361 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:45.724885 10421 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:45.421216 10321 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.235s
I20260709 12:25:45.642515 10313 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:45.868399  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 54169.2us
I20260709 12:25:45.874557  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3032) took 1802 ms. Trace:
I20260709 12:25:45.760124 10040 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:45.875133  1878 rpcz_store.cc:276] 0709 12:25:44.071682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.071848 (+   166us) service_pool.cc:224] Handling call
0709 12:25:44.072234 (+   386us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:44.072618 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:44.072647 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:44.073138 (+   491us) log.cc:844] Serialized 26 byte log entry
0709 12:25:44.307106 (+233968us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:44.323435 (+ 16329us) log.cc:844] Serialized 14 byte log entry
0709 12:25:44.323663 (+   228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.323741 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:44.375959 (+ 52218us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b1800352840. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:45.410232 (+1034273us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:45.874277 (+464045us) raft_consensus.cc:1685] finished
0709 12:25:45.874301 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.874468 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15985,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":59,"raft.queue_time_us":3121,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":6898,"spinlock_wait_cycles":10146304,"thread_start_us":233552,"threads_started":1,"wal-append.queue_time_us":343784}
I20260709 12:25:45.718600 10420 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:45.908039  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:45.813474 10428 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:45.810297 10068 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:45.928314  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 365671us
I20260709 12:25:45.899366 10061 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:45.900271 10400 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.911959 10040 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:45.812692 10386 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:45.912878 10324 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:45.913091 10299 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.874013 10112 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.057s
W20260709 12:25:46.044976  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 369431us
W20260709 12:25:45.998680  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.009s	sys 0.020s
W20260709 12:25:46.052790  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 123626us
W20260709 12:25:46.064793  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.009s	sys 0.021s
W20260709 12:25:46.065418  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 763860us
W20260709 12:25:45.948846 10433 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.224s	user 0.000s	sys 0.001s
I20260709 12:25:45.927675 10428 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:46.072090  1805 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:46.074167  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 161703us
I20260709 12:25:46.072224  1873 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:45.949064 10435 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.097651  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 222704us
I20260709 12:25:46.073338  1799 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
W20260709 12:25:45.947950 10417 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 12:25:46.079171  1878 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:45.952806 10074 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:46.080286  1872 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:45.928018 10421 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:45.997109  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1516) took 1429 ms. Trace:
W20260709 12:25:45.971599 10184 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.722s	user 0.013s	sys 0.000s
I20260709 12:25:45.972081 10436 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.130s
I20260709 12:25:45.999109 10066 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:25:46.056622  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 465529us
I20260709 12:25:46.064452  1846 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:46.117707  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:45.938575 10430 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.186s
W20260709 12:25:45.911945 10422 consensus_queue.cc:1175] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "123095127f334445889c21a42ba1d41e" responder_term: 26 status { last_received { term: 23 index: 15 } last_committed_idx: 15 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: [25.16-25.16]" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:25:46.129942  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:46.146032  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:46.132498  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 331585us
I20260709 12:25:46.132884  1799 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:46.123106  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:46.124010  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:46.123517  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:46.150823  1805 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:46.083014 10324 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:25:45.964062 10404 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.137128  1801 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:46.138026  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:46.139431  1872 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:25:46.177843  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:45.942261 10400 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:46.146602  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:46.153347  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 73295.1us
W20260709 12:25:46.179126  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.156975  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:45.941321 10068 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.160218  1799 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:25:46.166920  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 102586us
I20260709 12:25:46.180614  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:46.109614 10074 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:46.172435  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 68529.7us
I20260709 12:25:46.074445 10315 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:25:46.075234 10299 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:45.987046 10072 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.111269  2411 rpcz_store.cc:276] 0709 12:25:44.568145 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.593479 (+ 25334us) service_pool.cc:224] Handling call
0709 12:25:44.593946 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:44.594246 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:45.286561 (+692315us) log.cc:844] Serialized 14 byte log entry
0709 12:25:45.996552 (+709991us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:45.996636 (+    84us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:45.996704 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.996819 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:45.997015 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22537,"prepare.queue_time_us":710105,"prepare.run_cpu_time_us":131,"prepare.run_wall_time_us":8976,"raft.queue_time_us":4068,"raft.run_cpu_time_us":578,"raft.run_wall_time_us":8125,"spinlock_wait_cycles":63438208,"thread_start_us":1369500,"threads_started":3,"wal-append.queue_time_us":612775}
W20260709 12:25:46.184027  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65630.8us
I20260709 12:25:46.132359 10066 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:46.145457 10297 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.204270  1805 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:46.207108  1872 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:46.219141  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:46.220584  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:46.221670  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:46.227276  1801 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 27 index: 20. (term mismatch)
I20260709 12:25:46.227722  1801 pending_rounds.cc:85] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 19
I20260709 12:25:46.227943  1801 pending_rounds.cc:107] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 26.20
I20260709 12:25:46.228169  1801 raft_consensus.cc:2889] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:46.144589 10112 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.058s
I20260709 12:25:46.173978 10232 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:46.113866 10441 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.102s
I20260709 12:25:46.133581 10184 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:46.244941  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 65206.4us
I20260709 12:25:46.246071  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:46.208077 10304 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:46.248374  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 91689.3us
I20260709 12:25:46.164666 10421 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.075s
I20260709 12:25:46.108398 10435 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.264688 10264 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:46.264354  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:46.207914 10419 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:46.264231  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:46.237416 10340 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:46.262086 10444 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.325562  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 145044us
I20260709 12:25:46.181262 10040 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:46.337705  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 133578us
I20260709 12:25:46.327894 10462 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:46.306455 10461 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:46.316154 10232 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:46.361030  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 96763.9us
I20260709 12:25:46.264688 10304 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.348827  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:46.324877 10340 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:46.318290 10451 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.380854  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:46.335074 10420 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:46.271469 10449 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.104s
I20260709 12:25:46.305960 10264 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.389163  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:46.235414 10297 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.357798 10462 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.404742  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 2972) took 2284 ms. Trace:
I20260709 12:25:46.407395  1802 rpcz_store.cc:276] 0709 12:25:44.120144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.215030 (+ 94886us) service_pool.cc:224] Handling call
0709 12:25:46.404515 (+2189485us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b54003dbdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:46.404665 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":928895360}
I20260709 12:25:46.400243 10451 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.408931 10469 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.067s
I20260709 12:25:46.357380 10461 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:46.343032 10459 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:46.469319 10445 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.380861 10454 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.323215 10066 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 27.12, Last appended by leader: 12, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:46.529557 10245 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.148s	user 0.011s	sys 0.005s
I20260709 12:25:46.460878  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:46.471304 10264 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:46.378903 10383 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.074s
I20260709 12:25:46.479343 10482 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.480396 10457 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:46.411028 10360 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:46.375626 10304 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:46.330384 10444 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.468084 10438 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:46.465533 10448 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.394076 10232 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.410667 10462 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:46.481531 10480 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:46.349584 10074 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:46.400195 10340 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:46.532995  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:46.534613  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 270660us
I20260709 12:25:46.359664 10386 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:46.538703 10443 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.268s	user 0.000s	sys 0.001s
I20260709 12:25:46.469658 10446 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:46.575163   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:46.550681 10322 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:46.595417  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:46.601043  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.597330 10322 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:46.566208 10400 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:46.572697 10360 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.611814  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:46.612326  1878 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
W20260709 12:25:46.597611 10361 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.692s	user 0.014s	sys 0.000s
I20260709 12:25:46.579231 10461 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.604947  1619 maintenance_manager.cc:419] P 4002daa3304c4fb4be053abd4780c195: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033403
I20260709 12:25:46.579730 10477 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.074s
I20260709 12:25:46.604947 10451 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:46.554325 10454 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.647001  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:46.622308 10438 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:46.605432 10441 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:46.626117 10360 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:46.619109 10072 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.557700 10245 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.635598 10361 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.651429 10340 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:46.640393 10400 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.646546 10068 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:46.626682 10420 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:46.672983 10438 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.657514 10453 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.637857 10459 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:25:46.659674 10349 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:46.619403 10184 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:46.667394  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 206686us
W20260709 12:25:46.669869  1771 connection.cc:256] RPC call timeout handler was delayed by -1.2187380589998611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260709 12:25:46.564867 10482 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.640468 10435 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:46.580085 10448 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.611306 10483 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.063s
I20260709 12:25:46.633381 10446 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:25:46.687009 10475 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.687099  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:46.687846 10460 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.688437 10100 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.303s	user 0.015s	sys 0.001s
I20260709 12:25:46.665549 10461 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:46.690997 10478 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.019s
I20260709 12:25:46.694567 10444 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:46.684507 10072 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.655275 10476 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:46.723769 10451 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.107s
I20260709 12:25:46.719326 10453 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.668141 10264 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:46.701550 10400 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.628883 10480 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.062s
I20260709 12:25:46.715346  2411 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:46.691035 10297 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:46.702198 10340 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:46.727193 10383 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.009s
W20260709 12:25:46.736544 10458 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.105s	user 0.001s	sys 0.001s
I20260709 12:25:46.736452 10472 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.855854 10361 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:46.856233 10492 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.707857 10438 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:46.799217  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:46.699821 10468 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:25:46.719831 10420 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.663172 10322 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.808120 10460 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.683120 10457 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.590979 10445 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.760635 10488 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:46.804164 10100 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.840910  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:46.853000 10400 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:46.829914 10072 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:46.855350 10481 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:46.832787 10469 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.864192 10494 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.856148 10472 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.865163 10453 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:46.859355 10476 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.832015 10264 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:46.831872 10245 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:46.876101 10322 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:46.723501 10184 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.911660 10184 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.914300 10184 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:46.887545  2414 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.023s	sys 0.031s
W20260709 12:25:46.915278  2414 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.024s	sys 0.031s
I20260709 12:25:46.886234  1872 raft_consensus.cc:2376] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24. Re-sending same reply.
I20260709 12:25:46.878036 10232 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:46.765540 10435 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.896294  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 445829us
I20260709 12:25:46.856719 10297 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:25:46.857949 10340 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.879777 10488 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:46.923148  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:46.797416 10475 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.914736 10441 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.926445  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:46.891350 10478 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:46.930981  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:46.934923  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:46.925180 10428 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.823s	user 0.014s	sys 0.000s
I20260709 12:25:46.891723 10400 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:25:46.944782  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:46.877543 10420 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.942653  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:46.935966  2418 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:46.945816  1732 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:46.948751  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:46.863247 10492 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:46.943516  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:46.950857  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:46.943821  1806 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:46.944312  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:46.944653  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:46.978675  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:46.762290 10349 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.995709  1730 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:46.949311  1772 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:46.997689  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 48476.6us
I20260709 12:25:46.879565 10457 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.966492  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:46.950264 10419 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:46.983654  1732 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:46.982205  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:46.957405  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:46.985574  1805 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:47.010222  1723 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:46.985574  1726 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:46.919551 10453 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.028s
I20260709 12:25:46.940776 10478 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:46.990952  1801 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:25:46.986145  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.013638  1801 raft_consensus.cc:606] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:47.014093  2414 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:47.014139  1801 raft_consensus.cc:614] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:46.994915  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:46.995139  1724 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
W20260709 12:25:46.947021  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 136796us
I20260709 12:25:47.004717  1807 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:46.926877 10435 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.019075  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.024030  2418 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:25:47.021553 10492 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:47.022390 10478 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:25:46.908968 10469 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.993952 10297 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.026832  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:46.974191 10454 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.011620 10468 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:46.915750 10476 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.035426  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.037364  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:46.941991 10304 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.035643  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:46.936975 10441 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:46.909487 10481 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:46.948154 10428 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.034338 10419 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:47.044185 10499 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.183s
I20260709 12:25:47.049484 10482 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.049681 10476 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.074364 10501 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.005s
I20260709 12:25:47.053094 10472 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.054855 10457 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.058357  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.059863 10340 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.062044 10435 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.064102 10349 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:47.066608  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.066855 10469 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.077903 10419 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:47.069080 10061 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.132s	user 0.013s	sys 0.002s
W20260709 12:25:47.072785  2417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1612) took 2208 ms (client timeout 2197 ms). Trace:
I20260709 12:25:47.088721 10462 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.051529 10475 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:47.080669  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 81982.9us
I20260709 12:25:47.083825  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.074011 10508 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:47.046608 10468 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:47.103904 10503 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.105279 10468 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:47.003722 10264 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.077591 10454 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.092437 10419 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:47.081827 10480 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:47.110762  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:46.922048 10494 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:47.098397  2417 rpcz_store.cc:269] 0709 12:25:44.864737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.865090 (+   353us) service_pool.cc:224] Handling call
0709 12:25:47.072463 (+2207373us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b5400489754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:47.072644 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":64272640}
I20260709 12:25:47.015831 10438 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:47.103251  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66062.5us
I20260709 12:25:47.047243 10297 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.105741 10420 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.113109 10508 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:47.098618 10499 consensus_queue.cc:1243] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 23.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:25:47.098986 10100 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.116488  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:47.119397  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.105636 10503 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.126262 10249 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:47.092985 10453 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:25:47.127022 10297 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.115424 10445 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.128583  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:25:47.129261 10396 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.756s	user 0.003s	sys 0.011s
I20260709 12:25:47.129623 10232 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:47.131567 10482 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:47.132052 10482 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:47.133303 10480 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:47.116266 10454 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.135047 10100 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:47.134379 10480 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.138249 10494 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.139549  2412 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:47.135190  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:47.132967 10482 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.143155  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.142208  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.146677 10304 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.148694 10438 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:47.155613 10505 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:47.137254 10454 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.165154 10462 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.122632 10508 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:47.166159  1872 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:47.158059 10505 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:25:47.168491  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:47.168502  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:47.135445 10460 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.177961  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3047) took 1048 ms. Trace:
I20260709 12:25:47.150931 10061 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:47.178818  1873 rpcz_store.cc:276] 0709 12:25:46.130047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.150983 (+ 20936us) service_pool.cc:224] Handling call
0709 12:25:46.151289 (+   306us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:46.307446 (+156157us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:46.334518 (+ 27072us) log.cc:844] Serialized 14 byte log entry
0709 12:25:46.739760 (+405242us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b18003d9b40. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a 00007f16abde2a2a
0709 12:25:46.949049 (+209289us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:46.989081 (+ 40032us) log.cc:844] Serialized 26 byte log entry
0709 12:25:46.989257 (+   176us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:46.989297 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.989367 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:47.166408 (+177041us) raft_consensus.cc:1685] finished
0709 12:25:47.177658 (+ 11250us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:47.177886 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14577,"prepare.queue_time_us":29954,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":63,"raft.queue_time_us":60969,"raft.run_cpu_time_us":359,"raft.run_wall_time_us":57531,"spinlock_wait_cycles":4253056,"thread_start_us":348520,"threads_started":2,"wal-append.queue_time_us":674252}
I20260709 12:25:47.189075  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:47.186165  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:47.165714 10454 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.192231  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:47.197600  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.199090  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:47.201143  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.201438  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:47.201957  1802 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:47.202136  1805 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 27 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:25:47.188724 10482 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:47.205335  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:47.206187 10461 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:47.196497 10494 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:47.187845 10475 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:47.173139 10507 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.195498 10460 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:47.212699  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.223739  2416 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:47.224824  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.213953  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.214326  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.208724 10508 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.192783 10511 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:47.210824 10420 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:47.181779 10360 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:47.217833 10432 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.281s	user 0.004s	sys 0.000s
I20260709 12:25:47.218182 10441 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.227192  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:47.227284  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.207868 10232 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:47.233566  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.251407  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:47.235421 10484 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:25:47.238579  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.250003  1806 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:47.253911  1798 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:47.254611  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.230273 10460 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:47.262920  2416 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
I20260709 12:25:47.263327  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.264178  1798 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:47.265440  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.211432 10461 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.191597 10300 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 12:25:47.199930 10454 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.224458 10449 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.250087  2418 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.20-24.20]
I20260709 12:25:47.271719  2417 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:25:47.269233  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.269134  2416 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:47.269528  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.248518 10428 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.259744  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.236271 10420 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:47.278513  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:47.251061 10232 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:47.288815  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.289772  1801 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:47.286173  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.267179 10505 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:47.270908 10460 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:47.268128 10484 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:47.236801 10511 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:47.301411  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.307498  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.307965  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:47.288511 10420 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:47.272197 10461 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:47.228637 10507 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.300354 10505 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.308517  1801 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:47.294019 10232 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:47.322544 10515 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:47.301898 10508 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:47.309170 10511 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:47.312202  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:47.316460 10499 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:47.281015 10475 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.033s
W20260709 12:25:47.334824  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 114110us
I20260709 12:25:47.336163  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.334615  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:47.327672 10505 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:47.340922  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:47.341470  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114123us
I20260709 12:25:47.342986  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:47.309733 10510 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.347641  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:47.330395 10507 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.350056  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:47.337988 10100 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:47.299157 10449 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.372066 10468 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.344847 10445 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:47.348278 10300 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:47.350129 10475 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:47.375994  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:47.387809  1806 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:47.390669  1802 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:47.376920 10498 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:47.399518 10299 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.400597 10360 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.380043 10300 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:47.402225 10468 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.378333 10453 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:47.349849 10510 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:47.385519 10513 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:25:47.379222 10445 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.412456 10488 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.384050 10475 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.496578 10360 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.168s
I20260709 12:25:47.393357 10400 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:47.668257  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:47.570482  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.506228  1881 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:25:47.433004 10300 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:47.694527  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 80945.9us
W20260709 12:25:47.632516 10419 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.520s	user 0.005s	sys 0.008s
I20260709 12:25:47.667754 10475 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:47.665043 10445 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:47.445983 10454 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.665982 10510 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.692672 10488 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:47.440655 10453 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:47.689790 10400 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:47.694664 10347 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:47.495720 10449 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:47.699396 10061 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.503s	user 0.012s	sys 0.000s
I20260709 12:25:47.738600 10453 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:47.707634 10347 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:47.711455 10400 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:47.701573 10419 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.745009 10061 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:47.812798 10407 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 3.001s	user 0.020s	sys 0.018s
I20260709 12:25:47.795720 10449 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:47.810205 10347 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:47.832535 10526 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:25:47.860992  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:47.879418  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:47.881250  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:47.884277  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.888218  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:47.894647  1802 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:25:47.903223 10523 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
W20260709 12:25:47.913707 10460 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.603s	user 0.011s	sys 0.000s
I20260709 12:25:47.844784 10511 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.921808 10511 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.934084  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3872) took 1105 ms. Trace:
I20260709 12:25:47.839313 10449 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.934806  1872 rpcz_store.cc:276] 0709 12:25:46.828905 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.409331 (+580426us) service_pool.cc:224] Handling call
0709 12:25:47.409726 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:47.410075 (+   349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:47.410102 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:47.410553 (+   451us) log.cc:844] Serialized 26 byte log entry
0709 12:25:47.852591 (+442038us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:47.852636 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.852723 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:47.933822 (+ 81099us) raft_consensus.cc:1685] finished
0709 12:25:47.933847 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:47.934007 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3234,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":2442,"thread_start_us":439042,"threads_started":1,"wal-append.queue_time_us":520553}
I20260709 12:25:47.937023 10449 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.914250 10460 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:47.939904 10505 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.512s	user 0.008s	sys 0.004s
W20260709 12:25:47.942090 10232 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.609s	user 0.010s	sys 0.002s
W20260709 12:25:47.941905 10461 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.614s	user 0.005s	sys 0.006s
I20260709 12:25:47.952646  1801 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [18.13-21.14]
W20260709 12:25:47.959610  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:47.950245 10505 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.987982 10505 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:47.990180 10363 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:47.992728 10440 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.762s	user 0.000s	sys 0.009s
I20260709 12:25:47.995715  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1524) took 2372 ms. Trace:
I20260709 12:25:47.996124  2413 rpcz_store.cc:276] 0709 12:25:45.623269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.722692 (+ 99423us) service_pool.cc:224] Handling call
0709 12:25:45.733402 (+ 10710us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:46.077569 (+344167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:46.077618 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:46.078289 (+   671us) log.cc:844] Serialized 26 byte log entry
0709 12:25:46.114396 (+ 36107us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:46.114601 (+   205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.995303 (+1880702us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54003fc854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:47.995358 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:47.995415 (+    57us) raft_consensus.cc:1685] finished
0709 12:25:47.995427 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:47.995653 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101507,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":151,"spinlock_wait_cycles":308763904,"thread_start_us":244844,"threads_started":2,"wal-append.queue_time_us":280717,"wal-append.run_cpu_time_us":9380,"wal-append.run_wall_time_us":1252402}
I20260709 12:25:47.998634  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1267) took 1055 ms. Trace:
I20260709 12:25:47.999076  1807 rpcz_store.cc:276] 0709 12:25:46.943587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.943737 (+   150us) service_pool.cc:224] Handling call
0709 12:25:46.944047 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:47.016691 (+ 72644us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.998380 (+981689us) spinlock_profiling.cc:243] Waited 60.2 ms on lock 0x7b540009fbd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:47.998573 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":138435840}
I20260709 12:25:48.000479  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1259) took 2739 ms. Trace:
I20260709 12:25:48.000878  1804 rpcz_store.cc:276] 0709 12:25:45.260922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.359256 (+ 98334us) service_pool.cc:224] Handling call
0709 12:25:45.411731 (+ 52475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:45.533326 (+121595us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:45.533369 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:45.545815 (+ 12446us) log.cc:844] Serialized 26 byte log entry
0709 12:25:45.995418 (+449603us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:45.995461 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:45.995540 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:47.992110 (+1996570us) spinlock_profiling.cc:243] Waited 4.09 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:48.000190 (+  8080us) raft_consensus.cc:1685] finished
0709 12:25:48.000213 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.000420 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9411840,"thread_start_us":367648,"threads_started":1,"wal-append.queue_time_us":564145}
W20260709 12:25:47.983733 10420 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.655s	user 0.011s	sys 0.002s
I20260709 12:25:48.008611  1873 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20260709 12:25:48.007484  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3854) took 3460 ms (client timeout 1654 ms). Trace:
W20260709 12:25:48.010804  1879 rpcz_store.cc:269] 0709 12:25:44.547323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.572952 (+ 25629us) service_pool.cc:224] Handling call
0709 12:25:48.007251 (+3434299us) spinlock_profiling.cc:243] Waited 238 ms on lock 0x7b5400026054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:48.007403 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":546653440}
I20260709 12:25:47.954716 10232 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:47.975759 10461 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.005300 10475 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:48.032195  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3034) took 3959 ms. Trace:
I20260709 12:25:48.032683  1877 rpcz_store.cc:276] 0709 12:25:44.072290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.107221 (+ 34931us) service_pool.cc:224] Handling call
0709 12:25:44.425908 (+318687us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:44.426460 (+   552us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:44.426504 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:44.437346 (+ 10842us) log.cc:844] Serialized 26 byte log entry
0709 12:25:44.659294 (+221948us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:44.659366 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.659481 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:45.947074 (+1287593us) raft_consensus.cc:1685] finished
0709 12:25:45.947101 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.029983 (+2082882us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54002df0d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:48.032105 (+  2122us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5839,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":2165,"spinlock_wait_cycles":684546304,"thread_start_us":200006,"threads_started":1,"wal-append.queue_time_us":1102483,"wal-append.run_cpu_time_us":6976,"wal-append.run_wall_time_us":1428358}
I20260709 12:25:48.033984 10356 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:48.035645 10331 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:48.014149 10420 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:48.051519  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3858) took 3396 ms (client timeout 1517 ms). Trace:
W20260709 12:25:48.051970  1880 rpcz_store.cc:269] 0709 12:25:44.657355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.657612 (+   257us) service_pool.cc:224] Handling call
0709 12:25:48.051322 (+3393710us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b540001efd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:48.051443 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1809730816}
I20260709 12:25:48.034613 10356 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.036623 10300 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.060537 10445 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:48.095055  1618 tablet_replica.cc:1406] Time spent applying in-flights took a long time: real 0.914s	user 0.000s	sys 0.000s
I20260709 12:25:48.099032 10445 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.099413 10461 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.107389  1880 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:25:48.110966  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.112912 10445 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.113078  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.117616  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.118368  1876 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:48.119211  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.119853  1879 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:25:48.120703  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.121497 10356 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:48.117195 10445 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.122490 10356 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:48.122660 10475 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:48.123245 10356 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.124114 10445 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.121501 10300 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.125977 10356 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:48.126515 10537 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:48.126703 10356 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:48.128350 10356 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.132604  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.136070  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.138309  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3046) took 2011 ms. Trace:
I20260709 12:25:48.139019  1878 rpcz_store.cc:276] 0709 12:25:46.126999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.149201 (+ 22202us) service_pool.cc:224] Handling call
0709 12:25:46.150182 (+   981us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:46.620671 (+470489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.137946 (+1517275us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b540001ead4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:48.138226 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1061948672}
I20260709 12:25:48.143947  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.131269 10356 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:48.149336  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.151404  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:48.154181  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.152441  1872 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:48.156549  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:48.157132  1697 leader_election.cc:336] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.778s (SENT)
I20260709 12:25:48.157390  1878 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:48.157922  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:48.159216 10061 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:48.172760 10486 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.514s	user 0.002s	sys 0.009s
W20260709 12:25:48.174019  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 79591us
I20260709 12:25:48.142594 10400 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.175877  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.177199  1878 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:25:48.180603  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2996) took 1956 ms. Trace:
I20260709 12:25:48.180629 10254 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:48.181443 10232 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:48.181221  1799 rpcz_store.cc:276] 0709 12:25:46.224652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.224799 (+   147us) service_pool.cc:224] Handling call
0709 12:25:46.225127 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:46.225446 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:46.225472 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:46.235701 (+ 10229us) log.cc:844] Serialized 26 byte log entry
0709 12:25:46.580385 (+344684us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:46.580445 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:46.580514 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:48.172118 (+1591604us) spinlock_profiling.cc:243] Waited 6.63 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:48.180281 (+  8163us) raft_consensus.cc:1685] finished
0709 12:25:48.180316 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.180537 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":120451,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":9223,"spinlock_wait_cycles":15253376,"thread_start_us":264990,"threads_started":1,"wal-append.queue_time_us":387493}
I20260709 12:25:48.187098 10542 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:48.188382 10542 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:48.189337 10542 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.189724 10269 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:48.193167 10542 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:48.194631  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:48.199148  1771 leader_election.cc:336] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.529s (SENT)
I20260709 12:25:48.200451  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:48.193596 10184 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:48.203047 10184 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:48.203819  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.204232  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.204859  1872 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:48.205171  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.205937  1879 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:48.205801  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.206611  1872 raft_consensus.cc:2410] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 27 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:48.204432 10184 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.207510  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.193987 10269 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:48.210037  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:48.207727  1876 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:25:48.211112 10434 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:48.202553 10232 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:48.210680 10269 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.213012  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.210956 10184 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:48.213680 10434 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.208946  1879 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:48.216248 10400 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.183845 10347 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.221581  1729 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:48.234447  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.235150  1729 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:48.222263  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.237143  1723 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:48.222584  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.238400  1732 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 28: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:48.226954  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.227296  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.240428  1729 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:25:48.240558  1730 raft_consensus.cc:2393] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:48.239764  1724 raft_consensus.cc:2368] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:25:48.220100  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:48.243607  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.243772  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.244834  1724 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:48.245448  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:48.247510  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.248230  1730 raft_consensus.cc:2368] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
I20260709 12:25:48.229413  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:48.249063  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.249802  1732 raft_consensus.cc:2368] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:25:48.232861  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.218832 10434 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:48.251256  1726 raft_consensus.cc:2368] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27.
I20260709 12:25:48.251459  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.229950 10061 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
W20260709 12:25:48.254268 10413 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 2.995s	user 0.007s	sys 0.014s
I20260709 12:25:48.256537  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:48.258652  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:48.262319  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 86967us
I20260709 12:25:48.263274  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:48.218747 10347 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.264384  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:48.267403  1799 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.256578 10434 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:48.265920  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:48.266404  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 2975) took 3789 ms. Trace:
I20260709 12:25:48.275295  1799 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:48.281037  1803 rpcz_store.cc:276] 0709 12:25:44.477010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.477160 (+   150us) service_pool.cc:224] Handling call
0709 12:25:44.477549 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:44.478009 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:44.478358 (+   349us) log.cc:844] Serialized 14 byte log entry
0709 12:25:44.913989 (+435631us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:44.914561 (+   572us) log.cc:844] Serialized 26 byte log entry
0709 12:25:44.914725 (+   164us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:44.914767 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:44.914869 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:48.253447 (+3338578us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:48.262049 (+  8602us) raft_consensus.cc:1685] finished
0709 12:25:48.266101 (+  4052us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.266311 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12022,"prepare.queue_time_us":3495,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":452912,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":22913,"spinlock_wait_cycles":9105280,"thread_start_us":430558,"threads_started":2,"wal-append.queue_time_us":551873}
I20260709 12:25:48.282888  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.283905  1881 raft_consensus.cc:2410] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:25:48.247139  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:48.286252  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.286382  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.286949  1729 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:48.288009  1618 maintenance_manager.cc:643] P 4002daa3304c4fb4be053abd4780c195: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 1.628s	user 0.030s	sys 0.010s Metrics: {"bytes_written":4026,"delete_count":0,"lbm_write_time_us":5770,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":137}
I20260709 12:25:48.271586 10400 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:48.289638 10400 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:48.290061 10400 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:48.290306 10232 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:48.291500 10467 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:48.291913 10467 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:48.292807 10467 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.294427 10467 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:48.295863  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.296594  1726 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.298900 10061 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:48.299383  1724 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:25:48.299526 10061 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:48.300088 10061 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.302127  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:48.316733  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.318920 10478 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:48.319439 10478 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:48.319885 10478 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:48.320720 10484 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.322212 10484 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:25:48.304033  1619 maintenance_manager.cc:419] P 4002daa3304c4fb4be053abd4780c195: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.208232
W20260709 12:25:48.324776  1771 leader_election.cc:336] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.654s (SENT)
I20260709 12:25:48.333839  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1429) took 3067 ms. Trace:
W20260709 12:25:48.340701  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.341650 10219 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.342346 10219 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.343582  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.344184  1723 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:48.344973 10219 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.345234 10419 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:48.345521 10219 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.347160 10219 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.347637 10347 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:48.348289 10347 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.348901 10503 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.350397 10419 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:48.351668 10419 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:48.352056 10419 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:48.352150 10219 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.350454 10347 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.353735  1732 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.20-24.20]
I20260709 12:25:48.353404 10419 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.20, Last appended by leader: 20, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.355454  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.356077  1730 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:48.356758  2414 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:48.357964  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1623) took 1438 ms. Trace:
I20260709 12:25:48.358326  2414 rpcz_store.cc:276] 0709 12:25:46.919711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.090495 (+170784us) service_pool.cc:224] Handling call
0709 12:25:48.356541 (+1266046us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:48.357204 (+   663us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.357778 (+   574us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54004745d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:48.357909 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2910697216}
I20260709 12:25:48.334450  2412 rpcz_store.cc:276] 0709 12:25:45.266354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.266600 (+   246us) service_pool.cc:224] Handling call
0709 12:25:47.073322 (+1806722us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:47.142903 (+ 69581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.331950 (+1189047us) spinlock_profiling.cc:243] Waited 65.9 ms on lock 0x7b5400489754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:48.333469 (+  1519us) spinlock_profiling.cc:243] Waited 1.81 s on lock 0x7b5400489750. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:48.333721 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4305560192}
I20260709 12:25:48.341735 10481 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:48.370663  1880 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:25:48.376533 10448 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.356634  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1522) took 2792 ms. Trace:
I20260709 12:25:48.381979  2415 rpcz_store.cc:276] 0709 12:25:45.564186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.584112 (+ 19926us) service_pool.cc:224] Handling call
0709 12:25:48.356365 (+2772253us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54004745d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:48.356567 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3217193216}
I20260709 12:25:48.383042  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:48.384145  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:48.384644  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.384964  2416 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 17. Current term is 18. Ops: []
W20260709 12:25:48.385308  1845 connection.cc:256] RPC call timeout handler was delayed by -1.1960073619998184s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3607s for a response
I20260709 12:25:48.386937 10542 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.387408  1804 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
W20260709 12:25:48.348932  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 2.80142e+06us
I20260709 12:25:48.387451 10542 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.388824  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:48.389652  1803 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:48.389492  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:48.389833  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:48.390473  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.391744  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:48.393770  1806 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 27. Ops: []
I20260709 12:25:48.394119 10466 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:48.389896  1804 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:25:48.396064  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.396472  1806 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:25:48.401718  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.402384  1880 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:48.402458 10254 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.403173 10254 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.403487  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:48.404395 10542 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.405535 10254 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:48.410694 10541 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.248s	user 0.003s	sys 0.000s
I20260709 12:25:48.411345  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3871) took 1738 ms. Trace:
W20260709 12:25:48.411047  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.411825  1873 rpcz_store.cc:276] 0709 12:25:46.672763 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.189775 (+517012us) service_pool.cc:224] Handling call
0709 12:25:48.008154 (+818379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:48.010324 (+  2170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.411124 (+400800us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b5400026050. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:48.411279 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1881109888}
I20260709 12:25:48.414036  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:48.415454 10420 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:48.416594  2411 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1438) took 2036 ms (client timeout 1778 ms). Trace:
W20260709 12:25:48.417088  2411 rpcz_store.cc:269] 0709 12:25:46.380197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.380377 (+   180us) service_pool.cc:224] Handling call
0709 12:25:48.416334 (+2035957us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b54005230d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:48.416491 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":760024960}
I20260709 12:25:48.417497  2415 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:25:48.418567 10227 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:48.419127 10227 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:48.420193 10227 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.421792 10227 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:48.303982  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:48.427978 10450 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.429211  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2917) took 1191 ms. Trace:
I20260709 12:25:48.428905 10450 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.429585  1805 rpcz_store.cc:276] 0709 12:25:47.237433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.237692 (+   259us) service_pool.cc:224] Handling call
0709 12:25:47.238090 (+   398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:47.373500 (+135410us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:47.373531 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:47.374012 (+   481us) log.cc:844] Serialized 26 byte log entry
0709 12:25:47.503717 (+129705us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:47.503766 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:47.503870 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:48.428094 (+924224us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b24002b79a8. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:25:48.428892 (+   798us) raft_consensus.cc:1685] finished
0709 12:25:48.428916 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.429150 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":93462,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":6371,"spinlock_wait_cycles":35707264,"thread_start_us":242102,"threads_started":2,"wal-append.queue_time_us":439212}
I20260709 12:25:48.431137 10450 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:48.428884  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 124991us
I20260709 12:25:48.432173 10474 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.431962 10482 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:48.433115 10474 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.434688  1873 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:48.435348  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1261) took 3037 ms. Trace:
I20260709 12:25:48.435506 10474 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:48.435822  1800 rpcz_store.cc:276] 0709 12:25:45.398381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.400588 (+  2207us) service_pool.cc:224] Handling call
0709 12:25:45.401096 (+   508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:45.767096 (+366000us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:45.767144 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:45.767203 (+    59us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:45.767268 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.435113 (+2667845us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54002c0954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:48.435162 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.435285 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":841031424}
I20260709 12:25:48.440358  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.448150 10441 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:25:48.450434  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.453262 10335 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:48.453884 10335 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:48.404661  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:48.454986 10335 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.456182 10335 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.456260  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.458215  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:48.373957 10400 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.450399  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:48.464586  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 160616us
W20260709 12:25:48.346313  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44294.9us
I20260709 12:25:48.480448  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:48.482677  1723 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:25:48.484143  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:48.486119  1730 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:48.486832  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:48.487506  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.487814  2416 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.465415  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:48.505316  1800 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:48.508999  1729 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:48.509358  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:48.510628  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.519241  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:48.520828  1800 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:48.521332  1806 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:48.484174  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.524765  1723 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:48.520088  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:48.527395  1881 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.498638  1726 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:48.532982  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.536454  1730 raft_consensus.cc:2410] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 23 because replica has last-logged OpId of term: 21 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:25:48.531771  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:48.526544  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:48.539625  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:48.540045  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:48.540583  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.534333  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.548696  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.549681  1728 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.552701  1726 raft_consensus.cc:2393] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:48.554675  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.555076  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.556036  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.557183  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.556259  1726 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 27: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:48.558617  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.561053  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:48.562647  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:48.562867  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:48.563822  2412 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:48.564881  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:48.521554 10446 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:25:48.597463  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.586272  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.461831 10400 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:48.589293  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:48.489770 10441 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:48.613014  1881 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:48.590723  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:48.619525  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 105011us
I20260709 12:25:48.479254 10419 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:48.521485 10347 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:48.585086 10487 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.944s	user 0.024s	sys 0.049s
I20260709 12:25:48.549580 10542 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:48.634611  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.658352  1726 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:48.646409  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:48.645006  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82423.8us
I20260709 12:25:48.615473  1723 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:25:48.633976  1799 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 19.
I20260709 12:25:48.633417  1729 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:48.638193  1725 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:48.639276  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 66542.6us
I20260709 12:25:48.585410 10184 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:48.660566  2410 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 19.
I20260709 12:25:48.630970  2413 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:48.581225  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.609396  1728 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:25:48.645598  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:48.633895  2411 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.664100  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:48.665652  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.629487 10446 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:48.549835 10420 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.095s
I20260709 12:25:48.707067  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:48.707008  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61583.9us
I20260709 12:25:48.707657  9394 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:48.635941 10450 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:25:48.660132 10487 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.040s	user 0.024s	sys 0.050s
I20260709 12:25:48.585739 10363 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.084s
I20260709 12:25:48.712474  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:48.713510  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 77509.5us
I20260709 12:25:48.659528 10505 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:48.716388  1846 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:48.721215  9394 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:48.736691  9394 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:48.738023  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.740126  2410 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:48.710937 10446 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:48.744174  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:48.747853  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.709419 10040 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:25:48.750073 10339 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:48.750568 10040 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:48.750946 10040 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:48.751742 10040 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:48.752928 10040 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:48.721717 10324 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:25:48.758075  2416 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:48.762159 10478 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:48.763259 10540 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.595s	user 0.004s	sys 0.000s
I20260709 12:25:48.736779 10505 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:48.767364 10066 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:48.767544 10505 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:48.767693 10061 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.772153 10450 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:48.772711 10450 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:48.703936 10347 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.776544 10347 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:48.764006  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:48.777853  2419 raft_consensus.cc:2368] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:25:48.778445  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.785452  2413 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:25:48.787622  2410 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:48.789148  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.790957  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:48.793064  2411 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:25:48.793188  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:48.714745 10378 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.076s
I20260709 12:25:48.786805 10219 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:48.798537 10542 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:48.803066 10537 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:48.807477 10537 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:48.809092 10537 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:48.810096 10537 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:48.809739 10542 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:48.868794  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.868870  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.662545  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:48.871404  1723 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:48.873538  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.875094  1723 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:48.876523  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:48.877115  1697 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:48.879585  1725 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:48.879827  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:48.874163  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 211625us
I20260709 12:25:48.881223  1726 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:48.881433  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.883558  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:48.694079  1729 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:48.884907  1730 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:25:48.885684  1732 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27. Ops: []
W20260709 12:25:48.887811  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:48.888792  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 152062us
I20260709 12:25:48.893637  1807 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:48.894917  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 178439us
W20260709 12:25:48.900832  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.895663  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3935) took 1952 ms. Trace:
I20260709 12:25:48.878284  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:48.908704  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 188299us
W20260709 12:25:48.948805  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.037680  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 157483us
I20260709 12:25:49.040798  1725 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:49.045029  1725 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:25:49.047811  1728 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:49.048643 10478 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:49.048075 10553 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:49.048986 10564 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:25:49.051410  2417 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1628) took 1909 ms (client timeout 2158 ms). Trace:
I20260709 12:25:49.052296  2418 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:25:49.052301 10299 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:49.053974  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:48.958160  9394 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.100s
I20260709 12:25:49.057946 10074 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:49.059669  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:48.884888  1724 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:49.064299  1726 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:25:49.082345  1723 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:25:49.070739 10061 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:49.066776 10269 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:49.071635 10184 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:49.104394 10299 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:48.956929 10091 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:49.101545  2417 rpcz_store.cc:269] 0709 12:25:47.141687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.142914 (+  1227us) service_pool.cc:224] Handling call
0709 12:25:49.050761 (+1907847us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400840154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:49.051096 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":285575936}
I20260709 12:25:49.079850 10460 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:49.138511  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:49.148983 10456 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 2.489s	user 0.014s	sys 0.002s
I20260709 12:25:49.150489 10505 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.153748 10505 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:49.138564 10299 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:49.158500 10313 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:49.162909 10482 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:25:49.166525  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 520036us
I20260709 12:25:49.167008  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:49.172876  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:49.174583 10400 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:49.175467 10400 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:49.177376 10400 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.181006  1724 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:25:49.184129 10424 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:25:49.185026  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 595576us
W20260709 12:25:49.187340  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.189495  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 130105us
I20260709 12:25:49.189975  1725 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:25:49.190689 10461 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.191123 10232 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:49.192391 10461 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.192607 10232 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.190187 10324 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:49.195758 10324 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.196231  1728 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:49.197564  1807 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:25:49.198159  1728 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:49.191210 10437 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:49.199496  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:49.200573  1799 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:49.203364  1804 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:49.205060  1806 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:25:49.206770  1725 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 24. Ops: []
I20260709 12:25:49.206599  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:49.079376 10448 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.128s
I20260709 12:25:49.211822 10363 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:49.212333  1800 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:49.213299  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1452) took 1979 ms. Trace:
I20260709 12:25:49.213768  2418 rpcz_store.cc:276] 0709 12:25:47.233739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:47.292170 (+ 58431us) service_pool.cc:224] Handling call
0709 12:25:49.051325 (+1759155us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:49.118024 (+ 66699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.213005 (+ 94981us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b5400840150. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:49.213179 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4004414848}
I20260709 12:25:49.208400  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:49.214483 10232 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:49.215168  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 46734.2us
I20260709 12:25:49.215533  1802 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:25:49.038364  1731 rpcz_store.cc:276] 0709 12:25:46.943185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.943978 (+   793us) service_pool.cc:224] Handling call
0709 12:25:46.987299 (+ 43321us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:47.137274 (+149975us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:47.137307 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:47.137872 (+   565us) log.cc:844] Serialized 52 byte log entry
0709 12:25:47.190952 (+ 53080us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:47.190999 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:48.887492 (+1696493us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54003cd7d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:48.887596 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:48.887685 (+    89us) raft_consensus.cc:1685] finished
0709 12:25:48.887702 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:48.895394 (+  7692us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1498,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":97,"spinlock_wait_cycles":343895808,"thread_start_us":52645,"threads_started":1,"wal-append.queue_time_us":138663,"wal-append.run_cpu_time_us":23461,"wal-append.run_wall_time_us":1515296}
W20260709 12:25:49.219730  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.221175  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:49.208801  1770 leader_election.cc:400] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:25:49.211380  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:49.212929 10363 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.218686  1731 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:49.198683 10324 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:49.234359  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:49.235611  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49138.1us
I20260709 12:25:49.237890  1770 leader_election.cc:400] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
I20260709 12:25:49.238296  1770 leader_election.cc:403] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:49.191102 10184 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.241513  1723 pending_rounds.cc:85] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 14
I20260709 12:25:49.241977  1723 pending_rounds.cc:107] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:25:49.242309  1723 raft_consensus.cc:2889] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:49.243541 10448 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:49.244046 10448 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:49.245296 10448 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.248526  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:49.250039 10461 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:49.250229 10537 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:49.250744 10537 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:49.251230  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:49.252089  1806 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:49.251956 10537 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.253123 10537 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:49.234692  1725 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
W20260709 12:25:49.254417  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.254199  1725 pending_rounds.cc:85] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
I20260709 12:25:49.260337  1725 pending_rounds.cc:107] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 12:25:49.260676  1725 raft_consensus.cc:2889] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:49.250239  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:48.955776 10219 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.298990  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:49.299695  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.300150  1807 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:25:49.300635  1803 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:49.303283  1798 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:49.303957  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.308136  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.309689  2414 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:49.311605 10494 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:49.312608 10494 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.315804 10363 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:49.316947  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:49.320037  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:49.241547 10184 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:49.262264 10091 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.327458  1770 leader_election.cc:400] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:25:49.328009  1770 leader_election.cc:403] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:25:49.329322  1770 leader_election.cc:400] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27.
I20260709 12:25:49.329699  1770 leader_election.cc:403] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:49.329912 10091 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:49.330993  1730 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29. Ops: [28.19-28.19]
I20260709 12:25:49.333101  1730 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:49.333387  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:49.300097  1879 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:25:49.372232  1800 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:49.334331  1875 raft_consensus.cc:2376] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20. Re-sending same reply.
I20260709 12:25:49.278409 10448 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:49.395958  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.407976  2413 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:49.411834  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.412235  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1387) took 1048 ms. Trace:
I20260709 12:25:49.412886  1732 rpcz_store.cc:276] 0709 12:25:48.363425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.367794 (+  4369us) service_pool.cc:224] Handling call
0709 12:25:48.368166 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:49.109535 (+741369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.411773 (+302238us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b54000d2ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:49.412113 (+   340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1188583680}
I20260709 12:25:49.396929 10061 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:49.406893 10299 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.417246  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.418336  1875 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:49.419107  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.419363  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.385859 10184 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:49.381675  1724 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:49.419878  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.381224  1731 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 16. (term mismatch)
I20260709 12:25:49.421947  1731 pending_rounds.cc:85] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 15
I20260709 12:25:49.422248  1731 pending_rounds.cc:107] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 26.16
I20260709 12:25:49.422549  1731 raft_consensus.cc:2889] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:49.426908  1807 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
W20260709 12:25:49.384999 10227 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.963s	user 0.022s	sys 0.000s
W20260709 12:25:49.405010  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 131487us
I20260709 12:25:49.418170  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:49.428653  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.379041 10324 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:25:49.388391  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:49.438153  1802 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:49.420719  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:49.451273  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71681.1us
W20260709 12:25:49.453967  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.457608 10450 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.684s	user 0.002s	sys 0.020s
I20260709 12:25:49.469955  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:49.480055  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 204681us
I20260709 12:25:49.458238 10450 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.435544 10363 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27.
I20260709 12:25:49.493275  1878 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:49.504266  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.510717  1881 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:49.521051  1803 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:49.522666  1798 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
W20260709 12:25:49.530562  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.533958  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 214146us
I20260709 12:25:49.538249  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.545120  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.555892  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:49.581746  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.595428  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:49.596820 10321 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:25:49.599737  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 66479.8us
W20260709 12:25:49.601890  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.599699 10066 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.820s	user 0.008s	sys 0.013s
W20260709 12:25:49.605208  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 217147us
I20260709 12:25:49.583022 10436 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24.
I20260709 12:25:49.590691 10564 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:25:49.599663 10040 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.846s	user 0.012s	sys 0.009s
I20260709 12:25:49.601032 10184 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:49.605832 10299 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:49.622427  1881 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
W20260709 12:25:49.625571  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.622193 10076 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.634827  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:49.635681  1731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6388b1a5243d463589750794e8ac5511"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:49.663280  1731 raft_consensus.cc:606] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:49.663671  1731 raft_consensus.cc:614] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:49.641414 10461 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:25:49.642880  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53590.4us
W20260709 12:25:49.643971  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.647137  1876 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:25:49.631676  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:49.665104  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.650391 10494 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:49.666399  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:49.667470  1724 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 18.
I20260709 12:25:49.668311  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:49.669592 10269 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:25:49.671762 10573 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 12:25:49.674039  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3938) took 2727 ms. Trace:
I20260709 12:25:49.674304 10450 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:49.674553  1727 rpcz_store.cc:276] 0709 12:25:46.946623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:46.946769 (+   146us) service_pool.cc:224] Handling call
0709 12:25:46.947175 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:48.883549 (+1936374us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b18004237c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abdd0bfb 00007f16abddb64b
0709 12:25:48.884024 (+   475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:48.884055 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:48.950493 (+ 66438us) log.cc:844] Serialized 26 byte log entry
0709 12:25:49.225729 (+275236us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:49.225771 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.262335 (+ 36564us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:49.673449 (+411114us) raft_consensus.cc:1685] finished
0709 12:25:49.673476 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:49.673787 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10621,"raft.queue_time_us":88710,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":13771,"spinlock_wait_cycles":4972928,"thread_start_us":319320,"threads_started":2,"wal-append.queue_time_us":571139}
I20260709 12:25:49.676064  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:49.677371  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:49.680024 10232 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:49.680657 10232 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:49.681885 10232 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:49.683226  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 65750.2us
W20260709 12:25:49.686112  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.683825 10424 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:49.683259 10421 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:49.664763 10461 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:49.685148 10219 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:49.689630 10076 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:49.685148  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.684113  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 67979us
I20260709 12:25:49.692730  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:49.693751  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1389) took 1329 ms. Trace:
I20260709 12:25:49.697882  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:49.699175  1880 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:49.703104  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:49.703840  1723 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:49.705116 10421 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:49.706327 10421 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.707101  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3072) took 1501 ms. Trace:
I20260709 12:25:49.707644  1876 rpcz_store.cc:276] 0709 12:25:48.205902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.212746 (+  6844us) service_pool.cc:224] Handling call
0709 12:25:49.706738 (+1493992us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400222754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:49.706994 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3212844928}
I20260709 12:25:49.688333 10424 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:49.711195 10424 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:49.711323 10461 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.712805 10461 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:49.699395  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:49.685397 10074 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.552s	user 0.003s	sys 0.011s
I20260709 12:25:49.692824  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:49.688634  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 162202us
I20260709 12:25:49.687086  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:49.694206  1729 rpcz_store.cc:276] 0709 12:25:48.364110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.667157 (+303047us) service_pool.cc:224] Handling call
0709 12:25:48.677075 (+  9918us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:49.134303 (+457228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.693331 (+559028us) spinlock_profiling.cc:243] Waited 9.54 ms on lock 0x7b54003cc8d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:49.693540 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21942656}
I20260709 12:25:49.724287  1731 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:49.724431  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:49.725803 10581 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.055s	user 0.000s	sys 0.002s
I20260709 12:25:49.727762  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.727954  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:49.728983  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:49.729627  1874 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:49.730456 10537 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:49.731284 10537 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.731783 10537 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:49.732836  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.733295  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.730613 10269 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:49.734747 10313 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:49.735179  1877 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:25:49.735950 10219 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:49.738502 10439 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:49.739166 10439 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:49.739179 10040 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:49.739964 10040 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.741171  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:49.741611  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:49.741948  2412 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:49.742393 10040 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:49.743489 10066 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:49.740242  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:49.744277 10066 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:49.743347 10494 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:25:49.745430 10324 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:49.746405 10494 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:49.746842 10494 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:49.747104 10066 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:49.747911 10494 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.730475 10074 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:49.750393 10074 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:49.751089 10074 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:49.752215 10476 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:49.752820 10476 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:49.753188  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 48432.1us
I20260709 12:25:49.749643 10494 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:49.745173  1880 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
W20260709 12:25:49.750569  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71868.9us
I20260709 12:25:49.731396  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:49.753939 10476 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.756232  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:49.745570  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:49.768430  1874 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:25:49.765137  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.735193 10269 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:49.779297  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:49.780063 10269 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.755493 10076 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:49.763727 10299 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:49.788693  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:49.788846  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:49.797127  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.798256  1872 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 17.
W20260709 12:25:49.798889  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53443.5us
I20260709 12:25:49.799052  1876 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:49.799508  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.808984  1872 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:49.810143 10363 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:25:49.815464 10576 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.191s	user 0.004s	sys 0.000s
I20260709 12:25:49.816438 10461 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:49.789865 10219 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:49.787772 10299 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:49.821271  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.822093  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.822414  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.830811  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:49.755437 10439 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.838568  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:49.840065  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49903.1us
W20260709 12:25:49.843986  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 112694us
I20260709 12:25:49.847354  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:49.859274  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.830173 10299 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:25:49.871445  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.839159 10219 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:49.847900 10076 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:49.885181 10076 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:49.885768 10219 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 23.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:49.887393  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3148) took 1018 ms. Trace:
W20260709 12:25:49.891281  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 122586us
I20260709 12:25:49.892346  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:49.893270  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 137200us
I20260709 12:25:49.894047 10564 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:49.894726 10564 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:49.834834 10184 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:49.878000  1881 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:49.895705  1872 raft_consensus.cc:2376] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25. Re-sending same reply.
I20260709 12:25:49.902163 10263 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:49.902575 10446 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.131s	user 0.012s	sys 0.009s
I20260709 12:25:49.907164  1878 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:49.887920  1728 rpcz_store.cc:276] 0709 12:25:48.868600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.211133 (+342533us) service_pool.cc:224] Handling call
0709 12:25:49.211492 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:49.211973 (+   481us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:49.517450 (+305477us) log.cc:844] Serialized 14 byte log entry
0709 12:25:49.871913 (+354463us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:49.872009 (+    96us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:49.886937 (+ 14928us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.887058 (+   121us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:49.887273 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":375853,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":33039,"thread_start_us":604405,"threads_started":2}
I20260709 12:25:49.885735 10076 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:49.916343  1876 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
W20260709 12:25:49.895619 10580 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:25:49.871038 10461 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:49.937336  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:49.923702  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3138) took 1343 ms. Trace:
I20260709 12:25:49.959009  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:49.971207  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:49.971771  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:49.915153 10184 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:49.971122  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3057) took 1850 ms (client timeout 2083 ms). Trace:
I20260709 12:25:49.922186 10066 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:49.980427  1877 rpcz_store.cc:269] 0709 12:25:48.133303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:48.141546 (+  8243us) service_pool.cc:224] Handling call
0709 12:25:49.922738 (+1781192us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400116a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:49.970893 (+ 48155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3644070272}
I20260709 12:25:49.919591 10439 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:49.919235 10450 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:49.924274 10324 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:49.957871  1726 rpcz_store.cc:276] 0709 12:25:48.580179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.168698 (+588519us) service_pool.cc:224] Handling call
0709 12:25:49.169344 (+   646us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:49.241979 (+ 72635us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:25:49.242015 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:49.242733 (+   718us) log.cc:844] Serialized 26 byte log entry
0709 12:25:49.609834 (+367101us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:25:49.611512 (+  1678us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:49.915063 (+303551us) spinlock_profiling.cc:243] Waited 72.2 ms on lock 0x7b54002b7ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:49.915122 (+    59us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:49.915180 (+    58us) raft_consensus.cc:1685] finished
0709 12:25:49.915201 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:49.923600 (+  8399us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166042880,"thread_start_us":313053,"threads_started":1,"wal-append.queue_time_us":419504}
I20260709 12:25:49.875895 10436 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.100s
I20260709 12:25:49.999167  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:50.000205  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:49.999220  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 220027us
I20260709 12:25:49.976207 10363 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:50.003513  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 108790us
I20260709 12:25:49.983755 10184 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:49.986459 10066 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:50.010098  1872 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:49.957162 10446 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:50.022112  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 221843us
W20260709 12:25:50.027016  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:50.027118  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 124600us
W20260709 12:25:50.037626  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 219544us
I20260709 12:25:50.038352  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.019650 10066 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:50.083022  1804 raft_consensus.cc:2410] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:25:49.970435 10263 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.018694 10450 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:50.082823  1731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:50.094988 10537 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.098204 10537 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:50.103490  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:50.104959  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:50.105170  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:50.105685  1872 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:50.107079  1726 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:50.107374 10331 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:50.108951 10363 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.110962 10061 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.111722 10061 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.114121 10061 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:50.114754 10461 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:50.117211  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 146755us
I20260709 12:25:50.117825 10450 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:50.119225  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:50.120635 10450 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.003s
W20260709 12:25:50.127959  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 103297us
I20260709 12:25:50.129099  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:50.132922 10586 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.243s	user 0.000s	sys 0.003s
I20260709 12:25:50.133183  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:50.131685  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.134945  1731 raft_consensus.cc:606] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.136641  1731 raft_consensus.cc:614] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:25:50.133790  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 110203us
I20260709 12:25:50.137801 10594 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:50.138940  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 57521.6us
I20260709 12:25:50.140669 10313 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:50.144570 10446 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:50.146167 10494 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:50.034230 10400 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:50.147125  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.134137  2413 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 29 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
W20260709 12:25:50.130896 10448 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.692s	user 0.011s	sys 0.000s
I20260709 12:25:50.158421  1771 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:50.165609  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:50.175994  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.152177 10232 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:25:50.186021  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66963.6us
I20260709 12:25:50.162981 10494 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:50.189008  1770 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:50.199075  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49353.4us
I20260709 12:25:50.199589  9394 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:50.200245  9394 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:50.200685  9394 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:50.189867  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.173238 10061 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:50.173554 10400 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.207137  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:50.185715 10321 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.198738 10494 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:50.221927  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.223308 10534 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:50.216992 10448 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:50.231209 10450 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:50.212859 10232 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:50.244777  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:50.239676  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73778.8us
I20260709 12:25:50.233875 10400 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:50.164999 10424 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:50.238957  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 339480us
I20260709 12:25:50.233345  9394 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.221793 10597 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:50.219532 10263 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:50.255919  9394 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:50.238512 10534 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:50.265381  1772 leader_election.cc:400] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:25:50.220561 10074 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.237407 10321 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:50.281386  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:50.283329  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83130.7us
I20260709 12:25:50.289353 10424 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:50.254061 10040 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:25:50.307006  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76994.2us
W20260709 12:25:50.308126  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 155439us
I20260709 12:25:50.307248  1731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:50.309746  1731 raft_consensus.cc:606] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.310134  1731 raft_consensus.cc:614] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:50.302109  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:50.314368  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 60178.3us
W20260709 12:25:50.315729  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.300653 10446 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:50.303144 10436 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:50.323714  1875 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:25:50.324270  9394 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:50.326287  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:50.328073 10074 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:50.357096  9394 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.328523 10564 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.332906 10321 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:50.358590 10321 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:50.335187 10424 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:50.359536  9394 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:50.325789 10076 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.338203 10450 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:50.360627 10564 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:50.340444 10421 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:50.257823 10219 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.050s
I20260709 12:25:50.323743  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:50.347496  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:50.348299 10439 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:50.351598 10600 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:50.351738 10232 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:50.357575  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55597.1us
W20260709 12:25:50.338904  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.375211  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.362550 10076 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:50.361239 10066 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.363221 10446 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:50.359462 10321 consensus_queue.cc:1243] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:25:50.362264 10534 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.403053 10534 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:50.404064 10588 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.209s	user 0.005s	sys 0.000s
I20260709 12:25:50.307881 10040 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:50.361176  1877 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:50.373219  1881 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 24
W20260709 12:25:50.408135 10593 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.205s	user 0.002s	sys 0.000s
I20260709 12:25:50.408402 10331 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:50.408854 10331 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:50.409970 10331 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.414065  1725 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:25:50.378849  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2942) took 1137 ms. Trace:
W20260709 12:25:50.418282  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70904.6us
I20260709 12:25:50.388288 10424 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:50.388501  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.418535  1804 rpcz_store.cc:276] 0709 12:25:49.237057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:49.247989 (+ 10932us) service_pool.cc:224] Handling call
0709 12:25:50.373529 (+1125540us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b5400132c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:50.374312 (+   783us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1886355200}
W20260709 12:25:50.342603 10587 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.453s	user 0.003s	sys 0.002s
I20260709 12:25:50.414023  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:50.376791  1731 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:50.421885  1731 raft_consensus.cc:606] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.373895 10439 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:50.372504 10421 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:50.414448  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:50.416972  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:50.414295  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:50.392745 10066 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:50.422405  1731 raft_consensus.cc:614] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:25:50.390631  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.424108 10564 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:25:50.411913  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:50.447201  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58799.8us
I20260709 12:25:50.448611  1723 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:50.355396 10184 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.356277 10299 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.452849  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:50.454272  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.447014 10602 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:50.448827  1727 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
W20260709 12:25:50.456387  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.462320  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:50.447654 10439 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:50.471904  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 97652.6us
W20260709 12:25:50.456234  1846 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 2.041s (SENT) [suppressed 16 similar messages]
I20260709 12:25:50.476846  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:50.478917  1728 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:25:50.478920  1726 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:50.452677 10564 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:50.481208  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 141265us
W20260709 12:25:50.498380  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:50.499364  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 119514us
W20260709 12:25:50.503325  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47137.5us
W20260709 12:25:50.536535  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.494176 10363 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.485783 10564 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:50.449420 10446 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.502978 10461 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.453521 10040 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.582659  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:50.583235  1732 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 18. Current term is 19. Ops: []
W20260709 12:25:50.581223  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 93345.4us
I20260709 12:25:50.585381  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:50.596742  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.504096 10324 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:50.601349  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 107683us
I20260709 12:25:50.581727 10066 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.584419 10313 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:50.586293 10424 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.601972 10076 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:50.604130 10542 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:50.604295 10400 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:50.606339  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 247286us
I20260709 12:25:50.579416 10232 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:50.627702 10542 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:50.663748  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 46141.3us
I20260709 12:25:50.583536 10448 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.585170 10600 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.027s
I20260709 12:25:50.633988  1881 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:50.622862 10076 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:50.581562 10363 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.640618 10400 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.611163 10040 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:50.639925  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:25:50.734284  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 102846us
I20260709 12:25:50.627447 10594 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.012s
I20260709 12:25:50.621042 10476 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.017s
I20260709 12:25:50.621973 10537 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.025s
W20260709 12:25:50.610284  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 121994us
W20260709 12:25:50.673336  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 177216us
I20260709 12:25:50.674983  2413 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260709 12:25:50.676774  1804 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:50.682824 10339 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.744638  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:50.684119  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:50.628192 10313 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:50.687180 10609 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:25:50.647502 10424 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.689477  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:50.690376  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:50.691396  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:50.691915 10612 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:25:50.693161  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:50.668803 10542 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:50.694381  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:50.694448  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:50.696954  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:50.699846 10477 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:25:50.702772  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.666179 10602 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:50.727151  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:50.621184 10066 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.611514 10461 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.620908 10324 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.736129  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:50.739410  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 155677us
W20260709 12:25:50.739548  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76344.6us
W20260709 12:25:50.739687  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.600601 10421 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.724478 10076 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:50.747030 10331 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:50.764648 10611 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:50.752862  1728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "def0036446324543ba07e132e885133a"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:50.681974 10494 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:50.755347  1807 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:25:50.758234  1800 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:50.762270  1799 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:50.763427 10424 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:50.742827 10436 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:50.765661  2419 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:50.771759  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.773187  9394 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.009s
I20260709 12:25:50.758853 10450 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.040s
I20260709 12:25:50.808943  1727 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:50.753655 10476 consensus_queue.cc:1243] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 24.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:25:50.787047  2415 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:50.765395 10184 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:50.737818 10448 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.747624 10074 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:50.801005  1724 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:50.736644 10610 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.075s
I20260709 12:25:50.804454  2411 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:50.771528 10477 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:50.782020 10602 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:50.808223  1728 raft_consensus.cc:606] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.815150  1725 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:25:50.813916 10436 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:50.755008 10339 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.839651  1728 raft_consensus.cc:614] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:50.817150  1804 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:25:50.842900  1726 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:50.819420 10597 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.513s	user 0.008s	sys 0.003s
I20260709 12:25:50.807886 10611 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:50.863090  1729 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260709 12:25:50.787956 10601 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.035s
W20260709 12:25:50.864904  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72545us
I20260709 12:25:50.871323  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:50.884713 10299 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:50.855228 10436 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:50.890264  1725 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:25:50.842839 10602 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:50.891012  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:50.897955  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 56692.7us
I20260709 12:25:50.843478 10600 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.889648  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:50.897109  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 137439us
I20260709 12:25:50.867836 10611 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:50.912474  1727 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:50.885946 10597 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:25:50.806274 10618 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.080s
I20260709 12:25:50.914848  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:50.917121  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:50.924119  1806 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:25:50.905854 10421 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:50.916819  1799 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:50.927897  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:50.925678  1618 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.216s	user 0.005s	sys 0.001s
I20260709 12:25:50.955473 10324 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:50.955927 10339 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:50.959029 10616 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.959685 10616 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.964535  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:50.966287  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:50.967844 10066 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:50.927904 10600 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:50.969914  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:50.970366 10494 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:50.970968  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.971441  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:50.973791  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:50.974450  1876 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 17.
W20260709 12:25:50.959029  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 59742us
I20260709 12:25:50.976174  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:50.978204  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:50.954639  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:50.969189  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:50.982662  1872 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:50.977574  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:51.008634  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.033182  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:51.032372  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:51.048012  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:51.048790  1873 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:51.047645  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:51.050657  1874 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:51.031090  1806 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:51.062990  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:50.975155 10461 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:50.985509 10324 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:51.063007  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:51.051369  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:51.073597  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.081615  1807 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:51.031477 10363 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.044480 10614 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:51.045550 10450 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:51.095047  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.045697 10232 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:51.031174 10184 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:51.076879 10439 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.588s	user 0.013s	sys 0.000s
I20260709 12:25:51.116358  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:51.116245  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.171718  1876 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:51.163528  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:51.173081  1874 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:25:51.173511  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:51.063151 10601 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 12:25:51.029785 10627 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.089s
I20260709 12:25:51.166810  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.166903  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.167920  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.181972  1806 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
W20260709 12:25:51.182293  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 204724us
I20260709 12:25:50.943966 10477 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.171785  1798 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
W20260709 12:25:51.106928  1696 connection.cc:256] RPC call timeout handler was delayed by -2.1478215799997997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16790000000000002s for a response
W20260709 12:25:51.193408  1696 connection.cc:256] RPC call timeout handler was delayed by -1.9714215799999693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1875s for a response
W20260709 12:25:51.193794  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4180361180001455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16010000000000002s for a response
I20260709 12:25:51.029753 10624 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.335s
I20260709 12:25:51.118204 10450 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:51.192040  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:51.204574  1806 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:51.194074  1696 connection.cc:256] RPC call timeout handler was delayed by -1.3307361180000044s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1698s for a response
I20260709 12:25:51.197783  1800 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:51.152647 10610 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:51.205108  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1997766330000559s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
W20260709 12:25:51.206940  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0170656569998755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2601s for a response
I20260709 12:25:51.123428 10321 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
W20260709 12:25:51.171044 10534 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.767s	user 0.017s	sys 0.000s
I20260709 12:25:51.211447  1803 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260709 12:25:51.215130  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:51.176975 10618 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.163050 10040 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.050s
I20260709 12:25:51.224164  1875 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:51.226976  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:51.234807  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.121387 10614 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:51.252532  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 275160us
I20260709 12:25:51.202838 10537 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.260749  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:51.263577  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.221710 10313 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.192288 10184 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:51.300473  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.312469  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.291078  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.251781 10477 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:51.319458  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58788us
I20260709 12:25:51.209254 10610 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:51.202186 10439 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.324090  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:51.254681 10339 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:51.326565  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:51.328403  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 124487us
I20260709 12:25:51.334098  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:51.334997  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:51.329514 10564 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.731s	user 0.011s	sys 0.004s
I20260709 12:25:51.326364 10313 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.323436 10632 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:25:51.353434  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 136619us
I20260709 12:25:51.353993  1801 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:51.361647 10598 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:51.394277 10632 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:25:51.394639  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:51.365756  1803 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:51.371798  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:51.399159 10542 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.607s	user 0.011s	sys 0.000s
I20260709 12:25:51.399916 10542 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:51.374054 10639 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:25:51.375591 10648 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.404115 10648 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.405558  1729 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:25:51.375742  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:51.407938   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:25:51.408229 10074 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:51.408828 10074 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:51.409765 10597 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:51.410360 10597 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.410818 10597 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:25:51.411625 10638 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
W20260709 12:25:51.414448  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 80420.7us
I20260709 12:25:51.415269  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:51.376533 10061 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.377772  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:51.379020  1731 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:51.380417  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:51.417225  1725 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:51.417382  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:51.381127  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:51.380899 10651 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:51.381290  1730 raft_consensus.cc:1217] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Deduplicated request from leader. Original: 21.12->[22.13-25.15]   Dedup: 22.13->[23.14-25.15]
I20260709 12:25:51.428772  1877 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:51.382067 10652 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.383659  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:51.384443  1728 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:51.385546 10400 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.388763 10439 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:51.394563 10598 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:51.364566 10594 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:51.354903 10564 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.496467  2410 raft_consensus.cc:2368] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25.
I20260709 12:25:51.372884 10448 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.462540 10534 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:51.249374 10618 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:51.432556  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.366647  1724 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:25:51.313824 10537 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.438477  1727 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:51.442085  1806 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:51.573964  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.443497  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:51.443118 10652 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.418721 10450 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.435576   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:51.444348  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:51.445163  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:51.445367 10061 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:51.451143 10076 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.632s	user 0.011s	sys 0.001s
I20260709 12:25:51.452945  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.454613 10299 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:51.455618 10553 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:51.629812 10553 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:51.630968 10606 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:51.631234 10553 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.631740 10606 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:51.632606 10553 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:51.633004 10606 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.459125 10601 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:25:51.636437 10601 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:25:51.464104 10635 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.394s	user 0.000s	sys 0.001s
W20260709 12:25:51.466231 10646 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.200s	user 0.005s	sys 0.000s
I20260709 12:25:51.467922 10494 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.638705 10494 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.468899 10476 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:25:51.641981 10476 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:51.406698 10648 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.429536  1730 pending_rounds.cc:85] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
I20260709 12:25:51.643502  1730 pending_rounds.cc:107] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:25:51.457930  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:51.643839 10645 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.207s	user 0.000s	sys 0.000s
I20260709 12:25:51.644400  1725 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:51.645517  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:51.489674  1873 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:51.646123 10542 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:51.496162  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:51.647125  1798 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
W20260709 12:25:51.497833  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75874.1us
I20260709 12:25:51.420830  2382 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:51.409976 10074 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.499140  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:51.649118  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 228406us
I20260709 12:25:51.568277  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:51.648339  2414 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.747s	user 0.012s	sys 0.002s
W20260709 12:25:51.650017  2414 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.757s	user 0.013s	sys 0.002s
I20260709 12:25:51.650041  1724 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:51.643877  1730 raft_consensus.cc:2889] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:51.651670  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:51.648600  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:51.653962  1724 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:25:51.439450 10313 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.611588  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:51.612293  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:51.615909 10652 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.542258 10600 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:51.494053 10651 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.620410  1732 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:51.582484  2411 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:51.561543 10534 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:51.430421  1874 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
W20260709 12:25:51.610252  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.650584  1807 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:51.573998 10400 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:25:51.658670  1878 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.000s	sys 0.015s
I20260709 12:25:51.673651  2416 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 27: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:51.659921  2415 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:51.674546  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:51.674873  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.011s	sys 0.005s
I20260709 12:25:51.620519 10653 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:51.680434  1844 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.622s	user 0.010s	sys 0.009s
I20260709 12:25:51.457896  1622 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 17 to 19, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 19 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:51.694729  1844 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.010s	sys 0.009s
W20260709 12:25:51.695453  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:51.695453  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 280244us
W20260709 12:25:51.678222 10641 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.201s	user 0.000s	sys 0.000s
W20260709 12:25:51.692713  1881 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.003s	sys 0.015s
W20260709 12:25:51.690544  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.012s	sys 0.005s
W20260709 12:25:51.702198  1881 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.004s	sys 0.015s
W20260709 12:25:51.692344 10219 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.004s	sys 0.014s
W20260709 12:25:51.693785  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 296815us
W20260709 12:25:51.702519  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:51.703725  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:51.704221  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.677106 10439 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:51.684496 10232 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:25:51.698686 10331 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.890s	user 0.012s	sys 0.001s
I20260709 12:25:51.691046 10624 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:51.650849 10076 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.732947  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:51.734257  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 287572us
W20260709 12:25:51.734602  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.749722  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:51.733685  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:51.707954 10219 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.634s	user 0.004s	sys 0.014s
I20260709 12:25:51.735359  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:51.750579  1724 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:25:51.682068 10650 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260709 12:25:51.735980 10624 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:51.755929  1879 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:51.711483 10439 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.760056 10624 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:51.759922  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 54654.4us
I20260709 12:25:51.764290  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1547) took 1074 ms. Trace:
I20260709 12:25:51.764770 10076 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.766971 10642 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:51.767539 10649 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:51.772202 10666 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:51.774755 10602 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.855s	user 0.010s	sys 0.002s
I20260709 12:25:51.775318 10066 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.775559 10665 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:51.784727 10061 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.829248  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:51.695003 10651 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:51.793449  1799 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.853s	user 0.010s	sys 0.002s
I20260709 12:25:51.835799  1799 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:51.789461  2410 rpcz_store.cc:276] 0709 12:25:50.689587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.693885 (+  4298us) service_pool.cc:224] Handling call
0709 12:25:51.763933 (+1070048us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b54005609d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:51.764200 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1672405504}
W20260709 12:25:51.795194  1872 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.751s	user 0.011s	sys 0.001s
I20260709 12:25:51.723752 10232 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:51.751685 10074 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:51.735170 10494 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.750439 10184 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:51.819784  2419 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:51.794245 10439 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:51.823966  1724 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27. Ops: [26.18-26.18]
W20260709 12:25:51.833799 10436 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.920s	user 0.012s	sys 0.000s
I20260709 12:25:51.830544  2418 raft_consensus.cc:2368] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:25:51.708534 10400 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.804801  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:51.734122 10600 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:25:51.673862 10598 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:51.798969 10649 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.837930  1872 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:51.709908 10476 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:51.798030 10632 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.856528 10263 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:51.793829 10542 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.841851 10665 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.859177  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:51.859097  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:51.844026 10446 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.774s	user 0.005s	sys 0.009s
I20260709 12:25:51.859503  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:51.819983 10602 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:51.863034  1725 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:51.863117  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:51.941891  1873 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:51.830207 10614 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:51.864183  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:51.866582 10637 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.867216 10644 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:51.867831 10647 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:51.867242 10339 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.531s	user 0.012s	sys 0.000s
I20260709 12:25:51.989684 10339 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:51.839231 10424 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:25:51.875388 10673 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:51.991179 10673 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:51.992190 10313 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:51.992357 10673 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:51.897639  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.995275 10400 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:51.845458  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 110184us
I20260709 12:25:51.908694 10616 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:51.912151  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.912767 10671 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.997730 10671 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.850365 10610 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:51.999229 10640 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:52.000484 10640 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:52.001652 10643 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:52.002099 10643 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:52.003255  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:51.925386 10632 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:51.917479  9394 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:52.004482  9394 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:52.006017 10632 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:51.857527  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:51.829309 10066 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:51.919162  1727 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:51.917351 10263 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:52.017084  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:52.018601  1727 raft_consensus.cc:2410] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:25:51.879446 10672 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:51.888419 10564 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:51.947556  1877 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:51.831135 10534 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:51.946653 10537 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:51.894543 10649 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:51.977051  1726 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:52.040620  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:52.041407  1726 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:52.043001 10676 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:52.043586 10676 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.043963 10676 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:52.044543  1622 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 17 to 18, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 18 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:51.612140   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:25:52.048161 10662 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:25:51.916229 10324 consensus_queue.cc:1175] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 18 status { last_received { term: 18 index: 12 } last_committed_idx: 11 last_received_current_leader { term: 18 index: 12 } }
I20260709 12:25:52.049468  1622 catalog_manager.cc:5697] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:25:51.900848 10461 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:51.944419 10446 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.006s	sys 0.009s
I20260709 12:25:51.980726 10674 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:25:51.983860  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 164060us
I20260709 12:25:51.824285 10666 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:51.919987 10542 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:51.922086  1872 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:25:52.007164  1724 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:25:52.081123  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:52.081981  1724 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:52.006743  9394 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:52.011655  1880 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:52.024778  1805 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
W20260709 12:25:52.028112  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:51.865651 10436 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:51.906407  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 172855us
I20260709 12:25:52.030862  1730 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:52.035301  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:52.099524  1727 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:52.035910 10672 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:52.059788  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:52.076900  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3083) took 1099 ms. Trace:
I20260709 12:25:52.028862 10313 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.023079 10618 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:52.148586  1881 rpcz_store.cc:276] 0709 12:25:50.952980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.953234 (+   254us) service_pool.cc:224] Handling call
0709 12:25:50.991463 (+ 38229us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:51.703020 (+711557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:51.703076 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:51.758452 (+ 55376us) log.cc:844] Serialized 26 byte log entry
0709 12:25:52.051661 (+293209us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:52.051708 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.051784 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:52.051831 (+    47us) raft_consensus.cc:1685] finished
0709 12:25:52.051841 (+    10us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:52.052004 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25851,"raft.queue_time_us":182677,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":21598,"thread_start_us":824014,"threads_started":2,"wal-append.queue_time_us":249174}
I20260709 12:25:52.175139  1727 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:52.219858  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:52.220397  2383 connection.cc:256] RPC call timeout handler was delayed by -1.0927438560001974s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2174s for a response
I20260709 12:25:52.246539  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:52.247767  1802 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:52.057044 10674 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:52.056679 10653 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:52.024284 10644 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:52.271299  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:52.378438  1798 raft_consensus.cc:2410] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 25 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:25:52.272444  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:52.279034 10446 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:52.225344 10313 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:52.011406 10637 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:52.284941 10219 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.288978 10672 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:52.303270 10666 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:52.307461 10602 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:52.310500 10611 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.389s	user 0.010s	sys 0.004s
I20260709 12:25:52.312923 10671 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:52.313835 10263 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:52.316848  1621 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 259) took 34097 ms (client timeout 15000 ms). Trace:
I20260709 12:25:52.323719 10066 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:52.402981 10674 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:52.327625 10665 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:52.328878  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 79556.2us
I20260709 12:25:52.330862 10651 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:52.395632  1726 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:52.017212 10647 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:52.486145  1845 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.014s	sys 0.030s
I20260709 12:25:52.071749 10461 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:52.502424 10219 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:52.509007 10637 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:52.326062 10542 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:52.557694  1845 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.014s	sys 0.031s
W20260709 12:25:52.581023  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 806313us
I20260709 12:25:52.585319 10219 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:52.585675 10647 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:52.575342 10263 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:52.538619 10611 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:52.593751 10313 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:52.615507  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.593757 10461 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:52.625329  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:52.657891  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:52.658804  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.176220 10643 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:52.322268  1622 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 10 to 17, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 17 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:25:52.614179 10649 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.576s	user 0.013s	sys 0.000s
I20260709 12:25:52.724253  1728 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:52.745628  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 524125us
I20260709 12:25:52.750953  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:52.721242 10597 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.310s	user 0.008s	sys 0.009s
W20260709 12:25:52.756042  1724 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.000s
I20260709 12:25:52.725391 10461 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.756759 10542 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:52.763612 10677 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
I20260709 12:25:52.770135  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:52.779507  1724 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:25:52.761545  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:52.776156  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:52.774222 10421 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.829s	user 0.007s	sys 0.010s
I20260709 12:25:52.779884  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:52.485769 10446 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:52.816489  2414 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:52.819453  2418 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:25:52.814527  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.083s	user 0.040s	sys 0.131s
W20260709 12:25:52.822235  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.091s	user 0.040s	sys 0.132s
I20260709 12:25:52.748463 10649 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:52.814756  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:52.829161  1728 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
W20260709 12:25:52.829082  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48206us
I20260709 12:25:52.752365 10673 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
W20260709 12:25:52.792801 10606 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.743s	user 0.091s	sys 0.145s
I20260709 12:25:52.759087 10597 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:52.813241 10436 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:52.841673  1727 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.030s	sys 0.047s
W20260709 12:25:52.842281  1727 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.030s	sys 0.048s
W20260709 12:25:52.844673  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 83322.5us
I20260709 12:25:52.825469 10594 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.827953 10537 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:52.850449  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:52.853448  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:52.853957 10653 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:52.854434  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3912) took 1009 ms. Trace:
W20260709 12:25:52.836839 10606 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.092s	sys 0.145s
I20260709 12:25:52.836534 10421 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.857590  1873 rpcz_store.cc:276] 0709 12:25:51.844500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.987287 (+142787us) service_pool.cc:224] Handling call
0709 12:25:51.987792 (+   505us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:51.988214 (+   422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:51.988241 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:51.988902 (+   661us) log.cc:844] Serialized 52 byte log entry
0709 12:25:52.245974 (+257072us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:52.246040 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.246134 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:52.854155 (+608021us) raft_consensus.cc:1685] finished
0709 12:25:52.854178 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:52.854361 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1324,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"thread_start_us":213987,"threads_started":1,"wal-append.queue_time_us":754951}
I20260709 12:25:52.859905  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1292) took 1502 ms. Trace:
I20260709 12:25:52.860392  1801 rpcz_store.cc:276] 0709 12:25:51.353329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.353447 (+   118us) service_pool.cc:224] Handling call
0709 12:25:51.353815 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:51.394260 (+ 40445us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.855716 (+1461456us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54005003a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abddba02 00007f16abdd7dd6 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:25:52.855970 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":240454528}
I20260709 12:25:52.827473 10461 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:52.868193 10685 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.879s	user 0.020s	sys 0.028s
I20260709 12:25:52.869300 10461 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:52.868170 10649 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:52.870980  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3175) took 1075 ms. Trace:
I20260709 12:25:52.819602 10542 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:52.872326  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 3991) took 1462 ms. Trace:
I20260709 12:25:52.864234 10421 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:52.866917 10669 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:52.871414  1731 rpcz_store.cc:276] 0709 12:25:51.796080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.796228 (+   148us) service_pool.cc:224] Handling call
0709 12:25:51.822341 (+ 26113us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:51.834103 (+ 11762us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:51.834153 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:51.834755 (+   602us) log.cc:844] Serialized 26 byte log entry
0709 12:25:51.939631 (+104876us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:51.941236 (+  1605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.941325 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:52.867515 (+926190us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:52.870587 (+  3072us) raft_consensus.cc:1685] finished
0709 12:25:52.870615 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:52.870906 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15659,"raft.queue_time_us":11362,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":18040,"spinlock_wait_cycles":73131904,"thread_start_us":64315,"threads_started":1,"wal-append.queue_time_us":130034}
I20260709 12:25:52.850860 10690 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.872807  1729 rpcz_store.cc:276] 0709 12:25:51.409775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.409979 (+   204us) service_pool.cc:224] Handling call
0709 12:25:51.410402 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:51.692809 (+282407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:51.692874 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:51.698902 (+  6028us) log.cc:844] Serialized 26 byte log entry
0709 12:25:52.190870 (+491968us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:25:52.190911 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:52.190996 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:52.872077 (+681081us) raft_consensus.cc:1685] finished
0709 12:25:52.872103 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:52.872249 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":56947,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"thread_start_us":719061,"threads_started":2,"wal-append.queue_time_us":773165}
I20260709 12:25:52.852070 10537 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:52.888059  1725 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.905s	user 0.001s	sys 0.010s
I20260709 12:25:52.888895  1725 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:52.900331 10597 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:52.909529 10542 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:52.910475  2413 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:25:52.852283 10594 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:52.913412  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3086) took 1949 ms (client timeout 2020 ms). Trace:
W20260709 12:25:52.935308  1875 rpcz_store.cc:269] 0709 12:25:50.963982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.966005 (+  2023us) service_pool.cc:224] Handling call
0709 12:25:52.913136 (+1947131us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b5400260a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:52.913314 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":590904192}
I20260709 12:25:52.926265  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:52.937505  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:52.892277 10537 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:52.947135  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:52.929515 10624 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.099s	user 0.012s	sys 0.003s
I20260709 12:25:52.948827 10624 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:52.856211  1622 catalog_manager.cc:5697] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 10 to 19, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 19 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:52.951838 10446 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:52.891791 10690 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:52.859974 10436 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
W20260709 12:25:52.954581  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:52.909701 10643 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:52.893806 10669 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:52.914875 10611 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:52.952575 10477 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.616s	user 0.006s	sys 0.006s
I20260709 12:25:52.966955  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:52.967039 10606 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:25:52.919157 10644 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:52.974023 10706 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:52.983563  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:52.984506  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 149572us
W20260709 12:25:52.985522  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 119934us
I20260709 12:25:52.987663 10698 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:52.988456 10698 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:52.986579 10597 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:52.991827 10597 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:52.992240 10597 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:52.969470 10446 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:52.969666 10678 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:53.001578  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.001670 10678 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:52.969823 10671 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:53.006727  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.007225 10671 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.008831  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:53.009680  1801 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:53.009855  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.011025  1798 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:25:53.020994  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.021910  1798 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:53.038493  1801 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
W20260709 12:25:53.036469  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.049690  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:53.052603  1726 raft_consensus.cc:2368] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:25:52.996888 10672 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:25:52.999583 10611 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:53.046061 10624 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:53.065026 10439 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.185s	user 0.009s	sys 0.006s
I20260709 12:25:53.065757 10439 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.068046 10669 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.071089  1622 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 14 to 19. New cstate: current_term: 19 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:53.046725 10446 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.076996 10446 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:53.078310 10598 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:53.078725 10690 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:25:53.079751 10421 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:53.073793  1622 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 13 to 17, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 17 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:52.532562  1621 rpcz_store.cc:269] 0709 12:25:18.219573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.307904 (+ 88331us) service_pool.cc:224] Handling call
0709 12:25:52.316527 (+34008623us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:43.200066 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:43.284366 (+ 84300us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:43.284439 (+    73us) write_op.cc:435] Acquired schema lock
0709 12:25:43.284460 (+    21us) tablet.cc:662] Decoding operations
0709 12:25:43.303785 (+ 19325us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:43.303923 (+   138us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:43.303970 (+    47us) tablet.cc:685] Acquiring locks for 23 operations
0709 12:25:43.399928 (+ 95958us) tablet.cc:701] Row locks acquired
0709 12:25:43.399998 (+    70us) write_op.cc:260] PREPARE: finished
0709 12:25:43.400277 (+   279us) write_op.cc:270] Start()
0709 12:25:43.400480 (+   203us) write_op.cc:276] Timestamp: P: 1783599943400190 usec, L: 0
0709 12:25:43.400526 (+    46us) op_driver.cc:348] REPLICATION: starting
0709 12:25:43.402599 (+  2073us) log.cc:844] Serialized 9016 byte log entry
0709 12:25:44.387281 (+984682us) op_driver.cc:464] REPLICATION: finished
0709 12:25:45.697372 (+1310091us) write_op.cc:301] APPLY: starting
0709 12:25:45.711912 (+ 14540us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:47.216596 (+1504684us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:47.217589 (+   993us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:47.903553 (+685964us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:47.909018 (+  5465us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=46,key_file_lookups=46,delta_file_lookups=81,mrs_lookups=0
0709 12:25:47.909513 (+   495us) write_op.cc:312] APPLY: finished
0709 12:25:47.914685 (+  5172us) log.cc:844] Serialized 204 byte log entry
0709 12:25:48.109077 (+194392us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:48.117453 (+  8376us) write_op.cc:454] Released schema lock
0709 12:25:48.128070 (+ 10617us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:48.128594 (+   524us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:48.128670 (+    76us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3934126,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":1231957,"apply.run_cpu_time_us":61770,"apply.run_wall_time_us":2456621,"cfile_cache_hit":94,"cfile_cache_hit_bytes":100359,"num_ops":23,"prepare.queue_time_us":149537,"prepare.run_cpu_time_us":31434,"prepare.run_wall_time_us":454093,"raft.queue_time_us":352438,"raft.run_cpu_time_us":52081,"raft.run_wall_time_us":473946,"replication_time_us":946682,"thread_start_us":984102,"threads_started":5,"wal-append.queue_time_us":454453,"wal-append.run_cpu_time_us":48094,"wal-append.run_wall_time_us":2633337}]]}
I20260709 12:25:53.102272  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:53.105882  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 158837us
I20260709 12:25:53.107831  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.103797  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:53.113474  2414 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 26 because replica has last-logged OpId of term: 25 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 19.
I20260709 12:25:53.055370 10653 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:53.120081  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:53.125701  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:53.132074  2410 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:53.133538  2412 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:25:53.133843 10653 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:53.134318  1622 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 13 to 20. New cstate: current_term: 20 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:53.135358  2417 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:25:53.136411  1622 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 14 to 15, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 15 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:53.133452 10698 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:53.151470  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.160213 10439 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:53.161811 10676 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.117s	user 0.017s	sys 0.000s
I20260709 12:25:53.162341 10676 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.164803  1799 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:52.975584  1618 maintenance_manager.cc:643] P 4002daa3304c4fb4be053abd4780c195: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 4.652s	user 0.098s	sys 0.047s Metrics: {"cfile_cache_miss":8,"cfile_cache_miss_bytes":25285,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":126130,"dirs.run_cpu_time_us":787,"dirs.run_wall_time_us":1663,"lbm_read_time_us":2272,"lbm_reads_lt_1ms":24,"lbm_write_time_us":52623,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":8,"peak_mem_usage":251732,"reinsert_count":0,"thread_start_us":140204,"threads_started":3,"update_count":148,"wal-append.queue_time_us":124889,"wal-append.run_cpu_time_us":1106,"wal-append.run_wall_time_us":1103104}
I20260709 12:25:53.102496 10537 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.138759  1622 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 19 to 21, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 21 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:25:53.169800  2417 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260709 12:25:53.184660  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55714.5us
W20260709 12:25:53.168884  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:53.170168  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.181782  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:53.168115  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.173296  1619 maintenance_manager.cc:419] P 4002daa3304c4fb4be053abd4780c195: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
W20260709 12:25:53.186437  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.186829  1801 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:53.122476 10594 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:53.194406  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.209002  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:53.184113 10674 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.624s	user 0.013s	sys 0.000s
I20260709 12:25:53.199884  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:53.203032  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:53.118378 10477 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:53.211612  1807 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:53.213294  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.176106 10537 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:53.165689 10676 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:53.184465 10706 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:53.227784 10706 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.234905 10690 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.189335 10446 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:25:53.235555 10690 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.236735  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.218497  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.184664 10654 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:53.221920  1618 maintenance_manager.cc:643] P 4002daa3304c4fb4be053abd4780c195: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s	user 0.000s	sys 0.000s Metrics: {}
I20260709 12:25:53.224524 10477 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:53.237564  2412 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:25:53.240926  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:53.185863 10606 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:53.223730 10669 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:53.248957  1622 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 14 to 17, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 17 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:53.251672  1622 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 12 to 20. New cstate: current_term: 20 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:53.254492  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.244120  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.199258 10421 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.054s
I20260709 12:25:53.218873 10674 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.255363  1806 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:53.255861 10710 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:53.256532 10299 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.257403 10674 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:53.258041 10709 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.258625 10709 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.259737 10450 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:53.257179 10299 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.256500  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.255280  1800 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:53.261644  1801 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:53.202759 10698 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:25:53.306500  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:53.277680  1727 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:25:53.301132  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:53.297900  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.243579  2419 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:53.316099  1732 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20260709 12:25:53.318315  1725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 3166) took 1880 ms (client timeout 2247 ms). Trace:
W20260709 12:25:53.318817  1725 rpcz_store.cc:269] 0709 12:25:51.428045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.645164 (+217119us) service_pool.cc:224] Handling call
0709 12:25:53.308011 (+1662847us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54003ccdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:53.308134 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":498690944}
I20260709 12:25:53.326241  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:53.275146 10703 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.328795  1802 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:53.332271  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.335450  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:53.336208  1801 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:53.336813  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.254113  1622 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 7 to 12, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 12 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:25:53.306653 10710 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:53.342135  1807 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:53.351784  1801 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:25:53.353215  1801 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:53.354230  1801 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:53.321449 10537 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:53.346321  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:53.356954  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.358736  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:53.254931 10702 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.081s
I20260709 12:25:53.362162  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.362856  1800 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:53.255254 10654 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.372303 10710 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.346763 10703 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.359112 10653 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:53.360761 10643 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:53.366431 10642 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:53.378589 10706 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:53.370607 10537 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:53.382907 10703 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:53.360981 10597 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.419543  1798 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:25:53.406317 10710 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:53.417726 10653 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:53.407778 10642 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:53.414502 10674 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.380957 10477 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.429068 10654 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:53.377128 10606 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.405236 10643 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:53.435191  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 109047us
I20260709 12:25:53.442745  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:53.445490  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:53.447448  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:53.430956 10653 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.451603 10706 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:53.452114 10706 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:53.452486 10706 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:53.453843 10653 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:53.462864  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:53.447536 10710 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:53.461774 10703 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:53.468135  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:53.435184 10674 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.471859  1622 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 19 to 20, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 20 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:53.467159 10718 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:53.468575 10719 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
W20260709 12:25:53.517768 10717 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
W20260709 12:25:53.520982  2411 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1465) took 2537 ms (client timeout 1605 ms). Trace:
W20260709 12:25:53.521499  2411 rpcz_store.cc:269] 0709 12:25:50.983888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.984193 (+   305us) service_pool.cc:224] Handling call
0709 12:25:53.520753 (+2536560us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b54005438d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:53.520868 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1168761088}
I20260709 12:25:53.521883  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.522509  2418 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:53.520982  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.526136 10671 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:53.526738 10476 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:53.527926  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.467156 10537 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:53.534389 10636 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.638s	user 0.001s	sys 0.000s
W20260709 12:25:53.540468  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 72422.8us
I20260709 12:25:53.512717 10653 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:53.543869  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:53.480609 10642 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.548518  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:53.546463  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:53.556489 10723 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:25:53.554940 10719 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:53.561213  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1466) took 2190 ms. Trace:
I20260709 12:25:53.561700  2415 rpcz_store.cc:276] 0709 12:25:51.371288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.371480 (+   192us) service_pool.cc:224] Handling call
0709 12:25:53.560984 (+2189504us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b5400572e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:53.561145 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":602819456}
I20260709 12:25:53.562433 10710 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:53.562059 10720 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:53.563026  2410 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
W20260709 12:25:53.565677  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.571715 10706 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.574319 10706 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:53.572351 10663 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.569s	user 0.011s	sys 0.014s
I20260709 12:25:53.580699 10643 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:53.581225 10713 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.250s	user 0.002s	sys 0.000s
I20260709 12:25:53.581316 10690 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:53.585549 10299 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:53.585870  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.586654  2414 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:53.574319  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1297) took 1475 ms. Trace:
I20260709 12:25:53.587267  1799 rpcz_store.cc:276] 0709 12:25:52.099054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.099252 (+   198us) service_pool.cc:224] Handling call
0709 12:25:52.174976 (+ 75724us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:53.165247 (+990271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.574028 (+408781us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b5400167bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:53.574227 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2275900800}
I20260709 12:25:53.588522 10534 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:53.590324  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:53.591370 10720 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:53.592058 10720 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:53.593302 10720 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.597401  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.597504  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.598306  1731 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:53.598958  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.599531  1728 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:53.602136  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.563035 10710 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:53.604188  1799 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:25:53.603910 10710 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.598403  1730 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:25:53.607235  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:53.607332  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.608131  1806 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
W20260709 12:25:53.608860  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.610656  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:53.611029  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.611508  1806 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:53.614107  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.616776  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:53.617622  1807 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
W20260709 12:25:53.611680  2382 connection.cc:256] RPC call timeout handler was delayed by -1.1701888109996617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16180000000000003s for a response
I20260709 12:25:53.621767  1806 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:53.602833  1729 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
W20260709 12:25:53.622247  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 229118us
I20260709 12:25:53.620972  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:53.625200  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:53.625780  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.626812  2412 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:53.628357  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:53.631242 10611 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:53.632150 10611 consensus_queue.cc:1243] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 21.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:25:53.632695 10706 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:53.635519  2382 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:53.638271  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:53.626649  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:53.655130  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:53.655810  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1412) took 2927 ms. Trace:
I20260709 12:25:53.552675 10597 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:53.555402 10653 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:53.623240 10436 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:53.656493  1723 rpcz_store.cc:276] 0709 12:25:50.728370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.728488 (+   118us) service_pool.cc:224] Handling call
0709 12:25:50.740217 (+ 11729us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:50.758726 (+ 18509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:50.758799 (+    73us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:50.798706 (+ 39907us) log.cc:844] Serialized 78 byte log entry
0709 12:25:50.940406 (+141700us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:50.968728 (+ 28322us) log.cc:844] Serialized 14 byte log entry
0709 12:25:50.969127 (+   399us) log.cc:844] Serialized 14 byte log entry
0709 12:25:51.120937 (+151810us) log.cc:844] Serialized 14 byte log entry
0709 12:25:51.121128 (+   191us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:51.121908 (+   780us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:51.454250 (+332342us) spinlock_profiling.cc:243] Waited 8.18 ms on lock 0x7b4800041c58. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:25:51.467452 (+ 13202us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b4800041c58. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:25:52.896273 (+1428821us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x7b180033f460. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:53.533389 (+637116us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:53.655467 (+122078us) raft_consensus.cc:1685] finished
0709 12:25:53.655501 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.655725 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9168,"prepare.queue_time_us":31960,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":3783,"raft.queue_time_us":155847,"raft.run_cpu_time_us":9378,"raft.run_wall_time_us":1016741,"spinlock_wait_cycles":112729344,"thread_start_us":213708,"threads_started":2,"wal-append.queue_time_us":312590}
I20260709 12:25:53.664825  1802 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:25:53.649219  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:53.649363 10702 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:53.633167 10706 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:53.649744 10263 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:53.680985 10436 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:53.703375 10702 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:53.780766  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 143242us
W20260709 12:25:53.781666  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 98960.9us
W20260709 12:25:53.756831  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 101827us
W20260709 12:25:53.780128  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 130869us
I20260709 12:25:53.715444 10299 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:53.762245 10436 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:53.791518  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:53.781256 10714 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:53.790381  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:53.749231 10719 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:53.783463 10263 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.783718 10702 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:53.782368 10703 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:53.803097  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2945) took 2766 ms. Trace:
I20260709 12:25:53.792413  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:53.823307  1802 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
W20260709 12:25:53.824745  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.826522  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:53.788156 10659 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.026s
I20260709 12:25:53.791740 10720 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:53.797025 10299 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.827304  1730 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:53.807721  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:53.831519  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:53.831131 10719 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:53.821717  1805 rpcz_store.cc:276] 0709 12:25:51.031646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:51.048022 (+ 16376us) service_pool.cc:224] Handling call
0709 12:25:51.081574 (+ 33552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:52.174429 (+1092855us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.797913 (+1623484us) spinlock_profiling.cc:243] Waited 943 ms on lock 0x7b54003b88d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:53.798132 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2168600832}
W20260709 12:25:53.837605  2383 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.528s (SENT)
I20260709 12:25:53.839511  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:53.842842  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:53.813819 10714 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:53.816283 10731 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:53.847635  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.860334  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:53.851413  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:53.851861  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.862607  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:53.832213 10742 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:53.863628  1729 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:25:53.866817  2410 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 19: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:53.828531 10671 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.868366  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.871371  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.872211  1724 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
W20260709 12:25:53.876390  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.81652e+06us
I20260709 12:25:53.815290 10439 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:25:53.844578 10299 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:53.868300 10725 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.329s	user 0.002s	sys 0.000s
I20260709 12:25:53.861966 10714 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:53.915861  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:53.878756  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:53.932819  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.821817 10263 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.860204 10731 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.929003  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3096) took 2922 ms. Trace:
W20260709 12:25:53.941704  1844 connection.cc:256] RPC call timeout handler was delayed by -1.4592800240002362s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
I20260709 12:25:53.941829  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.942503  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:53.944492  1879 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:53.802389 10706 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:53.864825 10742 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:53.870610 10611 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:53.943593  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:53.934525  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:53.809913 10653 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:53.942198  1844 connection.cc:256] RPC call timeout handler was delayed by -1.4448800240002129s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
I20260709 12:25:53.861737 10640 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.069s
I20260709 12:25:53.960064  1880 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
W20260709 12:25:53.978874  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.979210  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:53.870394 10421 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.079s
I20260709 12:25:53.959111  1881 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 30 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:25:53.915823 10720 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:53.983013  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:53.973466  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:53.867512 10654 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.075s
I20260709 12:25:53.936646 10610 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:53.974362  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:53.913038 10740 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:54.068372  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:53.912523 10671 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.002682 10634 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:54.021311 10728 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.223s	user 0.000s	sys 0.002s
I20260709 12:25:53.953574 10665 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:53.943692 10731 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:53.967499 10637 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.003005  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.026466  1731 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:54.077668  1872 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:54.024403  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:53.959520 10647 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:54.004473 10727 log.cc:927] Time spent T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.272s	user 0.001s	sys 0.000s
I20260709 12:25:53.960937 10742 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:54.076663 10634 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.071660 10076 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:54.087168 10076 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.089656 10076 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:54.081579 10740 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:53.978964  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.091140  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:54.027779  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:54.093180 10637 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.095742 10637 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:54.096232 10746 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 11, Time since last communication: 0.015s
I20260709 12:25:54.068408 10631 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:54.029320  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:53.984160  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:53.983920  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.022368 10706 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:54.050884 10745 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:53.939652  1878 rpcz_store.cc:276] 0709 12:25:50.989943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:50.990210 (+   267us) service_pool.cc:224] Handling call
0709 12:25:51.062709 (+ 72499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:51.105208 (+ 42499us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:51.694131 (+588923us) log.cc:844] Serialized 14 byte log entry
0709 12:25:52.078167 (+384036us) log.cc:844] Serialized 14 byte log entry
0709 12:25:53.064792 (+986625us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b1800403ea0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1
0709 12:25:53.065575 (+   783us) spinlock_profiling.cc:243] Waited 93.8 ms on lock 0x7b1800403ea0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:25:53.873798 (+808223us) spinlock_profiling.cc:243] Waited 69.3 ms on lock 0x7b1800403ea0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:25:53.873954 (+   156us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:53.874003 (+    49us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:53.911895 (+ 37892us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:53.911985 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:53.912141 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1649633,"prepare.queue_time_us":270275,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":85,"raft.queue_time_us":1824564,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":36893,"spinlock_wait_cycles":455098240,"thread_start_us":831671,"threads_started":2,"wal-append.queue_time_us":543600,"wal-append.run_cpu_time_us":2304,"wal-append.run_wall_time_us":1007927}
I20260709 12:25:54.054306  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:54.055635  1844 connection.cc:256] RPC call timeout handler was delayed by -1.2297800240003198s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
W20260709 12:25:54.132799  1844 connection.cc:256] RPC call timeout handler was delayed by -1.1217800240001452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
W20260709 12:25:54.133278  1844 connection.cc:256] RPC call timeout handler was delayed by -1.0884800240000914s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
I20260709 12:25:53.942159 10674 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:54.141815  1846 leader_election.cc:400] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:25:54.142321  1846 leader_election.cc:403] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:25:54.069592  1876 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:25:54.128454 10746 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:53.983515 10659 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:54.143623 10746 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:54.143950 10659 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:54.144520 10659 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:54.144676 10746 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.145735 10709 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:54.146207 10746 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:54.074216  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.147356  1880 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:25:54.067023 10439 consensus_queue.cc:1243] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 24.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:25:54.148183  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:54.077682  1877 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:54.078883  1881 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:53.958345 10611 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:54.080258  1730 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
W20260709 12:25:54.065131  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 225630us
I20260709 12:25:54.154484  2382 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:54.083570  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.081305 10665 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:54.063205 10610 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:54.018453 10734 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:25:54.163261  2382 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:54.163976  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:54.164767  1878 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:54.068323  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 201972us
W20260709 12:25:54.065279  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 148908us
I20260709 12:25:54.102979  1879 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:54.133992  9394 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.149145  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:54.176137  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.177945  1872 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:25:54.103183 10708 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:54.169657  9394 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:54.171166  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:54.098043 10647 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.182860  1877 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:54.181350  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:54.188365  1881 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:25:54.189886  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.191658  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:54.192327  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 276194us
W20260709 12:25:54.143985 10730 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:25:54.130216 10637 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.192436  1881 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:25:54.225600 10763 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:54.226208 10763 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.206990  2382 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:54.181571  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:54.223707  1873 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
W20260709 12:25:54.229054  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47465us
I20260709 12:25:54.232628  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:54.235714 10654 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:54.236205 10654 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:54.236552 10654 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:54.239753  1881 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:25:54.153120 10331 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:54.241456  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.161952 10450 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:54.162719 10611 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:54.165120 10610 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:54.268062  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.180271 10750 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:54.164433 10321 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:54.272416  1874 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:54.243516  9394 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:54.272621  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 101541us
I20260709 12:25:54.164001 10040 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:25:54.226645 10763 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:54.275753  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.179941 10644 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.108s
I20260709 12:25:54.344308  1877 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
W20260709 12:25:54.274709  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94276.1us
I20260709 12:25:54.139855 10066 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.103s
I20260709 12:25:54.240952 10439 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.279470 10742 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:54.283845 10534 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.695s	user 0.016s	sys 0.000s
I20260709 12:25:54.287290  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:54.106359 10631 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.232734 10720 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 29 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:54.296357  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 147264us
I20260709 12:25:54.361373  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:54.325492  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.287712 10753 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:25:54.346624  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:54.356314  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69107.8us
I20260709 12:25:54.182683 10710 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.166s
I20260709 12:25:54.359645  1879 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
W20260709 12:25:54.350868 10436 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.542s	user 0.012s	sys 0.000s
I20260709 12:25:54.325577 10450 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.365264  1881 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:25:54.277385 10751 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:54.368083  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:54.343683 10642 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.033s
I20260709 12:25:54.288321 10331 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.272783 10637 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.350487 10750 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:54.371963 10666 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.273097 10757 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.121s
I20260709 12:25:54.326354 10597 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:25:54.364132  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:54.375316  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 147443us
I20260709 12:25:54.417294  1874 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:54.375034  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:54.382107  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:54.091187 10740 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.360616 10698 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:54.381319 10751 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:54.356484 10439 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:54.388912 10762 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.247s	user 0.001s	sys 0.000s
I20260709 12:25:54.244570 10708 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:54.391474  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:54.462474  1881 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:54.392006  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:54.326874 10714 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:54.399252  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:54.369174 10263 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:54.381166 10436 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:54.417534  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:54.421034  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:54.383523  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:54.423738  1872 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:54.383348 10745 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:54.425138 10777 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:25:54.425992 10742 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:54.426115  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.428741 10331 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:54.429366 10324 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:54.431360 10643 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:54.397101 10534 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:54.417408 10666 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.441434 10654 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:54.445629 10766 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.223s	user 0.000s	sys 0.001s
W20260709 12:25:54.495478  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 103561us
I20260709 12:25:54.419478 10669 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:25:54.504205  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86756.1us
W20260709 12:25:54.504737  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83812.1us
W20260709 12:25:54.461877 10299 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.518s	user 0.007s	sys 0.004s
W20260709 12:25:54.434758 10702 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.589s	user 0.000s	sys 0.008s
I20260709 12:25:54.606383  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:54.510186  1874 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:25:54.624751  1881 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:25:54.549203  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 151079us
W20260709 12:25:54.662134  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:54.605314  1872 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 25 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
W20260709 12:25:54.911010  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 527520us
I20260709 12:25:54.456743 10751 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:55.024740  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:55.016420  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:55.034857  1881 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
W20260709 12:25:54.911881  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 512725us
I20260709 12:25:54.911695  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:55.029640  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.012s	sys 0.037s
I20260709 12:25:55.029402  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:55.030906  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 75175.9us
W20260709 12:25:54.887104 10761 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.281s	user 0.002s	sys 0.000s
W20260709 12:25:55.037556  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 431303us
I20260709 12:25:55.043191  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:55.039048  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.012s	sys 0.038s
I20260709 12:25:55.047128  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:55.048074  1879 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:55.043771  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:54.894491 10768 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.363s	user 0.000s	sys 0.000s
I20260709 12:25:55.056030  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:55.037590  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:54.530205 10771 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:54.530486 10705 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.058029  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:55.058478  9394 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:55.059791  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:55.067358  2411 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:54.524739 10263 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:54.665710 10718 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:25:54.886773 10746 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.740s	user 0.016s	sys 0.006s
W20260709 12:25:54.891551 10774 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.502s	user 0.001s	sys 0.000s
I20260709 12:25:55.076742  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:55.125012  1725 raft_consensus.cc:2410] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:55.125530  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1432) took 1527 ms. Trace:
I20260709 12:25:55.125854  1725 rpcz_store.cc:276] 0709 12:25:53.597885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.599039 (+  1154us) service_pool.cc:224] Handling call
0709 12:25:55.076601 (+1477562us) spinlock_profiling.cc:243] Waited 7.15 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3eee 000000000036eb39 00007f16b0d271e0 00007f16b0df00c4 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:25:55.125461 (+ 48860us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16435968}
I20260709 12:25:55.068959  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:55.127272  1874 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:25:54.895136 10759 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.570s	user 0.001s	sys 0.000s
I20260709 12:25:54.893484 10678 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:54.761749 10597 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:55.000116 10757 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:54.908550 10665 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:55.071995  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:54.626556 10640 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:25:55.117920  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 74823.2us
I20260709 12:25:55.119238  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:54.531662 10644 raft_consensus.cc:2764] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:25:55.122907  9394 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:55.132488  1799 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:54.522395 10777 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:55.132844  2415 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:55.037106 10324 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.049111 10299 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.038211 10647 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:55.032768 10669 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.136946  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:55.139928  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:55.061576 10653 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:55.077230 10706 raft_consensus.cc:2764] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:25:55.196895  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:55.202121  1879 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:25:55.320504  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3934) took 1036 ms. Trace:
I20260709 12:25:55.205236  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:55.321985  1881 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:55.322664  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3928) took 1123 ms. Trace:
I20260709 12:25:55.323086  1881 rpcz_store.cc:276] 0709 12:25:54.199041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.204868 (+1005827us) service_pool.cc:224] Handling call
0709 12:25:55.322589 (+117721us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.115504 10718 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:55.185076  1878 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.020s	user 0.007s	sys 0.006s
I20260709 12:25:55.329159  1878 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:25:55.329900  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3114) took 1189 ms. Trace:
I20260709 12:25:55.179806  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:55.150177 10263 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:55.302743  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:54.469105 10720 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 27.12, Last appended by leader: 12, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:54.985109 10702 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.150619 10597 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:55.140331 10631 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:55.331418  1874 raft_consensus.cc:2410] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:25:54.895367 10775 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:55.332011  1723 raft_consensus.cc:2368] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:25:55.149799 10757 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:55.330322  1878 rpcz_store.cc:276] 0709 12:25:54.140932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.141140 (+   208us) service_pool.cc:224] Handling call
0709 12:25:55.329819 (+1188679us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.304911 10640 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:55.333030  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 136256us
I20260709 12:25:55.366469  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3929) took 1142 ms. Trace:
W20260709 12:25:55.378583  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 320611us
I20260709 12:25:55.382350  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:55.383694  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:55.321005  1879 rpcz_store.cc:276] 0709 12:25:54.284388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.997487 (+713099us) service_pool.cc:224] Handling call
0709 12:25:55.320413 (+322926us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.379544  1874 rpcz_store.cc:276] 0709 12:25:54.224225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.128308 (+904083us) service_pool.cc:224] Handling call
0709 12:25:55.366359 (+238051us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:55.441519  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59979.4us
I20260709 12:25:55.442601  1844 leader_election.cc:400] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:25:55.443993  1874 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:55.313153 10450 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:55.445114  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 407566us
I20260709 12:25:55.448576  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:55.449647  1879 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:55.450309  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3926) took 1252 ms. Trace:
I20260709 12:25:55.450769  1879 rpcz_store.cc:276] 0709 12:25:54.198326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.448234 (+1249908us) service_pool.cc:224] Handling call
0709 12:25:55.450231 (+  1997us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.469535  1730 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:25:55.472509  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:55.477108  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 421077us
I20260709 12:25:55.483279  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:55.484258  1878 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:55.394407 10757 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:55.037771 10750 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.122212 10705 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.377665 10597 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:55.491501  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 402846us
I20260709 12:25:55.485057  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:55.494086  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3927) took 1295 ms. Trace:
I20260709 12:25:55.507042  1878 rpcz_store.cc:276] 0709 12:25:54.198692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.440023 (+1241331us) service_pool.cc:224] Handling call
0709 12:25:55.493993 (+ 53970us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.365347 10790 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:55.485584  1879 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:55.523506  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1307) took 2497 ms. Trace:
I20260709 12:25:55.525208  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3932) took 1249 ms. Trace:
I20260709 12:25:55.526353  1879 rpcz_store.cc:276] 0709 12:25:54.275571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.461300 (+1185729us) service_pool.cc:224] Handling call
0709 12:25:55.485354 (+ 24054us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.524478 (+ 39124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.524816 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.529785  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:55.330535 10653 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.471433 10706 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:55.363332 10263 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.525921  1798 rpcz_store.cc:276] 0709 12:25:53.026313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.026469 (+   156us) service_pool.cc:224] Handling call
0709 12:25:53.026962 (+   493us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:53.420562 (+393600us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.523153 (+2102591us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b540004f6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:55.523425 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":901800832}
W20260709 12:25:55.495404 10764 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.005s	sys 0.014s
I20260709 12:25:55.540201  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:55.472013 10775 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:55.483879 10439 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:55.143617 10745 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.484333 10781 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:55.524219 10790 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.563657  1878 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 18. Current term is 19. Ops: []
I20260709 12:25:55.564534  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3116) took 1223 ms. Trace:
I20260709 12:25:55.564904  1878 rpcz_store.cc:276] 0709 12:25:54.340875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.541521 (+1200646us) service_pool.cc:224] Handling call
0709 12:25:55.563444 (+ 21923us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.564230 (+   786us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.564468 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.566828  1878 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:55.567636  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3117) took 1226 ms. Trace:
I20260709 12:25:55.568039  1878 rpcz_store.cc:276] 0709 12:25:54.341296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.566171 (+1224875us) service_pool.cc:224] Handling call
0709 12:25:55.566579 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.567299 (+   720us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.567567 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.522384 10450 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:55.572309 10450 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:55.573666 10450 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.534792 10637 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:55.522737 10436 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:55.540223 10764 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.124s	user 0.005s	sys 0.015s
I20260709 12:25:55.562122  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3938) took 1261 ms. Trace:
I20260709 12:25:55.587097  1879 rpcz_store.cc:276] 0709 12:25:54.285607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.531130 (+1245523us) service_pool.cc:224] Handling call
0709 12:25:55.546180 (+ 15050us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.546575 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:55.546641 (+    66us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:55.546728 (+    87us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:55.546831 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.546950 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.547198 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.580977  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:55.589349  1874 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:25:55.590174  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3115) took 1396 ms. Trace:
I20260709 12:25:55.590571  1874 rpcz_store.cc:276] 0709 12:25:54.193977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.440148 (+1246171us) service_pool.cc:224] Handling call
0709 12:25:55.440708 (+   560us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:55.589886 (+149178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.590113 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:55.543259 10765 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:25:55.600344  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 151850us
I20260709 12:25:55.523048 10640 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.532832 10647 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:25:55.576049 10781 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.584249 10790 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:55.620129 10534 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:55.650812 10534 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.657138 10770 raft_consensus.cc:2764] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
W20260709 12:25:55.657866 10767 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.262s	user 0.000s	sys 0.013s
W20260709 12:25:55.661332  1844 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.630s (SENT) [suppressed 6 similar messages]
I20260709 12:25:55.622025 10637 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:25:55.619751 10764 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:55.676609  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:55.617362 10793 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:55.622745 10654 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:55.643177 10781 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:55.648679 10436 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.684649  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1471) took 2915 ms. Trace:
I20260709 12:25:55.711545  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:55.618335 10765 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.719971 10798 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:25:55.723744 10789 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.392s	user 0.001s	sys 0.000s
I20260709 12:25:55.723498 10782 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:55.676468 10795 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:55.729717 10669 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:55.731596 10263 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:55.604949 10718 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.735291 10806 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:55.739014  2413 rpcz_store.cc:276] 0709 12:25:52.768901 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:52.769054 (+   153us) service_pool.cc:224] Handling call
0709 12:25:55.670998 (+2901944us) spinlock_profiling.cc:243] Waited 74.7 ms on lock 0x7b54007ce554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:55.684532 (+ 13534us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171798784}
I20260709 12:25:55.756343  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.757319  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:55.765406  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 104104us
I20260709 12:25:55.781674  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:55.724054 10791 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.368s	user 0.002s	sys 0.000s
I20260709 12:25:55.694427 10299 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:55.578202 10777 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.797937  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.798110  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:55.797338  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 413710us
I20260709 12:25:55.688359 10324 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:55.693181 10637 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:55.802376  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.802629  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.801124  1874 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:55.805174  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:55.817723  1723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 4003) took 1993 ms (client timeout 1800 ms). Trace:
W20260709 12:25:55.818327  1723 rpcz_store.cc:269] 0709 12:25:53.824813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.825468 (+   655us) service_pool.cc:224] Handling call
0709 12:25:55.817487 (+1992019us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54000d1bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:55.817636 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3434354432}
I20260709 12:25:55.820348  1729 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:25:55.818925  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:55.821969  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.781745 10678 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:55.825335  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.822810  1732 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:55.688057 10764 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.805078 10654 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:55.798437 10787 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:55.815776 10783 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.346s	user 0.001s	sys 0.001s
I20260709 12:25:55.859444  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:55.860025 10610 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.516s	user 0.008s	sys 0.006s
I20260709 12:25:55.828509  1726 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:25:55.826442 10718 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:25:55.809862 10794 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:55.860885 10794 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:55.861289 10794 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:55.861760  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.831043 10534 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:55.863279  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3113) took 1781 ms. Trace:
I20260709 12:25:55.839498 10678 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.863751  1875 rpcz_store.cc:276] 0709 12:25:54.082333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.092056 (+  9723us) service_pool.cc:224] Handling call
0709 12:25:54.092457 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:54.396748 (+304291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:54.396786 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:54.406565 (+  9779us) log.cc:844] Serialized 26 byte log entry
0709 12:25:54.579434 (+172869us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:54.579486 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.112355 (+532869us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b5400026f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:55.112449 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:55.862925 (+750476us) raft_consensus.cc:1685] finished
0709 12:25:55.862951 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.863213 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16899,"raft.queue_time_us":24214,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":13503,"spinlock_wait_cycles":659930496,"thread_start_us":163735,"threads_started":1,"wal-append.queue_time_us":888985}
I20260709 12:25:55.866007  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:55.859481 10787 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.867653  1731 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:55.868103 10765 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:55.868373 10764 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:55.869221 10808 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:55.869062 10654 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.869796 10808 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:55.870266 10808 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:55.871407 10654 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:55.871927 10659 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.727s	user 0.010s	sys 0.004s
I20260709 12:25:55.872380  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4018) took 1494 ms. Trace:
I20260709 12:25:55.872426 10659 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.872819  1724 rpcz_store.cc:276] 0709 12:25:54.377831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.378037 (+   206us) service_pool.cc:224] Handling call
0709 12:25:54.396443 (+ 18406us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:54.980019 (+583576us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:54.980062 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:55.042109 (+ 62047us) log.cc:844] Serialized 52 byte log entry
0709 12:25:55.531356 (+489247us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:55.563463 (+ 32107us) log.cc:844] Serialized 14 byte log entry
0709 12:25:55.610415 (+ 46952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.610544 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:55.872115 (+261571us) raft_consensus.cc:1685] finished
0709 12:25:55.872140 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:55.872299 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":141547,"prepare.queue_time_us":44169,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":676084,"raft.run_cpu_time_us":452,"raft.run_wall_time_us":85394,"thread_start_us":244266,"threads_started":1,"wal-append.queue_time_us":542247}
I20260709 12:25:55.833637  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.874749  1726 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 19: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:55.875808  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.872458 10634 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:55.801676 10702 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:55.798537 10263 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:55.877955 10263 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.789911 10798 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:55.753702 10436 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.841946  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:55.883613  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 78506.7us
I20260709 12:25:55.809842 10806 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.847986 10640 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:55.851317  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.853015 10653 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:55.853253 10801 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:25:55.853574 10812 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:25:55.853768 10637 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:55.855409 10792 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.256s	user 0.000s	sys 0.000s
I20260709 12:25:55.716995 10775 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.868479 10787 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:55.810005 10669 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
W20260709 12:25:55.858929 10731 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.774s	user 0.015s	sys 0.000s
I20260709 12:25:55.860186  1874 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:55.854377 10811 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:55.859192  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.692724 10793 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.886765  1732 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:55.887647  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51118.6us
I20260709 12:25:55.884693  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:55.874189 10232 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:55.930337  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45791.6us
I20260709 12:25:55.901216  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:55.860462 10610 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.904126  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.914337  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:55.895524 10763 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.535s	user 0.009s	sys 0.006s
I20260709 12:25:55.830216 10705 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:55.890818 10765 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:25:55.943287  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 155104us
W20260709 12:25:55.921473  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 97602.3us
I20260709 12:25:55.932067  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.940469 10647 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:55.933254 10321 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.585s	user 0.015s	sys 0.000s
I20260709 12:25:55.943187  1726 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:25:55.955564  1874 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:55.956821  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.965131  1876 raft_consensus.cc:2368] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23.
I20260709 12:25:55.935490 10787 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:55.977602  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:55.979036  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.920251 10597 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.985431  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.987977  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:55.993347  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.944000 10765 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:56.001230  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:56.002283  1875 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:55.951135 10647 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:55.897511 10798 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:55.996524  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:55.988584 10653 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.005270  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:56.005852 10800 raft_consensus.cc:2764] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:25:56.006685  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:56.008001  1805 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:56.009307 10299 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:55.977684 10321 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:25:56.010030 10299 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:55.957597 10263 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:56.005565 10653 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.049942  1875 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:25:56.012418 10597 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:55.934307 10731 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:56.013419 10764 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:56.014083 10040 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:25:55.934020 10811 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:56.014745 10436 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:55.986907 10705 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
W20260709 12:25:56.016314 10732 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Append to log took a long time: real 2.163s	user 0.011s	sys 0.001s
I20260709 12:25:56.016618 10678 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:56.017444 10647 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:56.018564  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:25:56.021023 10715 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 2.680s	user 0.009s	sys 0.007s
W20260709 12:25:56.067479 10757 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.537s	user 0.012s	sys 0.000s
I20260709 12:25:56.021281  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:56.022377 10810 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:56.061542 10659 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:55.999087  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:56.027005 10798 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:56.029455   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:56.036490  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:56.012295 10821 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 12:25:56.142403  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:55.983506 10232 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:56.038594  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:56.150558  1732 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:56.039510  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2977) took 2409 ms. Trace:
W20260709 12:25:56.042309  2383 connection.cc:256] RPC call timeout handler was delayed by -2.0147427270003391s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2141s for a response
I20260709 12:25:56.000034  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.143121  2413 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:25:56.152901  1874 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:25:55.987501 10703 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:56.001299 10665 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:56.053128  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:56.013298 10797 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:55.995411 10611 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.661s	user 0.012s	sys 0.003s
I20260709 12:25:56.060916  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.064908  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:56.162009  1875 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:56.162267  1807 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:56.033062 10765 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:55.998054 10787 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:56.032714 10674 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:56.030274 10794 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.069794  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:56.071601  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:56.072628  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:56.171495  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 99041.1us
I20260709 12:25:55.945762 10763 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:56.020010  1805 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
W20260709 12:25:56.022959 10769 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.792s	user 0.012s	sys 0.000s
I20260709 12:25:56.139612  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:56.149049 10669 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:56.139955  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:56.140846  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 119672us
I20260709 12:25:56.065945 10040 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:56.143244  2410 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:56.154633  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:56.159584  2383 connection.cc:256] RPC call timeout handler was delayed by -1.7528427270003704s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2432s for a response
I20260709 12:25:56.067260 10764 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:56.168206  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.196889  1874 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:56.168447  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:56.156289 10751 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.082s	user 0.006s	sys 0.008s
I20260709 12:25:56.202013 10610 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:56.203599 10777 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:56.151468  1804 rpcz_store.cc:276] 0709 12:25:53.629999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.630185 (+   186us) service_pool.cc:224] Handling call
0709 12:25:53.630549 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:53.631030 (+   481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:53.631064 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:53.631641 (+   577us) log.cc:844] Serialized 26 byte log entry
0709 12:25:53.716108 (+ 84467us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:53.750542 (+ 34434us) log.cc:844] Serialized 14 byte log entry
0709 12:25:56.031905 (+2281363us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540011e2a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd53733 00007f16abdd8dbf 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c
0709 12:25:56.031971 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:56.032081 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:56.032129 (+    48us) raft_consensus.cc:1685] finished
0709 12:25:56.039188 (+  7059us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:56.039423 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35756,"prepare.queue_time_us":71741,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":52,"raft.queue_time_us":30640,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":6203,"spinlock_wait_cycles":337527168,"thread_start_us":83954,"threads_started":1,"wal-append.queue_time_us":74692,"wal-append.run_cpu_time_us":7390,"wal-append.run_wall_time_us":1312991}
I20260709 12:25:56.177817 10798 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:56.178426 10706 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:56.179023 10782 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:56.179956 10702 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:25:56.221715 10832 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260709 12:25:56.222918 10825 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:56.182320 10731 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:56.182440  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:56.183226 10763 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:56.185992  2383 connection.cc:256] RPC call timeout handler was delayed by -1.7024427270002889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24880000000000002s for a response
W20260709 12:25:56.225149  2383 connection.cc:256] RPC call timeout handler was delayed by -1.5491588360000605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
I20260709 12:25:56.225395 10731 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:56.226049 10731 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.226473 10731 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:56.161286 10665 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.169322 10611 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.205442  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:25:56.206993 10655 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.183s	user 0.012s	sys 0.023s
I20260709 12:25:56.208194 10653 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:56.166000 10703 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.077392 10808 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:56.252341  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.253190  1874 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:56.254279  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:56.259160  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:56.260056  1723 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:56.217171 10751 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:56.247884 10655 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.294s	user 0.013s	sys 0.024s
I20260709 12:25:56.213481  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.263244 10765 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.264017  1875 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 18.
I20260709 12:25:56.265108  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.265825  1875 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:25:56.266145  1872 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3919) took 2087 ms (client timeout 1636 ms). Trace:
I20260709 12:25:56.141808 10757 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:56.172799  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:56.225582  2383 connection.cc:256] RPC call timeout handler was delayed by -1.3296473529999275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1704s for a response
I20260709 12:25:56.193528  1877 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:56.227198 10665 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:56.187196 10669 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:56.201815 10698 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:25:56.249330 10436 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:56.172396 10797 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:56.251434 10790 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.584s	user 0.008s	sys 0.004s
I20260709 12:25:56.252066 10666 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:56.143517 10810 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:56.284032  1804 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:25:56.284870  2383 connection.cc:256] RPC call timeout handler was delayed by -1.27375883600007s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19770000000000001s for a response
I20260709 12:25:56.272379  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:56.266619  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:56.293527  1875 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:56.254395 10808 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:56.262252  1724 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:56.211985  1732 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:25:56.287295  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.271395  1878 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:56.272339  2413 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
W20260709 12:25:56.266701  1872 rpcz_store.cc:269] 0709 12:25:54.178596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.180858 (+  2262us) service_pool.cc:224] Handling call
0709 12:25:56.265866 (+2085008us) spinlock_profiling.cc:243] Waited 236 ms on lock 0x7b5400224054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:56.266056 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":543920896}
I20260709 12:25:56.290185  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:56.291088  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 118398us
I20260709 12:25:56.267935  2414 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 18. (term mismatch)
I20260709 12:25:56.285386  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:56.285950 10669 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:56.229012 10798 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:56.370659  1877 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 18.
I20260709 12:25:56.293789  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:25:56.274052 10781 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.532s	user 0.012s	sys 0.001s
I20260709 12:25:56.382658  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:56.382855  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:56.368136  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:56.369011  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:56.322337  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50041.1us
I20260709 12:25:56.375321  2414 pending_rounds.cc:85] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 17
I20260709 12:25:56.376618  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:56.381844  1806 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
I20260709 12:25:56.383576  1804 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:56.389369  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:56.383841  2410 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:56.273447 10816 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:56.390187  1732 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:56.269608 10611 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:56.384833  1800 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:56.390367  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.385227  1807 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:25:56.386273  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.392699  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:56.391322  1803 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 12. Preceding OpId from leader: term: 29 index: 13. (index mismatch)
I20260709 12:25:56.386233  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:56.388957  2419 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29. Ops: []
W20260709 12:25:56.388501  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94799us
I20260709 12:25:56.406816  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:56.387311  2414 pending_rounds.cc:107] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 26.18
I20260709 12:25:56.284552 10698 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:56.407639  2414 raft_consensus.cc:2889] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:56.407646  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:56.406490  1729 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:25:56.417725  1806 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:25:56.424685  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:56.290181 10436 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:56.437666  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:56.383206 10690 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.239s	user 0.036s	sys 0.047s
I20260709 12:25:56.438388  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:25:56.302703 10610 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:56.441843  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:56.445245  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68805.2us
I20260709 12:25:56.446857  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:56.386888 10653 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:56.453593  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:56.472492  1874 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 25 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 12:25:56.446972  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:56.442613  1877 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:25:56.384326 10798 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:56.470283  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:56.396443 10816 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:56.481599  1872 raft_consensus.cc:2376] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23. Re-sending same reply.
W20260709 12:25:56.532255  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 151004us
I20260709 12:25:56.429689 10782 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:25:56.525612  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 78710us
I20260709 12:25:56.559535  1732 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:56.562145  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:56.381887  1875 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:56.409423 10745 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:25:56.442091 10690 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.298s	user 0.036s	sys 0.047s
I20260709 12:25:56.470324 10770 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:56.272707 10040 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:56.294442 10790 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.416523 10606 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:56.579002  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:56.665067  1771 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.185s (SENT)
I20260709 12:25:56.721014  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:56.561321  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:56.453207 10611 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:56.437234 10827 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:25:56.471148 10720 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:56.429380 10718 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.110s
I20260709 12:25:56.389057 10781 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:56.760160  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 198956us
I20260709 12:25:56.628158 10777 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:25:56.748293 10817 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:56.482491 10778 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:56.768814  2410 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:25:56.450795 10610 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.442023 10808 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.272s
I20260709 12:25:56.729504  1800 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
W20260709 12:25:56.814913  1723 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.554s	user 0.012s	sys 0.000s
I20260709 12:25:56.404228 10787 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.717097 10770 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:25:56.841706  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:56.769868 10821 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.018s	sys 0.037s
I20260709 12:25:56.860146  1799 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:56.851698  1877 raft_consensus.cc:2376] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27. Re-sending same reply.
I20260709 12:25:56.853899 10827 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:25:56.848006  1723 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:56.751336 10611 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.866302  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.784256 10666 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:56.870014  1877 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:25:56.772606 10777 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:56.859609 10821 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.019s	sys 0.038s
I20260709 12:25:56.844967 10778 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:56.752238 10775 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.147s
I20260709 12:25:56.862658 10770 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:56.851403 10610 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:56.931025  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.759232 10606 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.857060 10324 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:56.935549  2383 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.012s	sys 0.008s
I20260709 12:25:56.880652 10611 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:56.939688  2383 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.013s	sys 0.008s
W20260709 12:25:56.941277  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 508053us
I20260709 12:25:56.700273 10698 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.956820 10831 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:56.884222 10777 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:25:56.950058 10232 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.795s	user 0.010s	sys 0.003s
I20260709 12:25:56.950579 10324 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:56.926126 10778 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:56.868271 10827 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:57.007980 10827 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.009963 10827 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:57.018394 10797 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:57.021464  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:56.975795  1801 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
W20260709 12:25:57.030455  1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 4009) took 3164 ms (client timeout 2675 ms). Trace:
W20260709 12:25:57.044375  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:57.044142  1728 rpcz_store.cc:269] 0709 12:25:53.865699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.865874 (+   175us) service_pool.cc:224] Handling call
0709 12:25:57.030186 (+3164312us) spinlock_profiling.cc:243] Waited 2.06 s on lock 0x7b54000d16d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:57.030363 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4748709632}
I20260709 12:25:56.937148 10637 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:56.983418 10810 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:56.985473 10781 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:56.986295 10757 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:56.817334 10436 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:56.995801 10758 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:56.999840 10806 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.002317  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3197) took 1200 ms. Trace:
I20260709 12:25:56.936082 10764 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.025367  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:57.068589  2410 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:57.026481  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:56.959740 10816 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.028786  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:57.029096 10606 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.029512  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:57.029659  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3109) took 2976 ms (client timeout 3551 ms). Trace:
I20260709 12:25:57.076301  2417 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
W20260709 12:25:57.078714  1873 rpcz_store.cc:269] 0709 12:25:54.052973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.067863 (+ 14890us) service_pool.cc:224] Handling call
0709 12:25:57.029354 (+2961491us) spinlock_profiling.cc:243] Waited 68 ms on lock 0x7b5400051754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:57.029534 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":156387200}
I20260709 12:25:57.031119 10671 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.033350 10790 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.037832  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.092435  1875 raft_consensus.cc:2410] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 21 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:25:57.093290  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.040505  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4017) took 2664 ms. Trace:
I20260709 12:25:57.093883  1730 rpcz_store.cc:276] 0709 12:25:54.376135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.376513 (+   378us) service_pool.cc:224] Handling call
0709 12:25:54.376906 (+   393us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:55.469996 (+1093090us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.040205 (+1570209us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b5400436cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:57.040441 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2512156544}
I20260709 12:25:57.095494  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:57.096856  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:57.096976  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:57.097585  1730 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:57.041239 10647 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.900s	user 0.005s	sys 0.006s
W20260709 12:25:57.042484  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:57.104216  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.104008 10647 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:56.963837  1798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.006s	sys 0.066s
I20260709 12:25:56.959398  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:56.864988 10817 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.045763  1772 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:56.968809 10698 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:57.068213  1724 rpcz_store.cc:276] 0709 12:25:55.802975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.881662 (+ 78687us) service_pool.cc:224] Handling call
0709 12:25:55.899040 (+ 17378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.370104 (+471064us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.001952 (+631848us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b54000d25d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:57.002234 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":834773760}
W20260709 12:25:57.069090  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 404044us
I20260709 12:25:57.027215  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:57.051802 10805 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.054754 10594 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.062633 10782 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:57.084573 10743 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:57.091269  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:57.101372 10839 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:57.101927 10606 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.103107  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:57.132638  2413 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:57.103322  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3119) took 2761 ms. Trace:
I20260709 12:25:57.133249  1878 rpcz_store.cc:276] 0709 12:25:54.341985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.568853 (+1226868us) service_pool.cc:224] Handling call
0709 12:25:56.271113 (+702260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.372156 (+101043us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.103099 (+730943us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b5400224050. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:57.103259 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1614106880}
W20260709 12:25:57.104403  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3939) took 2818 ms (client timeout 2185 ms). Trace:
W20260709 12:25:57.135135  1876 rpcz_store.cc:269] 0709 12:25:54.285892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.089974 (+804082us) service_pool.cc:224] Handling call
0709 12:25:57.104127 (+2014153us) spinlock_profiling.cc:243] Waited 659 ms on lock 0x7b540021b6d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:57.104323 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1515729792}
I20260709 12:25:57.136602  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.107046  1877 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:57.137857  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3129) took 1154 ms. Trace:
I20260709 12:25:57.138273  1877 rpcz_store.cc:276] 0709 12:25:55.983863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.959083 (+975220us) service_pool.cc:224] Handling call
0709 12:25:57.137757 (+178674us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:57.139523  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.140301  1877 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
W20260709 12:25:57.046307  1804 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.653s	user 0.008s	sys 0.002s
I20260709 12:25:57.142140  1804 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:25:57.143421  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.066351 10758 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:57.112068  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:57.113137  1697 connection.cc:256] RPC call timeout handler was delayed by -1.2042541189998701s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.46660000000000007s for a response
I20260709 12:25:57.149737  1876 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:57.150863  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.153406  1876 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:25:57.156900  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:57.157953  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.158576  1878 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:25:57.159739  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:57.160996  1877 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:57.158957  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:57.126966 10594 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.163339 10647 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:57.163568 10758 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.165097 10758 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:57.165934 10647 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.167837  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:57.169970  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:57.122044  2410 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:57.059993 10637 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.129019  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:57.160962  1697 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:25:57.109484  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 338776us
I20260709 12:25:57.145732  1804 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40096
W20260709 12:25:57.130473  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 84829.8us
I20260709 12:25:57.126842 10743 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:25:57.150441 10731 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.882s	user 0.017s	sys 0.000s
I20260709 12:25:57.183620  1804 raft_consensus.cc:606] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:57.189646  1804 raft_consensus.cc:614] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:57.185428  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:57.191447  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:57.192334 10040 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:25:57.192925  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:57.194027  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1554) took 2214 ms. Trace:
I20260709 12:25:57.194429  2415 rpcz_store.cc:276] 0709 12:25:54.979341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.979474 (+   133us) service_pool.cc:224] Handling call
0709 12:25:57.193856 (+2214382us) spinlock_profiling.cc:243] Waited 83 ms on lock 0x7b5400840154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:57.193968 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":190850432}
I20260709 12:25:57.195894  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:57.196468 10637 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:57.197221 10637 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:57.198103 10637 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:57.198451 10637 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:57.198858 10702 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:57.200426 10743 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.201148 10794 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:57.201699 10743 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:57.203522  1872 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:57.186239  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:25:57.127836 10805 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.204253  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1310) took 1585 ms. Trace:
I20260709 12:25:57.204685  1697 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:57.205749 10671 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.206270  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:57.151930 10781 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.203261 10797 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:57.144997 10804 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:57.216352  2413 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:25:57.173884  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3130) took 1175 ms. Trace:
I20260709 12:25:57.217696  1876 rpcz_store.cc:276] 0709 12:25:55.987969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.150499 (+1162530us) service_pool.cc:224] Handling call
0709 12:25:57.162924 (+ 12425us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:57.208254  2416 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.204639  1872 rpcz_store.cc:276] 0709 12:25:55.618438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.202944 (+1584506us) service_pool.cc:224] Handling call
0709 12:25:57.203331 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.203995 (+   664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.204186 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:57.186553  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1321) took 1542 ms. Trace:
I20260709 12:25:57.222940  1877 rpcz_store.cc:276] 0709 12:25:55.645123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.141227 (+1496104us) service_pool.cc:224] Handling call
0709 12:25:57.186462 (+ 45235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:57.210412 10790 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:57.151099 10816 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.224146  1873 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
W20260709 12:25:57.235591  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76691.5us
I20260709 12:25:57.239470  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1313) took 1620 ms. Trace:
I20260709 12:25:57.236207  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:25:57.239743  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:57.239902  1873 rpcz_store.cc:276] 0709 12:25:55.619338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.090973 (+1471635us) service_pool.cc:224] Handling call
0709 12:25:57.239379 (+148406us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:57.154484 10839 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:57.244496  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:57.210955 10810 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:57.123317 10773 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.065s
I20260709 12:25:57.243212  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:57.166555 10647 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.241930  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:57.253338  1877 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27. Ops: [24.15-26.17]
I20260709 12:25:57.255386  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1314) took 1635 ms. Trace:
I20260709 12:25:57.255843  1877 rpcz_store.cc:276] 0709 12:25:55.619583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.223675 (+1604092us) service_pool.cc:224] Handling call
0709 12:25:57.253067 (+ 29392us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:57.255085 (+  2018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.255323 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:57.149408 10606 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.216114 10702 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:57.205780 10805 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:57.215677 10742 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:57.261113  2413 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:25:57.245730 10672 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 3.406s	user 0.024s	sys 0.026s
I20260709 12:25:57.206395 10671 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.264384  1772 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:57.243124 10232 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:57.247385 10831 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:25:57.259528 10653 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.776s	user 0.007s	sys 0.004s
I20260709 12:25:57.285089  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1318) took 1664 ms. Trace:
I20260709 12:25:57.287822  1877 rpcz_store.cc:276] 0709 12:25:55.620591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.256795 (+1636204us) service_pool.cc:224] Handling call
0709 12:25:57.284446 (+ 27651us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.284661 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:57.284698 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:57.284753 (+    55us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:57.284798 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.284880 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.285019 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:57.296241  1730 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260709 12:25:57.212383 10731 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.296762  1730 pending_rounds.cc:85] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 14
I20260709 12:25:57.262626 10804 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:57.279927 10672 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 3.449s	user 0.025s	sys 0.026s
I20260709 12:25:57.301309  1730 pending_rounds.cc:107] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260709 12:25:57.311878  1730 raft_consensus.cc:2889] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:57.249933 10782 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.239697 10781 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.247062 10807 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:25:57.323647  1726 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:57.312601  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1319) took 1677 ms. Trace:
I20260709 12:25:57.280776 10810 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:57.326138  1877 rpcz_store.cc:276] 0709 12:25:55.636035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.289021 (+1652986us) service_pool.cc:224] Handling call
0709 12:25:57.312069 (+ 23048us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.312252 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:57.312278 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:57.312318 (+    40us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:57.312351 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.312404 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.312541 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:57.299032 10653 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:57.304069 10804 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:25:57.322532 10690 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:57.348798  1732 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:57.302590 10666 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:57.322881 10831 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:57.322319 10232 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:57.351708  1877 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:57.357465  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1324) took 1555 ms. Trace:
I20260709 12:25:57.357884  1877 rpcz_store.cc:276] 0709 12:25:55.801937 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.328414 (+1526477us) service_pool.cc:224] Handling call
0709 12:25:57.349136 (+ 20722us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.357122 (+  7986us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.357392 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:57.348450 10804 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:57.305420 10637 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:57.387647  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.004s	sys 0.020s
I20260709 12:25:57.342329 10778 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:57.391376  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.004s	sys 0.020s
I20260709 12:25:57.354665 10831 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:57.394588  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:57.372946 10750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.521s	user 0.004s	sys 0.030s
I20260709 12:25:57.357630 10690 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:57.386852  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:57.403429  1877 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:57.386304 10653 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.014s
W20260709 12:25:57.404903  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 249292us
I20260709 12:25:57.372226 10232 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:57.403575 10750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.551s	user 0.005s	sys 0.032s
I20260709 12:25:57.398546 10831 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:25:57.406430 10665 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.114s	user 0.011s	sys 0.002s
I20260709 12:25:57.424535  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:57.346969 10839 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.449643 10781 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:57.451027  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53734.1us
I20260709 12:25:57.417328 10848 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.427639 10797 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:57.427953 10702 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.445497 10742 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:25:57.488922  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 245745us
I20260709 12:25:57.426081 10827 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:25:57.492753  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:57.493079 10610 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.523s	user 0.011s	sys 0.000s
I20260709 12:25:57.477037 10848 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.501762 10610 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.505601 10714 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.498615 10806 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:57.515311  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2980) took 1543 ms. Trace:
I20260709 12:25:57.474467 10647 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.432569 10665 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.485584 10606 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.484082 10742 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:57.511962  1804 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:25:57.540067  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:57.541195  1804 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:57.513006  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:57.513931  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:57.509573  1878 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:57.515846  1799 rpcz_store.cc:276] 0709 12:25:55.971559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.972040 (+   481us) service_pool.cc:224] Handling call
0709 12:25:55.992007 (+ 19967us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.860887 (+868880us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.508038 (+647151us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b54001e3a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:57.515200 (+  7162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1995790592}
I20260709 12:25:57.516649  2416 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:57.520009  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:57.521255  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.410554 10810 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.547832  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:57.537640  1877 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:57.500506  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:25:57.534315  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 269982us
I20260709 12:25:57.542438  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:57.553153  1807 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:57.542836  1800 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:57.545449  1805 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:25:57.553576  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:57.553690  1804 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:57.484758 10797 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.488004 10702 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.461895 10666 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.495251 10690 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.555297  1872 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:57.604614  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:57.624344  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:57.550904  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:57.555826  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1441) took 1478 ms. Trace:
I20260709 12:25:57.573941  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:57.626272  1803 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:57.534142 10772 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.627100  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:25:57.579185  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1433) took 1666 ms. Trace:
W20260709 12:25:57.633862  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83096.2us
I20260709 12:25:57.634613  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:25:57.565294  1805 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
W20260709 12:25:57.635058  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.633026  1723 rpcz_store.cc:276] 0709 12:25:55.912441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.879713 (+967272us) service_pool.cc:224] Handling call
0709 12:25:56.935219 (+ 55506us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:56.943713 (+  8494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.943774 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:56.944564 (+   790us) log.cc:844] Serialized 26 byte log entry
0709 12:25:57.356117 (+411553us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:57.356185 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.356287 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:57.578901 (+222614us) raft_consensus.cc:1685] finished
0709 12:25:57.578930 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.579117 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":32019,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":72189,"thread_start_us":291128,"threads_started":1,"wal-append.queue_time_us":416988}
W20260709 12:25:57.628742  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 130520us
I20260709 12:25:57.632011  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:57.636242  1727 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:57.637406  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:57.639815  1800 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:25:57.624029 10671 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.640542 10805 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:25:57.578737 10850 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:25:57.600088 10741 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:25:57.555038 10770 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.594s	user 0.009s	sys 0.006s
I20260709 12:25:57.642491  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:25:57.586931 10838 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.237s	user 0.002s	sys 0.000s
I20260709 12:25:57.579572 10778 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.544201 10848 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:57.582522 10720 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.812s	user 0.013s	sys 0.000s
I20260709 12:25:57.625433  1728 rpcz_store.cc:276] 0709 12:25:56.077104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.052087 (+974983us) service_pool.cc:224] Handling call
0709 12:25:57.052450 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.052681 (+   231us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:57.383546 (+330865us) log.cc:844] Serialized 14 byte log entry
0709 12:25:57.555464 (+171918us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:57.555522 (+    58us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:57.555567 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.555630 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.555770 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":424782,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":113,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":11633,"thread_start_us":490197,"threads_started":2}
I20260709 12:25:57.648171  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1317) took 2027 ms. Trace:
I20260709 12:25:57.543514 10742 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:57.600471 10777 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.555s	user 0.008s	sys 0.003s
I20260709 12:25:57.645767  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:57.601464 10793 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.622943 10709 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:25:57.708350 10076 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:57.708640 10778 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:57.636844 10606 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260709 12:25:57.709383 10777 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.584467 10610 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.652830  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1311) took 2034 ms. Trace:
I20260709 12:25:57.578841 10665 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.656322  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:57.637390 10862 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:57.658272  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:57.642549  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.717741  1877 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:25:57.652726  1846 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:57.637001 10672 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:57.640951 10843 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:57.661677  1873 rpcz_store.cc:276] 0709 12:25:55.620365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.240865 (+1620500us) service_pool.cc:224] Handling call
0709 12:25:57.241292 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:57.241654 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:57.241684 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:57.249453 (+  7769us) log.cc:844] Serialized 26 byte log entry
0709 12:25:57.484517 (+235064us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:57.484671 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.484841 (+   170us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:57.647918 (+163077us) raft_consensus.cc:1685] finished
0709 12:25:57.647947 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.648113 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2662,"raft.queue_time_us":152067,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":12235,"thread_start_us":187930,"threads_started":1,"wal-append.queue_time_us":345621}
I20260709 12:25:57.640974 10787 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.653579 10714 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:25:57.651345 10706 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:57.664134 10299 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.667394 10845 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
W20260709 12:25:57.668056 10669 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.137s	user 0.000s	sys 0.013s
I20260709 12:25:57.736969 10669 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.739339  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:57.740007  1877 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.740057 10669 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.741395 10843 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:57.742853 10843 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.744376 10804 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.668895 10831 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.670146  1875 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:25:57.670950 10702 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:57.651657 10741 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:57.674249 10708 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:57.674431 10798 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.135s	user 0.013s	sys 0.000s
I20260709 12:25:57.675921  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:57.676199 10758 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.511s	user 0.012s	sys 0.000s
I20260709 12:25:57.677554  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3200) took 1837 ms. Trace:
I20260709 12:25:57.758682  1881 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:57.658044 10805 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:25:57.683364 10720 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:25:57.684337 10698 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.570s	user 0.010s	sys 0.001s
W20260709 12:25:57.655045 10830 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.451s	user 0.006s	sys 0.003s
I20260709 12:25:57.684985 10751 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.763342  2415 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:25:57.532367 10232 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.685628 10797 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:57.685793  1731 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 3187) took 1929 ms (client timeout 1899 ms). Trace:
I20260709 12:25:57.686722  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:57.687783  1724 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:57.695850  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3915) took 3536 ms. Trace:
I20260709 12:25:57.698037  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:57.698031  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:57.769075  1807 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:25:57.659451 10611 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.708s	user 0.012s	sys 0.001s
W20260709 12:25:57.703536  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61116.3us
I20260709 12:25:57.644253 10647 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:57.713855  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:57.720981 10799 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.062s	user 0.019s	sys 0.010s
W20260709 12:25:57.721688  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84170.8us
W20260709 12:25:57.727049  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68882.2us
I20260709 12:25:57.732817  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 2988) took 1661 ms. Trace:
W20260709 12:25:57.733397  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 80879.3us
I20260709 12:25:57.733711  1727 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:25:57.734242 10787 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.731838 10606 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:57.736253 10714 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:57.650053 10671 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:57.711624  1876 rpcz_store.cc:276] 0709 12:25:55.618735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.219087 (+1600352us) service_pool.cc:224] Handling call
0709 12:25:57.219510 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.219803 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:57.449907 (+230104us) log.cc:844] Serialized 14 byte log entry
0709 12:25:57.652010 (+202103us) log.cc:844] Serialized 14 byte log entry
0709 12:25:57.652405 (+   395us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:57.652460 (+    55us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:57.652545 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.652624 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.652754 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3495,"prepare.queue_time_us":362111,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":89,"raft.queue_time_us":14031,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":2260,"spinlock_wait_cycles":10370176,"thread_start_us":416527,"threads_started":2,"wal-append.queue_time_us":195920}
I20260709 12:25:57.745533  1725 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:57.752302  1877 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:57.717242 10862 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:57.754130 10807 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
W20260709 12:25:57.754909  1845 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.085s (SENT)
I20260709 12:25:57.677733  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2994) took 1390 ms. Trace:
I20260709 12:25:57.759220  1732 rpcz_store.cc:276] 0709 12:25:55.840446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.713619 (+873173us) service_pool.cc:224] Handling call
0709 12:25:56.713980 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.351255 (+637275us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.677306 (+326051us) spinlock_profiling.cc:243] Waited 635 ms on lock 0x7b54002b7ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:57.677491 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1459492736}
I20260709 12:25:57.658859 10770 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.641458 10772 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:57.731374 10672 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:57.817291  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 2981) took 1838 ms. Trace:
I20260709 12:25:57.749101 10873 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:25:57.802270  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:57.910108  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:25:57.921862 10799 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.263s	user 0.021s	sys 0.013s
I20260709 12:25:57.891281  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.909410  1804 raft_consensus.cc:2368] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26.
I20260709 12:25:57.913712  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:57.975770  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.976598  1877 raft_consensus.cc:2376] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23. Re-sending same reply.
W20260709 12:25:57.918257  1731 rpcz_store.cc:269] 0709 12:25:55.756547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.756997 (+   450us) service_pool.cc:224] Handling call
0709 12:25:57.685561 (+1928564us) spinlock_profiling.cc:243] Waited 74.5 ms on lock 0x7b54002b6bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:57.685709 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171295488}
I20260709 12:25:57.818152  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1438) took 1829 ms. Trace:
I20260709 12:25:57.977608  1726 rpcz_store.cc:276] 0709 12:25:55.988710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.036475 (+1047765us) service_pool.cc:224] Handling call
0709 12:25:57.036794 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.324290 (+287496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.817871 (+493581us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b540030af54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:57.818086 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":659113728}
I20260709 12:25:57.978060  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:57.920501  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:57.909256  1798 rpcz_store.cc:276] 0709 12:25:56.287788 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.287917 (+   129us) service_pool.cc:224] Handling call
0709 12:25:56.290680 (+  2763us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:56.291110 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.291147 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:56.291689 (+   542us) log.cc:844] Serialized 26 byte log entry
0709 12:25:57.033972 (+742283us) spinlock_profiling.cc:243] Waited 73.3 ms on lock 0x7b48001c0158. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:25:57.106384 (+ 72412us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:57.106430 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.106510 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:57.677360 (+570850us) raft_consensus.cc:1685] finished
0709 12:25:57.677389 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.677660 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1784,"raft.run_cpu_time_us":13472,"raft.run_wall_time_us":737970,"spinlock_wait_cycles":168604032,"thread_start_us":663249,"threads_started":1,"wal-append.queue_time_us":888963}
I20260709 12:25:57.896790  1807 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:25:57.771086 10594 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.758569 10706 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:57.909832  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:57.946803  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.947801  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:25:57.956420  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:57.948957  1802 rpcz_store.cc:276] 0709 12:25:55.979140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.979463 (+   323us) service_pool.cc:224] Handling call
0709 12:25:55.997229 (+ 17766us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.033518 (+ 36289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.033570 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:56.033637 (+    67us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:56.033685 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.816945 (+1783260us) spinlock_profiling.cc:243] Waited 36 ms on lock 0x7b54002f1f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:57.817016 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.817213 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82862336}
I20260709 12:25:57.768970  1880 rpcz_store.cc:276] 0709 12:25:54.159565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:54.159719 (+   154us) service_pool.cc:224] Handling call
0709 12:25:54.183923 (+ 24204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:54.356769 (+172846us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:54.492668 (+135899us) log.cc:844] Serialized 14 byte log entry
0709 12:25:57.695359 (+3202691us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b180008e480. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:25:57.695479 (+   120us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:57.695530 (+    51us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:57.695577 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:57.695664 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:57.695798 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39674,"prepare.queue_time_us":756907,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":8450,"raft.queue_time_us":163371,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":12495,"spinlock_wait_cycles":41210240,"thread_start_us":723003,"threads_started":3,"wal-append.queue_time_us":947755,"wal-append.run_cpu_time_us":3044,"wal-append.run_wall_time_us":1396470}
W20260709 12:25:57.821069 10876 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:25:58.021189  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:57.761580 10798 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:57.969441  2413 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:58.019728  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:57.959017 10777 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.970811  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:58.037640  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.038317  1728 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:57.976543  2416 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:57.941867  1801 rpcz_store.cc:276] 0709 12:25:56.071117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.071281 (+   164us) service_pool.cc:224] Handling call
0709 12:25:57.732590 (+1661309us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b5400132754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:57.732743 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1849412480}
I20260709 12:25:57.894657 10811 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:57.979862  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:57.981842  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:57.982537  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.055469  2418 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:25:58.055856  1731 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:58.057665  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:57.910497 10714 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:25:57.995411  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 85382.2us
I20260709 12:25:57.996564  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:57.996803  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.021130  1876 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:57.967712  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:58.028931  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.909821  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.043180  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:58.080513  1723 raft_consensus.cc:2376] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29. Re-sending same reply.
I20260709 12:25:58.044840  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.081393  1802 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:25:58.060181  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.075551  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.079918  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:58.082279  1801 raft_consensus.cc:2368] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20.
I20260709 12:25:58.083961  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:57.999902 10671 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:57.944437 10647 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:58.101912 10647 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:57.944555 10611 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:58.104545 10611 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:58.105221 10611 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.105738 10611 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:58.087210  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.112692  1807 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:58.087237  1876 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:25:58.090955 10831 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:25:58.085709 10743 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.883s	user 0.016s	sys 0.000s
I20260709 12:25:58.091626  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:58.092730  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:58.124945  2416 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:58.093798 10714 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:58.090718  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:58.097270  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.099391  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:58.085340  1804 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:58.107021  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:58.128877  1880 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:57.819732 10741 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:58.109510  1726 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:58.113842  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.114758  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.134390  1807 raft_consensus.cc:2368] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25.
I20260709 12:25:58.134888  1731 raft_consensus.cc:2410] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:25:58.038480 10644 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:58.119016  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:58.091526  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.137384  1801 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:58.121629  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 366778us
I20260709 12:25:58.129839  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.140870  1802 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:58.131819  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.142182  1798 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:58.083937 10805 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:58.146065  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.146955  1727 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:25:58.148041  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.148801  1727 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:25:58.150203  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1447) took 1054 ms. Trace:
I20260709 12:25:58.149493 10706 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:58.115687 10799 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:58.155195  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.155133  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.155892  2410 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:25:58.110049 10671 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:25:58.084355  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:58.075780 10862 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:58.182428  1804 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:58.367612  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.177814  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:58.368930  1807 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:58.191632  1728 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:58.370177  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 3209) took 1159 ms. Trace:
I20260709 12:25:58.370311  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 4038) took 1028 ms. Trace:
I20260709 12:25:58.214565  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.373728  1732 rpcz_store.cc:276] 0709 12:25:57.341772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.127096 (+785324us) service_pool.cc:224] Handling call
0709 12:25:58.369713 (+242617us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.374293  1798 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:25:58.367664  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:25:58.368911  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 284613us
I20260709 12:25:58.166023  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:58.376287  1877 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:58.375689  1876 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:58.216245  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.370252  2416 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:58.150609  1727 rpcz_store.cc:276] 0709 12:25:57.095809 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.149369 (+1053560us) service_pool.cc:224] Handling call
0709 12:25:58.149689 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.149874 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:58.149899 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:58.149944 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:58.149977 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.150045 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:58.150152 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.372102  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.381953  1801 raft_consensus.cc:2368] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26.
I20260709 12:25:58.313606  1880 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:58.384440  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:58.386715  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.367933  1731 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:25:58.370958  1728 rpcz_store.cc:276] 0709 12:25:57.211264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.970405 (+759141us) service_pool.cc:224] Handling call
0709 12:25:58.370087 (+399682us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.387632  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.389549  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.390537  1807 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:58.391994  1728 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:25:58.393308  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3208) took 1182 ms. Trace:
I20260709 12:25:58.393963  1731 rpcz_store.cc:276] 0709 12:25:57.210936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.177477 (+966541us) service_pool.cc:224] Handling call
0709 12:25:58.367739 (+190262us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.393080 (+ 25341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.393246 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.394662  1804 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:58.385715  1880 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:25:58.395965  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4023) took 1181 ms. Trace:
I20260709 12:25:58.396117  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.397238  1728 rpcz_store.cc:276] 0709 12:25:57.214403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.391230 (+1176827us) service_pool.cc:224] Handling call
0709 12:25:58.391676 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:58.392773 (+  1097us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.395917 (+  3144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.367354 10714 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:58.400362  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.401022  1804 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 12.
I20260709 12:25:58.233749 10669 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.163296 10799 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:58.402743  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:58.410333  1801 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:58.403102  1728 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 19. Current term is 20. Ops: []
I20260709 12:25:58.410574  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:58.411248  1877 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:25:58.411381  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4029) took 1169 ms. Trace:
I20260709 12:25:58.176968 10873 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.407891  1731 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:58.371181 10772 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:58.416286  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:58.417652  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1334) took 1030 ms. Trace:
I20260709 12:25:58.418191  1880 rpcz_store.cc:276] 0709 12:25:57.386913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.402243 (+1015330us) service_pool.cc:224] Handling call
0709 12:25:58.417559 (+ 15316us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.335302 10770 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:58.376715 10602 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:58.262579 10804 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:58.397434  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.420341  1807 raft_consensus.cc:2368] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20.
I20260709 12:25:58.421716  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.424140  1807 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:58.411726  1728 rpcz_store.cc:276] 0709 12:25:57.242114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.399658 (+1157544us) service_pool.cc:224] Handling call
0709 12:25:58.402635 (+  2977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.411109 (+  8474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.411325 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.426110  1728 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:58.379163 10671 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:58.407768 10741 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:58.433310  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.432224  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.435657  1807 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:25:58.412835 10799 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:58.129189 10672 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:58.407531 10862 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:58.402093 10777 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:58.431433  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:58.447490  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:58.450999  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.414074  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4024) took 1199 ms. Trace:
I20260709 12:25:58.463522  1731 rpcz_store.cc:276] 0709 12:25:57.214649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.406345 (+1191696us) service_pool.cc:224] Handling call
0709 12:25:58.407613 (+  1268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.413647 (+  6034us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.413897 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.443394  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.437022  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.467831  1804 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:25:58.427438  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4033) took 1147 ms. Trace:
I20260709 12:25:58.470616  1728 rpcz_store.cc:276] 0709 12:25:57.280504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.425346 (+1144842us) service_pool.cc:224] Handling call
0709 12:25:58.425843 (+   497us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.426826 (+   983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.427351 (+   525us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.413064 10669 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:58.438901 10602 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:58.437763 10798 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:58.415529 10873 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:58.486056  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:58.487241  1804 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:58.474901  1798 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:58.475328  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.549167  1807 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:25:58.550757  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.472868  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4036) took 1158 ms. Trace:
I20260709 12:25:58.648957  1731 rpcz_store.cc:276] 0709 12:25:57.314490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.469374 (+1154884us) service_pool.cc:224] Handling call
0709 12:25:58.472036 (+  2662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.472362 (+   326us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:58.472401 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:58.472464 (+    63us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:58.472538 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.472618 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:58.472787 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.479573  1728 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:25:58.476579  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:58.658756  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1335) took 1268 ms. Trace:
I20260709 12:25:58.659333  1880 rpcz_store.cc:276] 0709 12:25:57.390588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.449830 (+1059242us) service_pool.cc:224] Handling call
0709 12:25:58.658661 (+208831us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.663118  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4037) took 1337 ms. Trace:
I20260709 12:25:58.663759  1728 rpcz_store.cc:276] 0709 12:25:57.325811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.476036 (+1150225us) service_pool.cc:224] Handling call
0709 12:25:58.476437 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.662682 (+186245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.663039 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:58.638855  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.645277  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:58.670200  1807 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:58.696100  1801 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:25:58.473372 10611 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:58.783775  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.788810  1731 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:58.789770  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3211) took 1456 ms. Trace:
I20260709 12:25:58.790388  1731 rpcz_store.cc:276] 0709 12:25:57.333570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.788259 (+1454689us) service_pool.cc:224] Handling call
0709 12:25:58.788628 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.789482 (+   854us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.789710 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.792876  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3213) took 1323 ms. Trace:
I20260709 12:25:58.793402  1731 rpcz_store.cc:276] 0709 12:25:57.469583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.791930 (+1322347us) service_pool.cc:224] Handling call
0709 12:25:58.792335 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.792536 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:58.792565 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:58.792608 (+    43us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:58.792647 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.792702 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:58.792821 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.638182 10841 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:25:58.795542 10841 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:25:58.796911  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.803444  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:25:58.804173  1880 raft_consensus.cc:2376] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27. Re-sending same reply.
I20260709 12:25:58.804705  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1336) took 1410 ms. Trace:
I20260709 12:25:58.805028  1880 rpcz_store.cc:276] 0709 12:25:57.394311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.660240 (+1265929us) service_pool.cc:224] Handling call
0709 12:25:58.804638 (+144398us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.453313 10777 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:58.759135 10666 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.023s	sys 0.044s
I20260709 12:25:58.834903  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.836323  1798 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:58.854523  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.855625  1801 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:58.758812 10798 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:58.871677 10819 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 3.017s	user 0.013s	sys 0.015s
I20260709 12:25:58.873785  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3940) took 3390 ms. Trace:
I20260709 12:25:58.874253  1879 rpcz_store.cc:276] 0709 12:25:55.483798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:55.596903 (+113105us) service_pool.cc:224] Handling call
0709 12:25:55.620347 (+ 23444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:55.648074 (+ 27727us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:55.648121 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:55.665516 (+ 17395us) log.cc:844] Serialized 26 byte log entry
0709 12:25:55.786166 (+120650us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:55.786703 (+   537us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:55.786905 (+   202us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:58.870602 (+3083697us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:58.873376 (+  2774us) raft_consensus.cc:1685] finished
0709 12:25:58.873409 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:58.873703 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27213,"raft.queue_time_us":28368,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":12648,"spinlock_wait_cycles":62845312,"thread_start_us":68933,"threads_started":1,"wal-append.queue_time_us":167284}
I20260709 12:25:58.876801  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:58.877792  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3146) took 1283 ms. Trace:
I20260709 12:25:58.878155  1879 rpcz_store.cc:276] 0709 12:25:57.594298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.876412 (+1282114us) service_pool.cc:224] Handling call
0709 12:25:58.877722 (+  1310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.884225  1807 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
W20260709 12:25:58.829303 10666 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.023s	sys 0.044s
W20260709 12:25:58.855942 10706 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.013s	sys 0.038s
I20260709 12:25:58.893505  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:25:58.893091  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.895967  1880 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:25:58.896301  1807 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:58.894161  1804 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
W20260709 12:25:58.856068 10805 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.014s	sys 0.039s
I20260709 12:25:58.897694  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3055) took 1131 ms. Trace:
I20260709 12:25:58.898319  1804 rpcz_store.cc:276] 0709 12:25:57.766759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.468954 (+702195us) service_pool.cc:224] Handling call
0709 12:25:58.897636 (+428682us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.898699  1879 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:58.899925  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1325) took 3085 ms. Trace:
I20260709 12:25:58.897185  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.900044  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.901729  1804 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:25:58.900671  1879 rpcz_store.cc:276] 0709 12:25:55.814816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.878954 (+3064138us) service_pool.cc:224] Handling call
0709 12:25:58.898464 (+ 19510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.899657 (+  1193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.899868 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.903352  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 1327) took 1213 ms. Trace:
I20260709 12:25:58.903888  1807 rpcz_store.cc:276] 0709 12:25:57.689450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.891888 (+1202438us) service_pool.cc:224] Handling call
0709 12:25:58.903280 (+ 11392us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.906816  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.907636  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3035) took 1217 ms. Trace:
I20260709 12:25:58.907950  1801 rpcz_store.cc:276] 0709 12:25:57.690473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.891373 (+1200900us) service_pool.cc:224] Handling call
0709 12:25:58.907579 (+ 16206us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:58.885969 10232 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.013s	sys 0.035s
I20260709 12:25:58.911952 10873 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:25:58.916752  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.918160  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.919045  1798 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:25:58.920011  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.921089  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:25:58.902014 10805 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.015s	sys 0.039s
I20260709 12:25:58.922820 10669 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:58.895700 10841 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:58.909415 10232 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.014s	sys 0.035s
I20260709 12:25:58.928028  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1315) took 1298 ms. Trace:
W20260709 12:25:58.933054 10742 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.209s	user 0.006s	sys 0.006s
I20260709 12:25:58.933714 10742 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:58.891882 10798 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:58.937340 10798 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:58.940127 10777 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:58.928766  1798 rpcz_store.cc:276] 0709 12:25:57.629460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.921881 (+1292421us) service_pool.cc:224] Handling call
0709 12:25:58.922181 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.922380 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:58.922408 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:58.922455 (+    47us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:58.922495 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.922680 (+   185us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:58.927951 (+  5271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.946949  1727 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:58.898090 10706 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.014s	sys 0.038s
I20260709 12:25:58.952276  1880 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:25:58.972726  1879 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260709 12:25:58.973377  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3943) took 3141 ms. Trace:
I20260709 12:25:58.956735  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:58.963991  1800 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: [25.20-25.20]
W20260709 12:25:58.965847  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:58.970871 10678 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:25:58.967958  1875 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3949) took 2950 ms (client timeout 2925 ms). Trace:
I20260709 12:25:58.978758  1804 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:58.977029  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1317) took 1347 ms. Trace:
W20260709 12:25:58.978300  1875 rpcz_store.cc:269] 0709 12:25:56.017862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.093037 (+1075175us) service_pool.cc:224] Handling call
0709 12:25:58.967704 (+1874667us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b5400073854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:58.967865 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1325266688}
I20260709 12:25:58.982137  1875 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:25:58.982872  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3124) took 3010 ms. Trace:
I20260709 12:25:58.983191  1875 rpcz_store.cc:276] 0709 12:25:55.972415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.981302 (+3008887us) service_pool.cc:224] Handling call
0709 12:25:58.981725 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.982530 (+   805us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.982810 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.985407  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3125) took 3012 ms. Trace:
I20260709 12:25:58.986089  1875 rpcz_store.cc:276] 0709 12:25:55.972729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.983889 (+3011160us) service_pool.cc:224] Handling call
0709 12:25:58.984188 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.984832 (+   644us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:58.984887 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:58.984958 (+    71us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:58.985025 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.985108 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:58.985330 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.974931  1880 rpcz_store.cc:276] 0709 12:25:55.832405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.897209 (+3064804us) service_pool.cc:224] Handling call
0709 12:25:58.951873 (+ 54664us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:58.972952 (+ 21079us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.973295 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.969861  1881 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:25:58.975473  1798 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:58.988310  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3122) took 3021 ms. Trace:
I20260709 12:25:58.988698  1881 rpcz_store.cc:276] 0709 12:25:55.967425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.968962 (+3001537us) service_pool.cc:224] Handling call
0709 12:25:58.969450 (+   488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.987972 (+ 18522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.988232 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:58.975346  1876 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.597s	user 0.013s	sys 0.000s
I20260709 12:25:58.990163  1876 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:25:58.991076  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3965) took 1351 ms. Trace:
I20260709 12:25:58.991516  1876 rpcz_store.cc:276] 0709 12:25:57.639846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.167001 (+527155us) service_pool.cc:224] Handling call
0709 12:25:58.991007 (+824006us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.979846  1800 rpcz_store.cc:276] 0709 12:25:57.629826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.963193 (+1333367us) service_pool.cc:224] Handling call
0709 12:25:58.963609 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:58.976611 (+ 13002us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.976948 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.994589  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1316) took 1364 ms. Trace:
I20260709 12:25:58.995114  1798 rpcz_store.cc:276] 0709 12:25:57.629654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.961260 (+1331606us) service_pool.cc:224] Handling call
0709 12:25:58.973410 (+ 12150us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.994262 (+ 20852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:58.994521 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.974498  1879 pending_rounds.cc:85] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 12
I20260709 12:25:58.997746  1731 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:59.001019  1879 pending_rounds.cc:107] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:25:59.001569  1879 raft_consensus.cc:2889] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:59.001741  1875 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:25:59.002470  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3121) took 3042 ms. Trace:
I20260709 12:25:59.003101  1879 rpcz_store.cc:276] 0709 12:25:55.959627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.903049 (+2943422us) service_pool.cc:224] Handling call
0709 12:25:58.953649 (+ 50600us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.002212 (+ 48563us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.002399 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:58.987447 10741 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.549s	user 0.006s	sys 0.012s
I20260709 12:25:59.002470  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3132) took 2816 ms. Trace:
I20260709 12:25:59.009410  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1442) took 2931 ms. Trace:
I20260709 12:25:59.020123  1730 rpcz_store.cc:276] 0709 12:25:56.077803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.107302 (+1029499us) service_pool.cc:224] Handling call
0709 12:25:57.107668 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.312689 (+205021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.009124 (+1696435us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x7b5400327dd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:59.009340 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":432795392}
I20260709 12:25:59.013814  1875 rpcz_store.cc:276] 0709 12:25:56.185687 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.000010 (+2814323us) service_pool.cc:224] Handling call
0709 12:25:59.001322 (+  1312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.002143 (+   821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.002400 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.022092  1730 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:25:59.022745  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3218) took 1385 ms. Trace:
I20260709 12:25:59.025573  1730 rpcz_store.cc:276] 0709 12:25:57.637773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.021342 (+1383569us) service_pool.cc:224] Handling call
0709 12:25:59.021738 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.022471 (+   733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.022693 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.024350  1881 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:59.027944  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3131) took 3020 ms. Trace:
I20260709 12:25:59.028389  1881 rpcz_store.cc:276] 0709 12:25:56.007354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.989483 (+2982129us) service_pool.cc:224] Handling call
0709 12:25:59.024157 (+ 34674us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.027531 (+  3374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.027858 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.030951  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1563) took 2922 ms. Trace:
I20260709 12:25:58.981065 10731 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:59.033342  2411 rpcz_store.cc:276] 0709 12:25:56.085822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.085944 (+   122us) service_pool.cc:224] Handling call
0709 12:25:56.086507 (+   563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:56.137054 (+ 50547us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:56.137101 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:56.137184 (+    83us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:56.137236 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.008358 (+2871122us) spinlock_profiling.cc:243] Waited 50.3 ms on lock 0x7b5400560c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:59.008407 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.008568 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":115577728}
I20260709 12:25:58.977598 10669 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.034149  1879 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:25:59.040462  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3134) took 2825 ms. Trace:
I20260709 12:25:59.040980  1879 rpcz_store.cc:276] 0709 12:25:56.214742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.007243 (+2792501us) service_pool.cc:224] Handling call
0709 12:25:59.033909 (+ 26666us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:59.039987 (+  6078us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.040372 (+   385us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.034108  1804 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:59.043488  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1320) took 1413 ms. Trace:
I20260709 12:25:59.043988  1804 rpcz_store.cc:276] 0709 12:25:57.630379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.004189 (+1373810us) service_pool.cc:224] Handling call
0709 12:25:59.033913 (+ 29724us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.043084 (+  9171us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.043407 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.046025  1804 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:25:58.977685 10678 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:25:59.038707  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:59.047958  1807 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:59.049469  1879 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:59.050611  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3033) took 1360 ms. Trace:
I20260709 12:25:59.046744  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1322) took 1389 ms. Trace:
I20260709 12:25:59.051071  1807 rpcz_store.cc:276] 0709 12:25:57.689982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.906944 (+1216962us) service_pool.cc:224] Handling call
0709 12:25:59.050534 (+143590us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.050710  1879 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:59.052204  1804 rpcz_store.cc:276] 0709 12:25:57.662590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.045013 (+1382423us) service_pool.cc:224] Handling call
0709 12:25:59.045522 (+   509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.046488 (+   966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.046689 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:58.942286 10602 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:59.016215 10706 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.068970  1881 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:59.071251  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3138) took 2781 ms. Trace:
I20260709 12:25:59.071730  1881 rpcz_store.cc:276] 0709 12:25:56.290172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.029451 (+2739279us) service_pool.cc:224] Handling call
0709 12:25:59.068728 (+ 39277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.070919 (+  2191us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.071181 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.098322  1804 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:25:59.098901  1807 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:59.099351  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1326) took 1410 ms. Trace:
I20260709 12:25:59.099498  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1325) took 1410 ms. Trace:
I20260709 12:25:59.099856  1807 rpcz_store.cc:276] 0709 12:25:57.688913 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.052571 (+1363658us) service_pool.cc:224] Handling call
0709 12:25:59.098548 (+ 45977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.099262 (+   714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.099446 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12672}
I20260709 12:25:59.099855  1804 rpcz_store.cc:276] 0709 12:25:57.689187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.053640 (+1364453us) service_pool.cc:224] Handling call
0709 12:25:59.096273 (+ 42633us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.098985 (+  2712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.099288 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.102632 10706 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:59.067157  1879 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:59.132043  1807 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:25:59.133142  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3045) took 1439 ms. Trace:
I20260709 12:25:59.133522  1807 rpcz_store.cc:276] 0709 12:25:57.693396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.100810 (+1407414us) service_pool.cc:224] Handling call
0709 12:25:59.131733 (+ 30923us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:59.132747 (+  1014us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.133064 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.137465  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1331) took 1773 ms. Trace:
W20260709 12:25:59.139287  1845 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.000s	sys 0.007s
W20260709 12:25:59.139729  1845 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.000s	sys 0.007s
I20260709 12:25:59.055070 10669 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:59.138051  2419 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.019s	sys 0.046s
I20260709 12:25:59.138096  1881 rpcz_store.cc:276] 0709 12:25:57.364223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.096489 (+1732266us) service_pool.cc:224] Handling call
0709 12:25:59.129857 (+ 33368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.130326 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:59.130370 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.137027 (+  6657us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:59.137104 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.137201 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.137383 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":425,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":4845}
W20260709 12:25:59.145807 10883 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:25:59.145426 10777 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:25:59.140033  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 696159us
I20260709 12:25:59.148387  1807 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: []
W20260709 12:25:59.148669  2419 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.019s	sys 0.046s
I20260709 12:25:59.163529  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3053) took 1468 ms. Trace:
I20260709 12:25:59.159157  1881 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:59.165480  1881 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:25:59.171113 10671 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.733s	user 0.008s	sys 0.004s
I20260709 12:25:59.171581 10671 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:59.171778 10669 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:59.158495 10869 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.160952 10659 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:59.164072  1807 rpcz_store.cc:276] 0709 12:25:57.695564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.134489 (+1438925us) service_pool.cc:224] Handling call
0709 12:25:59.148188 (+ 13699us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.163160 (+ 14972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.163453 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:59.185570 10882 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:25:59.169059 10742 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:59.191334  1845 leader_election.cc:400] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26.
I20260709 12:25:59.192224  1845 leader_election.cc:403] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:59.155511 10777 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:59.195904  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:59.197919  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:59.149580 10678 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:59.201208  1845 leader_election.cc:400] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20.
I20260709 12:25:59.205039  1845 leader_election.cc:403] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 19 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:59.206146  1807 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:59.206774  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:59.207213  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3054) took 1459 ms. Trace:
W20260709 12:25:59.204085  1728 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.002s	sys 0.010s
I20260709 12:25:59.202389  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3136) took 2954 ms. Trace:
I20260709 12:25:59.207957  1807 rpcz_store.cc:276] 0709 12:25:57.748146 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.188581 (+1440435us) service_pool.cc:224] Handling call
0709 12:25:59.205737 (+ 17156us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.206692 (+   955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.207103 (+   411us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.208616  1875 rpcz_store.cc:276] 0709 12:25:56.248310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.021579 (+2773269us) service_pool.cc:224] Handling call
0709 12:25:59.021886 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:59.023715 (+  1829us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:59.023749 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:59.024292 (+   543us) log.cc:844] Serialized 26 byte log entry
0709 12:25:59.191528 (+167236us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:59.191599 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.191695 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:59.202039 (+ 10344us) raft_consensus.cc:1685] finished
0709 12:25:59.202071 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.202302 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1369,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":7587,"thread_start_us":74197,"threads_started":1,"wal-append.queue_time_us":143235}
W20260709 12:25:59.211505  1728 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.002s	sys 0.010s
I20260709 12:25:59.214114  1875 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:59.187002 10671 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:59.199126 10873 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:59.216418 10703 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:59.203604 10631 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:59.213768 10666 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 18 FOLLOWER]: Advancing to term 20
I20260709 12:25:59.193075 10659 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:59.192145 10869 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:59.232908  1726 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 4028) took 2016 ms (client timeout 2075 ms). Trace:
W20260709 12:25:59.233335  1726 rpcz_store.cc:269] 0709 12:25:57.216781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.978641 (+761860us) service_pool.cc:224] Handling call
0709 12:25:59.232687 (+1254046us) spinlock_profiling.cc:243] Waited 54.4 ms on lock 0x7b54002b7fd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:59.232841 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":125037824}
I20260709 12:25:59.234890 10869 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:59.235767  1874 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 12. Preceding OpId from leader: term: 29 index: 13. (index mismatch)
W20260709 12:25:59.237538  1770 connection.cc:256] RPC call timeout handler was delayed by -1.4466748400000142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20260709 12:25:59.237918  1770 connection.cc:256] RPC call timeout handler was delayed by -1.4229575090002982s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20750000000000002s for a response
W20260709 12:25:59.238202  1770 connection.cc:256] RPC call timeout handler was delayed by -1.406618991999494s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2017s for a response
W20260709 12:25:59.239696 10862 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.792s	user 0.013s	sys 0.000s
I20260709 12:25:59.241300 10787 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:25:59.242615  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 144648us
I20260709 12:25:59.242836 10777 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:59.227583 10706 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:59.228639 10873 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:59.243211  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:59.219859 10678 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:59.244593  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:59.244532 10706 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:59.246161 10817 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:59.247189  1723 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:25:59.243358  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1444) took 2149 ms. Trace:
I20260709 12:25:59.248265 10709 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:59.248746 10709 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:25:59.246116  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:59.239737  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1435) took 3326 ms. Trace:
I20260709 12:25:59.250102  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3145) took 1683 ms. Trace:
I20260709 12:25:59.249750  1725 rpcz_store.cc:276] 0709 12:25:55.913187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.987436 (+1074249us) service_pool.cc:224] Handling call
0709 12:25:56.987880 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.909544 (+921664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.239505 (+1329961us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b5400212ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:59.239679 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1741758208}
I20260709 12:25:59.250818  1875 rpcz_store.cc:276] 0709 12:25:57.532238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.210819 (+1678581us) service_pool.cc:224] Handling call
0709 12:25:59.211129 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.214880 (+  3751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.215210 (+   330us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:59.238499  1770 connection.cc:256] RPC call timeout handler was delayed by -1.351608823999868s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20710000000000003s for a response
I20260709 12:25:59.247967  1724 rpcz_store.cc:276] 0709 12:25:57.094383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.133511 (+ 39128us) service_pool.cc:224] Handling call
0709 12:25:57.157029 (+ 23518us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.800826 (+643797us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.243128 (+1442302us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b54003ccdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:59.243304 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1220120960}
I20260709 12:25:59.253487  1807 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:59.256124 10709 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:59.257472 10709 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:25:59.238456 10714 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.830s	user 0.008s	sys 0.003s
I20260709 12:25:59.263384  1726 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:25:59.263996  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:59.264130  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3058) took 1161 ms. Trace:
I20260709 12:25:59.265149  1807 rpcz_store.cc:276] 0709 12:25:58.102682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.213062 (+1110380us) service_pool.cc:224] Handling call
0709 12:25:59.253240 (+ 40178us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.259086 (+  5846us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.264052 (+  4966us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.264541  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3969) took 1603 ms. Trace:
W20260709 12:25:59.266968  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.267596  1770 connection.cc:256] RPC call timeout handler was delayed by -1.2490851849997853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2643s for a response
I20260709 12:25:59.266420  1875 rpcz_store.cc:276] 0709 12:25:57.652290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.251886 (+1599596us) service_pool.cc:224] Handling call
0709 12:25:59.252322 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.252584 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:59.252622 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.254445 (+  1823us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:59.254708 (+   263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.254822 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.255141 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.270470  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1337) took 1563 ms. Trace:
I20260709 12:25:59.272389  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:59.273403  1730 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:25:59.274417  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:25:59.276423  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69757.4us
W20260709 12:25:59.277276  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.277936  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:25:59.278340  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:59.278658  2411 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:59.279429 10647 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:25:59.279856 10647 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:59.280200 10647 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:25:59.280535  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:59.281332 10851 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:59.281770  1845 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:59.282581 10851 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:59.287364  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:25:59.288759  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.289216  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.292865 10076 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:25:59.293583 10076 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:59.276432 10862 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.295354 10862 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:59.298861 10862 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:59.300150  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4041) took 1927 ms. Trace:
I20260709 12:25:59.301326  1807 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 12:25:59.301981  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 132084us
I20260709 12:25:59.302500  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3061) took 1198 ms. Trace:
I20260709 12:25:59.303187  1807 rpcz_store.cc:276] 0709 12:25:58.103771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.266049 (+1162278us) service_pool.cc:224] Handling call
0709 12:25:59.300755 (+ 34706us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.302126 (+  1371us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.302421 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.271324  1875 rpcz_store.cc:276] 0709 12:25:57.707666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.269432 (+1561766us) service_pool.cc:224] Handling call
0709 12:25:59.269815 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.270036 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:59.270085 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.270143 (+    58us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:59.270188 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.270256 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.270403 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.300647  1728 rpcz_store.cc:276] 0709 12:25:57.372689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.664837 (+1292148us) service_pool.cc:224] Handling call
0709 12:25:58.665269 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.665578 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:59.213958 (+548380us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.299728 (+ 85770us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.299793 (+    65us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:59.299846 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.299924 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.300079 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":546354,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"thread_start_us":633195,"threads_started":2}
I20260709 12:25:59.314708  1726 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:25:59.316054  1845 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:25:59.327034  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4022) took 2112 ms. Trace:
I20260709 12:25:59.327569  1727 rpcz_store.cc:276] 0709 12:25:57.214165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.381276 (+1167111us) service_pool.cc:224] Handling call
0709 12:25:58.403668 (+ 22392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.947768 (+544100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.326706 (+378938us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x7b54003cd2d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:59.326953 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1248526208}
I20260709 12:25:59.332701  1875 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
W20260709 12:25:59.334945  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.337101  2412 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1490) took 2244 ms (client timeout 2007 ms). Trace:
I20260709 12:25:59.342180  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1312) took 3723 ms. Trace:
W20260709 12:25:59.342581  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.342883  1872 rpcz_store.cc:276] 0709 12:25:55.619053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.223067 (+1604014us) service_pool.cc:224] Handling call
0709 12:25:57.223411 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.623717 (+400306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.341817 (+1718100us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b5400223b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:59.342094 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":762660096}
I20260709 12:25:59.344998  1872 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:25:59.345702  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1339) took 1200 ms. Trace:
I20260709 12:25:59.346071  1872 rpcz_store.cc:276] 0709 12:25:58.144847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.344150 (+1199303us) service_pool.cc:224] Handling call
0709 12:25:59.344563 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.345380 (+   817us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.345629 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.334242  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3150) took 1377 ms. Trace:
I20260709 12:25:59.381930  1875 rpcz_store.cc:276] 0709 12:25:57.957187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.331051 (+1373864us) service_pool.cc:224] Handling call
0709 12:25:59.331916 (+   865us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.333592 (+  1676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.334110 (+   518us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:59.388358  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3046) took 1694 ms. Trace:
W20260709 12:25:59.451118  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 135182us
I20260709 12:25:59.452247  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:59.337610  2412 rpcz_store.cc:269] 0709 12:25:57.092689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.167445 (+ 74756us) service_pool.cc:224] Handling call
0709 12:25:59.326174 (+2158729us) spinlock_profiling.cc:243] Waited 594 ms on lock 0x7b54006009d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:59.337008 (+ 10834us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1365906688}
W20260709 12:25:59.467818  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68936.2us
W20260709 12:25:59.468355  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98093.5us
W20260709 12:25:59.469017  1770 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.718s (SENT)
W20260709 12:25:59.470305  1770 leader_election.cc:336] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 1.800s (SENT)
I20260709 12:25:59.470674  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:25:59.310853 10851 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:59.472586  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:59.477981  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:25:59.511216  2411 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:59.479833 10831 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:25:59.479609 10873 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:59.482551 10869 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:59.476430 10903 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:59.504194 10851 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:59.506125 10867 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.523371 10436 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.521261 10862 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.529855 10765 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:59.535847 10903 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:25:59.535437 10647 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:59.542066 10867 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:59.556170 10805 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.544374 10436 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:59.582253 10631 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26.
W20260709 12:25:59.588590 10902 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:25:59.590265 10903 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:59.593757 10324 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:59.597812 10666 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 19. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20.
I20260709 12:25:59.547945 10862 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:25:59.600431  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 122538us
I20260709 12:25:59.601540 10862 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:59.602891 10805 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:25:59.608755 10436 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:59.609072 10647 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:25:59.610666 10799 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.160s	user 0.013s	sys 0.002s
I20260709 12:25:59.611222 10867 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:25:59.611430 10799 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:25:59.601529  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:59.816439  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:59.818101  1803 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:25:59.716058  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.822377  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.744352  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:59.744231  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:59.824010  1728 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:25:59.824497  1726 raft_consensus.cc:2410] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 27 because replica has last-logged OpId of term: 25 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 20.
I20260709 12:25:59.825847  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:59.826030  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:59.745159  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:25:59.827129  1728 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:25:59.827659  1727 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:25:59.828428  1728 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:25:59.829507  1728 raft_consensus.cc:606] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:25:59.829661  1727 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:59.830132  1728 raft_consensus.cc:614] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:59.744352  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:59.746884  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:25:59.763623  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 288202us
W20260709 12:25:59.763837  1729 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1439) took 3774 ms (client timeout 2184 ms). Trace:
W20260709 12:25:59.833638  1729 rpcz_store.cc:269] 0709 12:25:55.988973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:56.018272 (+ 29299us) service_pool.cc:224] Handling call
0709 12:25:59.762953 (+3744681us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b54002aa654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:59.763443 (+   490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":757164416}
I20260709 12:25:59.771916  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:25:59.772174  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3215) took 2200 ms. Trace:
I20260709 12:25:59.835232  1730 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:25:59.835541  1731 rpcz_store.cc:276] 0709 12:25:57.571886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.794425 (+1222539us) service_pool.cc:224] Handling call
0709 12:25:58.794693 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.005355 (+210662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.771447 (+766092us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54002a9c54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:59.771890 (+   443us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":456833408}
I20260709 12:25:59.781822  1802 raft_consensus.cc:2368] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28.
I20260709 12:25:59.782763  1807 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:25:59.389217  1804 rpcz_store.cc:276] 0709 12:25:57.693698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.101903 (+1408205us) service_pool.cc:224] Handling call
0709 12:25:59.141996 (+ 40093us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.142334 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:59.203743 (+ 61409us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.387775 (+184032us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.387876 (+   101us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:59.387950 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.388062 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.388268 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":56225,"prepare.run_cpu_time_us":117,"prepare.run_wall_time_us":8133,"raft.queue_time_us":232,"thread_start_us":234057,"threads_started":2}
I20260709 12:25:59.784788  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3207) took 2574 ms. Trace:
W20260709 12:25:59.839423  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 68030.9us
W20260709 12:25:59.841513  1770 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Timed out: RequestConsensusVote RPC to 127.0.140.65:35067 timed out after 2.675s (SENT)
I20260709 12:25:59.843310  1807 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:25:59.843586  1726 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:25:59.788467  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.844743  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.801656  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:59.803562  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:59.805255  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:25:59.848603  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.849354  1798 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:25:59.812048  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:59.812723  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:25:59.839035  1723 rpcz_store.cc:276] 0709 12:25:57.210546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.165029 (+954483us) service_pool.cc:224] Handling call
0709 12:25:59.246858 (+1081829us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.247561 (+   703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.784152 (+536591us) spinlock_profiling.cc:243] Waited 851 ms on lock 0x7b54003ccdd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:59.784531 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1957502208}
I20260709 12:25:59.784968  1801 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:59.848667  1806 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 19.
W20260709 12:25:59.816493  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 214851us
I20260709 12:25:59.787063  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4020) took 2573 ms. Trace:
I20260709 12:25:59.864554  1727 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:25:59.863694  1732 rpcz_store.cc:276] 0709 12:25:57.213635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.375102 (+1161467us) service_pool.cc:224] Handling call
0709 12:25:59.763401 (+1388299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.764207 (+   806us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:59.765020 (+   813us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.785480 (+ 20460us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.785570 (+    90us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:59.785630 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.785761 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.786617 (+   856us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54002aa650. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:59.786831 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3129405312,"thread_start_us":14023,"threads_started":1,"wal-append.queue_time_us":27777}
I20260709 12:25:59.865242  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:59.854576  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 3010) took 2261 ms. Trace:
I20260709 12:25:59.866084  1801 rpcz_store.cc:276] 0709 12:25:57.593391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.952865 (+1359474us) service_pool.cc:224] Handling call
0709 12:25:58.999028 (+ 46163us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.852587 (+853559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.854092 (+  1505us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b540026f054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:59.854471 (+   379us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1796738816}
W20260709 12:25:59.839355  1802 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3036) took 2148 ms (client timeout 1847 ms). Trace:
W20260709 12:25:59.868892  1802 rpcz_store.cc:269] 0709 12:25:57.690751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.159741 (+468990us) service_pool.cc:224] Handling call
0709 12:25:59.839000 (+1679259us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54001d4554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:25:59.839193 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3221631872}
I20260709 12:25:59.862377  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:25:59.870887  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:59.842324  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:25:59.616458  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3133) took 3413 ms. Trace:
I20260709 12:25:59.871461  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:25:59.871909  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.875797  1876 rpcz_store.cc:276] 0709 12:25:56.202926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.002553 (+2799627us) service_pool.cc:224] Handling call
0709 12:25:59.002941 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.003207 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:59.191967 (+188760us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.615904 (+423937us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:59.615977 (+    73us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:59.616048 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.616147 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:59.616305 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":206102,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":146,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":118,"thread_start_us":549659,"threads_started":2,"wal-append.queue_time_us":354536}
I20260709 12:25:59.877926  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:25:59.878545  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3141) took 2365 ms. Trace:
W20260709 12:25:59.879056  1880 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.003s	sys 0.010s
W20260709 12:25:59.879594  1880 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.003s	sys 0.010s
I20260709 12:25:59.879004  1874 rpcz_store.cc:276] 0709 12:25:57.512497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.200657 (+1688160us) service_pool.cc:224] Handling call
0709 12:25:59.219580 (+ 18923us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.236163 (+ 16583us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:59.877937 (+641774us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b5400222754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:25:59.878318 (+   381us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36339456}
I20260709 12:25:59.643591 10076 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:25:59.623956  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 294042us
I20260709 12:25:59.616257 10534 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:25:59.616585 10805 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:25:59.656345 10799 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
W20260709 12:25:59.887032  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:59.923280  2416 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.627s	user 0.000s	sys 0.016s
W20260709 12:25:59.986089  2416 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.001s	sys 0.017s
I20260709 12:25:59.987623 10647 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:59.988605  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:59.651947 10869 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:59.923372  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:59.964015 10709 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.972985  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:59.957572 10906 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:25:59.960496  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:59.973819 10764 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:59.974630 10873 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:59.975347  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:59.975994 10666 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:25:59.976672 10718 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:00.007890  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:25:59.990604  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:59.974349 10669 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.007547  1728 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 22 because replica has last-logged OpId of term: 20 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
I20260709 12:25:59.993229  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:25:59.994429 10709 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:25:59.973029 10690 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:25:59.971515 10807 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:59.976140 10602 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:00.035224 10631 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:26:00.036160  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.037171  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.036010 10602 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.038863  1807 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:26:00.039433  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.040373  1726 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:26:00.041750  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.043130  1876 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:00.044576  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.046700 10873 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:00.047224 10873 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:00.048193  1881 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.048892 10873 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.050092  1881 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:00.031863  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.032892  1727 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:00.034770  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:00.029474  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 100879us
I20260709 12:26:00.051855 10666 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:26:00.053000 10903 consensus_queue.cc:1175] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "075704438992440697732181e49ccd09" responder_term: 25 status { last_received { term: 23 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:59.970016 10777 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:00.063490  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.034626  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.064563 10764 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.025274  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.008934 10906 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:00.067624 10718 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.068156 10631 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:00.069111 10669 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.069581  2416 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:00.074949 10076 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:00.075168 10690 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:00.078183  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.078498  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.079766  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:00.080968 10886 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.954s	user 0.005s	sys 0.026s
I20260709 12:26:00.082278  1872 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:26:00.087337  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.089453 10873 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:00.095604 10706 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.100472 10666 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:00.126174  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.127038  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 255881us
I20260709 12:26:00.127283  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.128263  1875 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:26:00.129417  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.130478  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:00.130962  1874 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:00.135102 10777 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:00.138345  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 90214.2us
I20260709 12:26:00.138659 10706 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.141784  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:00.142314 10706 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.143692  1732 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:26:00.144297 10647 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:00.147311 10869 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:00.147905  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.149446 10869 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:00.150332 10869 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:26:00.151732  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.153118 10764 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.155907 10718 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:00.130702  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.112145 10906 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:00.162194 10914 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:00.161983 10918 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:26:00.163733  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53315.5us
I20260709 12:26:00.178160  1730 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:26:00.178929  1730 raft_consensus.cc:606] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 28 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:26:00.181190  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:00.178097  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114045us
I20260709 12:26:00.185835  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:00.200681  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:00.202020  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:00.207808 10666 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:00.211437  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.198684  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:00.223230  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.224434  1732 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:00.226682  1730 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:26:00.210711  1875 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:00.227867 10669 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.231256  1726 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793"
dest_uuid: "075704438992440697732181e49ccd09"
mode: GRACEFUL
new_leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
 from {username='slave'} at 127.0.0.1:40200
I20260709 12:26:00.232090  1726 raft_consensus.cc:606] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Received request to transfer leadership to TS 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:00.232494  1726 raft_consensus.cc:614] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:26:00.223518 10879 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 2.245s	user 0.015s	sys 0.017s
W20260709 12:26:00.233753  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.233083  1876 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
W20260709 12:26:00.241899  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.242528  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.245167  1878 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3120) took 4286 ms (client timeout 5025 ms). Trace:
W20260709 12:26:00.245675  1878 rpcz_store.cc:269] 0709 12:25:55.959142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.159309 (+1200167us) service_pool.cc:224] Handling call
0709 12:26:00.233156 (+3073847us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b5400051754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:00.244926 (+ 11770us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":778636544}
I20260709 12:26:00.246402  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:00.246968  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.247936  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.248775  2417 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:26:00.249778  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.250356  2417 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:26:00.250941 10915 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:00.251446 10915 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.251780 10915 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:00.255442   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:26:00.263639 10913 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:00.264423 10927 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:00.266331 10804 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:00.266543 10678 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:26:00.271281 10913 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:26:00.271831 10913 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:00.273206 10913 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.275138 10076 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:00.275713 10076 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.276070 10076 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:00.276296 10913 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:00.279536 10873 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:00.280220 10873 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:00.281482 10873 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.282648  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:00.283676  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:00.285372 10436 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:00.285431  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:00.286078 10436 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.283639  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3948) took 4293 ms. Trace:
I20260709 12:26:00.146731 10647 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
W20260709 12:26:00.301295  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68423.6us
W20260709 12:26:00.312446  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 79048.7us
W20260709 12:26:00.313259  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.312321  1875 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:00.294658 10436 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:00.318351  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:00.316157  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.320722 10926 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:00.321714 10923 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:26:00.322184  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.322443 10666 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:00.324596 10851 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:00.314226  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:00.327607  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.322072  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1667) took 3298 ms. Trace:
I20260709 12:26:00.332934  1881 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 19. (term mismatch)
I20260709 12:26:00.345857 10324 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:00.347023 10891 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 12:26:00.322220 10923 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.286760  1880 rpcz_store.cc:276] 0709 12:25:55.990149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.987593 (+2997444us) service_pool.cc:224] Handling call
0709 12:25:59.013019 (+ 25426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.029999 (+ 16980us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:59.215497 (+185498us) log.cc:844] Serialized 14 byte log entry
0709 12:26:00.237402 (+1021905us) log.cc:844] Serialized 14 byte log entry
0709 12:26:00.283004 (+ 45602us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:00.283085 (+    81us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:00.283209 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.283321 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:00.283552 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35111,"prepare.queue_time_us":408491,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":108563,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":279,"thread_start_us":927583,"threads_started":4,"wal-append.queue_time_us":443785}
I20260709 12:26:00.349931  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1566) took 2893 ms. Trace:
I20260709 12:26:00.366294  1732 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:26:00.303512 10669 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:00.333334  1881 pending_rounds.cc:85] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 18
W20260709 12:26:00.371253  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 155253us
I20260709 12:26:00.370415  2410 rpcz_store.cc:276] 0709 12:25:57.456354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.456520 (+   166us) service_pool.cc:224] Handling call
0709 12:25:57.457024 (+   504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:58.164721 (+707697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.349348 (+2184627us) spinlock_profiling.cc:243] Waited 673 ms on lock 0x7b540048a3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:00.349717 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1548749184}
I20260709 12:26:00.374262 10647 consensus_queue.cc:1243] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 24.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:26:00.372329  1881 pending_rounds.cc:107] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 24.19
I20260709 12:26:00.378448  1874 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:26:00.299494 10914 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.386932  1881 raft_consensus.cc:2889] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:00.391492 10831 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.391829 10647 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:26:00.392489 10831 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.393882  2416 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
W20260709 12:26:00.395236  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 264941us
I20260709 12:26:00.395687  1732 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:00.396057  1732 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:00.396265  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:00.397157 10647 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:00.398904 10921 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.399664 10921 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.402204 10921 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:00.406847 10920 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.407572 10920 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.402311  1732 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.409124  1732 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:00.409930 10920 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.410646 10669 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.386657 10631 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.412901 10669 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:00.372849 10923 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:00.416330  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.416544  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.417280  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.376713 10764 raft_consensus.cc:2764] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:26:00.329051  2414 rpcz_store.cc:276] 0709 12:25:57.023464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.026382 (+  2918us) service_pool.cc:224] Handling call
0709 12:25:57.026613 (+   231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:57.030086 (+  3473us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:57.352944 (+322858us) log.cc:844] Serialized 14 byte log entry
0709 12:26:00.007260 (+2654316us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b18004c02a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab97823a
0709 12:26:00.179998 (+172738us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:00.197167 (+ 17169us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:00.297231 (+100064us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.302201 (+  4970us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:00.321842 (+ 19641us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":448992,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":203686,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":234,"spinlock_wait_cycles":36878080,"thread_start_us":616860,"threads_started":3,"wal-append.queue_time_us":283531,"wal-append.run_cpu_time_us":1341,"wal-append.run_wall_time_us":1118454}
I20260709 12:26:00.350848 10906 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:00.421072  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 135739us
I20260709 12:26:00.427215  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1332) took 3062 ms. Trace:
I20260709 12:26:00.427738  1881 rpcz_store.cc:276] 0709 12:25:57.364515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.148372 (+1783857us) service_pool.cc:224] Handling call
0709 12:25:59.159020 (+ 10648us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:59.953131 (+794111us) spinlock_profiling.cc:243] Waited 5.39 ms on lock 0x7b4c00234d00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa0a4a20 00007f16aa0b5a29 00007f16aa07b644 00007f16aa0a2f2d 00007f16aa0a2b93 00007f16abdd318b 00007f16abdd3cb5 00007f16abdcef30 00007f16abdd07de 00007f16abddab49 00007f16abddb416
0709 12:26:00.426819 (+473688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.427115 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12387200}
I20260709 12:26:00.429109  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:00.434680  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 60558.4us
I20260709 12:26:00.435254 10924 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.438241  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:00.434283  1872 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
W20260709 12:26:00.436194  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49004.3us
I20260709 12:26:00.439610  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.443161  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:00.384284 10918 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:26:00.447124  1873 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:00.438401 10647 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:00.451097 10925 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.071s
I20260709 12:26:00.451467 10831 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.460827  1876 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:00.455654 10669 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:00.382970 10690 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.447021 10873 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:00.468708  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.451956  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:00.458652  1874 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:00.467727  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.466257  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.464811  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.466666  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.473663  2416 raft_consensus.cc:2410] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:26:00.466573 10924 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.476449  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.477851 10924 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.478214  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:00.476620  2415 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1493) took 3352 ms (client timeout 2889 ms). Trace:
I20260709 12:26:00.452809 10918 consensus_queue.cc:1243] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 20.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:26:00.476573  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:00.487115  2410 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
W20260709 12:26:00.479187  2415 rpcz_store.cc:269] 0709 12:25:57.124601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.195610 (+ 71009us) service_pool.cc:224] Handling call
0709 12:26:00.476241 (+3280631us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b54004993d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:00.476404 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1303339904}
I20260709 12:26:00.490361  2414 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:26:00.490819  2414 pending_rounds.cc:85] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 16
I20260709 12:26:00.462363 10669 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:00.493671  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.491536 10932 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:26:00.505326  1732 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:26:00.505690  2413 raft_consensus.cc:2376] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 25. Re-sending same reply.
I20260709 12:26:00.505455 10933 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.069s
I20260709 12:26:00.506556  2414 pending_rounds.cc:107] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:26:00.516700  2414 raft_consensus.cc:2889] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:00.512794 10884 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.411s	user 0.008s	sys 0.021s
W20260709 12:26:00.492328  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73788us
W20260709 12:26:00.519443  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56331.9us
I20260709 12:26:00.461829 10920 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.029s
I20260709 12:26:00.523156  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.529724 10862 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:00.530210 10862 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:00.531353 10862 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.513360 10915 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.533180  1878 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:00.533597  2413 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:00.535420 10906 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:00.535629  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.536803  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.538731 10703 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:00.539649 10931 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:00.540143 10931 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:00.541385 10931 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.542114 10703 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:26:00.543133  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 61700.1us
I20260709 12:26:00.534448 10915 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:00.544121  2410 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:26:00.518131 10669 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:00.543030  1878 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:26:00.547441  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.549474  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.549757  1876 raft_consensus.cc:2376] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21. Re-sending same reply.
I20260709 12:26:00.532742 10862 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:00.552029  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 121525us
I20260709 12:26:00.516481 10932 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.538343 10925 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.008s
I20260709 12:26:00.548383 10324 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:00.485675 10076 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.556497 10932 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:00.557621  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.582963  1878 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:00.543777 10706 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:26:00.584374  1878 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:26:00.587172  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.588303 10841 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:00.588876  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.589743  1723 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 25 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:26:00.591238  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.591894  1729 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:00.592787  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.593483  1729 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:00.594297  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.563645 10709 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.564124  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:00.601213 10534 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:00.565586 10931 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:00.565716  1874 raft_consensus.cc:2376] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24. Re-sending same reply.
I20260709 12:26:00.603374  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.604151  1731 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:00.604473  1875 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:00.597159  1725 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:26:00.605118  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.605908  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.606745  1725 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:00.607585  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.608482  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1666) took 3585 ms. Trace:
I20260709 12:26:00.608708  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.593483  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.567189 10324 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.610054  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.610661 10324 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:00.580438 10076 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:00.596997  1873 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:00.573803 10666 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.025s
I20260709 12:26:00.573951 10873 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:26:00.577643  1807 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:00.579021  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:00.601529  2410 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:26:00.570988  1879 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:00.613590  1728 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:26:00.605944  1731 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 12:26:00.609643  1725 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:00.611011  1876 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 20: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:00.575897  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:00.578155  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.590616  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.628768  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.630192  1880 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:00.612188 10914 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:00.632604 10915 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:00.633147  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.633911  2410 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:26:00.635900  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.612330 10076 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:00.653088  1873 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:26:00.609006  2419 rpcz_store.cc:276] 0709 12:25:57.023110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:57.025871 (+  2761us) service_pool.cc:224] Handling call
0709 12:25:57.026187 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:58.085410 (+1059223us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:58.085439 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:58.098314 (+ 12875us) log.cc:844] Serialized 26 byte log entry
0709 12:25:58.366517 (+268203us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:58.916186 (+549669us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:25:59.160515 (+244329us) log.cc:844] Serialized 14 byte log entry
0709 12:25:59.160786 (+   271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.607954 (+1447168us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400572e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:00.608023 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:00.608102 (+    79us) raft_consensus.cc:1685] finished
0709 12:26:00.608119 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:00.608285 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34318,"prepare.queue_time_us":813154,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":1589,"raft.queue_time_us":4914,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":19334,"spinlock_wait_cycles":2444629760,"thread_start_us":1034122,"threads_started":2,"wal-append.queue_time_us":685371,"wal-append.run_cpu_time_us":5142,"wal-append.run_wall_time_us":1481099}
W20260709 12:26:00.656368  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 104922us
I20260709 12:26:00.656862  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.657069  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.657290  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:00.657547  2419 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:00.642693  1881 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:26:00.658206  1730 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:26:00.651520  1878 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:26:00.652155  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.649590  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.561291 10602 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.668309  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.669591  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.666540  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.669711 10659 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:00.658690  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:00.676086  1727 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
W20260709 12:26:00.697583  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.689095  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.695695  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.700299  2413 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:26:00.712342  1725 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 13. (term mismatch)
I20260709 12:26:00.713008  1725 pending_rounds.cc:85] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 12
I20260709 12:26:00.713303  1725 pending_rounds.cc:107] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 24.13
I20260709 12:26:00.713583  1725 raft_consensus.cc:2889] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:00.716336  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:00.719163  1876 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:26:00.721000  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.724727  2410 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
W20260709 12:26:00.724112  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 186445us
I20260709 12:26:00.659605 10678 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:00.728605  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:00.729349  2413 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:00.736479  1798 raft_consensus.cc:2410] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:26:00.730656 10666 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:26:00.751890  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 93321.2us
I20260709 12:26:00.754913  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:00.755822 10799 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.755622 10436 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.732561 10672 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:00.760849  1728 pending_rounds.cc:85] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
I20260709 12:26:00.771458  1728 pending_rounds.cc:107] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 12:26:00.771700  1728 raft_consensus.cc:2889] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:00.772903  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.764358 10666 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.781968 10757 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:00.789811 10916 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.466s	user 0.001s	sys 0.000s
I20260709 12:26:00.792606  1725 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: [24.14-24.14]
W20260709 12:26:00.797390  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.800226  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:00.804138 10937 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:00.804802 10937 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:00.805255 10937 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:00.771052 10709 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.037s
W20260709 12:26:00.810949  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.818020  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:00.828657  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.770004 10672 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:00.832273  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63000.1us
I20260709 12:26:00.829828 10718 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.833675  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:00.838666  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.843309  1879 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:26:00.837020 10659 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.019s
W20260709 12:26:00.857127  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 179679us
I20260709 12:26:00.858667  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:00.859393 10851 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:00.860118 10851 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:00.864038 10936 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:26:00.864315 10718 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.773887 10799 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.882985 10777 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.885622  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:00.887562  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:00.892356  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:00.893033  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:00.835781 10672 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:00.895501  1872 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:00.895934  1872 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:00.896381  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:00.897086  1872 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.874154  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.893325  1803 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:00.871088  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:00.898746  1724 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:00.899366  1725 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:00.899863  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:00.900147  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.901258  1724 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:00.901592  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:00.894101  1804 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:00.905921  1723 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:26:00.908427 10436 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:00.923295 10915 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:00.924180 10915 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.925930 10610 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:00.926849  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.929150 10888 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.411s	user 0.005s	sys 0.027s
W20260709 12:26:00.936367  1771 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 2.153s (SENT) [suppressed 81 similar messages]
W20260709 12:26:00.944873  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 86323.3us
I20260709 12:26:00.946933  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:00.948176  1725 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:00.826424 10666 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:00.955613  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.956344  1878 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:00.956454  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.957036  1876 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:00.957300  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.957998  1878 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:26:00.958081  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.958886 10862 raft_consensus.cc:2764] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:26:00.959997 10742 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:00.960381 10742 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:00.960731 10742 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:00.963290  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:00.963948  1881 raft_consensus.cc:2376] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20. Re-sending same reply.
I20260709 12:26:00.964910 10637 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:00.964956  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:00.965829 10534 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.968513  1800 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:26:00.972049  1876 raft_consensus.cc:2410] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 12.
I20260709 12:26:00.952389  1802 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:26:00.953944 10891 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:00.954291 10914 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:00.969974 10906 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:00.951575  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1330) took 3507 ms. Trace:
I20260709 12:26:00.985678  1872 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:00.966341 10534 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:00.988153  1879 rpcz_store.cc:276] 0709 12:25:57.363689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.042068 (+1678379us) service_pool.cc:224] Handling call
0709 12:25:59.042533 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:00.570261 (+1527728us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x7b58007edad0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd340aa 00007f16abdb6447 00007f16abdcefe7 00007f16abdd07de 00007f16abddab49 00007f16abddb416 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4
0709 12:26:00.855572 (+285311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:00.870604 (+ 15032us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49034112}
W20260709 12:26:00.992537  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:00.996533  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:00.997241  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:00.997915  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 130780us
I20260709 12:26:00.998229  1878 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:00.998766  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:01.004247  1800 raft_consensus.cc:2368] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26.
I20260709 12:26:01.004043  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:01.009557  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.003547 10647 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.012465 10647 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.014945 10932 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:00.982079 10891 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.019126 10891 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:01.007686 10914 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.025084 10324 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.003229 10750 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.024s
I20260709 12:26:01.028508   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:01.028868 10718 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.030383   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:01.035281 10671 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.035943 10671 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.038280 10671 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:01.041011 10534 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:01.041812  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:01.042820  1800 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:01.045020  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:01.045276  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:01.046293  1800 raft_consensus.cc:2376] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23. Re-sending same reply.
I20260709 12:26:01.023046 10906 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.025528 10914 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:01.034188  2383 connection.cc:256] RPC call timeout handler was delayed by -2.8695736939998824s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
I20260709 12:26:01.062294  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:01.069027  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 181517us
I20260709 12:26:01.073230  1803 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:26:01.076378  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:01.077382 10671 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:26:01.079097 10867 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:01.080842  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:01.081689 10671 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:01.083235  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:01.084017 10671 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:01.084410 10671 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.084612  1872 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:26:01.084740 10671 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:01.085894  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:01.093757  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:01.103041  1770 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:01.110463  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.119797  1807 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:01.101326 10926 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:26:01.127179  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:01.130535  1807 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:01.130676  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:01.133301  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60685.2us
I20260709 12:26:01.134199  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:26:01.153339  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50364.2us
I20260709 12:26:01.164095 10324 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:01.168450 10937 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.170923 10867 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:26:01.172369  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:01.173470  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:01.245049  1800 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
W20260709 12:26:01.174603  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 56796.4us
I20260709 12:26:01.131443 10777 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:01.112530 10706 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:01.097510 10436 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.032s
W20260709 12:26:01.178062  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.181638  1804 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:26:01.187604  1724 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:01.191866  1725 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:01.186661 10757 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.009s
I20260709 12:26:01.131958 10647 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.195880 10841 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.187031 10862 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:01.207288 10923 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.773s	user 0.004s	sys 0.011s
W20260709 12:26:01.219561  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.230753 10602 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:26:01.203953 10669 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.642s	user 0.014s	sys 0.001s
I20260709 12:26:01.233086 10906 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.232967 10665 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:01.233325  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 99220.7us
I20260709 12:26:01.236902  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1500) took 1273 ms. Trace:
I20260709 12:26:01.237185  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:01.235059 10851 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.238205  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 3023) took 1391 ms. Trace:
I20260709 12:26:01.239459  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.249042  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:01.243184  1878 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:01.289328  1799 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:26:01.253654  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:01.260454  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 185525us
I20260709 12:26:01.262931  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:01.293210  1724 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:01.269290  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.291354  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:01.294106  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57011.7us
I20260709 12:26:01.270318  1730 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:26:01.289273  1798 rpcz_store.cc:276] 0709 12:25:59.847210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:59.850172 (+  2962us) service_pool.cc:224] Handling call
0709 12:26:00.198495 (+348323us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:26:01.237937 (+1039442us) spinlock_profiling.cc:243] Waited 504 ms on lock 0x7b5400067054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:01.238134 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1195837056}
I20260709 12:26:01.274524  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:01.296175  1806 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:01.279844  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:01.296653  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:01.283910  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.274497  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:01.298713  1877 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:01.298941  1802 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:01.299784  1725 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:26:01.300352 10851 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:01.300835  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:01.303681  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:01.304440  1731 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:26:01.305902 10647 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:01.296552 10703 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
I20260709 12:26:01.306569  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3160) took 1139 ms. Trace:
I20260709 12:26:01.306706 10647 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:01.307442  1875 rpcz_store.cc:276] 0709 12:26:00.167587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.362306 (+194719us) service_pool.cc:224] Handling call
0709 12:26:01.306275 (+943969us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54001f7554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:01.306463 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":424016256}
I20260709 12:26:01.283099  2417 rpcz_store.cc:276] 0709 12:25:59.963536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.279173 (+315637us) service_pool.cc:224] Handling call
0709 12:26:00.279731 (+   558us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:00.431789 (+152058us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:26:00.525634 (+ 93845us) log.cc:844] Serialized 14 byte log entry
0709 12:26:00.727915 (+202281us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:00.744619 (+ 16704us) log.cc:844] Serialized 26 byte log entry
0709 12:26:00.744894 (+   275us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:00.744951 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.235999 (+491048us) spinlock_profiling.cc:243] Waited 5.15 ms on lock 0x7b1800390100. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419
0709 12:26:01.236549 (+   550us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54005438d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:01.236604 (+    55us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:01.236651 (+    47us) raft_consensus.cc:1685] finished
0709 12:26:01.236665 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:01.236839 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25383,"prepare.queue_time_us":225239,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":57,"raft.queue_time_us":23034,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":11944,"spinlock_wait_cycles":307974272,"thread_start_us":242316,"threads_started":2,"wal-append.queue_time_us":302675}
I20260709 12:26:01.309370  1874 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:01.298166  1878 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:01.299732 10794 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.282290  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:01.244717 10742 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.283113 10777 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:01.314720 10851 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:26:01.361706  1845 leader_election.cc:400] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26.
I20260709 12:26:01.363684  1845 leader_election.cc:403] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:26:01.297313 10906 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:01.365111  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:01.367383  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57914us
I20260709 12:26:01.285467 10669 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:01.376343  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.316948 10703 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:26:01.284159 10923 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.384507  1728 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:01.377772  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:01.288479 10862 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:01.386050  2383 connection.cc:256] RPC call timeout handler was delayed by -2.0856983859998763s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19770000000000001s for a response
I20260709 12:26:01.386356  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:01.390092  2383 connection.cc:256] RPC call timeout handler was delayed by -1.6350260959998195s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15760000000000002s for a response
W20260709 12:26:01.390662  2383 connection.cc:256] RPC call timeout handler was delayed by -1.6257983860000422s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.24880000000000002s for a response
W20260709 12:26:01.391000  2383 connection.cc:256] RPC call timeout handler was delayed by -1.4307260959999439s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18030000000000002s for a response
W20260709 12:26:01.391299  2383 connection.cc:256] RPC call timeout handler was delayed by -1.4070983860001434s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2731s for a response
W20260709 12:26:01.391577  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1980091269997502s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28500000000000003s for a response
I20260709 12:26:01.327467 10932 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.294155 10841 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.383845 10637 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26.
I20260709 12:26:01.387897  1878 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:01.292001 10665 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.390416  1875 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:01.400673  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 99957.9us
I20260709 12:26:01.403453  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:01.407609  1878 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:01.415019  1725 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
W20260709 12:26:01.417946  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 178233us
I20260709 12:26:01.409538 10757 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:01.427278 10805 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:01.427770 10805 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:01.429152 10665 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:01.428844 10805 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.430156 10805 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:01.358175 10794 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.436473 10923 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.437191  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:01.437997  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:01.439515 10750 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:26:01.440689 10862 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:01.360737 10940 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.121s
I20260709 12:26:01.442008 10742 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.442760 10742 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:26:01.444466 10951 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:26:01.445024 10841 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:01.445848  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.447248  1729 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:01.440552 10750 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:01.380599 10666 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.451444  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.451912 10750 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:01.452474 10750 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:01.452922  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.454303  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:01.454691 10926 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:01.401001 10076 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:01.455235  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77586.5us
I20260709 12:26:01.455451 10926 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:01.456194  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:01.456705 10926 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.459555  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 3990) took 1229 ms. Trace:
I20260709 12:26:01.460022 10669 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:01.459945  1874 rpcz_store.cc:276] 0709 12:26:00.229777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.634695 (+404918us) service_pool.cc:224] Handling call
0709 12:26:01.309146 (+674451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:01.309826 (+   680us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:01.459309 (+149483us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b54001f7550. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:01.459488 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1502564736}
I20260709 12:26:01.385574 10436 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.076s
I20260709 12:26:01.458276  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:01.459151 10647 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:01.438002 10937 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:01.466142 10851 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:01.466918 10647 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:01.468384  1872 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:01.467995 10647 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.470446 10794 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:01.466729 10851 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.472661  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:01.472811 10851 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:01.474078  1723 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:26:01.453553 10750 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.465811 10891 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.475374  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.503193  1803 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:01.477169 10750 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:01.512779  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:01.503269  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 134102us
I20260709 12:26:01.515278  1732 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 24
W20260709 12:26:01.522179  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 118817us
I20260709 12:26:01.523248  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:01.526580  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 106694us
I20260709 12:26:01.527030  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:01.527910  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:01.440639 10799 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:01.536345  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 171947us
I20260709 12:26:01.465854 10666 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.539811  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:01.506108 10757 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:01.521601 10862 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:01.554482  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:01.543838  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:01.455704 10076 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.526124 10923 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.558151  2410 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:26:01.552678  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.556069  1724 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:01.536722 10706 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:01.569682  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:01.556628 10947 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.177s	user 0.000s	sys 0.002s
I20260709 12:26:01.573064  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:01.574070  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50933.6us
I20260709 12:26:01.579674  1725 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:01.581300  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:01.551558 10891 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:01.585186  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.572633 10940 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:01.536510 10669 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.036s
W20260709 12:26:01.576647 10672 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.682s	user 0.007s	sys 0.016s
I20260709 12:26:01.586807  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:01.587391  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:01.555173 10799 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.543494 10703 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 12:26:01.591152  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:01.593452  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61629.8us
I20260709 12:26:01.593678 10672 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:01.603314  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 3993) took 1372 ms (client timeout 1799 ms). Trace:
W20260709 12:26:01.603865  1880 rpcz_store.cc:269] 0709 12:26:00.230704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:00.378991 (+148287us) service_pool.cc:224] Handling call
0709 12:26:01.603085 (+1224094us) spinlock_profiling.cc:243] Waited 55.8 ms on lock 0x7b54002df5d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:01.603215 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":128379136}
I20260709 12:26:01.605561 10672 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:01.593012  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.568719 10777 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.572943 10706 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:01.611124 10706 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:01.614593  1877 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3151) took 3637 ms (client timeout 1904 ms). Trace:
W20260709 12:26:01.615119  1877 rpcz_store.cc:269] 0709 12:25:57.977393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:58.411873 (+434480us) service_pool.cc:224] Handling call
0709 12:26:01.614243 (+3202370us) spinlock_profiling.cc:243] Waited 2.78 s on lock 0x7b54001f8954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:01.614487 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6396835712}
W20260709 12:26:01.617914  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.619746  1772 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:01.620054 10757 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.621541 10906 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.621910 10867 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:01.596343  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:01.622522 10867 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.623116 10867 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:01.597502 10799 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.625682 10794 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:01.598814  2410 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:01.588408 10718 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:26:01.611124  1879 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:01.596554 10665 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:26:01.593845 10961 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:01.632670 10961 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.633044 10961 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:01.626713 10799 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.588907 10940 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:01.635231 10940 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:01.578712 10804 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:26:01.634397  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 43683.7us
I20260709 12:26:01.637897 10906 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.639673  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:01.589670 10671 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.640987  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:01.643775 10964 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:01.644392 10964 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:01.645388 10964 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.647737 10764 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:26:01.658367  1732 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:01.661222  1724 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:26:01.662083  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:01.672299  2413 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:01.580555 10923 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:01.673229  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 88824.8us
W20260709 12:26:01.685734  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 104526us
I20260709 12:26:01.686762  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:01.689198 10915 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.689929 10891 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:01.690538 10891 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.693111 10891 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:01.695742 10923 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:01.673537 10851 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.713810  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:01.721253 10923 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:01.724427  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:01.726327 10665 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:01.726922 10665 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.727344 10665 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:01.731901 10915 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 24.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:01.733603 10929 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:01.734161 10929 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:01.735550 10929 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:01.738360 10873 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:01.751888 10929 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:01.766623  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:01.766821  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.787794  1725 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:01.788726  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:01.783391  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 96741us
I20260709 12:26:01.790112  1771 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:01.792301  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 93288.4us
W20260709 12:26:01.791066  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 190875us
W20260709 12:26:01.793401  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 113895us
I20260709 12:26:01.796001  1727 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:01.798504  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.803081  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:01.809871  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:01.810719  1730 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:01.786823 10891 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:01.795106 10862 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.821122  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:01.822746  2410 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:01.801070 10672 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:01.824844  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.823987  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.832552  1728 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:26:01.800805 10804 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:26:01.837859  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.838321  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:01.863327  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 147827us
I20260709 12:26:01.873234  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:01.873917  1728 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:01.874682  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:01.875473  1727 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:01.878580  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50548.2us
I20260709 12:26:01.880760  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:01.886826  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:01.835372 10672 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:01.868994 10926 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:01.825627 10324 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
W20260709 12:26:01.891925  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63192.9us
I20260709 12:26:01.838333 10841 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 12:26:01.872267 10671 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:01.844713 10804 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:01.886274 10906 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:01.835634 10862 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:01.896095 10672 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:01.899029 10915 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:01.832063 10984 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:26:01.890867 10851 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:01.958480  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 45628.2us
I20260709 12:26:02.009725  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:02.011404 10669 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:02.012426 10669 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:02.013690 10669 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:02.014182 10669 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:01.955487  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:01.917868 10804 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:01.984115  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:01.953658 10076 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:01.989048 10867 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:01.990097 10805 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.560s	user 0.014s	sys 0.002s
I20260709 12:26:01.916114 10983 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.092s
I20260709 12:26:02.000432 10862 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:02.007881 10986 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.042s
W20260709 12:26:02.014928  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 193617us
I20260709 12:26:02.072914  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:02.014967  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:02.033876  1728 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
W20260709 12:26:02.064656  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 109308us
I20260709 12:26:01.912550 10926 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.034591 10757 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:02.028791 10919 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:02.021463 10915 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:02.087642  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:02.020172 10777 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:26:02.016790 10665 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.072170 10991 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:02.097606  1770 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:02.014743 10961 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.027315 10980 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:26:02.113969 10961 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.116043 10867 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.116763 10867 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:02.117161 10867 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:02.118952 10867 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.100675 10841 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:02.100535 10891 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.020s
I20260709 12:26:02.120463 10991 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:26:02.123157  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 108313us
I20260709 12:26:02.125315 10926 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:02.127004 10915 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.082753 10742 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:02.083508 10919 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:02.106976 10805 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.088199 10862 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:02.015628 10669 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.128217 10940 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.129901 10665 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:02.131062 10794 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:02.132339 10610 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.133140 10076 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.134094 10851 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:02.135542  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 271409us
I20260709 12:26:02.135607  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:02.144176  1723 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.006s	sys 0.021s
I20260709 12:26:02.081728 10757 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:02.159291 10436 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:26:02.153004  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 50996.6us
W20260709 12:26:02.153646  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56151.8us
I20260709 12:26:02.155251  1730 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
W20260709 12:26:02.160928  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 279882us
I20260709 12:26:02.215925  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:02.134694 10923 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.146129 10647 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.167919 10867 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:02.198072  1723 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.007s	sys 0.022s
I20260709 12:26:02.199417 10669 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.199743 10991 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.200171 10794 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.200922 10610 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.203622 10672 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:02.205096 10706 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.593s	user 0.007s	sys 0.007s
I20260709 12:26:02.205565 10076 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:02.207407  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 71905.3us
I20260709 12:26:02.208510  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:02.209323 10994 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:02.212284  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1333) took 1110 ms. Trace:
I20260709 12:26:02.218472  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:02.223547  1724 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:02.223595  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:02.253571  1728 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:02.253850  2413 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.220474  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:02.235630  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.236745  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.206372 10851 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:02.240376  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44192.8us
I20260709 12:26:02.225283 10742 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:02.218713 10862 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:02.256867  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:02.241446 10990 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:26:02.252585  1803 rpcz_store.cc:276] 0709 12:26:01.101964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:01.102099 (+   135us) service_pool.cc:224] Handling call
0709 12:26:01.115883 (+ 13784us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:01.506402 (+390519us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:02.211945 (+705543us) spinlock_profiling.cc:243] Waited 362 ms on lock 0x7b5400066654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:02.212190 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":831738240}
I20260709 12:26:02.266597  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:02.240367 10665 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:26:02.283468  1730 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:02.239167 10436 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:02.197168 10805 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.315918  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:02.263417 10671 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:26:02.261595 10929 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.316270  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.321446  1732 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:02.316680  1727 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.263396 10742 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:02.331346  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:02.303601  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46850.4us
I20260709 12:26:02.334098  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:02.318583  1728 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:26:02.308815  1723 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
W20260709 12:26:02.338934  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 72487.9us
I20260709 12:26:02.242285 10647 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.247578 10923 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.315017 10906 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:02.288718 10926 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.051s
I20260709 12:26:02.358925  1724 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:02.351009  2413 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:26:02.258448 10706 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.302117 10602 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:26:02.338831 10709 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:26:02.300626 10777 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 12:26:02.341655 10742 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:02.382304  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51052.5us
I20260709 12:26:02.367870 10986 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:02.361711 10906 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:02.391714 11018 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:02.395474 11005 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:26:02.399562 10076 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.401311 10706 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.402690 10851 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.314472 10919 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.389191  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:02.405768  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 71808.2us
I20260709 12:26:02.340130 10929 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.408197 10986 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:02.412668  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89274.5us
I20260709 12:26:02.412912 11015 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:02.413290 11017 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.416245  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:02.436285  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:02.428663  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 220219us
I20260709 12:26:02.372463 10969 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.428714  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:02.409256 10906 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:02.432859  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:02.456799  2413 raft_consensus.cc:2368] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 26. Current term is 27.
I20260709 12:26:02.441685  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:02.436067 10709 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.008s
I20260709 12:26:02.445921 10926 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:02.446210 10610 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:02.447211 10794 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.448840 10799 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:02.451426 10602 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:02.437024 10436 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.454419  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:02.480099  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.435943 10986 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.463407  1697 leader_election.cc:400] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 26. Current term is 27.
I20260709 12:26:02.483429  1697 leader_election.cc:403] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:26:02.380270 10757 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.484638  1723 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:02.435180 10991 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:02.473688 10805 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:02.484403 10610 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:02.464437 11017 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.475369 10799 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:02.475986 10602 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:26:02.459259  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.433947 10940 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.439271 11015 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:02.516009 10804 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.517143 11015 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:02.422621 11007 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 12:26:02.510321 10709 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 26. Current term is 27.
I20260709 12:26:02.497177 10672 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.499627 11018 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:02.500089 10647 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:02.503728  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68956.8us
I20260709 12:26:02.479688 10706 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:02.483790 10794 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.505846 10757 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:02.418082 10862 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:02.509011  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49917.4us
I20260709 12:26:02.507392 10610 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:02.512998  1727 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:02.514040  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.514883 11014 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:02.515476 10931 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:26:02.515921  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78156.3us
I20260709 12:26:02.519877 10923 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:02.523023  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:02.496055  1729 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:26:02.551177  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:02.527114  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:02.527791  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.531998  1723 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:26:02.538906 10706 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:02.512271 10665 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:02.545785  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:02.463260 10969 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.524994 10929 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:02.548005  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 184910us
I20260709 12:26:02.548708  1731 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:02.528587 10991 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:02.533645  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:02.582365  1725 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:02.555630  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.583449  1727 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:02.539029 11018 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.558645 10610 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:02.560228 11017 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:02.562192  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:02.563014 11020 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:02.546357 10794 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.563894 10742 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.564283 10804 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.549929 11014 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:02.547389 10324 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:26:02.566802 10706 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:02.514408 10799 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.570461 10665 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.584317  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:02.585638  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:02.588260  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:02.637342  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51795us
I20260709 12:26:02.591212  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:02.638176  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:02.596143  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:02.601707  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:02.604892  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55163.2us
I20260709 12:26:02.605320 10794 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:02.584671 10929 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.580586  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.622610  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:02.623620  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 61529.8us
I20260709 12:26:02.644524  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:02.644264  2415 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:26:02.643033  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 47045.5us
I20260709 12:26:02.582630 11015 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.650974 10984 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:02.595252 11024 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:26:02.606844 11017 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.652104  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:02.622133 11021 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:02.635314 10324 consensus_queue.cc:1243] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 26.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:26:02.622247 11014 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.704749  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:02.689383  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:02.700973  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56545.4us
I20260709 12:26:02.701182  1725 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:02.707253 10794 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:02.707821 10794 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:02.708209 10794 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:02.652010 10804 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:02.651829 10610 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:02.711719 10610 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:02.712246 10610 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:02.701803  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49802.2us
I20260709 12:26:02.719061 10706 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.722273 11018 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:02.648917 10962 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:26:02.736934 11026 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:02.745641  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 107536us
I20260709 12:26:02.750357  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:02.660629 10891 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:26:02.655973 11024 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:02.777097  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:02.750128 10799 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:02.762845  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58149.4us
I20260709 12:26:02.781527 10436 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:02.777797 11026 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.818312 11023 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.819047 11023 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:02.812156 10984 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:26:02.823532 10984 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:02.823930 10984 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:02.756614 11002 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.068s
I20260709 12:26:02.825192 11002 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:02.822863 10804 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:02.825618 11002 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:02.826241 10804 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:02.756258 10794 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.827167 11023 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:02.827852  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69256us
I20260709 12:26:02.829121 10794 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.829223 10804 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.660856 11017 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.751047 10929 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.881122  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:02.881901  1876 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:02.882215 11023 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:02.882659 11023 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:02.883077 11023 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.883188 10929 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.751304 10962 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:02.884248 10962 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:02.885807  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:02.833570  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:02.777664 10669 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.533s	user 0.014s	sys 0.000s
I20260709 12:26:02.835227  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:02.835599  1880 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:26:02.836056  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:02.836819  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:02.888679  1877 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:02.837519  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:02.891230  2413 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:02.891706  1876 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:02.834688  1879 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:02.823016 11026 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:02.783190 11024 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:26:02.861325  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:02.879629  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 190380us
I20260709 12:26:02.756824 10991 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.886566  1878 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:02.826951 10891 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:02.843791  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:02.902837  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:02.918577  1880 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:02.810351 11018 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:02.907982  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:02.900277  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:02.909193  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:02.900945  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:02.911022  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:02.916612  1877 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:26:02.917905  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:02.920449  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.778247 11021 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.131s
I20260709 12:26:02.941606  1878 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:02.942107  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:02.943650  1728 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:02.942337  1879 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:02.947540  1876 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:26:02.947868  2413 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:26:02.949509  1881 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:02.951282  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:02.961114 11018 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.826689 11002 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:02.966935 10929 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:02.968020 11002 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:02.901816 10867 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:26:02.980415  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.920848 11015 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:02.982544  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.983897  1725 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:26:02.993016  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.994115  1873 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:02.989038 10602 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:02.971083 10923 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:03.001806  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:03.008289  1874 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 25
I20260709 12:26:03.001938  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:03.002537  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 190992us
W20260709 12:26:03.024688  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.976964 10794 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.007488 10602 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:03.035413  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:02.990957 10805 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.051215  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:02.972894 10647 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:26:02.999589 10969 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:03.056674  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:02.971320 10706 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:03.057626  1723 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:02.996487 10742 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:03.047187 10906 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.539s	user 0.009s	sys 0.003s
I20260709 12:26:02.978368 11022 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.081s
I20260709 12:26:03.063953  1728 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 20.
I20260709 12:26:03.052243 11026 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:03.086874 11026 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:03.087638 10962 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.052449 10867 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:26:03.076663 11014 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.077493 10906 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:03.081624 10986 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.543s	user 0.011s	sys 0.000s
W20260709 12:26:03.061376 10757 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.001s
W20260709 12:26:03.083129  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 141628us
I20260709 12:26:03.088232 11026 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.089464 10891 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.090672 10436 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.064693  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:03.046981 10923 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:03.033167  1880 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:03.038704 10984 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.063369 10602 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:03.127384  1872 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:03.127406  1880 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:03.129686  1727 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 19. Current term is 20. Ops: []
W20260709 12:26:03.062664  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 163420us
I20260709 12:26:03.119823  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:03.130614  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.131800  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:03.120322  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.121760  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.102337 11018 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.133703  1728 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:26:03.133919  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69389.7us
I20260709 12:26:03.135620 11023 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.138393 11023 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:03.104159  1879 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:03.115669 10706 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.139452  1874 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:03.094004 11017 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.103133 10867 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:03.140491  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:03.149144  1729 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:26:03.151398  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.122371 10998 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:03.068367 10805 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.150283  1874 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:03.135695  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:03.158553 11020 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:03.164762  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.165025  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.166102  1798 raft_consensus.cc:2368] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24.
I20260709 12:26:03.167169  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.168908  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.169693  1802 raft_consensus.cc:2376] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 28. Re-sending same reply.
I20260709 12:26:03.179311 10998 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.154382 10706 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.066092 10794 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.187394 10867 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.123716 10986 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.189460  2414 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:03.190717 10986 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:03.192188 10969 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:03.192868 10969 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:03.193292 10969 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:03.148855 11002 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:03.195231 11002 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:03.196236 11002 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.139966 10436 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.138710 11021 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:26:03.201889 10984 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:03.202726 11026 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:03.206084 10962 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:03.148087 11007 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:03.182365 10998 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:03.208096 10923 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.226331  1805 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:03.227284  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.231411  1798 raft_consensus.cc:2410] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 24 because replica has last-logged OpId of term: 21 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 11.
W20260709 12:26:03.239490  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.243067  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.245195  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.246245  1802 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:03.247732  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:03.248646  1729 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:03.244037  1803 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:03.250322  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.254837  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:03.263497  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:03.300134  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45421.2us
I20260709 12:26:03.301030  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:03.244197 10906 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:03.307878  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:03.310114  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 182645us
W20260709 12:26:03.326979  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 179951us
I20260709 12:26:03.327554  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:03.328465  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:03.328850  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:03.347512 10436 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.349483  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:03.350426  2410 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:03.315594 11023 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:03.305383 11031 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:03.308247 11014 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.343809 10647 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:03.358399  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:03.362936  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.366125  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:03.370064  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.384146  1873 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:26:03.384757  1804 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:03.351634 10665 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:03.394932  1696 leader_election.cc:400] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24.
I20260709 12:26:03.377234  2410 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:26:03.378813  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.395675  1696 leader_election.cc:403] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:03.379380  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:03.381661  2418 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:03.376132  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.396549  1806 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:03.379127  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:03.364751 10804 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.361963 10906 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:03.403872  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.362304 10436 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.355165 11035 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:03.400583  1800 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:03.365361 10862 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.008s	sys 0.014s
W20260709 12:26:03.425483 10862 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.009s	sys 0.014s
I20260709 12:26:03.365718 11017 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:26:03.426205 11017 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.426470  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.381259 11023 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:03.427225  1804 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:03.411599 10804 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.419497  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:03.428504 11023 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.429756 10804 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:03.400234 10703 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.365036  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:03.419593 10805 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:03.420704 10610 consensus_meta.cc:287] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.708s	user 0.012s	sys 0.004s
I20260709 12:26:03.421238 10906 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.442288 10906 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.423156  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:03.411530 11035 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:03.423851  2444 heartbeater.cc:646] Failed to heartbeat to 127.0.140.126:43371 (7 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.140.126:43371 timed out after 15.000s (SENT)
I20260709 12:26:03.424068 10436 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.424888  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.426584 11017 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:03.419034  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:03.428359 11020 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24.
I20260709 12:26:03.460208 10602 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:03.467185 10799 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.560s	user 0.009s	sys 0.005s
I20260709 12:26:03.435503  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:03.469861  1800 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:03.436566  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71646.3us
I20260709 12:26:03.379065 10647 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:03.435111 10665 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.439697  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.439556 10703 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.440258 11031 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:03.435652  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.481580  1729 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:03.476862  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:03.462568 10602 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:03.494521  1798 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:03.508263  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.456714 11035 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:03.474942 10647 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:03.524932  1798 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:03.509814  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 108514us
I20260709 12:26:03.525394  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.527348  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:03.445660 10610 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.481719 11044 raft_consensus.cc:2764] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election decision vote started in defunct term 28: won
I20260709 12:26:03.536247  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.509745 11018 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:03.485225 10906 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:26:03.546370  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.549259  1806 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:03.547214  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.504745 10436 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:03.554278  1879 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:26:03.555598  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.556532  1798 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:03.524305 10926 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:26:03.484598 10665 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.567152  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:03.568084  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.585801  2418 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:26:03.548791 10929 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:03.546200 10915 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:03.592711  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:03.552215 11031 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.599267  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.601364  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.602176  1801 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:03.604964  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.605751  1801 raft_consensus.cc:2368] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25.
I20260709 12:26:03.606526  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.607770  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.609368  1798 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
W20260709 12:26:03.627368 11024 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.703s	user 0.013s	sys 0.000s
I20260709 12:26:03.627871 11024 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:03.630970  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52514.6us
I20260709 12:26:03.631479 10703 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:03.631913  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:03.596814 10915 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.636384 10915 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:03.636813  1800 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:26:03.637244  1800 pending_rounds.cc:85] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 14
I20260709 12:26:03.637643  1800 pending_rounds.cc:107] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:26:03.637884  1800 raft_consensus.cc:2889] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:03.612175 11031 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.640146 10841 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:03.640697 10841 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:03.640913 11031 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.641779 10841 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.621254 11001 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:03.643586 10794 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.645222 10665 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.648952  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.649803  1799 raft_consensus.cc:2376] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26. Re-sending same reply.
W20260709 12:26:03.651544  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.653318  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:03.653643  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:03.654881  2411 raft_consensus.cc:2410] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 15.
I20260709 12:26:03.607969 10929 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:03.657318 10929 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.658975 10324 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.659346 10841 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:03.659610 10324 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.660386  1806 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:03.662038 10324 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.663589  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:03.667701  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:03.664316  2417 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:03.669349  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.560276 10969 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.671177  1799 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:03.658097  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:03.680182  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.680811  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:03.681341  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.681496  1878 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:03.682703  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.684052  1799 raft_consensus.cc:2393] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:03.683179  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:03.686018  1876 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:03.678143 11026 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.688364 11026 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.669121 11024 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:03.677752 10610 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:03.691865  1879 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20260709 12:26:03.691984  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60158.6us
I20260709 12:26:03.678256 10703 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.019s
I20260709 12:26:03.696378  2416 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:03.699110  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:03.703579  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:03.705065  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:03.705837  1877 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:03.707803  1878 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
W20260709 12:26:03.709807  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.711402  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:03.677690 11007 raft_consensus.cc:2764] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:26:03.708180  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.676275 11002 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:03.684335 11031 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.707235 10324 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.725883  1877 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:26:03.690653 11026 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:03.727587  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:03.729540  1799 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:03.715660  1876 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:03.726970  1772 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:03.734632  1879 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:03.737704  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:03.739312 10961 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:03.751854  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:03.755007  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.732332 10602 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:03.757036  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:03.784170  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.757928  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:03.751931 10986 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.016s
I20260709 12:26:03.768256 11026 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:03.771497  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:03.773164  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:03.787252  1727 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:03.775365  1731 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:03.733901 10998 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.056s
I20260709 12:26:03.790211  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.790949  1731 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:03.779866 11002 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:03.744737 11007 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:03.794106 11007 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.796736 11007 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:03.754437  1723 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:26:03.797950  1723 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:03.799173  1724 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:26:03.799347  1723 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.801074  1723 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:03.781208 11035 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.811411  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.812835 10998 consensus_queue.cc:1243] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 28.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:26:03.813175  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.813261 11035 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:03.813562 10998 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:26:03.807909  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50093.3us
I20260709 12:26:03.815045  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:03.816004  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:03.817438  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.818159  1845 leader_election.cc:400] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25.
I20260709 12:26:03.818542  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.818601  1845 leader_election.cc:403] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:03.820608  1806 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:03.823339  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.801199  1731 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:26:03.823117  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 66171.9us
I20260709 12:26:03.814164 10998 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:03.831329  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:03.809355  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:03.833103  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:03.836997  1728 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:03.821513  1724 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:03.837984  1724 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:03.839612  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.840610  1729 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:03.841367  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:03.844722  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:03.839037 10926 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25.
I20260709 12:26:03.845693 10997 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:03.846230 10997 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:03.737046 10324 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.846608 10997 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:03.848718  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.849886  1726 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:26:03.850080  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:03.851457  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:03.854048  1799 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:03.854182  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:03.855363  1726 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:03.855891  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:03.856305  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.805440 11026 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:03.845608 11027 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.134s	user 0.000s	sys 0.004s
I20260709 12:26:03.742483 11003 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.090s
I20260709 12:26:03.819037 10665 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:03.819964 11023 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:03.860607  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.736631 11031 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.853935  1724 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.865175  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.866374  1724 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:03.842361 11002 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:03.861268 10764 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:03.838696 10742 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:03.866675  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.869127  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.848461 10794 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:03.858748 11026 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:03.872592  1799 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:03.868793 11002 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:03.876982 10764 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:03.878015  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.878245 10764 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.861083 10926 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.881515 10926 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.883267 11061 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.884074 10926 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:03.883982 11061 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.886399 11061 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:03.887326  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:03.887315 10906 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:03.888381 10906 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.888913 10906 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:03.889498 11031 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:03.884279 10324 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:03.890856 11063 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:26:03.891851  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.893318  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.893940  1806 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:03.896041 11007 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:03.896672 11007 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.861150 11001 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:03.876899  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:03.863076 10665 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:03.906308  1723 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:26:03.911629  1723 pending_rounds.cc:85] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 12
I20260709 12:26:03.912140  1723 pending_rounds.cc:107] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:26:03.912542  1723 raft_consensus.cc:2889] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:03.907788  1726 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:03.907699  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.913267  1724 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:03.921763  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:03.923969  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1531) took 1192 ms. Trace:
I20260709 12:26:03.924896  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.925166  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:03.925809  1799 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:26:03.924575  2414 rpcz_store.cc:276] 0709 12:26:02.718271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.718441 (+   170us) service_pool.cc:224] Handling call
0709 12:26:02.718821 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:03.189949 (+471128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:03.910462 (+720513us) spinlock_profiling.cc:243] Waited 470 ms on lock 0x7b54006b0154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:03.910747 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1081721216}
I20260709 12:26:03.943708  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.944450  1799 raft_consensus.cc:2376] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29. Re-sending same reply.
I20260709 12:26:03.877578 11002 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:03.948098  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:03.955854  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.921873 10665 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:03.898432 10647 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:03.922443 11023 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:03.958323  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:03.956707  1799 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:03.956569  1728 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:03.961429  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.962620  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.963320  1799 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:26:03.964380  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.901165 10742 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:03.976552 11031 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:26:03.977679 11031 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:03.980340  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.980906 10972 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:03.981626 10972 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.982178  1799 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
W20260709 12:26:03.983449  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70928.9us
I20260709 12:26:03.966674  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:03.941068 10986 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:03.989393  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:03.995906  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.946604 10929 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:04.008342  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:03.938246 10926 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:26:03.922961 11017 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:03.959646 11007 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:04.013211  1799 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:03.995952 10703 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:26:04.021231  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.020105  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.023712  1799 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:04.024369  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:03.986537  1806 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:04.016651 10742 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:04.039947  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.040894  1805 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:04.032037  1728 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:26:04.032759  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:04.042923  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.043387  1803 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:04.024627 10703 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:04.044503 10647 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:04.044597  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 4009) took 1212 ms. Trace:
I20260709 12:26:04.044986  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.045248  1872 rpcz_store.cc:276] 0709 12:26:02.831846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:02.834892 (+  3046us) service_pool.cc:224] Handling call
0709 12:26:04.044300 (+1209408us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b540021acd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:04.044520 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":549927680}
I20260709 12:26:04.046263  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.047392  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.020970 10929 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:04.063833  1728 pending_rounds.cc:85] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 14
I20260709 12:26:04.070716  1728 pending_rounds.cc:107] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:26:04.071044  1728 raft_consensus.cc:2889] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:04.010955 10972 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:04.073859  1803 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:26:04.079624  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.083539 10929 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.085192  1799 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:04.044507 10703 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:26:04.086007  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 225850us
W20260709 12:26:04.085134  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 126897us
I20260709 12:26:04.090502  1799 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:26:04.090273  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:04.079561 11037 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.027431 11035 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:26:04.094571  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:04.102415  1725 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:04.095342  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:04.099301 10972 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:26:04.099913  1799 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:26:04.108186  1803 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:26:04.109522  1799 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:04.111244 10841 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.115175 10841 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.117300 10841 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:04.119127 11007 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:04.119534 11007 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:04.120404 11007 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.121757 11007 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:04.126302  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:04.131178  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:04.131986 11067 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:26:04.131229  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:04.136255  1728 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:26:04.140202  1730 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:04.143239  1805 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:04.149252  1803 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:26:04.149576  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:04.149976  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.148926 10647 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:04.150484  1803 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:04.153239 10923 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:04.158387  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.160728  1877 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 26: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:04.148759 10076 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.186158 10076 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.187240 11002 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.187906  1803 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:26:04.188634 10076 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:04.092418 10906 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.243367  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.244380  1877 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:04.245270  1881 raft_consensus.cc:1240] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:04.252149  1872 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:26:04.257390  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.258314  1881 raft_consensus.cc:2376] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 23. Re-sending same reply.
I20260709 12:26:04.262003  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.264358  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.265143  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:04.267223  1881 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:04.123570 10742 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:26:04.287981  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.283825  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.292560  1872 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:04.243956 11072 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:04.319168 11072 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:04.319679 11072 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:04.112749 11037 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.323416 11037 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:04.324302  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:04.271332 10671 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.909s	user 0.052s	sys 0.089s
I20260709 12:26:04.287267 10076 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:04.295075 10972 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.295655 10764 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.301628  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.342635  1800 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:04.302870  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.308416  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.310767 11018 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:26:04.315555  1802 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: [25.17-25.17]
W20260709 12:26:04.269527  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 138416us
I20260709 12:26:04.357774  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:04.298378 10969 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:04.345082  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.357774  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.261098 11002 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:04.289408 10923 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:04.359951  1872 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:04.362385  1807 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 26: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:04.337386 10671 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.987s	user 0.053s	sys 0.089s
I20260709 12:26:04.364845  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:04.359812  1799 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
W20260709 12:26:04.367789  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.362169 10076 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.378672 10076 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:04.380457 10923 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:04.381011 10923 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:04.326334 11017 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:04.355080 11018 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:04.383808 11018 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.385454 11018 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:04.387135 10986 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:04.387392  1802 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: [18.13-23.15]
I20260709 12:26:04.361903 10742 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.391357 10742 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.394996 11017 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:04.395282 10906 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:04.395830  1802 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:26:04.397188  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.398233  1807 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:26:04.402187 11026 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.403205  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.404011  1799 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:04.404675 11026 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:04.404958  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.406072  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.406441 11001 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:26:04.361379 10764 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.360834 10972 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.417208 10972 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:04.418642 11003 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:04.419195 11003 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:04.379273 10671 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:04.427276 10671 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.438335  1879 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:04.413770 10764 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:04.442670  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.444363  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.447134  1880 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:26:04.449983  1874 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: [23.16-25.17]
I20260709 12:26:04.456570  1874 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:26:04.458823  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.459105  1800 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:04.462615  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.468729  1872 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:04.458015 10703 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.463426 11074 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:04.474975  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.450457 10742 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:04.464473  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:04.489750  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:04.498052  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:04.501827  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:04.505190  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:04.506105  1880 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:04.506193  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.507227  1800 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:04.509651  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.510365  1800 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
W20260709 12:26:04.511970  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:04.516757  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.527442  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.502813  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:04.528344  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:04.494091 11017 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:26:04.532598  2413 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:04.613577  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:04.522974  1772 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:04.611061  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 90183.1us
I20260709 12:26:04.485862 10706 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:04.612161  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 109463us
I20260709 12:26:04.616086  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:04.529313  2414 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
W20260709 12:26:04.624276  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 101388us
I20260709 12:26:04.625298  1772 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:04.504014 11073 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:26:04.501155 10076 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:04.630432  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:04.632664  1881 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 27. Ops: [25.16-25.16]
W20260709 12:26:04.633802  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.482995 11074 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.638551  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.639858  2416 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:04.636710  2419 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: []
I20260709 12:26:04.642452  1874 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:26:04.651916  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.652921  1800 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:04.648068  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:04.660715  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.499058 11026 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.130s
I20260709 12:26:04.611752 11001 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:04.676520  1873 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:26:04.677175  1880 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
W20260709 12:26:04.670542  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 89509us
I20260709 12:26:04.661717  1807 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:04.673142  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:04.635843 10926 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:04.639607 10436 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:04.635502 10647 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:04.734455  2419 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:04.529651 11002 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.138s
I20260709 12:26:04.680722  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:04.614702 10969 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:26:04.539217 11037 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.127s
I20260709 12:26:04.744169  1800 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:04.745878  1800 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:26:04.685575  1880 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:26:04.634686 11076 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:04.646044 11023 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.755287  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.759514  1807 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:04.760143  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:04.761021  2416 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 20.
I20260709 12:26:04.701503 10923 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.769129 10923 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:04.770108 11023 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.772552 11023 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:04.775254 10703 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:04.629081 11081 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.146s
I20260709 12:26:04.783970 11081 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:04.784505 11081 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:04.785597 11081 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.20, Last appended by leader: 20, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.787937  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:04.792483  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 3045) took 1091 ms. Trace:
I20260709 12:26:04.793114  1805 rpcz_store.cc:276] 0709 12:26:03.700682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.144474 (+443792us) service_pool.cc:224] Handling call
0709 12:26:04.145013 (+   539us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:04.153657 (+  8644us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:04.153689 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:04.189704 (+ 36015us) log.cc:844] Serialized 26 byte log entry
0709 12:26:04.497201 (+307497us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:04.497247 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:04.497318 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:04.792180 (+294862us) raft_consensus.cc:1685] finished
0709 12:26:04.792211 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:04.792401 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9574,"raft.queue_time_us":7860,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":5434,"thread_start_us":228025,"threads_started":1,"wal-append.queue_time_us":406190}
I20260709 12:26:04.644618 10706 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 29.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.097s
I20260709 12:26:04.646579 11045 consensus_queue.cc:1059] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.096s
I20260709 12:26:04.685803 10926 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:04.795495 10937 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:04.798368 11026 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:04.800014  1873 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:04.824632  1800 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:04.801047  1881 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:26:04.743350 10076 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:04.803486  1877 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: [28.16-28.16]
W20260709 12:26:04.739576 10997 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.892s	user 0.009s	sys 0.005s
I20260709 12:26:04.672127 10764 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:04.665630 11017 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:04.822901  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:04.844800  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 1.10963e+06us
I20260709 12:26:04.845815  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:04.847231  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.850881  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:04.830830  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:04.835249  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.848977  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:04.859524  1880 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 24 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:26:04.837598  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:04.837514  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:04.859665  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:04.835952  1807 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:26:04.861541  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:04.853482 11044 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:04.865229 11044 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:04.817883 10926 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:04.870615  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 189964us
I20260709 12:26:04.872915  2416 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:26:04.873605  2414 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:04.740146 11001 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.875809  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.828349 10764 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:04.878160  2413 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:26:04.878871  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:04.853428 10777 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:04.884790  2412 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:04.880738  1877 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:04.858273 11018 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:26:04.898417  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45451.5us
I20260709 12:26:04.899226  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:04.901247  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.902005  1877 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:04.904068  2411 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:04.907016  1877 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30. Ops: [29.13-29.13]
I20260709 12:26:04.915915 11076 raft_consensus.cc:2764] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election decision vote started in defunct term 26: won
I20260709 12:26:04.917141 10969 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:04.917886 10969 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.867527 10983 raft_consensus.cc:3055] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:04.919152 10983 raft_consensus.cc:740] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:04.921525 10969 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:04.866355 11044 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 29.13, Last appended by leader: 13, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.923687 11044 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:04.924453  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:04.925203  2411 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:04.926939  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:04.927603 11082 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:26:04.929268  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:04.930080  2411 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:04.859817 10647 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.087s
I20260709 12:26:04.839239 10997 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:04.977203  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:04.935160  1800 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:26:04.248253   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:04.973603  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.983325  1879 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 27: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:04.986560  2412 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:26:04.895506 10764 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:04.987519  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:04.990326  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.875001 11023 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:04.939894  2416 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
I20260709 12:26:04.996011  2416 pending_rounds.cc:85] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 13
I20260709 12:26:04.996323  2416 pending_rounds.cc:107] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:26:04.996582  2416 raft_consensus.cc:2889] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:04.968092  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.991255  1877 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:04.999476  1877 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:26:05.002574  2382 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:04.999572  2411 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
W20260709 12:26:04.942183 10665 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.952s	user 0.008s	sys 0.005s
W20260709 12:26:04.930678 11007 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.809s	user 0.008s	sys 0.004s
I20260709 12:26:05.009342  2417 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:04.854892 10076 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:04.974234 11075 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:26:04.965219 10986 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:26:05.002569 11094 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.084s	user 0.000s	sys 0.001s
I20260709 12:26:04.940831 11001 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:04.989338 11018 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:05.034085 10940 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:05.036423 10983 consensus_queue.cc:260] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 13, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.040696 10983 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:05.042532 10997 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:05.005051 11096 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:05.043856 11074 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:05.013056 10926 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.047611 10926 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:05.048910 11002 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:05.049557 11002 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:05.052165  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:05.055279 10923 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:26:05.057530 10923 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:05.058133 10923 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:05.059015 11080 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:05.059320 10923 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:05.061745  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.062412  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.042372  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:04.877612 11078 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.128s
I20260709 12:26:04.954348 10919 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:04.988308 10937 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:05.031569 10992 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:05.012305 10665 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:04.941809 11017 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.016259 11075 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:05.031675  1877 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:05.016672 11023 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:05.058192  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.025592  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 126428us
W20260709 12:26:05.069880  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.137635  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.139195 10983 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:05.143442  1874 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:26:05.143929  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:05.144595  1725 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:05.145309  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:05.146710  1730 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:05.148880  1730 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:05.152042  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:05.153761  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:05.154064  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:05.021127  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:05.155051  1879 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:26:05.066694  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:05.086390  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:04.941298 10706 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.084s
I20260709 12:26:05.033118 11018 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:05.026157 11072 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.706s	user 0.014s	sys 0.000s
I20260709 12:26:04.947599 10972 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.074s
I20260709 12:26:04.940605 10718 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.084s
I20260709 12:26:05.034586 10940 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:05.130097 10992 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:04.962677 10998 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.022941 10986 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:05.126173 11007 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.134239 10076 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:05.109272  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:05.207755  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:05.154443  1729 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:26:05.157559  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.157239  1730 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:05.072216  1880 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260709 12:26:05.159245  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:05.163787  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 161897us
I20260709 12:26:05.142274 10983 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:05.216636  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 107515us
I20260709 12:26:05.217484  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:05.171972  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 105410us
I20260709 12:26:05.172617  1881 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 20.
I20260709 12:26:05.174823  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:05.219465  1728 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:26:05.183496  1725 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
W20260709 12:26:05.223870  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64752.3us
I20260709 12:26:05.224421  1725 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:26:05.224677  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:05.180248 11041 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:26:05.196938  1873 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:05.185189  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:05.198414 11023 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.148381 11013 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:26:05.206135 10706 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:05.050554 11002 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.212020  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:05.211942  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:05.170650 11001 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:05.174096 10777 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:05.223003 10998 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:05.254627  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:05.229341  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:05.255419  1728 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:05.201879  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3110) took 1743 ms. Trace:
I20260709 12:26:05.228669 11044 raft_consensus.cc:3055] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:05.255950  1801 rpcz_store.cc:276] 0709 12:26:03.457967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.607351 (+149384us) service_pool.cc:224] Handling call
0709 12:26:05.201665 (+1594314us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002f2954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:05.201814 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2431621376}
I20260709 12:26:05.257169  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:05.257997  2413 raft_consensus.cc:2410] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 25 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:26:05.258230  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:05.223469 10076 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:05.249817  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:05.259647  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:05.261703  1724 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:05.223668 10742 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:26:05.263669  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:05.233273 11078 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:05.267001  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:05.216018 10997 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:05.253469 11100 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:05.234071  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48974.6us
I20260709 12:26:05.243010  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:05.258997  2411 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:26:05.179591 10436 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:26:05.181488 11072 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.260072 11002 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:05.263077  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:05.235083 11041 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.264348  1727 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:05.282997  2417 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:05.259610 11044 raft_consensus.cc:740] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:05.270361 10969 raft_consensus.cc:2764] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election decision vote started in defunct term 20: won
W20260709 12:26:05.268620  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 453883us
I20260709 12:26:05.268941  2382 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:05.265434 11001 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:05.246124 10706 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:05.216389 11007 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.230388 11061 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:26:05.261792  1729 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:26:05.279063 10718 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:05.279080 11021 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:05.280253  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 340143us
I20260709 12:26:05.291216  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:05.264102 10742 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:05.263628  1730 raft_consensus.cc:2410] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
W20260709 12:26:05.303611  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 79407.4us
I20260709 12:26:05.300750  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:05.222903 10937 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.240820 11075 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:05.335594  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57716.1us
W20260709 12:26:05.335918  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 179224us
I20260709 12:26:05.274432 11037 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:05.275041 11078 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:05.265601 11069 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.053s
I20260709 12:26:05.287014 11045 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.286727 11041 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:05.269065 10764 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:05.315213  1696 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.671s (SENT)
I20260709 12:26:05.333745  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:05.357098  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74901.3us
I20260709 12:26:05.316077  1876 raft_consensus.cc:2368] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26.
I20260709 12:26:05.313961 11100 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:05.337491 11013 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.004s
W20260709 12:26:05.342950  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.341857  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:05.372756  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:05.333957 11101 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:05.345036  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72966.2us
I20260709 12:26:05.326221 11001 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:05.290079 10986 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.358235  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:05.387528  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:05.314240 10076 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.044s
I20260709 12:26:05.392834 11097 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.117s
I20260709 12:26:05.393687  1802 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:05.394691 11059 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:05.395197 11059 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.395567 11003 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.397553 11003 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:05.271112 10940 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.399277 10940 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:05.399916 10742 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.401301 10742 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:05.401161 11078 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:05.368631 11075 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:05.287423 10992 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.407051 11072 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:05.291069 10799 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.049s
I20260709 12:26:05.355389 10926 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:05.389611  1732 raft_consensus.cc:1240] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:05.407210  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 92010us
W20260709 12:26:05.419580  1696 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Timed out: RequestConsensusVote RPC to 127.0.140.67:35761 timed out after 1.809s (SENT)
I20260709 12:26:05.334514 10718 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:05.413950  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.355845 10777 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:05.392889  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:05.333202 11021 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.377657 10436 raft_consensus.cc:2764] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
W20260709 12:26:05.440141  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.410036 11097 consensus_queue.cc:1243] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 24.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:26:05.395606 11059 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:05.416756 10926 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.390045 10997 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.051s
I20260709 12:26:05.448398 11007 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:05.387844 10923 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:05.369120 11045 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:05.441215 11072 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:05.410122 11106 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.049s
I20260709 12:26:05.410432 11013 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.042s
W20260709 12:26:05.448267  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 109154us
I20260709 12:26:05.483013  2410 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:05.480780  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3109) took 2023 ms. Trace:
I20260709 12:26:05.480147  2413 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:05.504622  2417 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:26:05.526553  2413 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:05.530061  1801 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:26:05.519762  1798 rpcz_store.cc:276] 0709 12:26:03.457637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.610643 (+153006us) service_pool.cc:224] Handling call
0709 12:26:04.955363 (+1344720us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:26:04.955686 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:04.956216 (+   530us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:04.956254 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:04.956930 (+   676us) log.cc:844] Serialized 26 byte log entry
0709 12:26:05.265073 (+308143us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:05.265116 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:05.265187 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:05.480337 (+215150us) raft_consensus.cc:1685] finished
0709 12:26:05.480371 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:05.480685 (+   314us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":20577,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":10420,"spinlock_wait_cycles":23679744,"thread_start_us":307689,"threads_started":1,"wal-append.queue_time_us":422928}
I20260709 12:26:05.453328 10926 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:05.386693 11101 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:05.541693  1800 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:26:05.475504 11110 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:26:05.470767 10764 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:05.533560 10799 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.553318 11106 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:05.553570 10799 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:05.496193 11021 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:05.525597 10718 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:05.556233 10799 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:05.544306  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:05.478216 10867 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:05.472103 11111 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:05.422055 10706 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.471060 11098 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:05.473495 11096 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.565238 11096 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:05.505232 11007 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.567874 11007 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:05.568284 11101 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:05.570885 11007 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:05.561786 11111 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:05.566869 10799 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:26:05.591017  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 215941us
W20260709 12:26:05.572224  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.613886  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51613.4us
I20260709 12:26:05.614267  1697 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:05.482257 11001 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.456823 10983 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:05.597654  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.449625 11100 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.125s
W20260709 12:26:05.589706  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.618871  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:05.648227  2417 raft_consensus.cc:2410] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 24 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:26:05.621300  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:05.620010  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.615022  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:05.631186  2412 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:26:05.612118  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:05.640833  1724 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
W20260709 12:26:05.643209  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.652935  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:05.653489  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.654078  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.655274 10718 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:05.655694 10718 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:05.656030 10718 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:05.656672  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 353175us
I20260709 12:26:05.482980 10777 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:05.644464  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 257033us
I20260709 12:26:05.653993 10940 raft_consensus.cc:2764] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:26:05.644343  1798 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:26:05.659688 11017 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:05.660336  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:05.661458 11114 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:05.661909 10923 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:05.664974  2412 raft_consensus.cc:1240] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:05.668262  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:05.638371  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:05.665166  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:05.683568  1801 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:05.685113  1800 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: [29.19-29.19]
I20260709 12:26:05.606621 11007 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.692230  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:05.692756  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:05.693403  2411 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:05.718878  1875 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: [23.16-25.17]
I20260709 12:26:05.728120  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:05.728927  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:05.729748  1877 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:05.693964  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 79086.9us
W20260709 12:26:05.735816  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70740.3us
I20260709 12:26:05.736162  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:05.737339  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.748164  1880 raft_consensus.cc:1217] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Deduplicated request from leader. Original: 25.17->[26.18-29.19]   Dedup: 26.18->[29.19-29.19]
I20260709 12:26:05.748834  1880 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: [26.18-29.19]
I20260709 12:26:05.751624  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:05.755708  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 87517.1us
I20260709 12:26:05.756868  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:05.763051  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:05.765285  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:05.767198  1879 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:05.768040  1880 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6388b1a5243d463589750794e8ac5511"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
I20260709 12:26:05.768743  1880 raft_consensus.cc:606] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:26:05.769176  1880 raft_consensus.cc:614] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:05.676923 11072 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.772827  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:05.780093  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:05.786247  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:05.792827  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:05.793514  1731 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:05.762754 10940 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:05.806190  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:05.807359  1877 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:05.787446 10851 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:26:05.763412 11013 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:05.763110 11069 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:05.693328 11017 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.102s
I20260709 12:26:05.819494  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:05.768918 11106 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:05.821631 11129 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:05.747778 10799 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.062s
I20260709 12:26:05.692366 11101 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.112s
I20260709 12:26:05.736162 11059 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:05.750186 11007 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:05.822221 11115 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.179s
I20260709 12:26:05.848364  1728 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:05.795567 11072 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:05.857736 11106 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:05.855017 11013 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:05.881382  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:05.882000  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.864306 11101 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.765398 11111 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.873153 11045 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:05.880828 11098 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:05.851570 11041 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:05.909153 11125 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
W20260709 12:26:05.914973 11075 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.507s	user 0.013s	sys 0.000s
I20260709 12:26:05.909600 10757 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:05.909454 10919 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:05.927488 11003 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.529s	user 0.014s	sys 0.000s
I20260709 12:26:05.902925 11118 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:26:05.944969  1803 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:05.913033 11101 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:05.917189 11123 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:05.949501  2418 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:05.914129 11117 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:05.951929 11118 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:05.952545 11116 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:26:05.968715  1731 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
W20260709 12:26:05.949147 10742 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.547s	user 0.012s	sys 0.003s
I20260709 12:26:05.975479 11117 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:05.907511 11132 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.306s
W20260709 12:26:05.996342 11002 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.703s	user 0.012s	sys 0.000s
I20260709 12:26:05.997159 11117 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:05.958420 11124 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:05.998936 11124 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:05.946728 10757 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:05.999500 11124 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:05.953807 11123 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:05.970146 11106 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 29.13, Last appended by leader: 13, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:05.937582 11112 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.232s
W20260709 12:26:06.038107  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:05.984436 11116 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:06.037213 10891 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.042681 11112 consensus_queue.cc:1243] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 24.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:26:06.042366 11129 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:06.070420  1731 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:06.076545 10919 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:06.079190 11129 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.080133 10665 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:06.081302 11127 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.099s	user 0.000s	sys 0.000s
I20260709 12:26:06.069375 11003 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.084481 11123 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.085628 11003 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.034936 10718 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.086318 11123 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:06.069134  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 303471us
I20260709 12:26:06.103698 11007 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:06.104509 11072 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:06.109062  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.114264  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:06.123423  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.126531  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:26:06.090474 10867 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.519s	user 0.008s	sys 0.004s
I20260709 12:26:06.068001 10891 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:26:06.132962  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.088696 11003 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:06.122984 10992 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.004s	sys 0.036s
I20260709 12:26:06.088613 11041 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.148860  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:06.107275 11059 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:06.132411 11096 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.563s	user 0.015s	sys 0.000s
I20260709 12:26:06.146235 11007 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:06.146745 10919 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.128211 11045 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:06.166396  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:06.148002 11013 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:06.168798  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:06.159840 10992 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.752s	user 0.004s	sys 0.037s
I20260709 12:26:06.132854 11097 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
W20260709 12:26:06.166157 10926 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.612s	user 0.010s	sys 0.003s
I20260709 12:26:06.176471 10926 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:06.177330 11128 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
I20260709 12:26:06.181218 11143 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:06.167855 11096 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.185235 11096 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:06.186429 10940 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:06.187320 10983 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.187922 10983 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.146070 11118 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.190164 10983 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:06.176723  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.172295 11045 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.192468  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.192723 11101 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:06.193509 11142 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:06.193876 11101 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:06.197256 10718 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:06.198004 10718 consensus_queue.cc:1059] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:06.199463 11013 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.191988 10891 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:26:06.200973  1804 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 22 because replica has last-logged OpId of term: 21 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:26:06.170750 11116 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.169183 11132 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:06.204600  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:06.203673  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:06.167467 11007 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:06.217005 10718 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:06.219523 10718 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.220191 10718 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:06.223559  2410 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1598) took 2859 ms (client timeout 1963 ms). Trace:
W20260709 12:26:06.224056  2410 rpcz_store.cc:269] 0709 12:26:03.364237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.404057 (+ 39820us) service_pool.cc:224] Handling call
0709 12:26:06.223346 (+2819289us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b5400591ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:06.223483 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4691017472}
I20260709 12:26:06.224651  2419 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:26:06.167708 11075 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
I20260709 12:26:06.214564  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.217346 11007 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:06.209723 11097 consensus_queue.cc:1243] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 24.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:26:06.210371 11132 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:06.173671 10919 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.239210 10926 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.232656  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:06.246479 11163 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:06.246959 10919 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:06.247819 11080 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:06.248292 11132 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:06.250284  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:06.251130  1725 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:26:06.252538  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:06.253532  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:06.255903  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:06.252182 11044 consensus_queue.cc:260] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.247037 11163 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:06.316699  1799 pending_rounds.cc:85] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 19
I20260709 12:26:06.271756 10919 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:26:06.317147  1799 pending_rounds.cc:107] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 24.20
I20260709 12:26:06.320562  1799 raft_consensus.cc:2889] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:06.317055  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61244.1us
I20260709 12:26:06.270961 10851 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:06.273718 10983 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:06.275635 11097 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:06.331749  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:06.332206  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.304085 11069 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:06.306174 11101 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:06.336174  1725 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:06.329032 11078 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:06.338119 11078 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.339331 11158 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:06.340123 11069 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.340917 11078 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.342178 10851 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.346617  1807 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 13. (term mismatch)
I20260709 12:26:06.305603 11116 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:06.347121  1807 pending_rounds.cc:85] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 12
I20260709 12:26:06.306130 11129 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:06.347414  1807 pending_rounds.cc:107] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260709 12:26:06.349375  1807 raft_consensus.cc:2889] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:06.346024 11076 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:06.352221 11076 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:06.353180 11076 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.354267 11076 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:06.319682 10919 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.349411  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:06.359579  1728 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:06.343066 10937 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.351395 11121 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.355621  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:06.356650  1798 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40096
I20260709 12:26:06.329399 11081 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:06.370039  1730 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:06.370172  1798 raft_consensus.cc:606] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.306573 11163 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:06.358645  1806 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:26:06.342546 11069 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.361950  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:06.376199  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3191) took 1574 ms. Trace:
I20260709 12:26:06.376627  1874 rpcz_store.cc:276] 0709 12:26:04.802206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.999276 (+197070us) service_pool.cc:224] Handling call
0709 12:26:05.019260 (+ 19984us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.143986 (+124726us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.372256 (+1228270us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b5400117454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:06.376116 (+  3860us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":284866816}
I20260709 12:26:06.355989 11044 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:06.381826  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:06.347672 11101 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:06.364665 10919 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:06.397333 11131 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:06.363011 10647 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:06.398160 11144 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.008s
I20260709 12:26:06.379153 10906 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.383392 11142 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:06.359566 11097 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:06.413412  1696 leader_election.cc:400] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26.
I20260709 12:26:06.418970  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3304) took 1234 ms. Trace:
I20260709 12:26:06.432875  1696 leader_election.cc:403] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:26:06.435634  1728 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:06.433010  1726 rpcz_store.cc:276] 0709 12:26:05.184612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.208490 (+ 23878us) service_pool.cc:224] Handling call
0709 12:26:05.208855 (+   365us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:05.247254 (+ 38399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.247289 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:05.247335 (+    46us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:05.247370 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.418594 (+1171224us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x7b54002b75d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:06.418647 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:06.418826 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87763712}
I20260709 12:26:06.391801 11101 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:06.440411  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:06.422571  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:06.426853  1874 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:26:06.442377  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.443553  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:06.443779  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.444718  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.442615  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:06.401185 11007 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.426179  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:06.359355 11129 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.413492 11117 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:26:06.448796  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.459293  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:06.449656  1697 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.959s (SENT) [suppressed 11 similar messages]
I20260709 12:26:06.461115  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.454011  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.462680   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:26:06.463215  1730 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:26:06.460109  1880 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:06.454535  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.465646  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.467108  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:06.467687  1880 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:06.468340  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.364058 11116 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.457067  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.373257 10937 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.432677 11142 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.471248  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.470733  1623 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 15 to 20, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 20 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:06.478297  1879 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:06.430927 10906 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.370657 11158 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:26:06.470117  1874 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:06.476264  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.480755  1877 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:06.446699 10703 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26.
W20260709 12:26:06.483660 11167 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.079s	user 0.000s	sys 0.001s
W20260709 12:26:06.484376 11155 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 12:26:06.483712 10703 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:26:06.489542 10742 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:06.489926 10742 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:06.491067 10742 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.492331 10742 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:06.498235 11045 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:06.500003 11162 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:26:06.506808 11132 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:06.508540  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54089us
W20260709 12:26:06.498463  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.122s	user 0.011s	sys 0.047s
W20260709 12:26:06.520531  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.144s	user 0.012s	sys 0.047s
I20260709 12:26:06.520805  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:06.521019  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.1447e+06us
I20260709 12:26:06.481809  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:06.430954 11131 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.377861 11121 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.516224  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:06.469676 11022 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:06.425848 10647 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.528565  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.546890  2416 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:26:06.551124  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.555886  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.480582 11142 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:06.541486  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 76880.7us
I20260709 12:26:06.455449 11144 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:26:06.576664  2410 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 17. (term mismatch)
I20260709 12:26:06.557572  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:06.577491  2410 pending_rounds.cc:85] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 16
I20260709 12:26:06.567709  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:06.577919  2410 pending_rounds.cc:107] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:26:06.576898  1874 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 20.
I20260709 12:26:06.578465  2410 raft_consensus.cc:2889] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:06.537089  2414 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 21: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:06.536742 11045 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.535557 11158 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:06.587153  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:06.584393  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:06.591073  1807 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20260709 12:26:06.577916 11098 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.649s	user 0.012s	sys 0.001s
I20260709 12:26:06.588056  1880 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:06.513891 10718 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.558751 10703 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:06.603111  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 153479us
I20260709 12:26:06.585824 11117 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:06.542130 11061 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:26:06.614501  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:06.618067  1726 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:06.588711 10923 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:06.620389  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:06.558763 11069 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
I20260709 12:26:06.621804  1726 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:06.623287  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:06.625720  1726 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:26:06.633838  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:06.609620 11098 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:06.639671  1728 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: [24.15-26.16]
I20260709 12:26:06.637763  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:26:06.641672  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84195.3us
I20260709 12:26:06.642517  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:06.643678  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:06.644512  1731 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 29 because replica has last-logged OpId of term: 27 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 13.
I20260709 12:26:06.619369 11158 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:06.598171 11045 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:06.648977  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83896.3us
I20260709 12:26:06.651325  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:06.637316 11017 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:06.654364  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:06.616695 10969 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:26:06.820731 11144 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.821430 11144 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.625432 11059 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:26:06.623904 10610 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:26:06.657580 11017 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.824250 11017 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:06.824841 11144 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.770922 11174 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:06.764631 10906 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.726475 10923 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:06.771394 11147 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.226s
I20260709 12:26:06.777604 11171 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:06.778426 11124 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.736s	user 0.009s	sys 0.005s
I20260709 12:26:06.779461 11121 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.780663 11181 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:06.781284 11178 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:06.649879 10915 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.115s
W20260709 12:26:06.798188 11133 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:26:06.840167 11174 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.815111 11184 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:06.816074 11132 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.767134 11117 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.651484  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:06.868638  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 217228us
I20260709 12:26:06.869805  2383 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:06.825457  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:06.761263  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 110000us
I20260709 12:26:06.783870 10926 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:06.837646  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1351) took 2393 ms. Trace:
I20260709 12:26:06.581583 11097 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:06.853942  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:06.858963  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 244505us
I20260709 12:26:06.833556 10937 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:06.813933 11007 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.843835 11129 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:06.878005 11131 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:06.919253  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 99012.2us
I20260709 12:26:06.920332 11045 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:06.920699 11109 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:06.921209 11109 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:06.921582 11109 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:06.920179 11111 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:06.877560  1878 rpcz_store.cc:276] 0709 12:26:04.440713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.440853 (+   140us) service_pool.cc:224] Handling call
0709 12:26:04.975954 (+535101us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:04.976346 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:26:05.265449 (+289103us) log.cc:844] Serialized 14 byte log entry
0709 12:26:05.765864 (+500415us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:05.778623 (+ 12759us) log.cc:844] Serialized 26 byte log entry
0709 12:26:05.779088 (+   465us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:05.795146 (+ 16058us) log.cc:844] Serialized 14 byte log entry
0709 12:26:06.247143 (+451997us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b18000bb780. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:26:06.247335 (+   192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:06.247412 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:06.257828 (+ 10416us) raft_consensus.cc:1685] finished
0709 12:26:06.257902 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:06.833797 (+575895us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b54002df0d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:06.833998 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11697,"prepare.queue_time_us":73292,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":103,"raft.queue_time_us":486610,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":238,"spinlock_wait_cycles":1238598144,"thread_start_us":738847,"threads_started":3,"wal-append.queue_time_us":178881}
I20260709 12:26:06.881227  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:06.881801  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:06.888832 11122 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:06.850858 10969 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:06.891119 10923 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:06.924475 11110 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:06.995559  2416 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:06.996013 11110 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:06.867834 11174 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:06.899051  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:06.861357 10647 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:06.904950  2418 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1542) took 2292 ms (client timeout 1959 ms). Trace:
I20260709 12:26:06.877497 10997 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:06.883579 11184 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 29 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:06.861267 11179 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:06.919437 10926 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:06.858593 10906 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:06.923520  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:06.925042  1873 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3192) took 2122 ms (client timeout 1519 ms). Trace:
I20260709 12:26:06.926667  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:06.927973  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:06.929405  1799 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
W20260709 12:26:06.989750  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 111009us
I20260709 12:26:06.860251 11171 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.023373 11184 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:07.018532  1873 rpcz_store.cc:269] 0709 12:26:04.802549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.838188 (+ 35639us) service_pool.cc:224] Handling call
0709 12:26:06.924809 (+2086621us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54001f7f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:06.924959 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":498432768}
W20260709 12:26:06.992856  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 108919us
I20260709 12:26:07.027050  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.027824  1873 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:26:07.028054  1881 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:06.894011 11013 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:06.861586 11178 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:06.862638 11181 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:26:07.006166  2418 rpcz_store.cc:269] 0709 12:26:04.612318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.613157 (+   839us) service_pool.cc:224] Handling call
0709 12:26:06.904726 (+2291569us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b54006009d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:06.904865 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3069940992}
I20260709 12:26:07.012883  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.029321  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:06.988425 11147 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:07.008066 11183 log.cc:927] Time spent T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.228s	user 0.000s	sys 0.031s
W20260709 12:26:07.005004 11081 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.627s	user 0.012s	sys 0.000s
I20260709 12:26:06.862824 11124 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.035254  2415 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:26:07.037269  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:07.012168 11044 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.621s	user 0.013s	sys 0.000s
I20260709 12:26:07.020151  1844 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:07.009496 10997 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:07.016683 11097 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:07.067420  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.059885  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:07.086179  1878 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:07.088074  1880 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:07.088477  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68442.3us
I20260709 12:26:06.989873 11017 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.091014  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.095722  1725 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20260709 12:26:07.040723 11101 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.585s	user 0.012s	sys 0.000s
I20260709 12:26:06.998744 10969 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.042663 11132 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:07.100193 11069 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:26:07.090291 10940 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.123342 10940 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.094199 10610 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:07.089391 10997 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:07.101213  2412 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:26:07.100998  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.102057  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:07.102650  2418 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:07.102735  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.017478 11179 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.103295 11095 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:07.089977 10915 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.102813 11097 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.104669 11158 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:07.105473 10919 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.681s	user 0.011s	sys 0.002s
I20260709 12:26:07.040918 11061 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.079s
I20260709 12:26:07.107846 10926 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.108450  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:07.090261  1881 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:07.103042  1878 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:26:07.174764  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.114878  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.030239 10906 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.023880 11171 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:07.041412 11178 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:07.107496 11076 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.753s	user 0.011s	sys 0.002s
I20260709 12:26:07.131793  1880 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:07.059195 11013 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:07.041214 11181 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:07.143443  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.034101 11123 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.087s
I20260709 12:26:07.146502 10919 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.121028 11101 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.176348  1725 raft_consensus.cc:2410] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 21 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 15.
I20260709 12:26:07.033942 10937 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.120620 11132 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:07.178704  2417 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:07.124461 10610 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:07.147337 10915 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.158977 11095 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:07.195794 11134 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.142s	user 0.003s	sys 0.010s
I20260709 12:26:07.197932 10992 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:07.060563 11147 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.210008  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.107355 10851 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:07.111807 11118 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.215662  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:07.274552  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58981.3us
I20260709 12:26:07.227687  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.275966  1731 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:26:07.230285  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.238477 10915 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:07.226018 11132 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:07.240234 10610 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.238755 11095 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:07.296056 11196 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
W20260709 12:26:07.297998 11195 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:26:07.241693 11154 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:07.299010 11154 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:07.300038 11154 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.242192 11139 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.244689 10919 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.218001 11013 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.247895 11017 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:07.217206 11163 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.832s	user 0.007s	sys 0.007s
I20260709 12:26:07.260380  1726 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:07.239168 11188 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20260709 12:26:07.282569  1877 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:26:07.284194 11112 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:07.284953  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.285532  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:07.348505  1876 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:07.288327 10906 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:07.288694 10718 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.291707  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.294929  1725 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 27: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:07.254753 10969 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.267483 10777 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:26:07.312361  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:07.427944  1731 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:07.301165 11154 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:07.255275 10992 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:07.324375 10757 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:07.326078 11179 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:07.328806 11147 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.330835  2413 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.439162  2413 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:07.333091 11013 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:07.337090  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1707) took 2577 ms. Trace:
I20260709 12:26:07.212389 11174 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.339291 11021 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.348938  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:07.406347  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:07.411427  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57295.3us
W20260709 12:26:07.412926  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.234843 10937 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.413933  1872 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:07.414480  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:07.415668  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.457425  2414 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:07.457868  2418 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 31 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:26:07.417418  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.459697  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.460397  1729 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:07.435463 11202 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:07.346520 11163 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.441717  2419 rpcz_store.cc:276] 0709 12:26:04.759627 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:04.760066 (+   439us) service_pool.cc:224] Handling call
0709 12:26:06.224295 (+1464229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:06.225029 (+   734us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.336851 (+1111822us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400591ad0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:07.337022 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3323389312}
I20260709 12:26:07.427543 10777 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:07.472100  2413 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:26:07.410598 11112 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:07.446276  1800 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:07.301822 11139 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.428025  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.459050  2412 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:07.460618  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:07.464162  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.292470 11118 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.466183  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.442188  2415 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:26:07.468530  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.474092  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1546) took 2151 ms. Trace:
I20260709 12:26:07.477238  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.513815  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1493) took 2339 ms. Trace:
I20260709 12:26:07.514467  1732 rpcz_store.cc:276] 0709 12:26:05.139962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.142046 (+  2084us) service_pool.cc:224] Handling call
0709 12:26:05.142354 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:05.417663 (+275309us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.478717 (+2061054us) spinlock_profiling.cc:243] Waited 247 ms on lock 0x7b54002d14d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:07.478960 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":568187648}
I20260709 12:26:07.515564  1729 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:07.517675  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:07.521730  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61240.2us
I20260709 12:26:07.521781  2419 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 24
W20260709 12:26:07.522879  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.500551  1731 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:07.526343  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:07.527185  1872 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:07.526119  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.527778  1729 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:26:07.528668  2418 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:07.479605  2416 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:07.541996  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.543576  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.544806  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.545730  2412 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:26:07.545732  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.513957  2413 rpcz_store.cc:276] 0709 12:26:05.322490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.322692 (+   202us) service_pool.cc:224] Handling call
0709 12:26:05.323172 (+   480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:05.480003 (+156831us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b5400525420. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd4fdc0 00007f16abdda1c0 00007f16abddb231 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c
0709 12:26:07.330407 (+1850404us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400525420. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd5606f 00007f16abdcef60 00007f16abdd07de 00007f16abddab49 00007f16abddb416 00007f16abdd7d8f 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62
0709 12:26:07.472576 (+142169us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.474044 (+  1468us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":830621824}
W20260709 12:26:07.501231 11037 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:26:07.472708 11013 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:07.478930 11112 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:07.548429  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.552552  2413 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:26:07.552983  1730 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 27 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:26:07.502480 10983 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.559559  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.560036  1731 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:07.467185 11124 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.501803 10926 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.462594 11021 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.563634  1725 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:26:07.569121  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.571655  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:07.572208  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:07.618896  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:07.619575  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:07.573338 11200 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.136s	user 0.000s	sys 0.000s
W20260709 12:26:07.574133 11197 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.296s	user 0.000s	sys 0.001s
W20260709 12:26:07.575721 11204 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:26:07.550853 10610 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.584203  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.641408  1806 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:07.584620  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.647348  1805 raft_consensus.cc:2393] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:07.649240  1805 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: [18.13-23.15]
I20260709 12:26:07.544688 11147 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:07.650818 11147 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.651850 11110 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.545078 11184 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.589375 10983 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.654524  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.655416  1725 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:07.656466  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:07.543376 11179 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:07.657719 11179 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:07.658165 11179 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:07.590811 10706 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:07.591758  1804 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3115) took 4122 ms (client timeout 1866 ms). Trace:
I20260709 12:26:07.471571 11202 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.593050 10997 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.594444  1803 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260709 12:26:07.680002  1803 pending_rounds.cc:85] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 15
I20260709 12:26:07.614084 11115 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:07.680410  1803 pending_rounds.cc:107] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260709 12:26:07.680851  1803 raft_consensus.cc:2889] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:07.600155  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:07.682706  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 82721.5us
I20260709 12:26:07.592288 10937 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:07.551335 11193 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.346s	user 0.001s	sys 0.000s
I20260709 12:26:07.609050  2419 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:07.622187  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.709971  1807 raft_consensus.cc:2410] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 27 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:26:07.624948  1872 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:07.619817  1798 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 19.
I20260709 12:26:07.633327  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.657091  1725 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
W20260709 12:26:07.664230  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.614753 11122 consensus_queue.cc:1059] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.067s
I20260709 12:26:07.614471 10742 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.066s
I20260709 12:26:07.713269 11122 raft_consensus.cc:3055] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:07.620976 10926 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.716974 10926 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:07.719848 11181 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.722915 10718 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:07.723909 11021 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.714555 10940 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:07.640259 11101 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:07.673846 11148 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.059s
I20260709 12:26:07.558931 11013 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.544894 11203 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.684856  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:26:07.765211  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 80444.8us
I20260709 12:26:07.599450  1876 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
W20260709 12:26:07.685923  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.667367  1805 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:26:07.768230  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:07.667958  1804 rpcz_store.cc:269] 0709 12:26:03.469080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:03.546904 (+ 77824us) service_pool.cc:224] Handling call
0709 12:26:07.591452 (+4044548us) spinlock_profiling.cc:243] Waited 2.65 s on lock 0x7b54001676d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:07.591641 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6093603200}
I20260709 12:26:07.769210  2414 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:07.770041  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.657338 10983 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:07.734368  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.708720 11189 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:07.672132  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.669595 10610 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.518889 10992 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.748881 10997 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.755580 11163 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.762634 11147 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:07.736838  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.790218  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.775472  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:07.777606  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:07.761865 11142 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.175s	user 0.014s	sys 0.000s
I20260709 12:26:07.744997 10436 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.802896  1876 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: []
I20260709 12:26:07.778735 10940 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:07.785372  1798 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:07.778402 11174 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.786552 11003 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:07.673988 11184 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.793833  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.794075  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.817603  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.819495  1799 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:07.799218  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.779718  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:07.804405  1873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.004s	sys 0.020s
W20260709 12:26:07.821722  1873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.004s	sys 0.020s
I20260709 12:26:07.782081  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:07.822625  1727 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:07.789671 11109 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.868s	user 0.014s	sys 0.000s
I20260709 12:26:07.810325  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.713893 11122 raft_consensus.cc:740] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:07.811509  1872 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:07.813050  1874 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:26:07.826367  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.779505 11189 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:07.829258 11189 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:07.804831 10915 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.834221  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3166) took 1273 ms. Trace:
I20260709 12:26:07.834843  1803 rpcz_store.cc:276] 0709 12:26:06.560525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.560633 (+   108us) service_pool.cc:224] Handling call
0709 12:26:07.594033 (+1033400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:07.681517 (+ 87484us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.833958 (+152441us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54001676d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:07.834143 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2308610816}
I20260709 12:26:07.808655  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:07.801563 10665 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.005s	sys 0.017s
I20260709 12:26:07.710135 11115 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.822492  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:07.810882  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.812014 11003 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:07.826272  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:07.858912  1723 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:26:07.859215  1800 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:07.807250 10940 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:07.864470  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.840822  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.867486  1799 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:07.871060  1799 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:26:07.853870  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:07.825438  1878 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:26:07.861634  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.868551  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:07.874482  1803 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:07.846568  1801 raft_consensus.cc:1275] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:26:07.842602  1805 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
W20260709 12:26:07.856976 10665 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.005s	sys 0.017s
I20260709 12:26:07.884194 11184 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:07.813583 11132 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.884349 11122 consensus_queue.cc:260] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.885833 11122 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:07.779531 11203 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.860285 11181 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:07.887799 11013 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.889137  1731 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:07.888989  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:07.805048 11142 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.903631  1727 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 31.
I20260709 12:26:07.856812 10706 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:07.904974  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.889075 11203 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:07.867949 11209 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:26:07.831599 11109 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.806896 10436 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.934476  2382 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:07.869494 10940 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:07.926118  1800 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:07.921413  1724 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:07.926551  1874 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:26:07.895753  1799 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:07.886629 11132 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.879541 11139 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.888374 11179 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.936646  1879 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:26:07.858536 10915 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:07.942333  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:07.941223 10969 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:07.944679  1726 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:07.920192 10919 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:07.948021  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:07.959414  1726 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:07.981230  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.984202  1726 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 16. (term mismatch)
I20260709 12:26:07.985055  1726 pending_rounds.cc:85] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 15
I20260709 12:26:07.985380  1726 pending_rounds.cc:107] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 25.16
I20260709 12:26:07.985718  1726 raft_consensus.cc:2889] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:07.937903 10706 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:07.989207  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:07.992328  1879 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 20: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:07.959023 10969 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 31 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:08.004760  1727 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:08.004990  1726 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:26:08.013113 11179 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:07.945343 11003 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:08.014972 11217 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:26:08.015373 11003 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:08.017489  1727 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:26:08.018427  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1507) took 1414 ms. Trace:
I20260709 12:26:08.018942  1723 rpcz_store.cc:276] 0709 12:26:06.603791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.604028 (+   237us) service_pool.cc:224] Handling call
0709 12:26:06.604485 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:07.859369 (+1254884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.018123 (+158754us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54000d2ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:08.018354 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2852555008}
I20260709 12:26:08.020654  1723 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:08.021018 11142 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.022007  1723 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:08.022589 10969 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.026206 10915 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:07.997859 11213 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.204s	user 0.000s	sys 0.002s
I20260709 12:26:08.028749  1731 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:08.034895  1726 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:08.030751  1874 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 12. Preceding OpId from leader: term: 30 index: 14. (index mismatch)
I20260709 12:26:08.031152   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:08.037685   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:07.956995 10919 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.040366  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.000129 10610 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:08.049381  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.051817  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.049343  1623 catalog_manager.cc:5697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 5 to 22. New cstate: current_term: 22 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:08.081750  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 215582us
I20260709 12:26:08.096408  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.097783  1872 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:08.098822  1623 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 16 to 17, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 17 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:08.104032  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:08.108888  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.110733  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.111171  1879 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:08.120850  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:08.113619  1623 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 21 to 27, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 27 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:08.126950  1623 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 15 to 21, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 21 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:08.088380 11160 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:08.094656 11179 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.135829  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 3058) took 2597 ms. Trace:
I20260709 12:26:08.138656  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:08.140909  1623 catalog_manager.cc:5697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 19 to 20, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 20 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:08.136548  1802 rpcz_store.cc:276] 0709 12:26:05.529049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:05.529159 (+   110us) service_pool.cc:224] Handling call
0709 12:26:05.549297 (+ 20138us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:05.549553 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:05.549588 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:07.328882 (+1779294us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b2c0005c178. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa6d1a11 00007f16aa6e1372 00007f16abe13a01 00007f16abdd843a 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2
0709 12:26:07.329455 (+   573us) log.cc:844] Serialized 26 byte log entry
0709 12:26:07.603596 (+274141us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:07.603648 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:07.603727 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:08.113611 (+509884us) raft_consensus.cc:1685] finished
0709 12:26:08.113636 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.126651 (+ 13015us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2323200,"thread_start_us":273740,"threads_started":1,"wal-append.queue_time_us":452702}
I20260709 12:26:08.144295 11215 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:08.144944 11215 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:08.144654  1623 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 21 to 22, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 22 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:08.145866 11215 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.146095  1623 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 20 to 26. New cstate: current_term: 26 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:08.147624  1623 catalog_manager.cc:5697] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 10 to 17. New cstate: current_term: 17 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:08.149377 11218 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:26:08.149065  1623 catalog_manager.cc:5697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 8 to 17, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 17 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:08.140940 11109 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.151350  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4135) took 1046 ms. Trace:
I20260709 12:26:08.150857  1623 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 18 to 23, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 23 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:08.151810  1729 rpcz_store.cc:276] 0709 12:26:07.104990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:07.529125 (+424135us) service_pool.cc:224] Handling call
0709 12:26:07.529515 (+   390us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:07.550113 (+ 20598us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:07.550164 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:07.558751 (+  8587us) log.cc:844] Serialized 52 byte log entry
0709 12:26:08.055419 (+496668us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:08.086386 (+ 30967us) log.cc:844] Serialized 14 byte log entry
0709 12:26:08.087185 (+   799us) log.cc:844] Serialized 14 byte log entry
0709 12:26:08.087501 (+   316us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.087622 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:08.151124 (+ 63502us) raft_consensus.cc:1685] finished
0709 12:26:08.151146 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.151295 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":74369,"prepare.run_cpu_time_us":123,"prepare.run_wall_time_us":6117,"raft.queue_time_us":9328,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":20364,"spinlock_wait_cycles":22546432,"thread_start_us":414542,"threads_started":1,"wal-append.queue_time_us":523183}
I20260709 12:26:08.154129  1623 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 22 to 23, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 23 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:08.155499 11097 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:08.160156 10986 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:08.159080  1623 catalog_manager.cc:5697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 19 to 21, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 21 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:26:08.162254 11154 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.721s	user 0.010s	sys 0.000s
I20260709 12:26:08.161799  1623 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 16 to 20, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 20 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:08.165473  1623 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 21 to 23. New cstate: current_term: 23 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:08.168136  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.166941  1623 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:08.177064  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:08.143275 11178 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.932s	user 0.007s	sys 0.007s
I20260709 12:26:08.176586  1623 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 15 to 24, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 24 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:08.174139  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.167656 11179 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.200170  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:08.187263 10665 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:08.186316 11218 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.142916 11160 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.209779  1623 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:08.216190 11097 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:08.227771 11216 log.cc:927] Time spent T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
I20260709 12:26:08.227459 11097 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.239051  1623 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 16 to 23, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 23 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:08.242946  1731 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 16. (term mismatch)
I20260709 12:26:08.243482  1731 pending_rounds.cc:85] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 15
I20260709 12:26:08.243724  1731 pending_rounds.cc:107] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 28.16
I20260709 12:26:08.243973  1731 raft_consensus.cc:2889] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:08.245347  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.245857  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.207304 10706 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:08.250561  1723 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:26:08.191402 11178 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:08.246755  1726 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
W20260709 12:26:08.253161  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.252221  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.254889  1622 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 274) took 73955 ms (client timeout 15000 ms). Trace:
I20260709 12:26:08.256946 11142 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.246647  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.270129  1731 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:26:08.260640  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.260843  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.261451  1623 catalog_manager.cc:5697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 16 to 18, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 18 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:08.264091  1874 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:08.264329  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.261482  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:08.273202  1872 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:08.257846 10436 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:08.273910  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:08.278052  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:08.278178  1732 raft_consensus.cc:1240] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:26:08.278931  1879 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:08.279633  1877 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:08.275408 10969 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:26:08.271456 11022 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.281123 11022 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:08.282344 11077 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:08.283519 11022 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:08.274205 11142 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:08.287882 11142 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.289837  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:08.291266  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 91261.4us
I20260709 12:26:08.292613  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:08.295679  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:08.217618 11218 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:08.291678  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:08.301292 10610 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:08.302717 11142 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.303656 11142 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.305511 11001 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:08.305996 11001 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:08.306221 11142 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:08.306396 11001 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:08.329763  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4120) took 1716 ms. Trace:
I20260709 12:26:08.325469  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3180) took 1180 ms. Trace:
W20260709 12:26:08.335582  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.360842  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.361552  2417 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:08.350706  1724 rpcz_store.cc:276] 0709 12:26:06.613336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.613524 (+   188us) service_pool.cc:224] Handling call
0709 12:26:06.613894 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:07.939641 (+1325747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.329453 (+389812us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54001bf3d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:08.329674 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3006404992}
I20260709 12:26:08.388617  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.389577  2417 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
W20260709 12:26:08.338382 11189 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.509s	user 0.011s	sys 0.001s
W20260709 12:26:08.336824 11112 consensus_meta.cc:287] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.720s	user 0.010s	sys 0.003s
I20260709 12:26:08.390935  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.389699  1623 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 20 to 25, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:08.376585 11117 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.308581 11179 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:08.391796 11232 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:26:08.333330 11118 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:08.400015  1802 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:26:08.334944 10919 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.393458 11017 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:26:08.411052 11112 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.394200 11072 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 12:26:08.311901 11097 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.306583 11178 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.379820 11077 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:08.421533 11116 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.401149 11007 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.378611 11080 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.041s
I20260709 12:26:08.431447  1623 catalog_manager.cc:5697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 10 to 25, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 25 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:08.436030 11077 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:08.363474 10937 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.407596 10610 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:26:08.442934 11160 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:08.455622  1623 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 12 to 17, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 17 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:08.460606  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:08.461051  1879 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 31.
I20260709 12:26:08.462252  1799 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:08.464705  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:08.466073  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:08.469337  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:08.499388  1801 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:26:08.350925  1807 rpcz_store.cc:276] 0709 12:26:07.144934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:07.726762 (+581828us) service_pool.cc:224] Handling call
0709 12:26:07.727162 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:07.889482 (+162320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:07.889528 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:07.890151 (+   623us) log.cc:844] Serialized 26 byte log entry
0709 12:26:08.109245 (+219094us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:08.109314 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.324819 (+215505us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54002c0454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:08.324885 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:08.324947 (+    62us) raft_consensus.cc:1685] finished
0709 12:26:08.324959 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.325359 (+   400us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1365,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122,"spinlock_wait_cycles":372493440,"thread_start_us":218539,"threads_started":1,"wal-append.queue_time_us":308403}
I20260709 12:26:08.489934  2411 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260709 12:26:08.409569 11189 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.510808  1623 catalog_manager.cc:5697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:08.416023 11117 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.512849 11160 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:08.519690  1875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.014s	sys 0.029s
W20260709 12:26:08.524134  1875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.632s	user 0.014s	sys 0.030s
I20260709 12:26:08.526530  1623 catalog_manager.cc:5697] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 8 to 22, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 22 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:08.533057  1623 catalog_manager.cc:5697] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 15 to 20, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 20 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:08.535638  2411 pending_rounds.cc:85] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 18
I20260709 12:26:08.539546  2411 pending_rounds.cc:107] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260709 12:26:08.540028  2411 raft_consensus.cc:2889] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:08.488574 11007 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.546965  1623 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 22 to 23, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 23 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:26:08.552768  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 211534us
I20260709 12:26:08.525434 11178 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.528200 11206 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:08.541097 11098 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.499123 10937 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:08.550348  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 258718us
W20260709 12:26:08.551937 11003 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.534s	user 0.014s	sys 0.001s
W20260709 12:26:08.551546 11122 consensus_meta.cc:287] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.665s	user 0.018s	sys 0.000s
I20260709 12:26:08.566946  1623 catalog_manager.cc:5697] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 19 to 23, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 23 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:08.550736 10969 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:08.526211 11160 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.590904  1724 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:26:08.585505  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.587353  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:08.582024 10986 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.012s
I20260709 12:26:08.594935 10986 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:08.595419 10986 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:08.596446 10986 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.598094 10986 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:08.600005 10992 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:08.269099  1622 rpcz_store.cc:269] 0709 12:24:54.299724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.300147 (+   423us) service_pool.cc:224] Handling call
0709 12:24:55.278932 (+978785us) spinlock_profiling.cc:243] Waited 311 ms on lock 0x7b5400002490. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb4640 00007f16b1b8b846 00007f16b1c9590e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:26:08.254494 (+72975562us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:54.389540 (+     0us) spinlock_profiling.cc:243] Waited 36 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16a8cce0ac 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16af80d16c
0709 12:25:54.531389 (+141849us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:55.284040 (+752651us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:55.284115 (+    75us) write_op.cc:435] Acquired schema lock
0709 12:25:55.284137 (+    22us) tablet.cc:662] Decoding operations
0709 12:25:55.627748 (+343611us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:55.664620 (+ 36872us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:55.676654 (+ 12034us) tablet.cc:685] Acquiring locks for 14 operations
0709 12:25:55.722321 (+ 45667us) tablet.cc:701] Row locks acquired
0709 12:25:55.722399 (+    78us) write_op.cc:260] PREPARE: finished
0709 12:25:55.722700 (+   301us) write_op.cc:270] Start()
0709 12:25:55.723116 (+   416us) write_op.cc:276] Timestamp: P: 1783599955722600 usec, L: 0
0709 12:25:55.723168 (+    52us) op_driver.cc:348] REPLICATION: starting
0709 12:25:55.735230 (+ 12062us) log.cc:844] Serialized 5499 byte log entry
0709 12:25:56.997895 (+1262665us) op_driver.cc:464] REPLICATION: finished
0709 12:25:57.477515 (+479620us) write_op.cc:301] APPLY: starting
0709 12:25:57.477639 (+   124us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:57.734399 (+256760us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:57.734452 (+    53us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:58.100888 (+366436us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:58.101300 (+   412us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=28,key_file_lookups=28,delta_file_lookups=0,mrs_lookups=0
0709 12:25:58.101361 (+    61us) write_op.cc:312] APPLY: finished
0709 12:25:58.101801 (+   440us) log.cc:844] Serialized 129 byte log entry
0709 12:25:58.661383 (+559582us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:58.811399 (+150016us) write_op.cc:454] Released schema lock
0709 12:25:58.858878 (+ 47479us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:58.938138 (+ 79260us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:58.938179 (+    41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1018169,"reported_tablets":50,"spinlock_wait_cycles":714433280,"child_traces":[["op",{"apply.queue_time_us":464682,"apply.run_cpu_time_us":108873,"apply.run_wall_time_us":1432953,"cfile_cache_hit":58,"cfile_cache_hit_bytes":62703,"cfile_init":1,"lbm_read_time_us":445,"lbm_reads_lt_1ms":4,"num_ops":14,"prepare.queue_time_us":391313,"prepare.run_cpu_time_us":29271,"prepare.run_wall_time_us":1250965,"raft.queue_time_us":670837,"raft.run_cpu_time_us":25095,"raft.run_wall_time_us":410441,"replication_time_us":1247248,"spinlock_wait_cycles":82687872,"thread_start_us":1535070,"threads_started":5,"wal-append.queue_time_us":971990,"wal-append.run_cpu_time_us":63798,"wal-append.run_wall_time_us":3091093}]]}
I20260709 12:26:08.607860  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.609150  1727 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:08.583755 11218 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:08.569079 11178 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:08.639003  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:08.634379  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.639726  1872 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:08.640071  1878 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:08.636519  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:08.644287  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 351751us
W20260709 12:26:08.630213 11171 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.306s	user 0.013s	sys 0.011s
I20260709 12:26:08.575829 11098 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.624575 11003 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:08.615918 11112 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.580289 11018 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.050s
I20260709 12:26:08.683473  1876 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
W20260709 12:26:08.692190  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.667013 11171 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.358s	user 0.013s	sys 0.012s
W20260709 12:26:08.705446  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.707221  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.711916 11007 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:08.721645 11206 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:08.688246 11003 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:08.723451 11003 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:08.728621  1622 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 260) took 18500 ms (client timeout 15000 ms). Trace:
W20260709 12:26:08.729291  1622 rpcz_store.cc:269] 0709 12:25:50.228149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.663309 (+18435160us) service_pool.cc:211] Skipping call since client already timed out
0709 12:26:08.728469 (+ 65160us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:26:08.732331 11160 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.733698  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1359) took 1702 ms. Trace:
I20260709 12:26:08.565235 11077 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:08.749274  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1721) took 1960 ms. Trace:
I20260709 12:26:08.624925 11122 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
I20260709 12:26:08.734333  1806 rpcz_store.cc:276] 0709 12:26:07.031678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:07.642455 (+610777us) service_pool.cc:224] Handling call
0709 12:26:07.642952 (+   497us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:07.660809 (+ 17857us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:07.660844 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:07.660889 (+    45us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:07.660929 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.733320 (+1072391us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b540026f554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:08.733394 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:08.733612 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":40437504}
I20260709 12:26:08.722510 11206 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:08.756664  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.750149  2411 rpcz_store.cc:276] 0709 12:26:06.788614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:06.790112 (+  1498us) service_pool.cc:224] Handling call
0709 12:26:06.827841 (+ 37729us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:08.540922 (+1713081us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.732349 (+191427us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54005f28d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:08.749151 (+ 16802us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3821116800}
I20260709 12:26:08.761066  1872 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:08.755180 10703 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:08.675493 10665 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.099s
W20260709 12:26:08.768594 11217 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:26:08.751538 11112 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:08.796029 10703 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:08.807484 10703 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:08.808007 11236 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:26:08.808804 10703 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:08.810063 11231 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:26:08.819038 11112 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:08.824909 10940 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.879s	user 0.009s	sys 0.005s
I20260709 12:26:08.804378 11189 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:08.832625  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:08.836483  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 190081us
W20260709 12:26:08.837273  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.856244  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.857300  2414 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:08.859001  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:08.859263  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.859736  2414 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:08.860170  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.862637  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.863605  2413 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:08.869803  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.870747  2413 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:08.866080  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:08.877592  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0469196549997832s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1521s for a response
I20260709 12:26:08.878749  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:08.900025  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.880311  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:08.881330  1730 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:08.886202  2414 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:08.887849  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.902261  2416 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:08.902715 10919 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:08.904227 10919 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:08.904755 10919 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:08.737094 10969 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.890288  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.881033 11117 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:08.885445  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:08.893306  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:08.849053 11203 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.825621 10940 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.868923 11106 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.953171 11106 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.956209 11106 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:08.895993 11178 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.897624 10937 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.956950 11115 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:08.957858 11115 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:08.879784 11142 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.898028 10986 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.893774 10998 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.005s
I20260709 12:26:08.879076  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.905979  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4131) took 1802 ms. Trace:
I20260709 12:26:08.967760  1732 rpcz_store.cc:276] 0709 12:26:07.103888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:07.523637 (+419749us) service_pool.cc:224] Handling call
0709 12:26:07.524042 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:08.278985 (+754943us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.905661 (+626676us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b54004362d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:08.905903 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1733679360}
I20260709 12:26:08.969825  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.907524  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.971959  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:08.909791  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:08.973556  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.974807  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.975649  1872 raft_consensus.cc:2393] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:08.911011  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:08.911881  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.976197  1877 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:08.912607  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 247808us
I20260709 12:26:08.978094  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:08.979295  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.914907  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.980428  1873 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:08.981789  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.916975  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.983211  1880 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:08.887796 11237 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.920787  2419 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:08.890893  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:08.923287  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:08.923532  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:08.989866  1874 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:08.894819 10777 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:08.888907  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:08.995726  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.906195 11097 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.067s
I20260709 12:26:08.925796  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:08.866861 11218 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
W20260709 12:26:08.936429  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 51108.2us
I20260709 12:26:08.939670  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:08.942027 11202 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:08.947387 11117 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:08.950579 11171 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:08.880075 11007 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:08.985147  1878 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 21: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:08.986696  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:08.990530  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.992928  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:08.939172 11147 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:08.935660 11189 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:09.102274  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 122432us
I20260709 12:26:08.837308  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:08.987560 11142 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:08.963292 11203 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.006592  1879 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
W20260709 12:26:09.032418  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:08.956977 11106 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:09.076094  1876 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:09.105700  1877 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:09.078168  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:09.111383  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:09.112563  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:09.075112  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:08.994380 10906 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:26:08.922613 10919 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.097608  1732 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:26:09.129678  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:09.099668  1730 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 26: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:08.999436  1725 raft_consensus.cc:2368] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27.
I20260709 12:26:08.980903 10986 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.077903 10777 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:09.082698 11117 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:09.005318  2411 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:09.115870  1874 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:26:09.092757 11171 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:09.147799  2383 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:09.133811  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 296566us
I20260709 12:26:09.149819  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:09.133984 10076 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:09.130649  1732 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:09.116883 10647 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:09.132455  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:09.143719  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:09.158198  1730 raft_consensus.cc:2410] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:26:09.108245 11080 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:09.117424 10906 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:09.127305 11118 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:09.166483  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:09.117637 11106 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:09.161502  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:09.160517 11242 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:09.174217  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:09.103220 11007 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:09.178905 11007 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:09.179550 11007 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:09.000408 11237 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.181777 11237 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:09.156757 11117 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:09.185256 11142 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:09.187194 11203 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:09.105185 11147 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:09.189533  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:09.190706 11147 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:09.192360 10940 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:09.167102 11178 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:09.193665 11178 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 26 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:09.194737 11178 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.166749 10906 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:26:09.198064  1725 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1516) took 1527 ms (client timeout 1630 ms). Trace:
W20260709 12:26:09.203130  1725 rpcz_store.cc:269] 0709 12:26:07.670975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:07.738269 (+ 67294us) service_pool.cc:224] Handling call
0709 12:26:09.197718 (+1459449us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54000d1bd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:09.197924 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2771616128}
I20260709 12:26:09.171911  1732 raft_consensus.cc:3060] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:09.106448 11189 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:09.173269 11106 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:09.161733 11080 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:09.173821 11122 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:09.212781 11122 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:09.213532 11122 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:09.197635  1726 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
W20260709 12:26:09.227833  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 345925us
I20260709 12:26:09.228790  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:09.169641 11233 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.044s
W20260709 12:26:09.221341  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.232117  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 339498us
W20260709 12:26:09.182561 11229 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.606s	user 0.004s	sys 0.002s
W20260709 12:26:09.209257 11001 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.884s	user 0.014s	sys 0.001s
I20260709 12:26:09.172504 11118 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:09.145961 11073 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.083s
I20260709 12:26:09.242061  1876 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:09.172356 10997 raft_consensus.cc:2804] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:09.246256  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:09.247058  2413 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
W20260709 12:26:09.250655  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 384630us
I20260709 12:26:09.215981 11080 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:09.254806  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.258170  1798 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:26:09.261255  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:09.260591  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.238022 11147 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:26:09.265784  1732 raft_consensus.cc:2468] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:09.266847  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:09.269062  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:09.272675  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:09.302814  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:09.303020  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:09.303666  1806 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:09.301522  1805 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:09.310269  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:09.319667  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:09.319749  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:09.321108  1804 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:09.253330 11118 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:09.334189  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:09.335126  1805 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:09.336783  1805 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:09.338114  1801 raft_consensus.cc:2393] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:09.321014  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:09.322530  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:09.330731  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:09.303834  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:09.346189  1799 raft_consensus.cc:2410] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:26:09.360113  1729 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:26:09.307035 11098 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:09.306798 10997 raft_consensus.cc:697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:09.345180 11147 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:09.254698 11001 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.344792 11189 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:09.375692 11003 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.652s	user 0.011s	sys 0.003s
W20260709 12:26:09.378549 11206 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.590s	user 0.012s	sys 0.000s
I20260709 12:26:09.376437 11147 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:09.437949  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 325496us
I20260709 12:26:09.441926  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:09.425436 10703 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:09.431906 11236 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:26:09.453246 11189 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:09.456507  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:09.436427 11147 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:09.466321  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:09.445441 11256 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:09.433768 11206 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.452075 10986 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:09.461139 10703 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:09.500917  1800 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.038s	sys 0.052s
I20260709 12:26:09.433048 11003 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.494201  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:09.502135  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.466301 11189 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:09.475008 10940 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:09.531764  2415 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.004s	sys 0.021s
I20260709 12:26:09.542658 11171 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.544245 11171 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:09.550967  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.553004 11257 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:26:09.502894 10997 consensus_queue.cc:237] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:09.558477 10992 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.958s	user 0.008s	sys 0.005s
I20260709 12:26:09.572402  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:09.523415   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:09.561470 11112 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:09.587177  1802 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 16. (term mismatch)
I20260709 12:26:09.587713  1802 pending_rounds.cc:85] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 15
I20260709 12:26:09.587936  1802 pending_rounds.cc:107] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 26.16
I20260709 12:26:09.588163  1802 raft_consensus.cc:2889] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:09.532326  2415 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.004s	sys 0.021s
W20260709 12:26:09.503846 11252 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.129s	user 0.001s	sys 0.000s
I20260709 12:26:09.568495  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:09.498260 11202 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:09.570246  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:09.574431  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 306670us
I20260709 12:26:09.575766  1873 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:09.577023  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:09.579504  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:09.526069  2419 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:26:09.656849  1880 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:09.590490  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:09.587249  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 454921us
I20260709 12:26:09.660210 10076 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.531872 11189 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:09.501931  1800 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.038s	sys 0.053s
I20260709 12:26:09.697510  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3186) took 1287 ms. Trace:
I20260709 12:26:09.654554  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:09.608469  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:09.573228 11254 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:09.644390  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 717180us
W20260709 12:26:09.699175  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44762.9us
I20260709 12:26:09.699290  1874 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:09.700670  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:09.568604 10940 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:09.594934  1877 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
W20260709 12:26:09.646564  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.591034 11116 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.647706  1697 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:09.644459  1879 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:09.660737 10076 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:09.663486  1731 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:26:09.664813  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:09.672034  2411 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.14-26.14]
I20260709 12:26:09.674813  2416 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:09.685529  2412 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:26:09.686654  2382 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:09.713474  1724 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:09.730666  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:09.504876 11256 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:09.735658  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 49091.2us
I20260709 12:26:09.736567  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:09.738408  2410 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:26:09.739642  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:09.742205 10076 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:09.745271  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 97709.8us
I20260709 12:26:09.746696  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:09.748008  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:09.729938 10906 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.510s	user 0.012s	sys 0.000s
I20260709 12:26:09.697901  1800 rpcz_store.cc:276] 0709 12:26:08.410010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.458987 (+ 48977us) service_pool.cc:224] Handling call
0709 12:26:08.459395 (+   408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:08.459634 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:26:08.696213 (+236579us) log.cc:844] Serialized 14 byte log entry
0709 12:26:09.697095 (+1000882us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:09.697158 (+    63us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:09.697218 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.697297 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:09.697431 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":436022,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":2256,"raft.queue_time_us":165518,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":7672,"thread_start_us":1198305,"threads_started":3,"wal-append.queue_time_us":411407}
I20260709 12:26:09.750460  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:09.751152  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:09.702139  2413 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:26:09.760835 11003 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:09.738404 11202 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:09.744796 10998 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:09.744701 11148 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:09.796083  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48778.8us
I20260709 12:26:09.798409  2382 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:09.800055  2419 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:09.753746 11007 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:09.746584 11001 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:09.815739  1724 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
W20260709 12:26:09.801676  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.000s	sys 0.037s
I20260709 12:26:09.804052  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:09.754035 11142 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:09.746402 11262 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:09.823153  1876 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
W20260709 12:26:09.825258  1772 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.060s	sys 0.124s
W20260709 12:26:09.826817  1772 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.061s	sys 0.125s
I20260709 12:26:09.739754 11116 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:09.828184  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 566921us
I20260709 12:26:09.734346 11254 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:09.819427 11261 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
I20260709 12:26:09.840615  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:09.802186 10998 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:09.757959 10906 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:09.823943 11233 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:26:09.864569 11266 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
W20260709 12:26:09.868768 11259 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:26:09.869381 10703 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:09.869999 10703 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:09.870538 10703 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:09.830492 11271 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:09.859189  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 47949.7us
I20260709 12:26:09.872335  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1560) took 1037 ms. Trace:
I20260709 12:26:09.818682 11003 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.048s
I20260709 12:26:09.809324 11202 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:09.877022  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.000s	sys 0.038s
I20260709 12:26:09.878903  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:09.900022  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 634605us
I20260709 12:26:09.900352  2419 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:09.819060 11237 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.043s
I20260709 12:26:09.862505 11206 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:09.858312 10998 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:09.858690 11080 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.551s	user 0.009s	sys 0.004s
W20260709 12:26:09.935552  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:09.897451 11131 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.050s
I20260709 12:26:09.948073 11131 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:09.948526 11131 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:09.949579 11131 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:09.955760 11269 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:26:09.956214 10906 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:09.861589 11233 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:09.877758 11112 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:09.875759  2417 rpcz_store.cc:276] 0709 12:26:08.834534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.867812 (+ 33278us) service_pool.cc:224] Handling call
0709 12:26:08.868176 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:09.182738 (+314562us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:09.182773 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:09.183485 (+   712us) log.cc:844] Serialized 26 byte log entry
0709 12:26:09.445132 (+261647us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:09.445185 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.760328 (+315143us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b5400572e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:09.760392 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:09.871982 (+111590us) raft_consensus.cc:1685] finished
0709 12:26:09.872009 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:09.872262 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6377,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"spinlock_wait_cycles":722446592,"thread_start_us":261132,"threads_started":1,"wal-append.queue_time_us":523816}
I20260709 12:26:09.873301 11262 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:26:09.987316  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 236913us
I20260709 12:26:09.854914 11267 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.115s
I20260709 12:26:09.989655  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:10.005628  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:09.844830 10940 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:09.872680 11263 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:26:10.013967  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 118308us
I20260709 12:26:09.882030 11271 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:09.844131 11178 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.010s	sys 0.009s
W20260709 12:26:09.876299  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 383045us
I20260709 12:26:09.873437 11218 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.136s
I20260709 12:26:09.859700 11148 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.149s
I20260709 12:26:09.987051 11233 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:10.042850  2419 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:09.874023 11272 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.154s
W20260709 12:26:10.038103  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.018276  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 278589us
W20260709 12:26:10.146394 11106 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.908s	user 0.009s	sys 0.005s
I20260709 12:26:10.048589 11263 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:10.012838 11131 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:09.958508 11080 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:10.039070 11178 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.012s	sys 0.009s
I20260709 12:26:10.026640 11271 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:10.005904 11112 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.037914 11262 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:10.106261 11077 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.008s	sys 0.041s
I20260709 12:26:10.025964 11206 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.087s
I20260709 12:26:10.124807  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3360) took 1136 ms. Trace:
I20260709 12:26:10.109109 11270 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:26:10.059650 10906 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.061s
W20260709 12:26:10.202996 11276 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:26:10.187160 11106 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:10.204571 11246 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 1.280s	user 0.001s	sys 0.004s
I20260709 12:26:10.205500 11106 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:10.125438 11256 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:10.135073 11254 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:10.207973 11112 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:10.137030 10610 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.209365 10610 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.212577 10610 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:10.138751 11203 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:26:10.139585 11122 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.889s	user 0.003s	sys 0.010s
W20260709 12:26:10.124971 11264 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.318s	user 0.000s	sys 0.001s
W20260709 12:26:10.128032 11118 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.766s	user 0.012s	sys 0.002s
I20260709 12:26:10.214540 11118 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:10.215461 11117 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.032s	user 0.010s	sys 0.005s
I20260709 12:26:10.216003 11117 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.146983  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:10.148533  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 247206us
I20260709 12:26:10.218345  1731 raft_consensus.cc:2368] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27.
I20260709 12:26:10.219046  1728 raft_consensus.cc:2368] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27.
I20260709 12:26:10.219306  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1562) took 1363 ms. Trace:
I20260709 12:26:10.219868  2415 rpcz_store.cc:276] 0709 12:26:08.856112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.856285 (+   173us) service_pool.cc:224] Handling call
0709 12:26:08.856727 (+   442us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:08.857186 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:08.857223 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:08.857789 (+   566us) log.cc:844] Serialized 26 byte log entry
0709 12:26:08.961494 (+103705us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b1800460ae0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ccde07 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ceac02 00007f16ab97220b 00007f16ab977d23 00007f16ab9780d1 00007f16abd93a0f
0709 12:26:09.648590 (+687096us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:09.648653 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.648769 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:10.218894 (+570125us) raft_consensus.cc:1685] finished
0709 12:26:10.218923 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.219221 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":476,"raft.queue_time_us":101,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":13503,"spinlock_wait_cycles":26177920,"thread_start_us":686523,"threads_started":1,"wal-append.queue_time_us":887103}
I20260709 12:26:10.223112  1725 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
W20260709 12:26:10.150008  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 314481us
W20260709 12:26:10.165187  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.181232 11007 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:10.224489  1772 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:10.226612  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:10.206184 11106 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.021s
I20260709 12:26:10.228773 11280 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:10.197211 11283 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:26:10.223155 11178 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:10.217564 11118 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:10.233151  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:10.235559  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:10.151973  1697 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:26:10.184901  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 179387us
I20260709 12:26:10.238883 11118 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:26:10.153187 11215 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.572s	user 0.028s	sys 0.022s
I20260709 12:26:10.152667 10703 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:10.239670 11118 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:10.239439  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:10.240756  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:10.195442  1730 rpcz_store.cc:276] 0709 12:26:08.988739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.227216 (+238477us) service_pool.cc:224] Handling call
0709 12:26:09.238601 (+ 11385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:09.246193 (+  7592us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:09.246227 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:09.259090 (+ 12863us) log.cc:844] Serialized 26 byte log entry
0709 12:26:09.803558 (+544468us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:09.803762 (+   204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:09.804079 (+   317us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:10.124502 (+320423us) raft_consensus.cc:1685] finished
0709 12:26:10.124531 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.124723 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":343,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":9792,"thread_start_us":440294,"threads_started":1,"wal-append.queue_time_us":703331}
I20260709 12:26:10.241145  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:10.194854 11077 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.009s	sys 0.041s
I20260709 12:26:10.241925  1880 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:10.200613 11202 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:10.201581 11080 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:10.202338  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:10.055740 11273 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.095s
I20260709 12:26:10.213512 11122 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.230882 11283 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:26:10.246006  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 94170.5us
I20260709 12:26:10.196907 11262 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:10.198249  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.240765 11171 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.696s	user 0.007s	sys 0.007s
I20260709 12:26:10.243196 11178 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:10.244011  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:10.256692  1872 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:26:10.244231  1876 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
W20260709 12:26:10.245200 11215 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.665s	user 0.028s	sys 0.023s
I20260709 12:26:10.262501  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:10.263240  1876 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:10.245592 10703 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:10.239413 11007 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.218393 11117 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:10.265648  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:10.266526  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:10.265095 11098 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:10.047649  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.279634  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:10.246346 11283 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:10.242291 11254 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:26:10.283414  2415 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:10.285544  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:10.294874  1876 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:10.317914  2413 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:10.320580  1876 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:26:10.322970  1876 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:10.330996  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:10.270861 10703 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:10.336715  2413 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:10.333312  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:10.337731  1876 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:10.240849 11118 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.280468 10992 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:10.347023  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:10.348299  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:10.256358 11285 consensus_queue.cc:1048] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 12, Time since last communication: 0.077s
I20260709 12:26:10.316726 11171 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:10.241063 11280 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.318727 10610 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:10.325084 11206 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.325989 11291 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:10.344787 10703 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:10.354900 11112 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:10.363206 10969 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:10.317831 11098 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.365094 10992 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:10.362939 11237 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:10.359633 11202 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:26:10.371656 11291 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:10.360880 10997 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:26:10.474857  1697 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:26:10.476754  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 210078us
I20260709 12:26:10.537669 11098 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:10.479998  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 242848us
I20260709 12:26:10.362162 11003 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.022s
W20260709 12:26:10.539665  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 65490.8us
I20260709 12:26:10.540001 11003 consensus_queue.cc:1243] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 28.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:26:10.419962 11271 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:10.370307 10610 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:10.554956  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 168285us
I20260709 12:26:10.556643  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:10.556865 11116 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:10.561545  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:10.484264  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 258308us
I20260709 12:26:10.490919  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3194) took 1964 ms. Trace:
I20260709 12:26:10.500463  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4144) took 1661 ms. Trace:
W20260709 12:26:10.503048  1728 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 4145) took 1663 ms (client timeout 2021 ms). Trace:
I20260709 12:26:10.508109  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:10.512359  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.433763 11280 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:10.516746 10777 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.358s	user 0.016s	sys 0.000s
I20260709 12:26:10.394236 11291 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:10.608325 11098 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:10.608917 11098 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:10.386503 10969 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.370118 11292 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:26:10.408951 11297 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:10.641803 11297 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:10.643050 10969 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:10.367429 11267 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.021s
W20260709 12:26:10.390012 11147 consensus_meta.cc:287] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.899s	user 0.015s	sys 0.000s
I20260709 12:26:10.366966 11171 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:10.655668 11147 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.372462 11206 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:10.385306 11273 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:10.372049 11283 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:10.633716 11106 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:10.379371 11288 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:10.362579 11290 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 12:26:10.374898 11299 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:10.478351  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3359) took 1489 ms. Trace:
W20260709 12:26:10.483738  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 313646us
I20260709 12:26:10.520699 10940 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:10.565784  1802 rpcz_store.cc:276] 0709 12:26:08.526649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.530222 (+  3573us) service_pool.cc:224] Handling call
0709 12:26:08.530755 (+   533us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:09.588577 (+1057822us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.490232 (+901655us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400132754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:10.490624 (+   392us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2428150656}
I20260709 12:26:10.566113  1729 rpcz_store.cc:276] 0709 12:26:08.839213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.841024 (+  1811us) service_pool.cc:224] Handling call
0709 12:26:09.354883 (+513859us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:09.360709 (+  5826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.499827 (+1139118us) spinlock_profiling.cc:243] Waited 513 ms on lock 0x7b54002b7fd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:10.500207 (+   380us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1180722560}
W20260709 12:26:10.566238  1728 rpcz_store.cc:269] 0709 12:26:08.839530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.841598 (+  2068us) service_pool.cc:224] Handling call
0709 12:26:10.502481 (+1660883us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540044a554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:10.502774 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2877446912}
I20260709 12:26:10.599395 11304 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:10.590854  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 83310.6us
I20260709 12:26:10.591923 11018 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:10.592564 11148 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.564271  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:10.378867 11112 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:10.601414  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.602363 11262 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:10.603135 11298 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.161s	user 0.001s	sys 0.001s
I20260709 12:26:10.603992  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:10.604898  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48769.8us
I20260709 12:26:10.606606  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:10.609396 11098 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:10.625779  1807 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:26:10.631126 10610 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:10.594621 11311 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.044s
I20260709 12:26:10.637979  1723 raft_consensus.cc:1275] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:10.589149 11293 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:10.676179  2382 leader_election.cc:400] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27.
I20260709 12:26:10.686455  2382 leader_election.cc:403] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:10.671157 10703 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:10.664750 11203 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:10.667303 11288 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:10.677846  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:10.660413 11122 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:10.677022  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:10.669795  1725 rpcz_store.cc:276] 0709 12:26:08.988498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:09.205724 (+217226us) service_pool.cc:224] Handling call
0709 12:26:09.206099 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:10.223755 (+1017656us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:10.477497 (+253742us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54000d20d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:10.478101 (+   604us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2337980160}
I20260709 12:26:10.666800 11106 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:10.666710 11283 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:10.673713  1770 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:10.681676  1807 pending_rounds.cc:85] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 12
W20260709 12:26:10.695114  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 56761.8us
I20260709 12:26:10.673730 11304 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:10.703737  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:10.674958 11148 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.704500 11288 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:10.676808 11112 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:10.671013 10940 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:10.710012  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 146072us
I20260709 12:26:10.711413  2415 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:10.674397 11018 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:10.658665 11171 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:10.661499 11206 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:10.662148 11273 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:10.696424 11203 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:10.720441  1807 pending_rounds.cc:107] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:26:10.740723  1807 raft_consensus.cc:2889] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:10.717259 11293 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:10.728384 10940 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:10.744055  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70635.8us
I20260709 12:26:10.740101 11116 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.019s
I20260709 12:26:10.750767 11116 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:10.751482 11116 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:10.752875 11285 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:10.753537 11285 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:10.754109 11285 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:10.755023  2410 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:26:10.755906 10919 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:10.728958 11297 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:10.744498  2412 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:26:10.765865 11148 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:10.767705 11313 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.296s	user 0.000s	sys 0.002s
I20260709 12:26:10.743752  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:10.771248 11292 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:10.771768 11292 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:10.771798 11271 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:10.772220 11292 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:10.775539  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 171785us
I20260709 12:26:10.775904 11254 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:10.767732 11018 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.776052  1879 raft_consensus.cc:1217] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Deduplicated request from leader. Original: 25.17->[26.18-31.20]   Dedup: 26.18->[29.19-31.20]
I20260709 12:26:10.776760  1879 pending_rounds.cc:85] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 18
I20260709 12:26:10.777134  1879 pending_rounds.cc:107] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 28.19
I20260709 12:26:10.777480  1879 raft_consensus.cc:2889] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:10.799732 11297 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:10.799976 11189 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.067s	user 0.011s	sys 0.011s
I20260709 12:26:10.801259 11001 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27.
I20260709 12:26:10.792743 11237 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:10.753310 11116 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.801846 11106 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:10.790372 11147 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:10.800624 11189 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.826134  1725 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:10.792954 11206 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:10.828331  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.854900  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:10.839076 11229 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.112s	user 0.002s	sys 0.002s
I20260709 12:26:10.833567  1697 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:10.832989 11302 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:10.858958  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:10.862996  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 187413us
I20260709 12:26:10.840335 11001 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:10.833140 11206 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.872524  1770 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:10.873473  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 103117us
I20260709 12:26:10.860924 11118 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:26:10.877792  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 70197.3us
I20260709 12:26:10.880103 11118 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:10.880650 11118 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:10.881017 11317 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.113s	user 0.000s	sys 0.002s
W20260709 12:26:10.883558  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49721.4us
I20260709 12:26:10.884982  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:10.886495 11323 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:10.887552 11293 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 25.20, Last appended by leader: 20, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.888562 11323 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:26:10.889247 10998 consensus_meta.cc:287] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.938s	user 0.022s	sys 0.004s
I20260709 12:26:10.890196 11323 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.899911 10703 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:10.901536 11098 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.904975 11098 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:10.828586 11237 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.908277 11189 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:10.909746 11325 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:10.834165 11254 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.077s
I20260709 12:26:10.911018  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:10.902431  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4042) took 3091 ms. Trace:
W20260709 12:26:10.912416 11318 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
W20260709 12:26:10.915014 11233 consensus_meta.cc:287] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.808s	user 0.016s	sys 0.012s
I20260709 12:26:10.915643 11233 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.918331 11233 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:10.836055 11271 consensus_queue.cc:1059] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:26:10.880832 11106 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:10.919891 11304 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:10.922765  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48515.5us
I20260709 12:26:10.923197  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:10.923447  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:10.925127  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:10.927603 11273 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:10.929531 11273 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:10.936251 11001 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:10.889845 10998 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:10.944201 11310 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.308s	user 0.004s	sys 0.000s
W20260709 12:26:10.946405  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:10.925089  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:10.912611  1881 rpcz_store.cc:276] 0709 12:26:07.810860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:07.810988 (+   128us) service_pool.cc:224] Handling call
0709 12:26:07.811392 (+   404us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:07.975012 (+163620us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:07.975055 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:07.979910 (+  4855us) log.cc:844] Serialized 52 byte log entry
0709 12:26:08.294594 (+314684us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:08.302571 (+  7977us) log.cc:844] Serialized 14 byte log entry
0709 12:26:08.401954 (+ 99383us) log.cc:844] Serialized 14 byte log entry
0709 12:26:08.407525 (+  5571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:08.597330 (+189805us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b180044bd80. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16afcac58f 00007f16a8ceb571 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd
0709 12:26:08.700356 (+103026us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:08.700443 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:09.181658 (+481215us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:10.603812 (+1422154us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b240023a968. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8b719c4 00007f16a8b6facb 00007f16a8d2adc9 00007f16a8d2aa55 00007f16a8d29fb0 00007f16ab9a0488 00007f16ab9a0be8 00007f16ab978398 00007f16ab97376a 00007f16ab972bb6
0709 12:26:10.901862 (+298050us) raft_consensus.cc:1685] finished
0709 12:26:10.901891 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:10.902175 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":131120,"prepare.queue_time_us":8409,"prepare.run_cpu_time_us":132,"prepare.run_wall_time_us":2007,"raft.queue_time_us":122541,"raft.run_cpu_time_us":633,"raft.run_wall_time_us":36603,"spinlock_wait_cycles":493409280,"thread_start_us":275384,"threads_started":1,"wal-append.queue_time_us":558547}
I20260709 12:26:10.964051  1725 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
W20260709 12:26:10.980368  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 92369.7us
I20260709 12:26:10.987113 11267 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:10.987493  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:10.988212 11302 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:10.991418  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:10.992803  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:10.993824  1799 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:10.995353  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:10.991627  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:10.996284  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 202618us
I20260709 12:26:10.999477  1806 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:11.000602  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:11.001039  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 137314us
I20260709 12:26:11.001472  1799 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:11.002048  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:11.003257  1727 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:11.003428 11299 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:11.004424 11299 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:11.005602 11271 raft_consensus.cc:3055] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:26:11.023626 11310 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:26:11.029929  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.037062 11292 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.038889  1802 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:11.040149 11291 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:11.042961  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.024263  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.045605  1804 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:11.006825 11098 consensus_queue.cc:1048] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:26:11.059321  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.060774  1801 raft_consensus.cc:2410] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:26:11.002480 10969 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.064074  2413 raft_consensus.cc:2368] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29.
W20260709 12:26:11.067139 11131 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.881s	user 0.016s	sys 0.006s
W20260709 12:26:11.071033  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.070955  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.072089  1724 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:11.081869 10998 leader_election.cc:290] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:11.095018  1727 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:11.009824 11271 raft_consensus.cc:740] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:11.108932  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:11.110921 10998 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.008585 11255 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:11.096226 11116 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:11.155256  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.088205 10969 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:11.032577 11001 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:11.161339  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 276687us
I20260709 12:26:11.185212  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.132207 11131 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.091172 11118 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:11.192924  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 190996us
I20260709 12:26:11.195026  2413 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:11.142627 10998 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.201185  1798 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 21.
I20260709 12:26:11.203749  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.090456 11203 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 30.14, Last appended by leader: 14, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:11.214501  1804 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:11.215541  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.219309 10969 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:11.221024 11118 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:11.211709 10906 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.222667  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.212349  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:11.227166 11292 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:11.227768  1728 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:26:11.228960  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.231415 11206 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:11.232868 10998 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:11.239956 11288 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.206982 10940 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.244403 11112 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.202339 11018 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:26:11.204496 11131 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.247324  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.248319  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:11.248989  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.249289  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:11.219400  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 128321us
I20260709 12:26:11.287187  1731 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:11.210280 11291 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:11.287139 10906 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:11.303599  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.287799 11203 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:11.332008  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:11.364159  1801 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:11.273890 11285 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.519s	user 0.013s	sys 0.004s
W20260709 12:26:11.363909  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75489us
I20260709 12:26:11.311080  1806 raft_consensus.cc:2410] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 28 because replica has last-logged OpId of term: 26 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
W20260709 12:26:11.334995  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 122764us
I20260709 12:26:11.291507 11283 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:11.381583  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.382472  1799 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:11.335547  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 256591us
I20260709 12:26:11.322499  1872 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:11.317912 10906 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:11.381155  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:11.390419  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.396867  1800 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:11.332340 11112 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:11.336691  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.395319  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.400308  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.388952  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.330013 10940 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:11.402751 11271 consensus_queue.cc:260] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:11.404608 11271 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:11.405019  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:11.406867 10919 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.407776  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.407649 10919 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.410459 10919 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:11.387370  1732 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:11.287246 11077 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.085s
I20260709 12:26:11.327491 10969 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.331604 11242 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:26:11.411393  1724 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:11.425978  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.427124  1724 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:26:11.430076 11131 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:11.430465 11273 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.500s	user 0.012s	sys 0.000s
I20260709 12:26:11.401039  1806 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:11.418980  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:11.424201 11309 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.006s	sys 0.004s
I20260709 12:26:11.379472 11285 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.438710  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:11.441004 11285 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:11.441473  1804 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:11.442262 11331 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:11.438591  1770 leader_election.cc:400] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27.
I20260709 12:26:11.445096  1770 leader_election.cc:403] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:26:11.444581 11309 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.836s	user 0.007s	sys 0.005s
W20260709 12:26:11.459560  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46813.8us
I20260709 12:26:11.460175  1726 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:11.461601  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.462321  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.461360  1799 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:11.467741  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.469539  1799 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:11.468446  1731 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
W20260709 12:26:11.464499  1845 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 2.834s (SENT) [suppressed 18 similar messages]
I20260709 12:26:11.466313 11273 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:26:11.486260  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.491727  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:11.505008 11262 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:11.505784 11262 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:26:11.505851  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 222474us
I20260709 12:26:11.507189  1805 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:11.511675  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:11.512585  2416 raft_consensus.cc:2368] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27.
I20260709 12:26:11.516029 11288 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:11.521862  1803 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:11.524997 11288 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.525285 10906 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:11.525902 10906 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:11.487682 11283 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:11.506448  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:11.531750  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:11.532979  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 318075us
I20260709 12:26:11.537570  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:11.539058  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.540132  1875 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:11.531410 11283 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:11.555030  1799 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:11.532302 11288 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:11.555042 11283 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:11.570709  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.554721 10906 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.571563  1878 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:11.572009 10906 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:11.562381  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:11.566869  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:11.582167  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75848.2us
I20260709 12:26:11.507228 11262 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:11.563167 11293 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:11.596772  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:11.598920  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:11.601372  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:11.605774  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.616699  2412 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:26:11.619505  1879 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:11.589772 10969 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:11.629196  1800 raft_consensus.cc:2468] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 31.
I20260709 12:26:11.634053  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:11.640826  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:11.590039 11348 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:11.642938  1846 leader_election.cc:400] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29.
I20260709 12:26:11.643738  1846 leader_election.cc:403] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:26:11.645536  1880 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:26:11.652871  1878 raft_consensus.cc:3060] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:11.654037  2412 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:11.659273  1881 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:11.661087  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.666896  1728 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:11.668282  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.671296  2413 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:26:11.668987  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:11.673578  1728 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:11.674734  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.675068  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.675925  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.675925  1880 raft_consensus.cc:2410] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:26:11.676187  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.691632  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:11.692214  1875 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:11.692382  2416 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:11.664951 11293 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:11.666932 11256 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:26:11.708071  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:11.809377  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 182665us
W20260709 12:26:11.700245 11346 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
W20260709 12:26:11.817695  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 213682us
I20260709 12:26:11.819796  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:11.806252 11256 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:11.831322  1728 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:26:11.843371  1724 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:11.844511  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:11.848999  2415 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:11.857828  1879 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:11.847560  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:11.859453  1728 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:11.860525  2410 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:26:11.864027  1806 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:26:11.865597  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.832010 11203 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:11.870937  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.872193  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.872400  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.873253  1873 raft_consensus.cc:2368] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
I20260709 12:26:11.873467 11171 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:11.873657  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.874230 11171 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:11.874855  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:11.875883  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.876529 11171 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:11.877049 11171 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:11.877305  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.879175  1880 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 22. Current term is 23. Ops: [20.12-22.13]
I20260709 12:26:11.874305  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:11.883178  1806 raft_consensus.cc:2368] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29.
I20260709 12:26:11.883632  2418 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:26:11.885046  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:11.886729  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.887434  1804 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:11.888283  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:11.888402 11342 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.255s	user 0.000s	sys 0.001s
I20260709 12:26:11.888887  1804 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:11.889806  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.890410  1804 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:11.890444  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.904814  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:11.905694  2418 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:11.905869  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.892386 11171 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:11.911345  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.908314  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:11.916276  1881 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:11.891388  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:11.918881  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.920014  1802 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
W20260709 12:26:11.920874  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.922909  1772 leader_election.cc:400] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27.
I20260709 12:26:11.921000  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.924726  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.925690  1802 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 25 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:26:11.927685  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3372) took 1032 ms. Trace:
I20260709 12:26:11.930663  1804 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
I20260709 12:26:11.928256  1730 rpcz_store.cc:276] 0709 12:26:10.895365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.926941 (+ 31576us) service_pool.cc:224] Handling call
0709 12:26:11.910662 (+983721us) spinlock_profiling.cc:243] Waited 3.28 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:26:11.921224 (+ 10562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.927096 (+  5872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:11.927131 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:11.927196 (+    65us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:11.927238 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:11.927328 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:11.927483 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7546112}
I20260709 12:26:11.922433 10610 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:11.943651 10610 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:11.944854  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:11.947249  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:11.948473  1804 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:26:11.949951  1804 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 16. (term mismatch)
I20260709 12:26:11.950314  1804 pending_rounds.cc:85] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 15
I20260709 12:26:11.950565  1804 pending_rounds.cc:107] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 25.16
I20260709 12:26:11.968874  1804 raft_consensus.cc:2889] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:11.953792 11324 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Append to log took a long time: real 1.079s	user 0.000s	sys 0.003s
I20260709 12:26:11.882047 11288 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:11.973070  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:11.977144  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 115017us
I20260709 12:26:11.921531 11077 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29.
I20260709 12:26:11.926543  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:11.982545 11351 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:26:11.985859  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:11.985167  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 93921us
I20260709 12:26:11.972311 11256 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:11.992909  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 70173.2us
I20260709 12:26:11.984771  1802 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:11.995734  1879 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:11.928540 11203 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:12.001935  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 140525us
I20260709 12:26:12.004670  1804 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:12.005136  1804 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:12.006080  1804 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.007335  1804 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:12.014613  1880 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 30. Current term is 31. Ops: [29.13-30.14]
I20260709 12:26:12.016148  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:12.017169 11203 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:12.017530  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:12.018601  1806 raft_consensus.cc:2410] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 30 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:26:12.019860  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:26:12.021665 11273 consensus_meta.cc:287] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.529s	user 0.004s	sys 0.004s
I20260709 12:26:12.022150 11273 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27.
I20260709 12:26:12.023579 11069 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:12.023641  1804 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:26:12.027696  1881 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: [23.16-26.17]
I20260709 12:26:12.028492 10906 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:12.030048  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:12.031185 11242 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:12.031682 11242 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.002130 11348 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:26:12.027550  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:12.046439  2416 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:26:12.046677  2417 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:12.050276  2411 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:12.032559 11355 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:26:12.053440  2413 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:12.062142  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:12.059162  1875 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:12.063112  1804 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:12.052258  1878 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:12.064491  1880 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 26. Current term is 27. Ops: [26.19-26.19]
I20260709 12:26:12.064673  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:12.044462 11242 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:12.070647  2418 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:12.074548  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:12.071131  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:12.075659  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:12.080152  1806 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:12.052670 11273 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:12.092883  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:12.092708  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:12.054353 11288 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.031s
I20260709 12:26:12.051445 11283 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.066191 11203 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.021s
W20260709 12:26:12.136704  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44078.1us
I20260709 12:26:12.090406 11148 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.092198 11322 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:12.164494  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:12.162128  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 134699us
I20260709 12:26:12.084704 11018 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.079s
I20260709 12:26:12.156371 11320 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:12.182716 11202 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.137s
I20260709 12:26:12.184165 11202 consensus_queue.cc:1243] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 123095127f334445889c21a42ba1d41e log is divergent from this leader: its last log entry 27.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:26:12.184870 11202 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:12.185324 11202 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:12.186705 11359 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:12.187455 11359 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.187922 11359 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:12.191166 10610 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:12.191562  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:12.197861  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 1376) took 1198 ms. Trace:
I20260709 12:26:12.198259 11131 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:12.198449  1803 rpcz_store.cc:276] 0709 12:26:10.999775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.008047 (+  8272us) service_pool.cc:224] Handling call
0709 12:26:12.197560 (+1189513us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b54002c0954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:12.197724 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1129480576}
W20260709 12:26:12.209764 11357 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.147s	user 0.000s	sys 0.001s
I20260709 12:26:12.170233 10919 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.041s
I20260709 12:26:12.212951  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1550) took 1100 ms. Trace:
I20260709 12:26:12.213541  1726 rpcz_store.cc:276] 0709 12:26:11.112668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.463002 (+350334us) service_pool.cc:224] Handling call
0709 12:26:11.473538 (+ 10536us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:11.501586 (+ 28048us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:11.501639 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:11.559794 (+ 58155us) log.cc:844] Serialized 26 byte log entry
0709 12:26:11.836820 (+277026us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:11.836862 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:11.836933 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:12.212600 (+375667us) raft_consensus.cc:1685] finished
0709 12:26:12.212633 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:12.212862 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9340,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":18583,"thread_start_us":244802,"threads_started":1,"wal-append.queue_time_us":363598}
I20260709 12:26:12.217031  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1400) took 1951 ms. Trace:
I20260709 12:26:12.217628  1872 rpcz_store.cc:276] 0709 12:26:10.265212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.265383 (+   171us) service_pool.cc:224] Handling call
0709 12:26:12.216687 (+1951304us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:12.216925 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1857290112}
I20260709 12:26:12.199062 11131 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:12.220479 11131 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.224932 11018 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:12.223400  1727 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1541) took 1584 ms (client timeout 1508 ms). Trace:
W20260709 12:26:12.223269  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 140722us
I20260709 12:26:12.168033 11322 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:12.225028 11131 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:12.248519  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1551) took 1015 ms. Trace:
I20260709 12:26:12.249012  1731 rpcz_store.cc:276] 0709 12:26:11.233324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.372256 (+138932us) service_pool.cc:224] Handling call
0709 12:26:12.248279 (+876023us) spinlock_profiling.cc:243] Waited 32.2 ms on lock 0x7b540035d754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:12.248425 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":74058112}
I20260709 12:26:12.244536  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:12.233829  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3197) took 3545 ms. Trace:
I20260709 12:26:12.168074 11347 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:12.235438  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:26:12.249929  1727 rpcz_store.cc:269] 0709 12:26:10.647694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:10.702947 (+ 55253us) service_pool.cc:224] Handling call
0709 12:26:12.223174 (+1520227us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b5400436cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:12.223311 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":620483968}
I20260709 12:26:12.226608 11171 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:12.252367  1807 rpcz_store.cc:276] 0709 12:26:08.688935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.697392 (+  8457us) service_pool.cc:224] Handling call
0709 12:26:08.697824 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:10.792713 (+2094889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.233315 (+1440602us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b54001d5454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:12.233706 (+   391us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4404746112}
I20260709 12:26:12.257012  1872 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
W20260709 12:26:12.260537  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 179599us
I20260709 12:26:12.225870 11080 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:12.178148 11273 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:12.243541 11320 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.238289 11288 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:12.279072 11288 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:12.279211  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.279994  1727 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:12.280349 11288 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.281816 11288 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:12.268399 11080 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:12.283995 11080 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.285142  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:12.285640 11080 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:12.286285  2410 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:12.168732 11148 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.286926  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.288481 11148 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:12.289535 11361 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:26:12.256493 11347 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.290673  2411 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:26:12.231256 11202 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.263556 11291 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:12.274907 11320 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:12.276312  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.252408 11322 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:12.255916 11262 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:12.230337 11283 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:12.267124  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:12.293004  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.291121  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.297024 11202 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:12.297798 10906 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:12.299168  1731 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 27: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:12.290727  1771 leader_election.cc:400] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
I20260709 12:26:12.302831  1771 leader_election.cc:403] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:12.303241  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:12.291179 11273 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:12.307662  2411 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:26:12.308066  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:12.308979  1878 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:12.290911 11347 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:12.321882  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.322854  1727 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 27: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:12.297200 11262 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.324087  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.325599 11116 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 20. Current term is 21.
I20260709 12:26:12.331302 11256 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:12.333781 11273 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.335354 11273 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:12.336506  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.338085 11283 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:26:12.338950 11358 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:26:12.303654 11271 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:12.341802  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.341764  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:26:12.342705  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 66474.1us
I20260709 12:26:12.343286  1878 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:12.298403 10906 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.318925 11291 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:12.353513  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.300307 11203 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:26:12.302656 11077 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.044s
I20260709 12:26:12.363874  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.319250 11069 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:26:12.367061  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.367364 11262 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:12.355566 11358 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:12.370025  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:12.372268  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.373045 10906 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:12.373288 11118 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:26:12.373466 11325 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:12.374346 11325 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:12.375456  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:12.376390  1732 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 30: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:12.378571  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:12.379261 11271 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:12.379262  1723 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:12.373072  1726 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:12.395248  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.396637  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:12.397449  1729 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 27: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:12.398648  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.399737  1732 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 30: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:12.399400 11242 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.400262  2413 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:26:12.375478 11325 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.401328  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:12.401932  2418 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:12.399434  1770 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.408682  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:12.407094  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.389098 11131 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:12.424026  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54115.8us
I20260709 12:26:12.425522  1726 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:12.431393  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:12.432870 11116 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:12.433283 11116 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:12.432894  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.434166 11116 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.435559  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.435809 11116 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:12.436834  1770 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.437539 11283 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:12.439383 11359 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.440696 11322 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.441591 11359 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:12.443576 11242 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:12.443457 11347 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.430599 11358 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:12.445238 11358 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:12.446352 11347 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:12.447543 11322 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:12.446233 11358 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.451107  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:12.423539 11271 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:12.467823 11271 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:12.468480 11271 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:12.459375 11322 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.016s
I20260709 12:26:12.475016  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:12.476477 11348 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:12.484988 11116 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:12.486027 11116 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:12.491070  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:12.456431 11131 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.416770 11291 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.056s
W20260709 12:26:12.541512  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 348643us
I20260709 12:26:12.504120 11018 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:12.580085  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 108591us
I20260709 12:26:12.507058 11322 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:12.506004 11262 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:12.590564  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:12.608865  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 433542us
I20260709 12:26:12.579108 11118 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:26:12.631126 11290 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.128s
I20260709 12:26:12.546756 10969 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:12.603106 11293 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:12.688889  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 252097us
I20260709 12:26:12.689983  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.693202  1799 raft_consensus.cc:2410] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:26:12.695129  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:12.605984 11320 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.590104 11368 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.089s
W20260709 12:26:12.704438  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 71050.7us
I20260709 12:26:12.706166  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:12.688114 11347 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:12.713251  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 389774us
I20260709 12:26:12.713553  2412 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:12.658499 11018 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:26:12.660250 11322 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.632314 11262 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.667240 10969 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.702099 11131 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:12.762686  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:12.714131 11372 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:12.747426 11242 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:12.707156 11293 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.781172  1845 leader_election.cc:400] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29.
I20260709 12:26:12.777506  1723 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:12.770558  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.784242  1724 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 31.
I20260709 12:26:12.707284 11007 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.213s
I20260709 12:26:12.788004  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.797137  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:12.797920  1724 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:26:12.800668  1726 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 27: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:26:12.803243  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44198.7us
W20260709 12:26:12.804442  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 143210us
I20260709 12:26:12.792881  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:12.781738 11242 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:12.809790 11242 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:12.817896  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:12.818611  1727 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:12.820574  1878 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:12.782923 11372 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:12.821478  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:12.822887  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 196379us
I20260709 12:26:12.788069 11290 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:12.844105  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:12.844604  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:12.760416 11347 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:12.843851  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:12.843444  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 489992us
I20260709 12:26:12.852233  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:12.867738 11383 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.868558 11383 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.870098  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:12.871675 11369 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.075s	user 0.001s	sys 0.001s
I20260709 12:26:12.872362 11293 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:12.873119  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:12.874147  1877 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:12.876549 11116 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.877226 11116 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:12.916158  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:12.919173  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:12.921979  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 100330us
W20260709 12:26:12.869230 11288 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.587s	user 0.014s	sys 0.000s
I20260709 12:26:12.839987 11372 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:12.925961  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 303344us
I20260709 12:26:12.925050  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:12.926292  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:12.928267  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:12.929167  1872 raft_consensus.cc:2410] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:26:12.948731  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:12.949774  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:12.948976  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:12.913962 11273 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.578s	user 0.014s	sys 0.000s
I20260709 12:26:12.853577 11290 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 31 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:12.960564  1878 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:12.949025  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3213) took 1066 ms. Trace:
I20260709 12:26:12.964882  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:12.964949  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 275056us
I20260709 12:26:12.968219  1770 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.980052  1876 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:26:12.926965 11309 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:12.984809  1807 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:12.985239  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:12.989920  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.942327 11118 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:12.991549  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:12.954984 11007 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:12.990527  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.022876  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:13.024268  1727 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
W20260709 12:26:13.025462  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 158793us
I20260709 12:26:13.026213  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:13.027747  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:13.030381  1803 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:13.031877  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:12.967070  1801 rpcz_store.cc:276] 0709 12:26:11.860023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.958504 (+ 98481us) service_pool.cc:224] Handling call
0709 12:26:11.968611 (+ 10107us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.979912 (+ 11301us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:12.265383 (+285471us) log.cc:844] Serialized 14 byte log entry
0709 12:26:12.925871 (+660488us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:12.925943 (+    72us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:12.926004 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:12.926092 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:12.926257 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":123404,"prepare.queue_time_us":241900,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":262179,"thread_start_us":712629,"threads_started":3,"wal-append.queue_time_us":532448}
I20260709 12:26:13.003187  1875 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
W20260709 12:26:13.040210  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50372.7us
I20260709 12:26:13.041471  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.042927  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:13.044154  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.049918  1872 raft_consensus.cc:2393] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:13.052028  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:13.060472  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:13.060866  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.061467  1878 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
W20260709 12:26:13.031744 11202 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.734s	user 0.004s	sys 0.010s
I20260709 12:26:12.986416 11273 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:12.996042 11203 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:13.066840  1879 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:13.003914 11379 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.070678  2410 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:13.074532  1804 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:12.991771 11288 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:13.082598  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:13.031765 11383 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.085325  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:13.051102 11388 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:13.091939  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:13.102808  1872 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
W20260709 12:26:13.091104  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 159077us
I20260709 12:26:13.104216  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:13.097478  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 56139.6us
I20260709 12:26:13.070312 11320 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.079900 10919 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:13.095996  1806 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:13.117177  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:13.108783  1770 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.098837  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.112035  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:13.101560  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.085378 11202 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.081053 11116 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.094513 11288 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:13.126962  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 75867.7us
I20260709 12:26:13.053809 11007 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.105976 11256 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:13.124598  1876 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
W20260709 12:26:13.129829  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 106545us
I20260709 12:26:13.138592  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:13.080260 11389 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:13.085009 11401 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:13.102118 11395 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:13.084405 11273 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.127496  1877 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:13.108343 11203 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:13.151481 11203 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.152752  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:13.154857 11116 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:13.155303 11116 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.155666 11116 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:13.155872  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:13.156131  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:13.157071  1801 raft_consensus.cc:2393] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:13.157181  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:13.157956  1803 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:13.100760 11148 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:26:13.114149 11388 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.159277  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:13.159502 11148 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:13.159775  1798 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:13.160043  1803 raft_consensus.cc:2393] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:13.160215 11148 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:13.160961 11388 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.117116 11322 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.161370  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:13.161602 11148 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.070129 11387 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.067s
I20260709 12:26:13.163022 11387 consensus_queue.cc:1243] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 26.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:26:13.125742 10919 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.164455 10919 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:13.165149 11271 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.696s	user 0.014s	sys 0.000s
I20260709 12:26:13.165642 11271 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.128544 11018 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:13.168968 11358 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:13.148249 11347 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.149153  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:13.171995 11373 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.250s	user 0.001s	sys 0.001s
I20260709 12:26:13.098464 11379 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.172598 11007 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.172760  1801 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:26:13.125218 11383 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:26:13.174517  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 57955.3us
I20260709 12:26:13.180460  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3215) took 1293 ms. Trace:
I20260709 12:26:13.183084 11392 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:13.183629 11399 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:26:13.184266  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 114301us
I20260709 12:26:13.184619 11240 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:13.134932 11288 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:13.181111  1799 rpcz_store.cc:276] 0709 12:26:11.887184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.892333 (+  5149us) service_pool.cc:224] Handling call
0709 12:26:13.180138 (+1287805us) spinlock_profiling.cc:243] Waited 786 ms on lock 0x7b54001e3a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:13.180367 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1808274816}
I20260709 12:26:13.183650 11392 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:13.195880 11283 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.758s	user 0.014s	sys 0.000s
W20260709 12:26:13.148411 11377 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
I20260709 12:26:13.171214 11118 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.197332 10906 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:13.199375  1875 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:26:13.173370 11370 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:26:13.190943  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:26:13.197441  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58968.9us
I20260709 12:26:13.204550  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.208783  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3203) took 1380 ms. Trace:
I20260709 12:26:13.140439 11385 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:13.203357 11387 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:13.222725  1801 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 27 index: 21. (index mismatch)
I20260709 12:26:13.234580  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:13.225032  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 116314us
I20260709 12:26:13.216848 11283 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:13.237273  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.136006 11202 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.206537 11148 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:13.219136  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:13.227244  1875 pending_rounds.cc:85] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 15
I20260709 12:26:13.240072  1875 pending_rounds.cc:107] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:26:13.240403  1875 raft_consensus.cc:2889] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:13.099505 11290 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 29.13, Last appended by leader: 13, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.207240 11392 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:13.166766 11242 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.240377 11401 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:13.248451 11394 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:13.249065 11394 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.248646 11401 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.249500 11394 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:13.249879 11242 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:13.251120  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:13.251338  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.251662 11401 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.252184  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:13.245378 11399 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.253976 11399 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.254616  1872 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:26:13.242286  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:13.228366  1805 rpcz_store.cc:276] 0709 12:26:11.828639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:11.828789 (+   150us) service_pool.cc:224] Handling call
0709 12:26:11.841690 (+ 12901us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:11.892209 (+ 50519us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:26:12.059766 (+167557us) log.cc:844] Serialized 14 byte log entry
0709 12:26:12.556760 (+496994us) log.cc:844] Serialized 14 byte log entry
0709 12:26:12.816394 (+259634us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b18002b1020. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:26:13.199809 (+383415us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b18002b1020. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287
0709 12:26:13.199936 (+   127us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:13.199995 (+    59us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:13.208510 (+  8515us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.208589 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:13.208728 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":77742,"prepare.queue_time_us":319395,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":104,"raft.queue_time_us":681030,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":38681,"spinlock_wait_cycles":20893824,"thread_start_us":655819,"threads_started":3,"wal-append.queue_time_us":633185}
I20260709 12:26:13.245942  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.256445 11399 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.253120 11400 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.257644 11400 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.245280 11370 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:13.243033 11368 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:13.261337 11370 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.262974 11393 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.146s
I20260709 12:26:13.264490 11273 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.246068  1879 pending_rounds.cc:85] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 15
I20260709 12:26:13.265020 11404 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:13.238708 11387 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.267184 11387 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:13.267599 11202 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.268206 11240 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:13.253084  2419 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:26:13.270488  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.272091 11395 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:13.265534  1879 pending_rounds.cc:107] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 26.16
I20260709 12:26:13.274770  1879 raft_consensus.cc:2889] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:13.276026 11271 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.266048 11404 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:13.277384 11386 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
W20260709 12:26:13.279716 11391 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260709 12:26:13.279908 11407 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:13.280673 11407 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.281741 11388 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.282413 11388 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.223064 11384 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.064s
I20260709 12:26:13.240710 11171 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:13.288604 11171 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.289142 11171 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:13.291273  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:13.291939  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:13.292721  2411 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:13.292583  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:13.297336 11388 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.297747 11404 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.300805  1801 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
W20260709 12:26:13.301223  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64039.2us
W20260709 12:26:13.303027  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60837.8us
I20260709 12:26:13.227114 10906 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:13.321478  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 111876us
I20260709 12:26:13.301294  1801 pending_rounds.cc:85] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 14
I20260709 12:26:13.323220  1801 pending_rounds.cc:107] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:26:13.323513  1801 raft_consensus.cc:2889] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:13.321915  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:13.327293  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:13.304221  2411 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:13.304989  1724 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
W20260709 12:26:13.306416  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 70345.5us
I20260709 12:26:13.335839  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:13.340916  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.343127  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.344451  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.345238  1881 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:13.344271  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.350477  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:13.343894  1873 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:13.353645  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.354485  1877 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:13.357542  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.359791  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.360395  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.362087  1874 raft_consensus.cc:2393] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:13.334092 11202 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:13.364907  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.375504  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:13.367584  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.362037  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.344234 11384 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.349977 11393 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:13.367728  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.368819  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:13.352159 11242 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:13.381096  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46071.7us
I20260709 12:26:13.345625 11395 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.378474  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:13.362690 11379 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.384231  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:13.395136 11242 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:13.394918 10919 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.395432 11395 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.396253 11242 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.397895 11395 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.392170 11398 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:13.382591 11393 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.367008 10906 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.393694  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:13.398715  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 52614.5us
I20260709 12:26:13.401563  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.401856  1726 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:13.402222  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.398873 11384 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.400028 10906 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:13.405987 11405 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.142s	user 0.003s	sys 0.001s
I20260709 12:26:13.399910  1729 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:13.370594 11388 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:13.408988  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.410059 11401 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:13.400787 11271 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:13.411474 11271 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.411943 11271 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:13.401422 11322 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.011s
I20260709 12:26:13.413151 11400 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.414686 11262 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.400542 11007 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:13.411667  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.400827  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:13.402839  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.400214  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:13.417924  1727 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:13.399329  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:13.349748 11347 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:26:13.400858  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.416262  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.436609  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.437848  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:13.392796 11256 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:13.415460 11007 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:13.438570  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.439282  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:13.446560  1727 raft_consensus.cc:2376] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24. Re-sending same reply.
I20260709 12:26:13.449519  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:13.451577  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:13.399307 11393 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:13.457388  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:13.461220  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:13.443979  1877 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:13.473171 11407 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.473202 11390 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:13.475270 11288 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.477833  2414 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 30. Current term is 31. Ops: [30.20-30.20]
I20260709 12:26:13.429705 11395 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:13.479063  1800 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 28: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:13.479297 11395 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:26:13.481769 11372 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.500s	user 0.012s	sys 0.001s
I20260709 12:26:13.482276 11372 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.458159 11240 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.029s
I20260709 12:26:13.444893 11206 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:26:13.484454  2419 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:13.485302  2413 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:13.485978 11018 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:13.484941  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 82808.3us
I20260709 12:26:13.401875 11408 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:13.427111 11398 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:13.444285 11400 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:13.505800  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 97581.8us
I20260709 12:26:13.444980 11401 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:13.508911  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89543.3us
I20260709 12:26:13.442888 11384 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.452836 11007 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:13.523710  1807 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.519s	user 0.010s	sys 0.002s
I20260709 12:26:13.524295  1807 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:13.477867 11288 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.427280 11202 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.522047 11402 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:13.494987 11290 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:26:13.494153 11390 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:13.494323 11408 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:13.550671  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 122834us
I20260709 12:26:13.560098  1876 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 20. Current term is 21. Ops: []
I20260709 12:26:13.524376 11407 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.573511 11407 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.573916 11271 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.571585 11390 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.522061 11394 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.589143  1802 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:13.586452  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:13.591531  1807 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:13.587356  1801 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:13.593693  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.596517  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.554692 11384 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:13.607208  2410 raft_consensus.cc:2368] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:26:13.588742 11400 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:13.576975 11408 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:13.604009 10998 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:13.586411 11116 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.562369 11402 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.688372  1805 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:13.666191 11118 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:26:13.697685  1726 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:13.619805 11400 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.706043  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:13.707842  1731 raft_consensus.cc:2393] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:13.707505 11007 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.709133  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.710070  1731 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:13.710202 11007 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.616680 11392 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.711771  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:13.704605  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 118510us
I20260709 12:26:13.612164 11419 consensus_queue.cc:1059] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.082s
I20260709 12:26:13.724380 11116 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.670809 11018 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:13.715440  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:13.720389 11392 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:13.696087 11118 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:13.731912  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.732942  1726 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:26:13.734841  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 207515us
I20260709 12:26:13.742954 11385 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:13.742844 11018 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.744240  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:13.745219 11018 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.746500 11116 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:13.746660  2417 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:26:13.746974 11116 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:13.747352 11116 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:13.749056  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:13.728899  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47901.8us
W20260709 12:26:13.735641 11413 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:26:13.736991 11118 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.751896 10998 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:13.752364 10998 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:13.752444 11320 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:13.752825 11118 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:26:13.737428 11394 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.683822 11407 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.656944 11384 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.752825 10998 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:13.755167  2383 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:13.756834 11384 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.756945  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:13.740604 10969 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:13.741644  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 178746us
I20260709 12:26:13.753000 11320 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:13.757889  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:13.739429 11171 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.696213 11372 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.775657  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:13.775880  1731 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
W20260709 12:26:13.778905 11421 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:26:13.684067 11417 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.091s
I20260709 12:26:13.790957 11438 raft_consensus.cc:2764] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:26:13.794932  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:13.795555 11171 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.797549 11202 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.765190 11148 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:13.778618 11432 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:13.785288  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:13.799787  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:13.801031 11018 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:13.801520 11400 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.787858  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:13.784029  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.777383 11262 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:13.769727 11398 raft_consensus.cc:2804] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:13.818933 11398 raft_consensus.cc:697] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 31 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:13.820338 11398 consensus_queue.cc:237] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.821025  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:13.821985  1876 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:13.805457  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:13.784646 11403 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.522s	user 0.000s	sys 0.001s
I20260709 12:26:13.824589  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:13.825218  1877 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 27
W20260709 12:26:13.807636  2383 connection.cc:256] RPC call timeout handler was delayed by -1.3850209570000516s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
W20260709 12:26:13.826208  2383 connection.cc:256] RPC call timeout handler was delayed by -1.2896346060001633s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20260709 12:26:13.826637  2383 connection.cc:256] RPC call timeout handler was delayed by -1.1434155990000363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2919s for a response
W20260709 12:26:13.827072  2383 connection.cc:256] RPC call timeout handler was delayed by -1.0443845590002638s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
I20260709 12:26:13.828497  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:13.787593  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.788512 11402 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:13.829850 11419 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:13.817180  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.817584 11262 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.825572  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3396) took 1056 ms. Trace:
I20260709 12:26:13.816684 11148 raft_consensus.cc:2764] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:26:13.831847  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.832347 11419 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:26:13.833294 11262 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.833778 11419 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:13.831575  1732 rpcz_store.cc:276] 0709 12:26:12.768723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.781020 (+ 12297us) service_pool.cc:224] Handling call
0709 12:26:12.846713 (+ 65693us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:26:12.847215 (+   502us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:12.914469 (+ 67254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:12.914526 (+    57us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:12.955610 (+ 41084us) log.cc:844] Serialized 52 byte log entry
0709 12:26:13.104991 (+149381us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:13.141058 (+ 36067us) log.cc:844] Serialized 14 byte log entry
0709 12:26:13.175155 (+ 34097us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.175245 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:13.544303 (+369058us) spinlock_profiling.cc:243] Waited 7.06 ms on lock 0x7b1800419bc0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:26:13.783958 (+239655us) spinlock_profiling.cc:243] Waited 61.6 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:13.825339 (+ 41381us) raft_consensus.cc:1685] finished
0709 12:26:13.825361 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:13.825502 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3089,"prepare.queue_time_us":32347,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":3437,"raft.queue_time_us":96686,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":10044,"spinlock_wait_cycles":162510592,"thread_start_us":100188,"threads_started":1,"wal-append.queue_time_us":246690}
W20260709 12:26:13.815286 11387 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.548s	user 0.012s	sys 0.003s
I20260709 12:26:13.835786  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:13.836316 11387 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.837450  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.837545 11390 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:13.838064  1724 raft_consensus.cc:2368] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:26:13.838229  1726 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:13.838348 11390 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:13.833316  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:13.840044 11387 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:13.815220 11388 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:13.840741  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.841032 11388 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:13.836755  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:13.841677 11320 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.842373  1731 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:13.843327 11432 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:13.844077 11393 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:13.845714 11432 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:26:13.846459  1872 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:13.849519  1725 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:26:13.839530 11390 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.850328  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.856509  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:13.839427 11406 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.544s	user 0.003s	sys 0.000s
I20260709 12:26:13.869369  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:13.854804  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:13.857801  1696 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.872821  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:13.883798  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:13.891520  2383 leader_election.cc:304] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:13.893253  2417 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:13.825202 11401 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:13.840760 11387 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:26:13.918197  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45459.3us
I20260709 12:26:13.929710  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:13.935834  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.938236 10969 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:13.939810  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:13.874248 11402 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:13.946916  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:13.944636  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:13.946741  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:13.927651 11237 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:13.930608 11385 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:13.930569 11400 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:13.955696  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:13.956802  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1588) took 1188 ms. Trace:
I20260709 12:26:13.957183 11433 consensus_queue.cc:1059] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.008s
I20260709 12:26:13.946012 11206 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:13.947263 11402 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:13.939675 11372 raft_consensus.cc:2804] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:13.969718  1876 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:14.038872 11402 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:14.038476 11372 raft_consensus.cc:697] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:14.041755 11372 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.038188 11400 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:14.043233  1723 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 3403) took 1270 ms (client timeout 1571 ms). Trace:
W20260709 12:26:14.043879  1723 rpcz_store.cc:269] 0709 12:26:12.772895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.796654 (+ 23759us) service_pool.cc:224] Handling call
0709 12:26:14.042930 (+1246276us) spinlock_profiling.cc:243] Waited 681 ms on lock 0x7b540044aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:14.043146 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1566787968}
I20260709 12:26:13.969766  2418 rpcz_store.cc:276] 0709 12:26:12.768200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.768418 (+   218us) service_pool.cc:224] Handling call
0709 12:26:12.794070 (+ 25652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:12.993193 (+199123us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:12.993250 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:13.039349 (+ 46099us) log.cc:844] Serialized 26 byte log entry
0709 12:26:13.144555 (+105206us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:13.144674 (+   119us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:13.144772 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:13.838659 (+693887us) spinlock_profiling.cc:243] Waited 8.76 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:13.956351 (+117692us) raft_consensus.cc:1685] finished
0709 12:26:13.956384 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:13.956674 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40400,"raft.queue_time_us":197634,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":23620,"spinlock_wait_cycles":20158848,"thread_start_us":208891,"threads_started":2,"wal-append.queue_time_us":171830}
I20260709 12:26:13.972891  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:13.975281  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.049122  1872 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:13.979820  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:13.975323  1877 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:13.977002 11359 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:14.050709 11359 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:13.983600 11262 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:13.973696 11237 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:13.984508 11404 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:14.052162 11404 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:14.051843 11359 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.052735 11237 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.053144 11404 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.053166 11359 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:14.054494 11404 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:13.978302 11206 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:14.056298 11206 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:13.991824 10610 raft_consensus.cc:3055] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:14.058981 10610 raft_consensus.cc:740] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:14.060029 10610 consensus_queue.cc:260] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:13.996048  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.002460 11117 raft_consensus.cc:3055] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:14.061266 11400 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.020279 10998 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.063154 11293 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:14.063738 11293 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:14.066358 11293 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:13.977270 11401 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.022718  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:14.023388  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:14.024870  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:14.025720 11419 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.029861  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:14.033818  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 3100) took 1079 ms. Trace:
I20260709 12:26:13.949848 11393 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:14.035684 10919 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:13.984894 11116 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.061339 11117 raft_consensus.cc:740] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:14.049124  2417 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:14.020674  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.071185  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:14.019266  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:14.077807  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.080731  2411 raft_consensus.cc:2368] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:26:14.154258  2414 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:14.080438  1800 rpcz_store.cc:276] 0709 12:26:12.955314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.963751 (+  8437us) service_pool.cc:224] Handling call
0709 12:26:14.033456 (+1069705us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b5400132754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:14.033702 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1101507968}
I20260709 12:26:14.132555  2413 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:14.133492  1729 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:14.156955  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:14.149978  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3398) took 1380 ms. Trace:
W20260709 12:26:14.093808  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 114073us
I20260709 12:26:14.161031  1725 rpcz_store.cc:276] 0709 12:26:12.769464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:12.780518 (+ 11054us) service_pool.cc:224] Handling call
0709 12:26:13.343709 (+563191us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b5800003098. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:26:13.343914 (+   205us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:13.849979 (+506065us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:14.149516 (+299537us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b54003cd2d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:14.149861 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1185916928}
W20260709 12:26:14.162803  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 142195us
I20260709 12:26:14.163511  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:14.164719  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:14.165231  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:14.166365  1725 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 27: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:14.166831 11438 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:14.167567 11438 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:14.168668 11438 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.186220  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:14.188491  1731 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:14.191757  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:14.192663  1723 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:14.192854  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:14.193797  1730 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:14.195462  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 124387us
W20260709 12:26:14.197283  1875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.020s	sys 0.018s
W20260709 12:26:14.199090  1875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.021s	sys 0.018s
I20260709 12:26:14.063735 10998 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:14.202121  1846 leader_election.cc:400] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:26:14.017473 11388 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.202402  1876 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:14.211699  1846 leader_election.cc:403] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:14.151618 11202 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.153106 11283 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:14.222221  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.223125  1876 raft_consensus.cc:2368] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28.
I20260709 12:26:14.201064  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.223934  1803 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:14.184692 11288 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.195139 10969 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:14.208702 11242 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:14.207027 11407 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.239363 11171 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:14.229422 10906 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:26:14.160625 11401 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.160796 11419 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.207002 11400 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:14.212309 11358 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:14.277706  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 419961us
W20260709 12:26:14.281886  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80917.7us
I20260709 12:26:14.269448 11283 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:14.292485  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 349406us
I20260709 12:26:14.256322 11242 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:14.325268 11242 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:14.290347  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.291162  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:14.240306 11202 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.251835 11288 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:14.319387  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:14.335398 11435 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.081s	user 0.000s	sys 0.001s
I20260709 12:26:14.336064 11447 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:14.292948 11256 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:14.340770  1723 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:14.344729  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.285799 11320 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:14.295373 11387 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 30 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:14.353376  1730 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
W20260709 12:26:14.331699 11118 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.578s	user 0.013s	sys 0.000s
I20260709 12:26:14.358755  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:14.360419  1770 leader_election.cc:400] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:26:14.360814  1770 leader_election.cc:403] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:26:14.325768 11116 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:14.338204 11394 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:14.295986 10919 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.055s
I20260709 12:26:14.294162 11358 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.291043 11407 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.296192 11283 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.336850 11447 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:14.345302 11359 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.294435 11400 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.391170  2417 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:14.389107  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:14.369352 11443 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:26:14.394584  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:14.283447 11117 consensus_queue.cc:260] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:14.418594  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.428495  2414 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:26:14.428849  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:14.429188  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 97379.8us
W20260709 12:26:14.376696 11441 log.cc:927] Time spent T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:26:14.432569  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.433534  1876 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:14.435105  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.443255  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:14.444077  1881 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:14.446300  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.448665  1877 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: [28.21-28.21]
I20260709 12:26:14.450512  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:14.451368  1876 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:14.453095  1878 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
W20260709 12:26:14.458055  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63586.8us
I20260709 12:26:14.351754 11256 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.370450 11449 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:14.455902  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.431545  2382 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:14.469259  2413 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:14.397487 11116 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.481391 11116 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.482406 11283 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.482641 11446 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:14.422375 11359 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.483541 11116 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.473165  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:14.484402 11450 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:26:14.485265 11359 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.473179  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:14.415125 11447 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:14.389885 11080 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:14.489145 11080 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:14.482502 11358 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:14.489867 11080 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:14.491076 11408 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.827s	user 0.012s	sys 0.002s
I20260709 12:26:14.491582 11408 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.493960 11408 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:14.388732 11118 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.110s
I20260709 12:26:14.497885 11080 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:14.499478 11439 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:26:14.500834 11434 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:14.500957 11080 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.501590 11434 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.504578 11434 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:14.505264  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73813.4us
W20260709 12:26:14.515669 11445 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20260709 12:26:14.530966 11390 raft_consensus.cc:2804] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:14.486456 11359 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:14.534646  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:14.518951 11444 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.203s	user 0.001s	sys 0.000s
I20260709 12:26:14.534838 11359 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:14.520386 11288 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:14.521307 11453 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.178s	user 0.001s	sys 0.000s
I20260709 12:26:14.520952  1876 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:14.523803 11458 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.538065 11458 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.524209 11256 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.541133  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:14.541217 11458 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.525774 11407 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.525436 11447 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.524487  1881 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:14.520987 11401 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:14.548905  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 188646us
W20260709 12:26:14.547432 11402 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.508s	user 0.003s	sys 0.009s
I20260709 12:26:14.466401 11388 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:14.468905 11347 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:26:14.477823 11405 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:26:14.565699  2413 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:26:14.478181 11456 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.546154 11206 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.573413  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.457864 11451 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.152s
I20260709 12:26:14.556630 11116 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:14.572674 11402 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:14.599623 11452 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.244s	user 0.001s	sys 0.000s
I20260709 12:26:14.600950 11402 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:14.602728 11171 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:14.603432 11400 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.565335 11288 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:14.604565 11288 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:14.551910 11283 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.608273 11283 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.561472 11390 raft_consensus.cc:697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:14.615309 11390 consensus_queue.cc:237] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.574945 11407 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:14.619652  1803 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 3113) took 1466 ms (client timeout 1850 ms). Trace:
I20260709 12:26:14.620162 11407 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:14.626503 11442 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.267s	user 0.001s	sys 0.000s
I20260709 12:26:14.581867 11404 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.630282 11404 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.632710 11404 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:14.716357  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:14.620254  1803 rpcz_store.cc:269] 0709 12:26:13.153539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:13.161077 (+  7538us) service_pool.cc:224] Handling call
0709 12:26:14.619436 (+1458359us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b54001e3f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:14.619572 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1899662080}
W20260709 12:26:14.665130  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 92128.3us
I20260709 12:26:14.557982 11387 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.556994 11358 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:14.700547 11202 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:14.701633  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 241065us
I20260709 12:26:14.579692 11434 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:14.592690 11456 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.603463 11171 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:14.708613  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 279867us
I20260709 12:26:14.737123  1881 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:14.701786 11417 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:14.562647 11408 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:14.575358 11401 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:14.714028  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 58659.8us
I20260709 12:26:14.744813  1881 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:26:14.715273  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 81198.7us
I20260709 12:26:14.550902 11359 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.718050  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:14.719310  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:14.751009  1800 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:14.727481  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:14.727958  1804 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:14.729060  1801 raft_consensus.cc:1240] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 30. Current term is 31. Ops: [29.19-30.20]
I20260709 12:26:14.719156  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:14.753136  1798 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:14.753716  1805 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 31: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:14.733795  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.737077  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:14.597769 11419 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:14.741380  1799 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 20. (index mismatch)
I20260709 12:26:14.743093  1732 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:26:14.747480  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:14.748759  1806 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:14.748916  1723 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:26:14.761319  1807 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:14.764503  1874 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:14.766713  1771 leader_election.cc:400] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28.
I20260709 12:26:14.767341  1771 leader_election.cc:403] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:14.720722 11358 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:14.769248  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.778059  1807 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:14.735868 11368 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:14.786582  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:14.787405  1801 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:14.787708  1799 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:14.744216 11417 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:14.793627  1800 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:26:14.794922  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:14.795728  1877 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:14.793540  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.729584 11385 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.060s
W20260709 12:26:14.827308  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:14.828085  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.803836  1799 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948"
dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
mode: GRACEFUL
new_leader_uuid: "a20e99e125b84513a44e88860667ed61"
 from {username='slave'} at 127.0.0.1:40096
W20260709 12:26:14.833045  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63915us
W20260709 12:26:14.833627 11460 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:26:14.834022  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:14.789513 11469 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:26:14.835408  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.769454 11407 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:14.777388 11288 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:26:14.840059  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.777769 11408 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:14.779564 11358 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:14.857602  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.857700  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.777923 11395 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:14.784427 11359 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:14.887467  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.759114 11434 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:14.882407  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:14.874651  1877 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
W20260709 12:26:14.894933  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 59640.5us
I20260709 12:26:14.895897  1696 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:14.886337  1880 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:14.888435  1872 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:26:14.793304 11417 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:14.893975  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:14.898309  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:14.899178  1874 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:14.766631 11401 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.784696 11283 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:14.903519  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:14.905303  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.904049  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:14.908600  1800 raft_consensus.cc:2410] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:26:14.903589  1801 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:26:14.909345  1801 pending_rounds.cc:85] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 16
I20260709 12:26:14.909587  1801 pending_rounds.cc:107] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:26:14.909886  1801 raft_consensus.cc:2889] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:14.859050 11449 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:14.883157 11118 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:14.928462  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:14.934906  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:14.937201  1799 raft_consensus.cc:606] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Received request to transfer leadership to TS a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.949646  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.950911  1799 raft_consensus.cc:614] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:26:14.951212  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:14.952667  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.840857 11451 consensus_queue.cc:1059] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.090s
I20260709 12:26:14.790017 11368 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:14.955735  1732 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:26:14.960858  1732 pending_rounds.cc:85] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 14
I20260709 12:26:14.961123  1732 pending_rounds.cc:107] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:26:14.961383  1732 raft_consensus.cc:2889] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:14.911118 11395 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:14.964728 11395 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:14.966107 11242 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.629s	user 0.007s	sys 0.002s
I20260709 12:26:14.883358 11419 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:26:14.920636 11408 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:14.960299  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:14.974200  1801 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:26:14.934497 11440 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:14.969053  1806 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:14.962723  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.887624 11393 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.063s
I20260709 12:26:14.963304 11118 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:14.981060  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.981236  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:14.987726  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:14.982744  1798 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:14.989271  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:14.988360  1875 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:14.984588  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:14.986712  1723 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:26:14.990015  1881 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:26:14.991447  1772 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:14.991573  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:14.987350  1872 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:14.993650  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:14.994374  1876 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:14.967938 11283 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:14.918557 11404 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:26:14.932155 11449 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:15.008567 11449 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:15.008740  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.010114  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:14.999475  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:15.010761  1802 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:15.013447  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:15.013962  1802 raft_consensus.cc:3060] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:26:15.016316 11456 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:15.016584 11320 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.659s	user 0.008s	sys 0.004s
I20260709 12:26:15.017118 11320 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:15.017891 11473 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:15.018522 11473 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.017922  2418 raft_consensus.cc:1240] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:15.012050  1801 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:15.024184 11401 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.025332  1802 raft_consensus.cc:2468] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:15.026379 11434 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.030040 11116 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25.
I20260709 12:26:15.030449 11206 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.031769 11117 raft_consensus.cc:3060] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:14.990872 11118 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:14.990568 11419 consensus_queue.cc:1243] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:26:14.985781 11271 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.005218 11385 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:15.006248 11080 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.007102 11408 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.007750 11368 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.002339 11425 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:15.026537  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:14.997483 11293 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:15.040956  1803 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:15.044188  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.044839  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:15.035892  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:14.966634 11242 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.048657  1801 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 22.
I20260709 12:26:15.050069  2417 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:26:15.059994  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.060864  2411 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:15.059175  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.061851  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.065619  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.000736 11471 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:15.069366  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.070489  1876 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 32.
I20260709 12:26:15.037370 11473 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:15.072060  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.073875  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.071286 11271 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:15.076761  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.086143  1879 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:15.086508  1802 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
W20260709 12:26:15.086580  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 190807us
I20260709 12:26:15.079694  2413 raft_consensus.cc:2368] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28.
I20260709 12:26:15.035593 11283 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.088443  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.030041 11171 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.095763  1876 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: [20.12-23.13]
I20260709 12:26:15.049026 11080 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.096421  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:15.084664 11456 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.078436 11401 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.108736  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:15.108736  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.096683 11271 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.096740 11393 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.017s
I20260709 12:26:15.115393  1872 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 32: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:15.119521  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.120275  2415 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:26:15.114302 11393 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:15.122773  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:15.123062 11393 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:15.124078 11393 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.125012  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.125379 11393 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:15.125878  1799 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:15.132593 11080 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.134702 11283 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.138180 11393 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:15.139657 11393 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.140071 11393 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:15.137502  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:15.141660 11404 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:15.148648  1806 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:15.149178 11404 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:15.149434 11447 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.149839  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.152741 11447 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:26:15.153592 11477 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:26:15.092391 11434 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.023s
I20260709 12:26:15.156867 11469 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28.
I20260709 12:26:15.110303 11456 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.158767  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.168212  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:15.177912  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:15.188270  1876 raft_consensus.cc:2410] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:26:15.188148  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:15.159214  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.168895  1874 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:15.165307  1727 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:15.192376  1875 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:15.150506  1806 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
W20260709 12:26:15.134742  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.194952  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:15.173256  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.094085 11440 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:15.175180  2417 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:15.179771  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.171948 10969 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.010s
I20260709 12:26:15.159616 11007 raft_consensus.cc:3055] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:15.179217 11398 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.004s
I20260709 12:26:15.164034 11171 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.165858 11271 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.185005 10919 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:15.209263 10919 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.212163 10919 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:15.212733 10610 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:15.179253 11116 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.214347 11116 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.217010 11449 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.218525  1728 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:15.185482  1730 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 20. (term mismatch)
I20260709 12:26:15.222302  1730 pending_rounds.cc:85] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 19
I20260709 12:26:15.222649  1730 pending_rounds.cc:107] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 30.20
I20260709 12:26:15.222963  1730 raft_consensus.cc:2889] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:15.228552  1803 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:26:15.230252  1803 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:26:15.172153 11463 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:26:15.116842 11401 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.234675  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.248667  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.257112  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:15.288157  1801 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:15.196986  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:15.269358  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:15.292658  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 95790.6us
I20260709 12:26:15.294305  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:15.190471  1772 leader_election.cc:400] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28.
W20260709 12:26:15.287572  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.314669  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:15.312353  1801 raft_consensus.cc:1275] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:26:15.250895  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.318387  1806 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
W20260709 12:26:15.233467 11482 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 12:26:15.204340 11283 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:15.258627  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:15.321252  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.312297 11473 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:15.341866  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83392us
W20260709 12:26:15.342969  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.233035 11237 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:15.344125  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.343176  1803 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 19. (term mismatch)
I20260709 12:26:15.345124  1803 pending_rounds.cc:85] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 18
I20260709 12:26:15.345364  1803 pending_rounds.cc:107] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 12:26:15.345762  1803 raft_consensus.cc:2889] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:15.217088 11116 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.323058  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.350912 11469 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:15.351634 11469 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:15.352682 11469 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.338600 11171 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:15.355381 11171 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.355924 11171 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:15.295742  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.358273  1805 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:15.361230 11449 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.361323 11293 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:15.362273  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:15.363130  1874 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:26:15.364326  1805 pending_rounds.cc:85] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 18
I20260709 12:26:15.207316 11425 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:26:15.365628 11393 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.367095  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:15.368475 11417 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.368903  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:15.370707  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:15.310855 11431 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:15.230963 11242 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.326228  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.326902  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:15.320387  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.231985 11451 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:15.340287  1729 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:15.321889  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.343484  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:15.383343  1799 raft_consensus.cc:2393] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 32: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:15.383527  1804 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:26:15.384002  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:15.203936 11007 raft_consensus.cc:740] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:15.322150  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:15.385447  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63386.5us
I20260709 12:26:15.247375  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:15.323117  1806 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:26:15.359553  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:15.386955  1881 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:15.328756  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 138429us
I20260709 12:26:15.365427  1874 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:26:15.201761  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:15.372542  1803 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 29. Current term is 30. Ops: []
W20260709 12:26:15.390775  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 189090us
I20260709 12:26:15.391021  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:15.357426  1798 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:15.373768  1724 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:15.373971  1805 pending_rounds.cc:107] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 26.19
I20260709 12:26:15.320710  1801 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
I20260709 12:26:15.324503 11446 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:15.381925  1725 raft_consensus.cc:2368] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28.
I20260709 12:26:15.380710  2412 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:15.392171  1805 raft_consensus.cc:2889] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:15.391999  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:15.331749  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:15.407215  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:15.408324  1880 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
W20260709 12:26:15.412346  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 165118us
I20260709 12:26:15.417474  1806 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:26:15.408596  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:15.424431  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:15.422077  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:15.427382  2415 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:15.429199  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 45304us
I20260709 12:26:15.430363  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:15.346621 11434 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.230666 11358 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.464178  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.466109  1726 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 31: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:15.433390  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:15.461930  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:15.433342  2383 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:15.468515  1731 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:15.469111  1806 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:15.455185  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:15.378572 11417 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.480036  1804 raft_consensus.cc:1240] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:15.423555 11283 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:15.492221  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.505357  1725 raft_consensus.cc:2368] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29.
I20260709 12:26:15.502506  1730 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:15.231491 11440 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:15.463687 11482 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.116s	user 0.000s	sys 0.001s
I20260709 12:26:15.382257 11451 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:15.515625  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.531677  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.532878  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.533464  1730 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:15.516747  1881 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:15.422905 11349 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:15.518438  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:15.465248 11271 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:15.421770 11449 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:15.432763 11432 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:15.432108 11468 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:15.555481  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:15.529523  1800 raft_consensus.cc:2368] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27.
I20260709 12:26:15.535231  1804 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:15.475564 11388 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:15.474968 11398 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:26:15.512027 11283 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.466768 11488 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:15.527881  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.494181 11466 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:15.376878 11237 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.504582 11431 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:15.559648  1730 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 32: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:15.504531 11203 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:15.550467  2415 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:15.574473  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:15.465085 10969 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.081s
I20260709 12:26:15.548774 11394 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:15.534341 11408 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:15.583575  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:15.512032 11463 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:26:15.500021 11434 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.612366  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:15.603982  1804 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:15.546814 11271 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.592427  1806 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:15.619094  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.619319  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.582273 11283 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:15.515820 11372 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.074s
I20260709 12:26:15.625605  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.547770 10919 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.064s
I20260709 12:26:15.626579  1730 raft_consensus.cc:2410] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 32 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:26:15.580076 11432 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:15.713821  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:15.588892 11203 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.709858  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:15.715061  1729 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:26:15.717234  1730 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:26:15.594925 11488 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 26 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:15.721212  1806 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:15.595618 11394 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:15.624312 11451 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.626255 11271 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:15.740187  1731 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:15.646114 11358 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.605850 11466 raft_consensus.cc:697] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:26:15.733925 10610 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.520s	user 0.011s	sys 0.003s
I20260709 12:26:15.749922  1806 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:15.751811  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.775610  1806 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:26:15.777910  1806 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:15.613960 11468 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.139s
I20260709 12:26:15.756834  1731 raft_consensus.cc:1240] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:15.604024 11007 consensus_queue.cc:260] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.620735 11408 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:15.753824 11118 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.720s	user 0.012s	sys 0.000s
W20260709 12:26:15.783030  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 327909us
I20260709 12:26:15.613797 11431 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.719990 11203 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:15.817060 11488 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.868603  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:15.820216  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.824568  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:15.827355 11495 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:26:15.874480  1806 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:26:15.875082  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:15.876008  1874 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:15.876289  1875 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:15.881186 11431 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.883143 11474 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:15.883999 11474 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.887395 11474 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.868468 11359 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.889158  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:15.888991 11359 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.889933  1802 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 31: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:15.891440  1802 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:26:15.832224 11434 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.903455  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:15.836980  1798 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:26:15.838295  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 4130) took 1037 ms. Trace:
I20260709 12:26:15.906134  1875 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 31: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:15.843593 11432 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:15.848206 11449 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.847752 11466 consensus_queue.cc:237] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.849198 11417 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.944919 11417 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.850031 11393 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.779428 11473 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.772795 11451 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:15.948314 11417 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.860982 11408 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:15.952818 11419 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:26:15.953559 11419 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:26:15.861048  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.955865 11237 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:15.956355  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:15.865322  1727 raft_consensus.cc:1275] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 20. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:26:15.866529  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:15.972118  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 105727us
I20260709 12:26:15.974129  1770 leader_election.cc:400] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28.
I20260709 12:26:15.867240  1877 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:26:15.867461  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:15.870980  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:15.871855  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:15.893985 11456 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:15.897152 11394 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.898674  1807 raft_consensus.cc:1240] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 24. Current term is 25. Ops: []
W20260709 12:26:15.900905  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 426010us
I20260709 12:26:15.902491  1801 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:26:15.999758  1875 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:26:16.001436  1874 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:26:15.904878  1879 rpcz_store.cc:276] 0709 12:26:14.800551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.927669 (+127118us) service_pool.cc:224] Handling call
0709 12:26:15.837948 (+910279us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b54002062d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:15.838179 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":240587008}
I20260709 12:26:15.905984 11431 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:15.831880 11007 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:15.924588  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:15.910671  1696 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:15.937503  1806 raft_consensus.cc:1240] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 26. Current term is 27. Ops: [26.20-26.20]
I20260709 12:26:15.939175  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:15.976879  1770 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:15.985064  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 117671us
I20260709 12:26:15.910368  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3228) took 1039 ms. Trace:
I20260709 12:26:16.048338  1803 rpcz_store.cc:276] 0709 12:26:14.872738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.363640 (+490902us) service_pool.cc:224] Handling call
0709 12:26:15.363966 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:15.390094 (+ 26128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:15.910012 (+519918us) spinlock_profiling.cc:243] Waited 8.34 ms on lock 0x7b54002c0454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:15.910282 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19178368}
I20260709 12:26:15.909916 11390 consensus_queue.cc:1059] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.126s
I20260709 12:26:16.050734  1877 raft_consensus.cc:1240] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:16.050621  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:16.034760  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:16.052911  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.002370  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3242) took 1044 ms. Trace:
I20260709 12:26:16.014196  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3229) took 1141 ms. Trace:
W20260709 12:26:16.055655  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 145027us
I20260709 12:26:16.052075  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:15.875015 11118 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:16.064320  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 48808.7us
I20260709 12:26:16.060098  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:16.066717  2411 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:16.060981  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:16.061905  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:16.053399  1804 rpcz_store.cc:276] 0709 12:26:14.957961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.721764 (+763803us) service_pool.cc:224] Handling call
0709 12:26:15.755005 (+ 33241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:15.864098 (+109093us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:15.864147 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:15.864208 (+    61us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:15.864258 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.002039 (+137781us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540004d8d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:16.002106 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.002293 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":250223488}
I20260709 12:26:15.995741 11358 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:15.987039 11408 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:16.065820  1729 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:26:16.069850  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:16.001952 11425 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:16.071028  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:16.029253  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:16.127276  1873 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.011s	sys 0.021s
W20260709 12:26:16.128104  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 98982.4us
W20260709 12:26:16.128876  1873 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.012s	sys 0.022s
I20260709 12:26:16.077127  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:16.078750  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.069681 11358 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:15.984289 11419 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:15.935953 10919 consensus_queue.cc:1048] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.117s
W20260709 12:26:16.079197  1805 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.502s	user 0.005s	sys 0.020s
I20260709 12:26:16.092444 11359 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:16.049896 11393 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.151664 11393 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.152724 11283 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.154291 11393 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.082549  1877 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c"
dest_uuid: "a20e99e125b84513a44e88860667ed61"
mode: GRACEFUL
new_leader_uuid: "075704438992440697732181e49ccd09"
 from {username='slave'} at 127.0.0.1:37240
W20260709 12:26:16.137806 11496 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:26:16.155153  1877 raft_consensus.cc:606] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Received request to transfer leadership to TS 075704438992440697732181e49ccd09
I20260709 12:26:16.155247  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3424) took 1393 ms. Trace:
I20260709 12:26:16.156003  1724 rpcz_store.cc:276] 0709 12:26:14.762152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.762398 (+   246us) service_pool.cc:224] Handling call
0709 12:26:14.762773 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:15.392128 (+629355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.155001 (+762873us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b5400414954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:16.155186 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1404460928}
I20260709 12:26:16.159885 11283 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:16.083716  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.055303  1800 rpcz_store.cc:276] 0709 12:26:14.873321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:14.913554 (+ 40233us) service_pool.cc:224] Handling call
0709 12:26:16.013867 (+1100313us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b54001d4f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:16.014104 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1288726784}
I20260709 12:26:16.137108  1730 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:16.058748 11080 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.162746  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:16.137286  2383 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.003s	sys 0.031s
I20260709 12:26:16.163273 11080 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:16.163993  2383 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.004s	sys 0.031s
W20260709 12:26:16.164391  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 648591us
I20260709 12:26:16.165424 11080 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.133963  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3277) took 1033 ms. Trace:
I20260709 12:26:15.997697 11237 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.102s
I20260709 12:26:16.165992 11080 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:15.983918 11449 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.169307 11449 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.070005 11474 raft_consensus.cc:3055] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:16.170042 11474 raft_consensus.cc:740] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:16.164418  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:16.171300 11474 consensus_queue.cc:260] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.171900 11401 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.172497 11018 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:16.172752 11474 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:16.173218 11018 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:16.071074 11387 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:16.173714 11018 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:16.142234 11171 consensus_meta.cc:287] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.782s	user 0.012s	sys 0.003s
I20260709 12:26:16.174453 11171 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.175716  1881 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:16.083784 11349 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.176599 11349 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.177034 11171 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.174075 11387 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:16.178015 11387 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:16.179495  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:16.136045 10610 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:26:16.080410 11390 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.136152 11358 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:16.181478 10610 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:16.181371 11390 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.148097 11472 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:16.150537 11293 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:16.161177 11395 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:16.132002 11425 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:16.104498  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:16.183029  1873 rpcz_store.cc:276] 0709 12:26:15.102212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.181468 (+ 79256us) service_pool.cc:224] Handling call
0709 12:26:15.202529 (+ 21061us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:15.202989 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:15.203023 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:15.203604 (+   581us) log.cc:844] Serialized 26 byte log entry
0709 12:26:15.580091 (+376487us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:16.131528 (+551437us) log.cc:844] Serialized 14 byte log entry
0709 12:26:16.133545 (+  2017us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.133645 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.133699 (+    54us) raft_consensus.cc:1685] finished
0709 12:26:16.133716 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.133902 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17350,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":17066,"thread_start_us":743650,"threads_started":2,"wal-append.queue_time_us":555109}
I20260709 12:26:16.051389 11431 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:16.078181 11398 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:16.003836 11203 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:16.203918  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:16.194147 11472 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:16.221676  2418 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.000s	sys 0.028s
W20260709 12:26:16.225736  2418 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.520s	user 0.000s	sys 0.028s
I20260709 12:26:16.231665  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:16.232849  1807 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:16.235283  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:16.197005 11349 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.240661  1802 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:16.199115 11425 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:16.245894  1798 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:26:16.192096 11372 consensus_queue.cc:1059] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:26:16.203336 11393 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:16.196192 11359 consensus_queue.cc:1059] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:26:16.196007 11503 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:26:16.238117 11117 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:16.201320 10998 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.048s
I20260709 12:26:16.268507  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3232) took 1335 ms. Trace:
I20260709 12:26:16.269013  1799 rpcz_store.cc:276] 0709 12:26:14.932722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.383935 (+451213us) service_pool.cc:224] Handling call
0709 12:26:15.463958 (+ 80023us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:15.464370 (+   412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:15.464398 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:15.464917 (+   519us) log.cc:844] Serialized 26 byte log entry
0709 12:26:15.934444 (+469527us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:16.003335 (+ 68891us) log.cc:844] Serialized 14 byte log entry
0709 12:26:16.003568 (+   233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.003642 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.268252 (+264610us) raft_consensus.cc:1685] finished
0709 12:26:16.268274 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.268431 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12687,"prepare.queue_time_us":64540,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":15518,"raft.run_cpu_time_us":331,"raft.run_wall_time_us":64872,"spinlock_wait_cycles":70678400,"thread_start_us":386799,"threads_started":1,"wal-append.queue_time_us":636399}
I20260709 12:26:16.206127 11438 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.051s
I20260709 12:26:16.270925 11117 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:16.273761 11237 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:16.274189 11237 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:16.274719 11237 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:16.222239 11468 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.042s
I20260709 12:26:16.254878 11349 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:16.211457 11080 consensus_queue.cc:1059] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 12:26:16.242430 11505 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.050s
I20260709 12:26:16.255487 11393 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:16.293872 11203 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:16.294525 11390 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:16.296294 11271 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.525s	user 0.001s	sys 0.009s
I20260709 12:26:16.296446 11472 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.296686 11463 consensus_queue.cc:1048] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:26:16.298270 11431 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:16.300240 11018 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:16.301110  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 249123us
I20260709 12:26:16.277987 11359 raft_consensus.cc:3055] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:16.271826 10610 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:26:16.318403  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:16.324910  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:16.305459 11456 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:16.314877 11358 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:16.317848  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 341050us
I20260709 12:26:16.287822 11372 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:26:16.323583 11388 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:16.337927 11390 raft_consensus.cc:3055] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:16.338290 11471 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:16.318809 11349 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:16.348040  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.319931 11501 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:16.350194 11203 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:26:16.340169 11359 raft_consensus.cc:740] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:16.351373 11118 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:16.348624  1770 leader_election.cc:400] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29.
I20260709 12:26:16.323815 11271 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.374150  1770 leader_election.cc:403] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:16.354197 11456 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:16.357227 11390 raft_consensus.cc:740] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:26:16.366465  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 237060us
I20260709 12:26:16.373643 11503 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:16.359695 11440 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.017s	sys 0.031s
I20260709 12:26:16.342871 11398 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:16.426486  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 78055.6us
I20260709 12:26:16.359830 11388 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:16.431499  1770 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:16.477380 11417 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.478155 11417 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.480618 11417 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.484500 11171 raft_consensus.cc:3055] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:16.485018 11171 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:16.485941 11171 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:16.491075 11117 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:16.492369 11117 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:16.495353 11472 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:16.495808 11472 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:16.511795  1726 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:16.515156 11171 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:16.523164 11394 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:16.443323 11503 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.583648 11358 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.585793 11503 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:16.587056 11359 consensus_queue.cc:260] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:16.588637 11359 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:16.600351  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 155931us
I20260709 12:26:16.601423  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:16.601588 11425 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.356184 11393 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 31.14, Last appended by leader: 14, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:16.377628 11507 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:16.603320  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 171902us
I20260709 12:26:16.603799 11456 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 30.14, Last appended by leader: 14, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:16.604772  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:16.606972 11401 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:16.608212 11425 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:16.583706 11506 log.cc:927] Time spent T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.262s	user 0.000s	sys 0.001s
I20260709 12:26:16.609859 11358 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:16.610417 11349 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.603564 11507 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.611757  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3227) took 1741 ms. Trace:
I20260709 12:26:16.613960 11507 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:16.612264  1805 rpcz_store.cc:276] 0709 12:26:14.870522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.361251 (+490729us) service_pool.cc:224] Handling call
0709 12:26:15.361645 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:15.431225 (+ 69580us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:26:15.503695 (+ 72470us) log.cc:844] Serialized 14 byte log entry
0709 12:26:16.318834 (+815139us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:16.321157 (+  2323us) log.cc:844] Serialized 26 byte log entry
0709 12:26:16.321321 (+   164us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:16.321583 (+   262us) log.cc:844] Serialized 14 byte log entry
0709 12:26:16.327681 (+  6098us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.327793 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.497301 (+169508us) spinlock_profiling.cc:243] Waited 6.36 ms on lock 0x7b18004d4220. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb557 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:26:16.498289 (+   988us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b18004d4220. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:26:16.611470 (+113181us) raft_consensus.cc:1685] finished
0709 12:26:16.611502 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.611687 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":294042,"prepare.queue_time_us":82821,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":95,"raft.queue_time_us":183943,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":14748,"spinlock_wait_cycles":32627712,"thread_start_us":565230,"threads_started":1,"wal-append.queue_time_us":738496}
I20260709 12:26:16.615042 11408 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:16.616166 11446 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:16.615856 11408 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.615200  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:16.610533 11358 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:16.620432 11018 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.621923 11358 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:16.406502 11237 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.624786  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:16.625442 11431 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:26:16.624828 11237 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:16.627628 11388 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:16.628783 11408 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:16.629667 11408 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:16.603514 11390 consensus_queue.cc:260] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:16.626317  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:16.632018 11390 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:16.633015  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:16.636658  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:16.448068 11440 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.020s	sys 0.031s
I20260709 12:26:16.644266  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:16.646821  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:16.649016  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:16.655227  1724 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:16.659284  1727 raft_consensus.cc:2410] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 30 because replica has last-logged OpId of term: 28 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 20.
I20260709 12:26:16.659476  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:16.657987  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:16.660996  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 422222us
I20260709 12:26:16.651191  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:16.445178 11501 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29.
I20260709 12:26:16.681675  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:16.682807  1697 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.015s	sys 0.008s
W20260709 12:26:16.683333  1697 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.015s	sys 0.008s
W20260709 12:26:16.683651  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 617903us
I20260709 12:26:16.685384  1732 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:16.687568  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:16.637890 11408 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:16.699786  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:16.637679 11358 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:16.707557  1729 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
W20260709 12:26:16.710114  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 469415us
I20260709 12:26:16.749967  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:16.741955  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:16.719511  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:16.715515 11466 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.014s	sys 0.010s
I20260709 12:26:16.771092  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:16.699301 11516 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:16.764547 11387 consensus_meta.cc:287] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.556s	user 0.005s	sys 0.007s
W20260709 12:26:16.766830 11451 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.819s	user 0.013s	sys 0.000s
I20260709 12:26:16.771396  1732 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:16.709787 11503 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:16.777606 11446 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20260709 12:26:16.781561 11007 consensus_meta.cc:287] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.722s	user 0.008s	sys 0.004s
I20260709 12:26:16.787395 11080 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:16.787227 11387 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:16.788595  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.788985 11509 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:16.608983 11425 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.187s
W20260709 12:26:16.797964 11474 consensus_meta.cc:287] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.625s	user 0.012s	sys 0.002s
W20260709 12:26:16.771533 11466 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.674s	user 0.016s	sys 0.011s
I20260709 12:26:16.773592 11358 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:16.884065  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:16.812662  2418 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.021s	sys 0.034s
W20260709 12:26:16.887465  2418 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.021s	sys 0.034s
I20260709 12:26:16.888710  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1671) took 1621 ms. Trace:
W20260709 12:26:16.890936  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.890878  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:16.819311  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.009s	sys 0.017s
I20260709 12:26:16.892118  2411 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
W20260709 12:26:16.892620  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.836s	user 0.009s	sys 0.017s
W20260709 12:26:16.833276  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:16.893033  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 836422us
I20260709 12:26:16.845779  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:16.846407  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:16.847470  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:16.852109  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:16.895756  1805 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
I20260709 12:26:16.856199  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:16.897563  1806 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:16.811601 10969 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:16.914644  1723 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 21.
I20260709 12:26:16.915210  1799 raft_consensus.cc:1240] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:26:16.904184  1727 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
W20260709 12:26:16.908185  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.914882  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:16.889223  2418 rpcz_store.cc:276] 0709 12:26:15.267698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:15.290358 (+ 22660us) service_pool.cc:224] Handling call
0709 12:26:15.290772 (+   414us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:15.291314 (+   542us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:15.332013 (+ 40699us) log.cc:844] Serialized 14 byte log entry
0709 12:26:16.227268 (+895255us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:16.242418 (+ 15150us) log.cc:844] Serialized 52 byte log entry
0709 12:26:16.242653 (+   235us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:16.888051 (+645398us) log.cc:844] Serialized 14 byte log entry
0709 12:26:16.888312 (+   261us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:16.888407 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:16.888462 (+    55us) raft_consensus.cc:1685] finished
0709 12:26:16.888475 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:16.888650 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":46209,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":44533,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":18720,"thread_start_us":1464159,"threads_started":3,"wal-append.queue_time_us":445846}
W20260709 12:26:16.916116  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.916914  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:16.917644  1802 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:16.918138  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:16.921301  1696 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.66:40361 timed out after 1.557s (SENT) [suppressed 5 similar messages]
I20260709 12:26:16.811645 11451 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:16.925843  1846 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.793s	user 0.031s	sys 0.102s
I20260709 12:26:16.926537  1729 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:26:16.931696  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:16.933132  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:16.944197  2411 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:16.932503  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:16.945535  1846 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.032s	sys 0.102s
W20260709 12:26:16.953394  1803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.016s	sys 0.028s
I20260709 12:26:16.813256 11503 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:16.940114  1804 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.001s	sys 0.029s
I20260709 12:26:16.915884 11447 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:16.982838  1804 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.002s	sys 0.029s
W20260709 12:26:16.896759 11483 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.009s	sys 0.020s
I20260709 12:26:16.959759  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:16.989488  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:16.961627  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:16.962803  1771 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.008s	sys 0.020s
I20260709 12:26:16.990487  2410 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:16.935189  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.011s	sys 0.016s
W20260709 12:26:16.991346  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.011s	sys 0.016s
W20260709 12:26:16.975386  1803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.016s	sys 0.028s
I20260709 12:26:16.849694 11438 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.094s
I20260709 12:26:16.914026 10969 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:16.902969 11320 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:16.715296 11472 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:16.989333 11483 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.010s	sys 0.020s
W20260709 12:26:16.994149 11497 log.cc:927] Time spent T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:26:16.954452  1801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.011s	sys 0.014s
W20260709 12:26:16.990725  1771 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.009s	sys 0.020s
W20260709 12:26:17.021987  1801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.012s	sys 0.014s
W20260709 12:26:17.022091  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 677642us
W20260709 12:26:16.963104  1800 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.597s	user 0.003s	sys 0.026s
I20260709 12:26:16.993273  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:16.938413 11358 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:16.989868  2418 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:16.999794  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:17.004199  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:17.006956  1806 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:17.024999  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:17.008914  1807 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:16.888028 11080 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.018936  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:16.937620 11417 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:26:17.023442  1800 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.003s	sys 0.026s
I20260709 12:26:17.024793  2416 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 30.
I20260709 12:26:17.022840  1771 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:17.029011  1802 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:17.030364  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:17.030215  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:17.010699 10969 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:17.021116 11448 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.004s	sys 0.016s
I20260709 12:26:17.009961 11501 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.017650 11408 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:16.994462 11447 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.038108  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.020710 11349 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:17.027909 11401 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:17.025243 11471 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:17.028213 11446 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:17.047465  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:17.077589  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:16.962177 11451 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.019984 11320 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:17.051411  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 366699us
W20260709 12:26:17.042351 11171 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.522s	user 0.010s	sys 0.000s
I20260709 12:26:17.064990  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:17.028051 11515 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:17.042054 11448 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.004s	sys 0.016s
I20260709 12:26:17.039448 11469 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:17.103746 11469 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:17.104226 11469 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:17.104564  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.045704 11466 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:17.028152 11358 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.106995 11466 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:17.108917 11358 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:17.109679 11358 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:17.069643 11472 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:17.112900 11358 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:17.108381 11466 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.119606 11466 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:17.120379 11358 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:17.091972  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 158919us
I20260709 12:26:17.124297 11358 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.129645  2417 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:17.053669 11349 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:17.134092  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
W20260709 12:26:17.133244  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 85684.8us
I20260709 12:26:17.130389  1726 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:17.069051 11529 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:17.077224 11446 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:17.141477  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47814.7us
I20260709 12:26:17.139544  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:17.050308 11501 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.087226 11528 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:17.053718 11408 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.157189  2410 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:17.102721 11515 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.136191 11432 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.143489 11440 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:17.158337 11171 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:17.173990  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:17.161963 11544 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:17.176712  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 255396us
I20260709 12:26:17.174041  1798 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:17.175545 11521 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.177963  2417 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:17.162148 11529 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:17.162382 11387 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:17.185191  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45723us
W20260709 12:26:17.187408  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.162602 11515 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:17.191017  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:17.191954  1878 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:17.147251 11417 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.026s
I20260709 12:26:17.193822  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:17.193965  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:17.194743  1874 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:17.194489  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.169919 11440 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.198825  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:17.199676  1875 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:17.200734  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:17.201449  1879 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:26:17.203226  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:17.203771  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:17.204233  1873 raft_consensus.cc:2410] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:26:17.204666  1880 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:17.205642  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:17.207772 11501 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.196257 11483 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:17.208770  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:17.185191 11521 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.185182 11543 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:17.241401 11543 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:17.241632 11540 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:26:17.241935 11543 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:17.180850 11541 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:17.243436 11541 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.212577  1874 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:17.213228 11473 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.245745 11521 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:17.246233 11541 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:17.213735  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:17.246716 11534 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:26:17.248188 11320 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.169730 11432 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.246006 11473 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.251220 11446 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.192665 11529 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.214676  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:17.216176  1880 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:26:17.219383 11408 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.194586  1877 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:17.235335  1872 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:17.237712 11080 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:17.259821 11419 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:17.272536  1732 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
W20260709 12:26:17.262458 11537 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:26:17.253122 11473 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:17.283883 11393 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:17.253334 11171 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:17.289872 11171 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:17.290441 11171 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:17.189832 11544 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.253162 11432 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.297089 11544 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:17.162637 11528 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.300200 11528 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.178220 11349 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.196885 11440 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:17.302901 11271 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:17.263073 11483 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.307250  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:17.307474  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:17.308784 11510 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:26:17.309286 11510 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:26:17.309434 11531 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
I20260709 12:26:17.308992  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:17.312011  2413 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:17.311959  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1419) took 1083 ms. Trace:
I20260709 12:26:17.312741  1803 rpcz_store.cc:276] 0709 12:26:16.228542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.228718 (+   176us) service_pool.cc:224] Handling call
0709 12:26:16.324036 (+ 95318us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:16.324453 (+   417us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.324486 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:16.346166 (+ 21680us) log.cc:844] Serialized 52 byte log entry
0709 12:26:16.995752 (+649586us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:17.020659 (+ 24907us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.020895 (+   236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.020976 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:17.212932 (+191956us) spinlock_profiling.cc:243] Waited 4.56 ms on lock 0x7b48001c0158. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b0d01aa1 00007f16b0cf8b37 00007f16b0cf89ca 00007f16b0df026f 00007f16b0deff37 00007f16b0e0455c 00007f16b0e04508 00007f16b0e04488 00007f16b0e04443 00007f16b0e036ec 00007f16afd2da96
0709 12:26:17.311592 (+ 98660us) raft_consensus.cc:1685] finished
0709 12:26:17.311723 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:17.311891 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":599,"prepare.queue_time_us":29125,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":884,"raft.run_cpu_time_us":4740,"raft.run_wall_time_us":885655,"spinlock_wait_cycles":21210112,"thread_start_us":629865,"threads_started":1,"wal-append.queue_time_us":856531}
I20260709 12:26:17.325137  1630 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 17 to 20, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 20 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:17.333446 11510 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.211901 11431 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
W20260709 12:26:17.276458 11390 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.644s	user 0.010s	sys 0.003s
I20260709 12:26:17.385321  1798 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:17.385671  1872 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
W20260709 12:26:17.431850  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 217235us
W20260709 12:26:17.432301  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.342577 11510 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:17.329337  1630 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 17 to 21, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 21 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:17.439427  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 245097us
I20260709 12:26:17.440428  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.432955  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.309274 11483 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.436014  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:17.444489  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.430676 11468 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:17.447357  1623 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 256) took 42388 ms (client timeout 15000 ms). Trace:
I20260709 12:26:17.447667 11448 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:17.438844 11372 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.448542 11417 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:26:17.451149  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.456171  2417 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
W20260709 12:26:17.456779  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.456918  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.457267 11408 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:17.457661 11432 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:17.460414  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.520882  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:17.519651  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:17.521302  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60984.5us
I20260709 12:26:17.461086 11501 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:17.461324 11527 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.376s	user 0.000s	sys 0.000s
I20260709 12:26:17.462230 11548 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:17.462580 11387 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:26:17.465072 11518 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.401s	user 0.000s	sys 0.001s
I20260709 12:26:17.465423  1630 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 16 to 22. New cstate: current_term: 22 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } }
W20260709 12:26:17.462206  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.466773  1878 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:17.434029 11451 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:17.487737  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.553432  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:17.497135  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.478097 11419 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.494225  1730 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:26:17.522424  1771 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:17.528838  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.557931  1771 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:26:17.558125  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.558434  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:17.520380 11408 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.487064 11271 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:17.391213 11543 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.487591 11293 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.563442  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:17.489576 11417 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:17.519402 11519 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:17.566969  1844 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:17.492496 11448 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.532292 11431 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.555750  1697 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:17.579947  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:17.530910 11472 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:17.529654 11283 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:26:17.546895 11456 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.908s	user 0.003s	sys 0.027s
I20260709 12:26:17.591398  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:17.592440  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:17.566599  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:26:17.593542  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 430182us
I20260709 12:26:17.576608  1877 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:17.548363 11544 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.595084  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:17.595808  1729 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:17.581354  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:17.569113  1630 catalog_manager.cc:5697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 10 to 19. New cstate: current_term: 19 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:17.577122  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:17.579720 11293 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.607627  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:17.576860 11501 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:17.583622  1880 raft_consensus.cc:1240] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:17.584283 11448 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.584821 11551 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:17.569360 11541 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:17.570389  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:17.592883  1873 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:17.594153  1771 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:17.522239 11468 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.569811 11408 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:17.520808 11469 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.568897 11419 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.610239 11271 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:17.568907 11451 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:17.569016 11528 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:17.612051  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.576577 11417 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:17.688386  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:17.492215 11372 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.548580 11548 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:17.657634  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45332us
W20260709 12:26:17.663533  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 206681us
I20260709 12:26:17.606958 11393 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:17.674314  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 80246.4us
W20260709 12:26:17.481271  1623 rpcz_store.cc:269] 0709 12:25:35.058331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:40.298258 (+5239927us) service_pool.cc:224] Handling call
0709 12:26:17.447037 (+37148779us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:09.494185 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:09.494995 (+   810us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:09.495064 (+    69us) write_op.cc:435] Acquired schema lock
0709 12:26:09.495088 (+    24us) tablet.cc:662] Decoding operations
0709 12:26:09.870826 (+375738us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:09.870943 (+   117us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:09.870972 (+    29us) tablet.cc:685] Acquiring locks for 28 operations
0709 12:26:10.050640 (+179668us) tablet.cc:701] Row locks acquired
0709 12:26:10.050700 (+    60us) write_op.cc:260] PREPARE: finished
0709 12:26:10.126448 (+ 75748us) write_op.cc:270] Start()
0709 12:26:10.138661 (+ 12213us) write_op.cc:276] Timestamp: P: 1783599970115186 usec, L: 0
0709 12:26:10.138715 (+    54us) op_driver.cc:348] REPLICATION: starting
0709 12:26:10.200795 (+ 62080us) log.cc:844] Serialized 10947 byte log entry
0709 12:26:10.438689 (+237894us) op_driver.cc:464] REPLICATION: finished
0709 12:26:10.557275 (+118586us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd3e28 00007f16af80e581 00007f16af80dd58 00007f16af8129f1 00007f16af8129a9 00007f16af812939 00007f16af812901 00007f16af811b2d 0000000000388634 0000000000388479
0709 12:26:10.803103 (+245828us) write_op.cc:301] APPLY: starting
0709 12:26:10.803295 (+   192us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:11.247367 (+444072us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:11.247422 (+    55us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:12.330075 (+1082653us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:12.330593 (+   518us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=56,key_file_lookups=56,delta_file_lookups=0,mrs_lookups=0
0709 12:26:12.330747 (+   154us) write_op.cc:312] APPLY: finished
0709 12:26:12.392157 (+ 61410us) log.cc:844] Serialized 244 byte log entry
0709 12:26:12.686424 (+294267us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:12.687191 (+   767us) write_op.cc:454] Released schema lock
0709 12:26:12.687317 (+   126us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:12.801215 (+113898us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:12.806731 (+  5516us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2786857,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":350798,"apply.run_cpu_time_us":84818,"apply.run_wall_time_us":1948829,"cfile_cache_hit":114,"cfile_cache_hit_bytes":121279,"num_ops":28,"prepare.queue_time_us":564809,"prepare.run_cpu_time_us":90699,"prepare.run_wall_time_us":1063408,"raft.queue_time_us":17942,"raft.run_cpu_time_us":48273,"raft.run_wall_time_us":287602,"replication_time_us":299650,"spinlock_wait_cycles":7121664,"thread_start_us":1013175,"threads_started":5,"wal-append.queue_time_us":573963,"wal-append.run_cpu_time_us":16061,"wal-append.run_wall_time_us":2028816}]]}
I20260709 12:26:17.606083 11431 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:17.609225  1770 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.021s	sys 0.044s
I20260709 12:26:17.682233 11521 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:17.592133 11456 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.957s	user 0.003s	sys 0.027s
I20260709 12:26:17.669523 11551 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:17.722203  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:26:17.689873 11562 log.cc:927] Time spent T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
W20260709 12:26:17.721909  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114366us
I20260709 12:26:17.575426 11473 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.081s
I20260709 12:26:17.674005 11501 raft_consensus.cc:697] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:26:17.674496  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 79919.2us
W20260709 12:26:17.733486  1770 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.022s	sys 0.044s
I20260709 12:26:17.695641 11555 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:17.615670 11544 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.763610  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:17.667514 11509 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.060s
W20260709 12:26:17.697023 11466 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.577s	user 0.005s	sys 0.008s
W20260709 12:26:17.766601  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44761.2us
I20260709 12:26:17.772186 11393 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:17.765772  1630 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:17.691354 11451 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:17.773124  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 708144us
I20260709 12:26:17.872563  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:17.874552  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:17.588600 11519 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.815428  1732 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 32.
I20260709 12:26:17.817215  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:17.728806 11387 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:17.732229 11372 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.853502  1872 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:17.769706 11551 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:17.924199  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:17.925786  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:17.926608  2418 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:17.727023 11271 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:17.701942 11554 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:17.776692 11448 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:17.691566 11432 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:17.851608 11440 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.549s	user 0.013s	sys 0.000s
W20260709 12:26:17.850574 11515 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.641s	user 0.011s	sys 0.000s
I20260709 12:26:17.867668  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.737798 11080 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.119s
I20260709 12:26:17.727803 11171 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.887667  2419 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:17.885746  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:17.929543 11456 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.884763  1630 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 17 to 19, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 19 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:17.869195 10969 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.823s	user 0.017s	sys 0.000s
I20260709 12:26:17.903972 11543 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:17.904208 11544 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:17.882268 11387 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.887374 11468 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.878095 11118 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:17.908021 11548 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.909657 11393 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.879377 11283 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:17.911055 11501 consensus_queue.cc:237] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.912017 11293 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:17.912359 11557 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.237s	user 0.000s	sys 0.000s
W20260709 12:26:17.914037 11561 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:26:17.880854 11519 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:17.877220 11558 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:17.770088 11521 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.912019 11419 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:17.925091  2413 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
W20260709 12:26:17.931190  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114086us
I20260709 12:26:17.862138 11559 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:17.864163 11472 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:18.024515 11569 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:17.875312 11451 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:17.944099 11469 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:17.954711  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 87139.3us
I20260709 12:26:17.956324  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:17.957161  2417 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:26:17.878118  1770 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:18.068001  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:26:18.068485  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 190438us
I20260709 12:26:17.964898 11515 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.069270  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.933552 11448 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:18.070696 11515 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.071058 11448 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:18.071609 11469 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:18.072088 11469 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:18.072539 11469 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:18.073098  2418 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:18.075013 11283 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:26:18.075766 11283 consensus_queue.cc:1243] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 23.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:26:17.971808  1802 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:26:17.976114  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:17.963117 10969 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.981664 11503 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:17.989315  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.082264  1799 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:17.993252  1874 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:17.986203  1630 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:18.013978  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:17.977178  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:17.939411 11554 raft_consensus.cc:697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:17.937309 11541 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:18.019850  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:18.024358  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.027067  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.101222  2416 raft_consensus.cc:1240] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:18.101646  1880 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
W20260709 12:26:18.101438  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.028837  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:18.029457  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:17.958474 11572 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:17.873632 11555 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.953127 11440 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:17.979156 11118 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.020644 11577 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:17.946015 11432 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.028148 11431 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:17.935720  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 73476.5us
I20260709 12:26:18.077310  1806 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:18.077908  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:17.968255 11456 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:17.992667 11171 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.136025  1880 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:18.084100 11515 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:18.096712 11358 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.064164 11559 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:18.089171 11425 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.065s	user 0.021s	sys 0.022s
I20260709 12:26:18.103209  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.142446  1727 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:26:18.104225  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.143224  1732 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:18.104809  1876 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:18.105144  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.106714  1726 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:18.108152  1724 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:26:18.084854 11519 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:18.079979 11569 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.115274  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.034224 11271 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.092090 11503 raft_consensus.cc:2764] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:26:18.098958 11543 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.120177  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.106809 11526 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.081513 11516 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
W20260709 12:26:18.135926  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58068.1us
I20260709 12:26:18.152719  1877 raft_consensus.cc:2410] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 31 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
W20260709 12:26:18.089531  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 112472us
I20260709 12:26:18.153822  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:18.153384  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:18.145697  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86531.4us
I20260709 12:26:18.117130 11577 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:18.144904 11425 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.125s	user 0.021s	sys 0.022s
I20260709 12:26:18.151408   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:26:18.134851 11473 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:18.135269 11283 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.170321  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.164031  2383 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:18.164641  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.143340 11419 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:26:18.162757   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:18.171964  1880 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
W20260709 12:26:18.174033  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.144778 11559 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:18.146858 11358 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:18.149536 11522 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:18.146620 11519 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:18.151438 11575 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.154729 11529 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.186187  1730 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:18.151664 11515 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:18.160634 11560 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.162097 11293 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:18.176370  1877 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:18.173341 11473 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.165848 11468 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:18.166054 11548 consensus_queue.cc:1059] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.044s
I20260709 12:26:18.199519  1880 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:26:18.147513 11472 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.202886 11568 consensus_queue.cc:1059] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
W20260709 12:26:18.215358  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55076.6us
I20260709 12:26:18.160569 11543 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.210826 11575 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.180032 11283 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.215689 11529 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.247836 11568 raft_consensus.cc:3055] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:18.248389 11568 raft_consensus.cc:740] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:18.249677 11529 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.249361 11568 consensus_queue.cc:260] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.193487 11358 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.166165 11526 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.192691 10919 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:18.226977 11468 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.238253  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.224478 11515 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.170332 11510 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.043s
I20260709 12:26:18.254091  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.201997 11519 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:18.250950 11586 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.062s
I20260709 12:26:18.259377  1877 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:18.257826 11293 raft_consensus.cc:3055] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:18.251370 11546 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.161130 11554 consensus_queue.cc:237] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.267712 11432 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.269958 11510 raft_consensus.cc:3055] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:26:18.270573 11579 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.149s	user 0.001s	sys 0.001s
W20260709 12:26:18.254737 11576 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.188s	user 0.002s	sys 0.000s
I20260709 12:26:18.354841   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:26:18.280819  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:18.355768  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.202236 11522 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.260842 11468 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:18.244415 11448 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:18.352151 11408 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.674s	user 0.013s	sys 0.000s
I20260709 12:26:18.353093 11283 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.255949 11521 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.353761 11555 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.276482 10919 raft_consensus.cc:697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:18.294915 11293 raft_consensus.cc:740] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:18.221961 11560 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.271833 10969 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:18.275825 11372 consensus_queue.cc:1059] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.121s
I20260709 12:26:18.333832 11529 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:18.256132 11473 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.341984 11546 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.405529 11408 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.342533 11568 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:18.261917 11548 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.124s
I20260709 12:26:18.346877 11510 raft_consensus.cc:740] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:18.392659  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.357148 11456 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.478433  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.478605 10919 consensus_queue.cc:237] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.479496 11007 raft_consensus.cc:3055] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:18.484314 11448 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.485272 11440 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.487138 11515 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.489204 11526 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.489874  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1611) took 1361 ms. Trace:
W20260709 12:26:18.491128  1798 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 3152) took 1485 ms (client timeout 1701 ms). Trace:
I20260709 12:26:18.497782  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.423718 11283 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:18.384691 11398 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.226591 11577 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.587392  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.385398 11117 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.128s
I20260709 12:26:18.588523  1880 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 32.
I20260709 12:26:18.523176 11573 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.515851 11529 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.585994  1726 raft_consensus.cc:2393] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:18.524083 11543 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.575081  2416 rpcz_store.cc:276] 0709 12:26:17.128088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.128777 (+   689us) service_pool.cc:224] Handling call
0709 12:26:17.129277 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:18.106292 (+977015us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.489550 (+383258us) spinlock_profiling.cc:243] Waited 972 ms on lock 0x7b54001c3ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:18.489782 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2234710784}
I20260709 12:26:18.590900  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:18.575258  1798 rpcz_store.cc:269] 0709 12:26:17.006027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.017902 (+ 11875us) service_pool.cc:224] Handling call
0709 12:26:18.490933 (+1473031us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b54001eb4d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:18.491060 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":320638336}
I20260709 12:26:18.604975  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:18.606420  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.573712 11007 raft_consensus.cc:740] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:18.606562  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.590487 11398 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.587601 11526 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.634663 11577 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:18.612999  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.667222  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.667650 11447 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.664870 11575 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:18.664817 11526 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:18.660776  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 404369us
I20260709 12:26:18.672724  1726 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:26:18.675072  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:18.681834  1726 raft_consensus.cc:1217] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Deduplicated request from leader. Original: 19.11->[20.12-24.14]   Dedup: 20.12->[22.13-24.14]
I20260709 12:26:18.683898  1726 pending_rounds.cc:85] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 12
W20260709 12:26:18.685148  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 198121us
I20260709 12:26:18.684420  1726 pending_rounds.cc:107] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260709 12:26:18.686017  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:18.686057  1726 raft_consensus.cc:2889] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:18.681238  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:18.677721  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 169360us
I20260709 12:26:18.588090 11283 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:18.673821 11569 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:18.696236  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:18.686677 11417 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.998s	user 0.014s	sys 0.002s
I20260709 12:26:18.575057 11293 consensus_queue.cc:260] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.700620  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.701987  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3288) took 1132 ms. Trace:
I20260709 12:26:18.702430  1872 rpcz_store.cc:276] 0709 12:26:17.569280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.570115 (+   835us) service_pool.cc:224] Handling call
0709 12:26:18.701811 (+1131696us) spinlock_profiling.cc:243] Waited 155 ms on lock 0x7b54002df0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:18.701920 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":357396224}
I20260709 12:26:18.684782 11548 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:18.708456  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.712921  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:18.715873  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.713670  1806 raft_consensus.cc:2376] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 26. Re-sending same reply.
I20260709 12:26:18.716635  1805 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:18.609912 11573 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.729317  1805 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: [25.17-27.18]
I20260709 12:26:18.727689  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.731556  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.731840  1802 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 21.
I20260709 12:26:18.732372  1872 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:18.709367 11592 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:18.707011 11417 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:18.733369  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.742717  1872 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:18.706423 11591 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.050s
I20260709 12:26:18.674310 11510 consensus_queue.cc:260] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.739470  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.758881  1799 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
W20260709 12:26:18.752895  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.762167  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.762532  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.763073  1798 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:18.764114  1872 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:26:18.766633  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.743747 11293 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:18.779938  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.785321  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:18.789060  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.751626 11592 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:18.763036 11473 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:18.780822 11590 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.207s	user 0.002s	sys 0.000s
I20260709 12:26:18.793390  1807 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:18.760553 11456 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:18.794654  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:18.739017 11596 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 12:26:18.747318  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:18.789769 11510 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:18.814625  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 67449.7us
W20260709 12:26:18.800657 11425 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.002s	sys 0.008s
W20260709 12:26:18.811726  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 115581us
I20260709 12:26:18.799273 11408 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.820389  1807 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:26:18.825034  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:18.825845  1798 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:18.799090 10969 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.016s
W20260709 12:26:18.828240  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:18.855808  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.812177 11509 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:26:18.817391 11559 consensus_meta.cc:287] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.625s	user 0.008s	sys 0.006s
W20260709 12:26:18.821118 11425 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.003s	sys 0.008s
I20260709 12:26:18.835222  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.861716  1875 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:18.863054  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.863750  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:18.816838 11548 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:18.869095  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:18.864881  2383 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:18.867813  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.879254  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:18.880283  1732 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:18.869386  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.872033 11440 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:18.873456 11602 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:26:18.881785  1875 raft_consensus.cc:2393] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:18.829195 11595 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:26:18.820228 11398 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.824857 11417 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:18.839780 11529 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.885336  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:18.905380  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:18.907879  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:18.914891 11395 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:18.915514 11395 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:18.915840 11573 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:18.915911 11395 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:18.916644 11448 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:18.840164 11522 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.917102 11448 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:18.917620 11448 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:18.839262 11592 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:18.924180 11606 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:26:18.925958 11395 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:18.932574 11395 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.951713 11401 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:18.952647 11401 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:18.954542  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3458) took 1356 ms. Trace:
I20260709 12:26:18.955164  1729 rpcz_store.cc:276] 0709 12:26:17.598005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.598604 (+   599us) service_pool.cc:224] Handling call
0709 12:26:17.599040 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.763090 (+164050us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:17.763124 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:17.763169 (+    45us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:17.763219 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.954186 (+1190967us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54002d1ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:18.954254 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.954453 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":376705920}
I20260709 12:26:18.957696  1729 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 31 index: 14. Preceding OpId from leader: term: 32 index: 15. (index mismatch)
I20260709 12:26:18.962054 11522 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.860843 11509 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:18.963093 11509 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:18.964416  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.965306  1724 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:18.965435 11203 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:18.966250 11203 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.962730 11522 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:18.971526  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1595) took 2085 ms. Trace:
I20260709 12:26:18.972321  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:18.973316  1806 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:18.973855  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.978232 11446 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:18.997584  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:18.972087  1731 rpcz_store.cc:276] 0709 12:26:16.886513 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.887058 (+   545us) service_pool.cc:224] Handling call
0709 12:26:16.924165 (+ 37107us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:17.101903 (+177738us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:17.101937 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:17.101978 (+    41us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:17.102018 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.971200 (+1869182us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54002aa154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:18.971301 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:18.971455 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":408108928}
I20260709 12:26:18.983062  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.998373  1875 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:18.872074 11560 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:18.988106 11522 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:18.987735 11447 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:18.859172 11393 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.130s
W20260709 12:26:18.990598 11451 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.901s	user 0.009s	sys 0.008s
I20260709 12:26:18.995163  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:19.001938  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:19.003347  1724 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:19.023809  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.024699  1804 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:18.999794  1723 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:19.025183  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.997929  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:19.025876  1731 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 31.
W20260709 12:26:18.910840 11519 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.557s	user 0.011s	sys 0.002s
W20260709 12:26:19.008767  1630 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.040s	sys 0.036s
I20260709 12:26:19.025076  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:18.893294 11372 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:18.838414 11593 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.174s
W20260709 12:26:19.030992  1630 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.041s	sys 0.037s
I20260709 12:26:19.012375  1875 raft_consensus.cc:1240] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:19.002938 11548 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:19.062688  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:19.015226  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:19.063711  2416 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:19.016047  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:19.064285  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3291) took 1483 ms. Trace:
I20260709 12:26:18.995597 11440 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:18.866381 11559 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.030139  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:18.892745 11515 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:19.032258  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:19.046613  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:19.064847  1876 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:19.078729  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:19.051115 10919 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.032259 11372 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:19.060312 11469 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.987s	user 0.013s	sys 0.000s
W20260709 12:26:19.061993 11468 consensus_meta.cc:287] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.655s	user 0.008s	sys 0.003s
I20260709 12:26:19.031865 11417 consensus_queue.cc:1059] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.077s
I20260709 12:26:19.061826  1880 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:19.013988  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.065940  1878 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:19.115450  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.116156  1801 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:19.077270 11515 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:19.117663 11515 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:19.065893  1731 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:19.119771  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:19.120431  1731 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:19.121310  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:19.107041 10919 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.124835 10919 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:19.125902 11568 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.599s	user 0.011s	sys 0.001s
I20260709 12:26:19.035105 11519 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.128496 11519 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:19.131148 11468 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.132231 11559 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:19.132987 11469 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.134115 11468 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:19.072476 11440 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.135130 11440 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:19.139770 11532 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.985s	user 0.000s	sys 0.007s
I20260709 12:26:19.135128  1802 raft_consensus.cc:2393] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:19.010354  1732 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:26:19.064761  1874 rpcz_store.cc:276] 0709 12:26:17.580829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.581075 (+   246us) service_pool.cc:224] Handling call
0709 12:26:19.064089 (+1483014us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b5400051754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:19.064216 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":910573312}
I20260709 12:26:19.080024  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.066072  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:19.089141  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:19.075630  1723 raft_consensus.cc:2368] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 30. Current term is 31.
I20260709 12:26:19.033094 11483 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:26:19.143348  2415 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:19.032856 11522 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:19.090745  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:19.144047  1874 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:19.144474  1724 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 21.
W20260709 12:26:19.057408 11580 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.989s	user 0.003s	sys 0.003s
I20260709 12:26:19.032671 11451 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.073086 11473 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.155457  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1416) took 2928 ms. Trace:
I20260709 12:26:19.115598 11118 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:19.113207 11551 consensus_meta.cc:287] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 1.179s	user 0.006s	sys 0.010s
I20260709 12:26:19.011608 10969 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.057s
I20260709 12:26:19.037046 11203 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:19.146502  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 55920.6us
W20260709 12:26:19.140791 11599 log.cc:927] Time spent T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.319s	user 0.000s	sys 0.000s
I20260709 12:26:19.149689  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:19.084529  1806 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:19.151815  2416 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
W20260709 12:26:19.155651  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 245330us
I20260709 12:26:19.135259 11469 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:19.143225  1875 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:19.108006 11372 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:19.193862 11451 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:19.146209 11522 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:19.195075 11522 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:19.195328  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1427) took 1228 ms. Trace:
I20260709 12:26:19.195935  1803 rpcz_store.cc:276] 0709 12:26:17.967470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.967697 (+   227us) service_pool.cc:224] Handling call
0709 12:26:17.985182 (+ 17485us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:17.985713 (+   531us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:26:17.985747 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:17.986415 (+   668us) log.cc:844] Serialized 52 byte log entry
0709 12:26:18.034630 (+ 48215us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:18.034678 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.034766 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:19.056585 (+1021819us) spinlock_profiling.cc:243] Waited 7.96 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:19.194874 (+138289us) raft_consensus.cc:1685] finished
0709 12:26:19.194904 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.195237 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1487,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":103,"spinlock_wait_cycles":18315008,"thread_start_us":45983,"threads_started":1,"wal-append.queue_time_us":81825}
I20260709 12:26:19.063984 11488 consensus_queue.cc:1059] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.077s
I20260709 12:26:19.161038  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.201188  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.201745 11398 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.202422  1804 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:19.203953 11524 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 2.116s	user 0.000s	sys 0.013s
I20260709 12:26:19.177323  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.180529  1725 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:19.063159 11548 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:19.184144 11609 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
I20260709 12:26:19.185827  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.187511  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1447) took 1028 ms. Trace:
I20260709 12:26:19.188230  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.189368  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.155390 11516 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:19.190598  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1449) took 1020 ms. Trace:
I20260709 12:26:19.191493  1876 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:26:19.146028 11483 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:19.194108  2416 pending_rounds.cc:85] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 18
I20260709 12:26:19.162703  1801 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:19.220806  2416 pending_rounds.cc:107] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260709 12:26:19.219509  1872 rpcz_store.cc:276] 0709 12:26:18.179155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.782880 (+603725us) service_pool.cc:224] Handling call
0709 12:26:18.783617 (+   737us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:18.792863 (+  9246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:18.792912 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:18.799950 (+  7038us) log.cc:844] Serialized 26 byte log entry
0709 12:26:19.041355 (+241405us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:19.041393 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.041589 (+   196us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:19.190368 (+148779us) raft_consensus.cc:1685] finished
0709 12:26:19.190395 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.190524 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8568,"raft.queue_time_us":8706,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":5269,"thread_start_us":219565,"threads_started":1,"wal-append.queue_time_us":370120}
I20260709 12:26:19.221312  2416 raft_consensus.cc:2889] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:19.156375  1800 rpcz_store.cc:276] 0709 12:26:16.227470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:16.239574 (+ 12104us) service_pool.cc:224] Handling call
0709 12:26:16.254893 (+ 15319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:16.338125 (+ 83232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:16.338159 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:16.338715 (+   556us) log.cc:844] Serialized 26 byte log entry
0709 12:26:17.029262 (+690547us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:17.159340 (+130078us) log.cc:844] Serialized 14 byte log entry
0709 12:26:17.173230 (+ 13890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:17.173316 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:19.139002 (+1965686us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:19.147686 (+  8684us) raft_consensus.cc:1685] finished
0709 12:26:19.155180 (+  7494us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.155356 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":82773,"prepare.queue_time_us":24571,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":67,"raft.queue_time_us":150852,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":1968,"spinlock_wait_cycles":33078528,"thread_start_us":775160,"threads_started":2,"wal-append.queue_time_us":726456}
I20260709 12:26:19.148752 11588 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.205842  1727 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:19.179520  1697 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:19.214129  1877 rpcz_store.cc:276] 0709 12:26:18.158882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.332873 (+173991us) service_pool.cc:224] Handling call
0709 12:26:18.343345 (+ 10472us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:18.376343 (+ 32998us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:18.376410 (+    67us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:18.401889 (+ 25479us) log.cc:844] Serialized 78 byte log entry
0709 12:26:18.695728 (+293839us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:18.695796 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.695906 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:19.140089 (+444183us) spinlock_profiling.cc:243] Waited 8.37 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:19.187129 (+ 47040us) raft_consensus.cc:1685] finished
0709 12:26:19.187155 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.187435 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48201,"raft.queue_time_us":40669,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":33092,"spinlock_wait_cycles":19252608,"thread_start_us":181258,"threads_started":1,"wal-append.queue_time_us":260586}
I20260709 12:26:19.207695  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:19.207901  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.224517  1803 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
W20260709 12:26:19.204363 11587 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.936s	user 0.000s	sys 0.002s
I20260709 12:26:19.189780 11401 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.208542  1723 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:19.167830 11473 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.232749  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:19.233098 11488 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:19.233372  1723 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:19.234421  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:19.213693  1799 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:19.225575 11483 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:19.207080  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:19.202672 11398 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.169986  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:19.207080 11529 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.181880 11468 raft_consensus.cc:2764] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:26:19.226404  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.170348 11551 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.228952  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.156556 11448 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.236927  1723 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
W20260709 12:26:19.223719  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 44308.8us
I20260709 12:26:19.236799  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:19.283062  1803 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
W20260709 12:26:19.283634  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46943.7us
I20260709 12:26:19.283852  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:19.218070  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.240978  1805 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:19.279781  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 109891us
I20260709 12:26:19.279721 11529 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.239632  1731 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
W20260709 12:26:19.279031 11271 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.012s	user 0.020s	sys 0.021s
I20260709 12:26:19.287848  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:19.286413  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 254219us
I20260709 12:26:19.291184  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:19.217537 11451 raft_consensus.cc:3055] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:19.293604  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:19.296072  1723 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 31: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:19.323135  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.327853  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:19.324612  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:19.332134  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:19.431893  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:19.435817  2410 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:19.431943  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:19.430271 11447 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:19.437126 11447 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.437512 11447 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:19.439543  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4160) took 1420 ms. Trace:
I20260709 12:26:19.432957  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:19.433774  1872 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:19.333490  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:19.262298 11568 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:26:19.443956  1877 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:19.440119  1873 rpcz_store.cc:276] 0709 12:26:18.011500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.020173 (+  8673us) service_pool.cc:224] Handling call
0709 12:26:18.024873 (+  4700us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:18.033540 (+  8667us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:18.033582 (+    42us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:18.063554 (+ 29972us) log.cc:844] Serialized 78 byte log entry
0709 12:26:18.147602 (+ 84048us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:26:18.191361 (+ 43759us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.534885 (+343524us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:18.535002 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:19.203458 (+668456us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:19.431022 (+227564us) raft_consensus.cc:1685] finished
0709 12:26:19.431170 (+   148us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.431373 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40428,"prepare.queue_time_us":44205,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":69948,"raft.run_cpu_time_us":241,"raft.run_wall_time_us":2774,"spinlock_wait_cycles":73868672,"thread_start_us":360750,"threads_started":2,"wal-append.queue_time_us":146431}
I20260709 12:26:19.445194  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:19.447578  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 115547us
I20260709 12:26:19.235687 11588 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.235889 11488 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:19.450085  1806 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:19.451978  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:26:19.469287  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.470273  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:19.463920  2410 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:19.464900  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:19.324285 11271 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.058s	user 0.022s	sys 0.022s
W20260709 12:26:19.331794 11609 log.cc:927] Time spent T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:26:19.455770 11596 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:19.487510 11237 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:19.488497 11237 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.489305  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:19.489715  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4219) took 1007 ms. Trace:
I20260709 12:26:19.466343  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.293951 11521 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.491235 11521 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.491694 11456 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:19.492378 11237 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:19.492259 11456 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.493582 11521 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:19.477190  2410 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:26:19.493944  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.483250 11388 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:19.495011 11388 consensus_queue.cc:1243] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 31.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:26:19.432047 11203 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:19.496616  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.485101 11622 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:19.497434 11622 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.497792  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.498446  1807 raft_consensus.cc:2376] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24. Re-sending same reply.
I20260709 12:26:19.330004 11473 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:19.499296  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.500023  1807 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:19.490468  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:19.489724 11621 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:19.501394  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.501776  1799 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:19.484902 11358 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:19.486605 11271 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:19.502139  1804 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:19.510565  1723 raft_consensus.cc:2393] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:19.513126  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.513466  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.511672  1872 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:19.330744 11516 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:19.490096  1726 rpcz_store.cc:276] 0709 12:26:18.481989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.675136 (+193147us) service_pool.cc:224] Handling call
0709 12:26:18.681534 (+  6398us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:18.700461 (+ 18927us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:26:18.790816 (+ 90355us) log.cc:844] Serialized 14 byte log entry
0709 12:26:18.986725 (+195909us) spinlock_profiling.cc:243] Waited 15 ms on lock 0x7b18004c3000. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:26:19.012363 (+ 25638us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:19.013093 (+   730us) log.cc:844] Serialized 52 byte log entry
0709 12:26:19.013283 (+   190us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:26:19.013329 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.013422 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:19.489403 (+475981us) raft_consensus.cc:1685] finished
0709 12:26:19.489498 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:19.489656 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16657,"prepare.queue_time_us":35335,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":22138,"raft.run_cpu_time_us":333,"raft.run_wall_time_us":30994,"spinlock_wait_cycles":34558208,"thread_start_us":295973,"threads_started":2,"wal-append.queue_time_us":258799}
I20260709 12:26:19.529618  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:19.530514  1726 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:19.332862 11425 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:19.433514 11451 raft_consensus.cc:740] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:26:19.500195 11293 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.709s	user 0.010s	sys 0.003s
I20260709 12:26:19.445969 11408 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:19.531623  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 207119us
I20260709 12:26:19.499799 11622 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:19.509114 11469 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:19.527653  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:19.515379 11592 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:19.507531 11568 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:19.547174  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.571236  1807 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:19.572583  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 342162us
I20260709 12:26:19.492270 11448 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:19.518229 11271 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.442095 11509 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.519587 11593 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:19.548449 11358 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:19.544451  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.489861 11007 consensus_queue.cc:260] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.595624  1807 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:19.567780 11469 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.601773  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:19.603729  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:19.604848  1807 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:19.600962  1804 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:19.623102 11425 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:19.633620  1804 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
W20260709 12:26:19.696476  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.585619 11271 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:19.670257 11007 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:19.569684 11408 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.630111 11358 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.681342 11349 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:19.684515 11283 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:26:19.691493  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:19.691325  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:19.691793  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:19.692926 11546 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:19.547238 11516 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.715794  2418 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:19.696362  1806 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:19.588430 11620 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:19.703475  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 265414us
W20260709 12:26:19.696307 11446 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.662s	user 0.012s	sys 0.000s
I20260709 12:26:19.710168  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:19.756806  1799 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:19.627696 11469 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:19.665737 11521 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:19.714073  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:19.757290  1806 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:19.681453 11456 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:19.758278  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44302.1us
I20260709 12:26:19.765241  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:19.766445  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:19.766579  1802 raft_consensus.cc:1240] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:19.702765 11520 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:19.770175  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:19.789557 11446 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:19.792307 11551 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:19.624380 11593 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.770694  1806 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:19.787088  1803 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:19.787773  1726 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:19.771229 11563 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:19.703236 11425 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.772650  1879 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:19.756345 11543 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:19.768808 11620 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.772223 11521 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:19.805668 11483 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.570s	user 0.005s	sys 0.005s
I20260709 12:26:19.599377 11488 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.771034 11431 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:19.630474 11451 consensus_queue.cc:260] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 17, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:19.859920   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:26:19.843986  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 367352us
W20260709 12:26:19.809957  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 339775us
I20260709 12:26:19.863830  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:19.865097  1770 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:19.811259  2416 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.018s	sys 0.038s
W20260709 12:26:19.865846  2416 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.019s	sys 0.038s
W20260709 12:26:19.709232 11522 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.507s	user 0.006s	sys 0.008s
W20260709 12:26:19.811764 11618 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.370s	user 0.001s	sys 0.001s
I20260709 12:26:19.839890  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:19.840710  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 287132us
I20260709 12:26:19.834574 11563 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:19.880798  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:19.875579  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.763017 11568 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:19.877170  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:19.902470  1727 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.012s	sys 0.040s
W20260709 12:26:19.903029  1727 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.676s	user 0.012s	sys 0.041s
I20260709 12:26:19.833568 11529 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:19.879684  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.880558  2412 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
W20260709 12:26:19.890173  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:19.906121  1697 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:19.845172 11548 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:19.942413 11633 log.cc:927] Time spent T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:26:19.946843 11509 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:19.975704 11623 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.188s	user 0.001s	sys 0.000s
I20260709 12:26:19.976493 11509 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:19.976635 11521 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:19.977521 11624 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
I20260709 12:26:19.939481 11432 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.880646 11456 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:19.902154 11283 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:19.881145 11431 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:19.943179 11619 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.370s	user 0.001s	sys 0.001s
I20260709 12:26:19.902724 11563 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:19.904500 11438 raft_consensus.cc:3055] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:20.014760  1729 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4229) took 1163 ms. Trace:
W20260709 12:26:20.036231  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 448224us
I20260709 12:26:19.978698 11509 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:19.940125 11451 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:19.978302 11432 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.023757 11358 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:20.024787  1730 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57234 (request call id 4214) took 1921 ms (client timeout 1595 ms). Trace:
W20260709 12:26:20.051915  1730 rpcz_store.cc:269] 0709 12:26:18.103714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.104888 (+  1174us) service_pool.cc:224] Handling call
0709 12:26:20.024493 (+1919605us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b540030aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.024699 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91832704}
I20260709 12:26:20.025509 11593 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.026949 11548 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:19.875164 11483 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.029366  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:20.030829 11515 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.913s	user 0.013s	sys 0.000s
I20260709 12:26:19.979988 11398 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.039927 11408 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:19.944238 11440 consensus_meta.cc:287] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.808s	user 0.014s	sys 0.000s
I20260709 12:26:20.034492  1729 rpcz_store.cc:276] 0709 12:26:18.851325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.958693 (+107368us) service_pool.cc:224] Handling call
0709 12:26:19.003145 (+ 44452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:19.009379 (+  6234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:19.009428 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:19.010063 (+   635us) log.cc:844] Serialized 26 byte log entry
0709 12:26:19.161929 (+151866us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:19.161972 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.162056 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:20.014446 (+852390us) raft_consensus.cc:1685] finished
0709 12:26:20.014471 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.014670 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5716,"raft.queue_time_us":7365,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":6856,"thread_start_us":151520,"threads_started":1,"wal-append.queue_time_us":216008}
W20260709 12:26:20.039687  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 464368us
I20260709 12:26:19.908268 11596 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:20.022612 11372 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.830s	user 0.005s	sys 0.007s
I20260709 12:26:20.137979 11372 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.017971  1806 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
W20260709 12:26:20.082425  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53185.5us
I20260709 12:26:19.881856 11522 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.044872 11509 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:20.078760 11456 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.091429 11431 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:20.092228 11626 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:26:20.092357 11620 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.092610 11515 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.104382 11432 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.034233 11438 raft_consensus.cc:740] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:20.109068 11483 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.090490 11358 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:20.119226 11636 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:20.117674  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:20.119079 11440 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.045735 11516 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.121223 11596 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:20.139832  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 373416us
I20260709 12:26:20.079420 11592 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 29 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:20.090433 11593 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:26:20.163072  1723 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.007s	sys 0.037s
W20260709 12:26:20.241392  1723 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.007s	sys 0.038s
I20260709 12:26:20.167640  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3277) took 1082 ms. Trace:
I20260709 12:26:20.242559  1806 rpcz_store.cc:276] 0709 12:26:19.084977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.000793 (+915816us) service_pool.cc:224] Handling call
0709 12:26:20.017725 (+ 16932us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:20.167289 (+149564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.167562 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:20.168442  2415 raft_consensus.cc:2368] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30.
I20260709 12:26:20.089165 11548 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.139645 11635 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:26:20.226816  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 109106us
I20260709 12:26:20.228974  1845 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:20.230007  1724 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 3470) took 1211 ms. Trace:
W20260709 12:26:20.202589 11387 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.002s	sys 0.034s
W20260709 12:26:20.224282 11447 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.764s	user 0.015s	sys 0.000s
I20260709 12:26:20.253274 11425 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:20.251760 11592 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:26:20.250530 11631 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:26:20.222857 11358 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.201452 11408 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:20.251838  1806 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:49336 (request call id 0) took 1161 ms. Trace:
I20260709 12:26:20.395574  1806 rpcz_store.cc:276] 0709 12:26:19.090296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.249625 (+1159329us) service_pool.cc:224] Handling call
0709 12:26:20.251685 (+  2060us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:20.292521  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 522402us
I20260709 12:26:20.241935 11630 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:20.167922 11456 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:20.283229  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 377202us
W20260709 12:26:20.326678 11349 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.613s	user 0.010s	sys 0.002s
I20260709 12:26:20.241155 11593 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:20.297989  1724 rpcz_store.cc:276] 0709 12:26:19.018678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.078372 (+ 59694us) service_pool.cc:224] Handling call
0709 12:26:20.229742 (+1151370us) spinlock_profiling.cc:243] Waited 64.8 ms on lock 0x7b54000d0cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.229920 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":149091712}
I20260709 12:26:20.245405 11637 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:20.300248  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 71394.8us
W20260709 12:26:20.178367 11474 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.122s	user 0.005s	sys 0.020s
I20260709 12:26:20.308307  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1444) took 2150 ms. Trace:
I20260709 12:26:20.302531 11555 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:20.309453 11447 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.296841 11635 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.339543 11596 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.343051  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1439) took 1339 ms. Trace:
I20260709 12:26:20.420459  1802 rpcz_store.cc:276] 0709 12:26:19.003970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.673999 (+670029us) service_pool.cc:224] Handling call
0709 12:26:19.766305 (+ 92306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.767364 (+  1059us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.342696 (+575332us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b540004f6d0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.342946 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211462400}
W20260709 12:26:20.344060  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58100 (request call id 1458) took 1834 ms (client timeout 1734 ms). Trace:
W20260709 12:26:20.344931 11271 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.635s	user 0.014s	sys 0.000s
I20260709 12:26:20.345530 11468 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.346771  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:20.349483 11401 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.353958  1725 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3454) took 2765 ms. Trace:
W20260709 12:26:20.363379 11634 log.cc:927] Time spent T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:26:20.367055 11639 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:20.369973  1723 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1608) took 1267 ms. Trace:
I20260709 12:26:20.458132  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1773) took 1264 ms. Trace:
W20260709 12:26:20.374012 11469 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.587s	user 0.011s	sys 0.000s
I20260709 12:26:20.376082 11440 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.376565  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:20.467689  1730 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:26:20.376963  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.377508 11521 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:20.378119  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:20.471359  1728 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.020s	sys 0.032s
W20260709 12:26:20.471788  1728 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.020s	sys 0.032s
I20260709 12:26:20.427281 11528 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:20.382084  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.384001  1879 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 4157) took 2388 ms (client timeout 1595 ms). Trace:
W20260709 12:26:20.472711  1879 rpcz_store.cc:269] 0709 12:26:17.996015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.996580 (+   565us) service_pool.cc:224] Handling call
0709 12:26:20.383738 (+2387158us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b540021b1d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.383877 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3839492480}
I20260709 12:26:20.382635  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57228 (request call id 3469) took 1364 ms. Trace:
I20260709 12:26:20.474308  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:20.403577 11408 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.323460  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3273) took 1240 ms. Trace:
W20260709 12:26:20.329420 11007 consensus_meta.cc:287] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.002s
I20260709 12:26:20.387889  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:20.331228 11558 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:20.331699 11372 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.334534 11522 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.336813 11515 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.387485  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1623) took 1089 ms. Trace:
W20260709 12:26:20.329507  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:20.497789  1846 connection.cc:256] RPC call timeout handler was delayed by -1.0071012869998412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1554s for a response
I20260709 12:26:20.461265 11639 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:20.391220  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:20.392154  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:20.398276  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:20.504106  2410 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
W20260709 12:26:20.504490  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 106304us
I20260709 12:26:20.417063 11637 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.401743  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:20.395179 11358 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:20.409198  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:20.378302 11583 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:20.411056  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:20.413056  1878 rpcz_store.cc:276] 0709 12:26:18.158048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.158731 (+   683us) service_pool.cc:224] Handling call
0709 12:26:19.065756 (+907025us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.114674 (+ 48918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.308005 (+1193331us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b5400051750. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.308212 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2035025280}
W20260709 12:26:20.515465  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 106371us
W20260709 12:26:20.421916  1880 rpcz_store.cc:269] 0709 12:26:18.509585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:18.590746 (+ 81161us) service_pool.cc:224] Handling call
0709 12:26:20.343795 (+1753049us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b5400073d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.343957 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":896708480}
I20260709 12:26:20.480681 11645 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.411901 11593 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:26:20.434772  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 88144.7us
I20260709 12:26:20.529588  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:20.529904  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:20.441346  1881 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:20.457472  1723 rpcz_store.cc:276] 0709 12:26:19.103114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.574531 (+471417us) service_pool.cc:224] Handling call
0709 12:26:19.574928 (+   397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:19.575155 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:19.575182 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:19.575723 (+   541us) log.cc:844] Serialized 26 byte log entry
0709 12:26:20.242077 (+666354us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:20.242145 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.242271 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:20.369677 (+127406us) raft_consensus.cc:1685] finished
0709 12:26:20.369705 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.369889 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":665929,"threads_started":1,"wal-append.queue_time_us":791885}
I20260709 12:26:20.459018  2416 rpcz_store.cc:276] 0709 12:26:19.193263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.193482 (+   219us) service_pool.cc:224] Handling call
0709 12:26:19.193891 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:19.222158 (+ 28267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:19.222224 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:19.236637 (+ 14413us) log.cc:844] Serialized 26 byte log entry
0709 12:26:19.866246 (+629609us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:26:19.866291 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.866374 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:20.457786 (+591412us) raft_consensus.cc:1685] finished
0709 12:26:20.457820 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.458042 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":629301,"threads_started":1,"wal-append.queue_time_us":845849}
I20260709 12:26:20.534363  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:20.534967  1724 raft_consensus.cc:2410] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260709 12:26:20.535099  1723 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:26:20.536381  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:20.407457 11425 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:26:20.380622  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3266) took 1336 ms. Trace:
W20260709 12:26:20.313616 11632 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
I20260709 12:26:20.475252  1874 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:20.475994  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.319989 11526 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:20.475747  1799 rpcz_store.cc:276] 0709 12:26:19.082746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.503142 (+420396us) service_pool.cc:224] Handling call
0709 12:26:20.323200 (+820058us) spinlock_profiling.cc:243] Waited 243 ms on lock 0x7b54002f1f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.323370 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":558402432}
I20260709 12:26:20.469051  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:20.558835  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:20.558955  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:20.479108  2412 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:20.479568  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.432649 11547 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.489277  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.433817 11588 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.408208 11630 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.391626 11438 consensus_queue.cc:260] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.461067 11596 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:20.498730  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:20.588428  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 89813.3us
I20260709 12:26:20.389264 11548 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.439136 11620 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.416630 11563 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.501299  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.502609  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.489032 11521 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.532652 11007 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:26:20.298002  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 417267us
I20260709 12:26:20.598646  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.506737  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:20.514097  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:20.510074  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 108481us
I20260709 12:26:20.521055  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:20.521667  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:20.600135  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 93532.3us
I20260709 12:26:20.479033 11516 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.523245  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:20.456966 11468 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.433825  1725 rpcz_store.cc:276] 0709 12:26:17.588739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.590747 (+  2008us) service_pool.cc:224] Handling call
0709 12:26:17.663984 (+ 73237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.212726 (+1548742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.353617 (+1140891us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400002154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:20.353852 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3487083904}
I20260709 12:26:20.556521 11584 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:20.511373 11394 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.643s	user 0.031s	sys 0.043s
I20260709 12:26:20.471984  1729 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:20.469262 11469 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.537446  1804 rpcz_store.cc:276] 0709 12:26:19.044903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.696581 (+651678us) service_pool.cc:224] Handling call
0709 12:26:19.696931 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.769290 (+ 72359us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:19.769621 (+   331us) log.cc:844] Serialized 14 byte log entry
0709 12:26:20.150466 (+380845us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:20.150555 (+    89us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:20.150625 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.380378 (+229753us) spinlock_profiling.cc:243] Waited 72.1 ms on lock 0x7b54001d5954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:20.380423 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.380563 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":591349,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":145,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":9599,"spinlock_wait_cycles":165743360,"thread_start_us":380353,"threads_started":1,"wal-append.queue_time_us":589236}
I20260709 12:26:20.433033 11271 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.473975  1731 rpcz_store.cc:276] 0709 12:26:19.018298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.121068 (+102770us) service_pool.cc:224] Handling call
0709 12:26:20.382403 (+1261335us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54002d0ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.382557 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":270656768}
I20260709 12:26:20.561849  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:20.509984 11639 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:20.514494 11522 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:20.567937  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:20.525651 11171 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:20.570544  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.644493  1880 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
W20260709 12:26:20.570931  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3290) took 2994 ms (client timeout 1701 ms). Trace:
W20260709 12:26:20.645932  1881 rpcz_store.cc:269] 0709 12:26:17.576549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.579664 (+  3115us) service_pool.cc:224] Handling call
0709 12:26:20.570668 (+2991004us) spinlock_profiling.cc:243] Waited 2.82 s on lock 0x7b5400260a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.570820 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6497344256}
I20260709 12:26:20.647226  1872 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:20.647972  1805 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:20.576547  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:20.649384  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.579445   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:20.537773 11515 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.088s
I20260709 12:26:20.581934  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.579054  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:20.497442  2415 rpcz_store.cc:276] 0709 12:26:19.297640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.432683 (+135043us) service_pool.cc:224] Handling call
0709 12:26:20.387114 (+954431us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b5400580f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:20.387312 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1667732352}
I20260709 12:26:20.662171  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1607) took 1587 ms. Trace:
I20260709 12:26:20.648257 11007 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:20.493432 11528 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.598897  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:20.415810 11474 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.360s	user 0.006s	sys 0.021s
I20260709 12:26:20.600077  1876 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 22: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:20.606024  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:20.606485  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:20.594725  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:20.552656 11501 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:26:20.623070  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:20.617465  1874 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:20.730440 11408 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:20.778502  1807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.013s	sys 0.022s
I20260709 12:26:20.637476  2417 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:26:20.608018 11651 raft_consensus.cc:2764] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:26:20.499867 11558 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.643371  2418 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 33: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:26:20.562547 11586 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.876s	user 0.008s	sys 0.016s
I20260709 12:26:20.587594 11596 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:20.534632 11645 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.526010 11583 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.683467  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.666002  1801 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:26:20.611974 11555 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:20.667363  1873 raft_consensus.cc:2410] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:26:20.589178 10969 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:26:20.678900  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:20.783422 11643 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:20.813133  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:20.792295 11586 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.109s	user 0.008s	sys 0.016s
I20260709 12:26:20.814112  1872 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:20.815279  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.816609  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:20.678959  1805 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3272) took 1597 ms. Trace:
I20260709 12:26:20.817512  1805 rpcz_store.cc:276] 0709 12:26:19.082024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.792973 (+710949us) service_pool.cc:224] Handling call
0709 12:26:19.793340 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:20.648901 (+855561us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.678601 (+ 29700us) spinlock_profiling.cc:243] Waited 854 ms on lock 0x7b54003b79d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:20.678880 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1965031808}
I20260709 12:26:20.681083  1725 raft_consensus.cc:1275] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:26:20.684095  1727 rpcz_store.cc:276] 0709 12:26:19.075421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.223884 (+148463us) service_pool.cc:224] Handling call
0709 12:26:19.224332 (+   448us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:19.225739 (+  1407us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:19.225777 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:19.226345 (+   568us) log.cc:844] Serialized 26 byte log entry
0709 12:26:19.903446 (+677101us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:19.903494 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:19.903572 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:20.661786 (+758214us) raft_consensus.cc:1685] finished
0709 12:26:20.661823 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.662078 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":926,"raft.queue_time_us":9764,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":676731,"threads_started":1,"wal-append.queue_time_us":938135}
I20260709 12:26:20.686560  1731 pending_rounds.cc:85] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 13
I20260709 12:26:20.821365  1731 pending_rounds.cc:107] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 31.14
I20260709 12:26:20.821717  1731 raft_consensus.cc:2889] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:20.644026 11171 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.669970  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.688184 11630 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.692637 11635 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:20.693636 11661 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.125s	user 0.001s	sys 0.000s
W20260709 12:26:20.694628 11644 log.cc:927] Time spent T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
W20260709 12:26:20.695449 11648 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.229s	user 0.001s	sys 0.000s
I20260709 12:26:20.706357 11469 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.707858 11456 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.583148 11547 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.643519 11522 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:20.711719 11271 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.650156 11447 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.640255 11584 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:20.626149 11516 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.618597 11620 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:20.656001 11621 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.144s	user 0.013s	sys 0.003s
I20260709 12:26:20.721030 11283 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:20.723003 11526 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:26:20.724997 11563 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:20.626948 11394 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.763s	user 0.031s	sys 0.044s
I20260709 12:26:20.536618 11451 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.563257 11488 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.735910 11501 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.778738 11548 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:20.799626 11596 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:20.809684 11555 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.836977 11630 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:20.864859 11622 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.012s	sys 0.037s
I20260709 12:26:20.873574 11469 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:20.871758 11522 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:20.886919 11637 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.891490 11528 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:20.894415 11621 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:20.893625 11526 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:20.917244 11501 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:20.921555 11622 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.013s	sys 0.037s
I20260709 12:26:20.918531 11630 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:20.921300 11555 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:20.926803 11548 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:20.947492 11630 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:21.001488 11664 log.cc:927] Time spent T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:26:20.653851   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:26:20.736471  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 137685us
W20260709 12:26:21.037776  1732 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.000s	sys 0.011s
W20260709 12:26:21.038513  1732 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.000s	sys 0.012s
I20260709 12:26:20.735723  1723 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:20.693018  1726 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4234) took 1547 ms. Trace:
I20260709 12:26:21.041319  1732 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:48886 (request call id 1609) took 1813 ms. Trace:
I20260709 12:26:20.738418  1697 leader_election.cc:400] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30.
W20260709 12:26:20.648105  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet def0036446324543ba07e132e885133a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:21.038132  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:20.779222  1807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.013s	sys 0.022s
W20260709 12:26:20.735045  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 128671us
I20260709 12:26:21.041971  1732 rpcz_store.cc:276] 0709 12:26:19.227552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.977832 (+750280us) service_pool.cc:224] Handling call
0709 12:26:19.978175 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.978428 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:26:20.507005 (+528577us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.039814 (+532809us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.040037 (+   223us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:21.040221 (+   184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.040686 (+   465us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.041137 (+   451us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":622846,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":54,"thread_start_us":1030473,"threads_started":2}
I20260709 12:26:21.079480  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:21.060026  1800 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.012s	sys 0.022s
W20260709 12:26:21.081763  1800 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.013s	sys 0.026s
I20260709 12:26:21.082046  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:21.043157  1728 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.010s	sys 0.013s
I20260709 12:26:21.061133   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:26:20.871861  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 459235us
W20260709 12:26:21.044499  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 523294us
W20260709 12:26:21.084563  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 345360us
I20260709 12:26:21.052016  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3261) took 2038 ms. Trace:
W20260709 12:26:20.736358  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 130430us
W20260709 12:26:20.738168  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 451385us
I20260709 12:26:21.094208  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:21.095572  2410 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:21.090492  1878 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 12. Preceding OpId from leader: term: 32 index: 15. (index mismatch)
W20260709 12:26:21.083608  1728 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.012s	sys 0.018s
I20260709 12:26:20.737843  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:20.825997  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.103996  1873 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:21.104403  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:21.105490  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.831265  1879 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:20.826745  1881 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:20.849637  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:21.109416  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.796630  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.109540  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.829974 11171 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:21.170435 11474 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:21.172151 11528 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:21.173306  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:21.042089  1726 rpcz_store.cc:276] 0709 12:26:19.145473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.836493 (+691020us) service_pool.cc:224] Handling call
0709 12:26:19.836944 (+   451us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:19.842816 (+  5872us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:19.842872 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:19.843522 (+   650us) log.cc:844] Serialized 26 byte log entry
0709 12:26:20.297506 (+453984us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:20.297554 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:20.297636 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:20.692595 (+394959us) raft_consensus.cc:1685] finished
0709 12:26:20.692637 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:20.692910 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9614,"raft.queue_time_us":161282,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":9787,"thread_start_us":411314,"threads_started":1,"wal-append.queue_time_us":786837}
W20260709 12:26:21.067005  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 467709us
W20260709 12:26:21.174990 11471 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.005s	sys 0.014s
I20260709 12:26:20.920229 11468 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.102305  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:21.111747  1615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 657670us
I20260709 12:26:21.088575  1697 leader_election.cc:403] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:26:21.175818 11471 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.005s	sys 0.014s
W20260709 12:26:21.082075  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44587us
I20260709 12:26:21.175662  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:20.911968 11451 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:21.195756  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114032us
W20260709 12:26:20.996796 11663 log.cc:927] Time spent T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.165s	user 0.002s	sys 0.000s
I20260709 12:26:20.929071 11521 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:21.178470 11548 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.231055  1872 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:21.236423  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:21.237375 11547 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:21.238135  2418 raft_consensus.cc:2368] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 30. Current term is 31.
W20260709 12:26:21.244432 11431 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.003s	sys 0.022s
I20260709 12:26:21.244743 11526 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.247931 11516 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.204334 11474 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:21.250540 11521 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:21.214577 11425 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:21.283604  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.285027  2417 raft_consensus.cc:2393] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:21.251488 11568 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.251665  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.264916 11586 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.286410  2415 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:21.252076 11558 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.252128  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.253113  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.247812  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:20.900863 11007 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:21.244434  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149525us
I20260709 12:26:21.260433  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.237982 11620 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:20.914916 11488 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:21.272621 11431 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.003s	sys 0.024s
I20260709 12:26:21.259686 11672 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.003s
W20260709 12:26:21.273519  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44367.9us
I20260709 12:26:21.268831  1881 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:21.238451 11547 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:21.280158 11358 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.735s	user 0.011s	sys 0.001s
I20260709 12:26:21.281877  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:21.319566  1872 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:21.319196 11358 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:20.971594 11645 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:21.318306 11488 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.304454  1844 leader_election.cc:400] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 30. Current term is 31.
I20260709 12:26:20.903519 11283 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.297508  2412 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:26:21.267761 11699 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:21.286952 11586 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.287824 11521 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.286172 11568 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.316385 11431 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:21.316965  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:20.930779 11469 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.282375 11583 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.327250  1844 leader_election.cc:403] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:26:21.090127  1798 rpcz_store.cc:276] 0709 12:26:19.013495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.686312 (+672817us) service_pool.cc:224] Handling call
0709 12:26:19.686761 (+   449us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:19.687197 (+   436us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:19.687227 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:19.687819 (+   592us) log.cc:844] Serialized 26 byte log entry
0709 12:26:20.118613 (+430794us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:20.573590 (+454977us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.051091 (+477501us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.051316 (+   225us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:21.051455 (+   139us) raft_consensus.cc:1685] finished
0709 12:26:21.051472 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.051772 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14727,"prepare.queue_time_us":242676,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":210846,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":9694,"thread_start_us":1286248,"threads_started":3,"wal-append.queue_time_us":619588}
I20260709 12:26:21.331843 11526 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:21.332310  2415 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:26:21.343873  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 114659us
I20260709 12:26:21.333240  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:21.334807 11283 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.334437 11474 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 29, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.335041 11554 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:21.361095  1724 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:21.318382 11451 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.297219 11440 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:26:21.373258  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 143306us
I20260709 12:26:21.376606  1844 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:21.377633  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:21.378482 11586 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.379623  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:21.380787  1732 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:21.382155  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:21.383462 11469 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.383773  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:21.356281  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 618063us
W20260709 12:26:21.308004  1730 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.574s	user 0.009s	sys 0.019s
W20260709 12:26:21.383674  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:21.343420  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68692.7us
I20260709 12:26:21.313547 11547 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.327057 11358 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:21.356168  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.005s	sys 0.040s
I20260709 12:26:21.350060 11521 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:21.351750 11431 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:21.362310  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1443) took 2028 ms. Trace:
W20260709 12:26:21.394042  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.005s	sys 0.041s
I20260709 12:26:21.396660  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.402570  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:21.397311  1730 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.749s	user 0.010s	sys 0.019s
I20260709 12:26:21.404500  1723 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:21.408691 11645 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:21.409410 11645 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:21.409988 11645 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:21.411114 11558 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:21.411861 11558 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:21.412578 11558 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:21.411701  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:21.399843  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83188.1us
I20260709 12:26:21.391026 11547 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:21.424733  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.425381  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57228 (request call id 3475) took 1048 ms. Trace:
I20260709 12:26:21.426062  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.434166  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:21.435057  2412 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:21.443472  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.444677 11568 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.445261  2417 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:21.361095 11699 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.448515 11643 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.449492 11699 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.449357 11643 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.437237  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.434613  1881 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:21.450362 11699 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:21.461630  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.462525  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:21.465402  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:21.467077 11447 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:26:21.467558  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:21.468379  1724 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:26:21.468468  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:21.469185  1804 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:21.469606 11586 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:21.469709  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:21.470609 11586 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.471074 11681 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.472064 11681 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.472786 11621 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:21.475528 11681 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.475703 11668 consensus_queue.cc:1059] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:21.476953 11668 consensus_queue.cc:1059] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:21.478006 11668 raft_consensus.cc:3055] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:21.478379 11668 raft_consensus.cc:740] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:21.392349  1800 rpcz_store.cc:276] 0709 12:26:19.327288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.346243 (+1018955us) service_pool.cc:224] Handling call
0709 12:26:20.346640 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:20.346939 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:26:21.082591 (+735652us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.266284 (+183693us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.276994 (+ 10710us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:26:21.301046 (+ 24052us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.313210 (+ 12164us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.355859 (+ 42649us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26335,"prepare.queue_time_us":14616,"prepare.run_cpu_time_us":137,"prepare.run_wall_time_us":135,"raft.queue_time_us":66074,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":205,"thread_start_us":833802,"threads_started":2}
I20260709 12:26:21.467913  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:21.485841  2416 raft_consensus.cc:2410] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:26:21.459704  1873 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:26:21.487205  1872 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:21.462714  1798 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:21.432411  1730 rpcz_store.cc:276] 0709 12:26:20.377545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.638800 (+261255us) service_pool.cc:224] Handling call
0709 12:26:20.640038 (+  1238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:20.640317 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:26:20.647864 (+  7547us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.424568 (+776704us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.424709 (+   141us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:21.424874 (+   165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.424996 (+   122us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.425301 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11972,"prepare.queue_time_us":66522,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":17365,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":197,"thread_start_us":756140,"threads_started":1}
W20260709 12:26:21.508525  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.507928  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:21.421819 11583 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.090s
I20260709 12:26:21.531046  1805 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:21.544377  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.455371 11563 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.114s
I20260709 12:26:21.552070 11546 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:21.554934  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:21.555431  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 87653.5us
I20260709 12:26:21.556586  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:21.557713 11546 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:21.562984 11690 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:21.550179  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.577894  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:21.551460  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 113198us
I20260709 12:26:21.580698  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.549547 11469 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:21.585206  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4232) took 2489 ms. Trace:
I20260709 12:26:21.591815 11686 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:21.595564 11639 consensus_meta.cc:287] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.937s	user 0.007s	sys 0.016s
I20260709 12:26:21.596176 11639 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.597675  2413 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:21.602932 11584 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:21.604719  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.571208 11431 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.607278 11643 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.600579  1873 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:21.607478 11431 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:21.620215  1723 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 33.
I20260709 12:26:21.631248  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.589295  1728 rpcz_store.cc:276] 0709 12:26:19.096010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.568337 (+472327us) service_pool.cc:224] Handling call
0709 12:26:19.574273 (+  5936us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.857136 (+282863us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:26:20.472263 (+615127us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.239880 (+767617us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.240054 (+   174us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:21.240266 (+   212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.571493 (+331227us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b54002a9754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:21.571549 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.585125 (+ 13576us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":70259,"prepare.queue_time_us":671341,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":10002,"raft.queue_time_us":72192,"raft.run_cpu_time_us":294,"raft.run_wall_time_us":295,"spinlock_wait_cycles":649308416,"thread_start_us":1186433,"threads_started":2,"wal-append.queue_time_us":798366}
W20260709 12:26:21.639602  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51298.7us
W20260709 12:26:21.647852  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 189367us
W20260709 12:26:21.649559 11509 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.466s	user 0.012s	sys 0.014s
I20260709 12:26:21.650243 11509 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.651163  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.652027  1873 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:21.652969 11509 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.653256  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.654172  1873 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:21.663343 11501 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.665158 11700 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:21.651437  2416 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 28: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:21.650622 11698 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:21.578424 11699 consensus_queue.cc:1059] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:26:21.673520  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:21.679682  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.666137 11501 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.686915  2412 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:21.621855 11686 consensus_queue.cc:1048] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.055s
W20260709 12:26:21.693359 11555 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.739s	user 0.015s	sys 0.002s
I20260709 12:26:21.696009  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:21.670975 11698 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:26:21.703589  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 58475.6us
I20260709 12:26:21.709036  1881 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:21.710376  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3304) took 1053 ms. Trace:
I20260709 12:26:21.710776  1881 rpcz_store.cc:276] 0709 12:26:20.657384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.332757 (+675373us) service_pool.cc:224] Handling call
0709 12:26:21.710291 (+377534us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:21.693378 11630 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.731s	user 0.011s	sys 0.005s
I20260709 12:26:21.715073 11408 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:21.715971 11408 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.720614 11672 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 30. Current term is 31.
I20260709 12:26:21.721771  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1444) took 2392 ms. Trace:
I20260709 12:26:21.723115  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4166) took 1087 ms. Trace:
I20260709 12:26:21.722239  1801 rpcz_store.cc:276] 0709 12:26:19.329280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.352712 (+1023432us) service_pool.cc:224] Handling call
0709 12:26:20.353181 (+   469us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:20.808269 (+455088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.721173 (+912904us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b54002f2e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:21.721584 (+   411us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":718686080}
I20260709 12:26:21.718508 11408 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.723711  1877 rpcz_store.cc:276] 0709 12:26:20.590415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.590564 (+   149us) service_pool.cc:224] Handling call
0709 12:26:20.598536 (+  7972us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:20.599030 (+   494us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:20.599062 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:20.599614 (+   552us) log.cc:844] Serialized 26 byte log entry
0709 12:26:21.088919 (+489305us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:21.089050 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.089220 (+   170us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:21.666442 (+577222us) raft_consensus.cc:1685] finished
0709 12:26:21.666464 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.678103 (+ 11639us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11627,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":24230,"thread_start_us":488582,"threads_started":1,"wal-append.queue_time_us":940471}
W20260709 12:26:21.723137  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:21.734130 11586 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.708993 11698 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:21.712261 11358 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:21.743834  2412 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:26:21.747241  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:21.748273  2415 raft_consensus.cc:2368] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 30. Current term is 31.
I20260709 12:26:21.748911  1731 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4239) took 1193 ms. Trace:
I20260709 12:26:21.749256  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:21.750845 11690 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:21.751271 11672 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:21.751782 11672 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:21.751951  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4178) took 1043 ms. Trace:
I20260709 12:26:21.752345 11358 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.752475  1873 rpcz_store.cc:276] 0709 12:26:20.708594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.688652 (+980058us) service_pool.cc:224] Handling call
0709 12:26:21.719213 (+ 30561us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.751538 (+ 32325us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:21.751586 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.751653 (+    67us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:26:21.751701 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.751770 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.751890 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16828,"raft.queue_time_us":17305,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122}
I20260709 12:26:21.758014 11558 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.758343 11349 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:21.749454  1731 rpcz_store.cc:276] 0709 12:26:20.555832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.685953 (+130121us) service_pool.cc:224] Handling call
0709 12:26:20.686352 (+   399us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:21.043561 (+357209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:21.043650 (+    89us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:21.045698 (+  2048us) log.cc:844] Serialized 52 byte log entry
0709 12:26:21.223715 (+178017us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:26:21.223821 (+   106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.275006 (+ 51185us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:21.748516 (+473510us) raft_consensus.cc:1685] finished
0709 12:26:21.748539 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:21.748722 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":220527,"raft.queue_time_us":371503,"raft.run_cpu_time_us":326,"raft.run_wall_time_us":333,"thread_start_us":167705,"threads_started":1,"wal-append.queue_time_us":596150}
I20260709 12:26:21.750998 11699 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
W20260709 12:26:21.761672 11596 consensus_meta.cc:287] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.836s	user 0.002s	sys 0.008s
I20260709 12:26:21.763649 11358 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.764364 11690 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.765906 11438 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:21.710049 11555 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.766440  1801 raft_consensus.cc:1240] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: [25.17-27.18]
I20260709 12:26:21.766626  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:21.767076 11680 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.767097 11516 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:21.768349  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.769304  1800 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:26:21.770581  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.770836 11586 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.766757  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:21.771377 11447 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30.
I20260709 12:26:21.823342  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.777801 11583 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:21.781594 11425 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.781911  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3294) took 1307 ms. Trace:
I20260709 12:26:21.786360  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.787940  2414 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:21.791353  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.824658  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:21.800905  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:21.802132  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:21.788851 11692 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.355s	user 0.001s	sys 0.001s
W20260709 12:26:21.802943 11171 consensus_meta.cc:287] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.540s	user 0.018s	sys 0.002s
W20260709 12:26:21.787808 11694 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.361s	user 0.003s	sys 0.000s
I20260709 12:26:21.809738  1872 raft_consensus.cc:1240] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:21.834365  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4177) took 1126 ms. Trace:
W20260709 12:26:21.771867 11593 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.220s	user 0.007s	sys 0.019s
I20260709 12:26:21.819567  1806 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:21.752323 11672 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:21.799448 11554 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:21.843182 11639 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.843549 11554 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.844775 11516 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.844959 11521 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.846101  1798 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:21.847574 11516 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.838192 11171 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.828658  1879 rpcz_store.cc:276] 0709 12:26:20.474099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.474661 (+   562us) service_pool.cc:224] Handling call
0709 12:26:21.781348 (+1306687us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400117954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:21.781623 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":550244608}
I20260709 12:26:21.826063  1803 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:21.834520  1802 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:21.834836  1872 rpcz_store.cc:276] 0709 12:26:20.708260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.698499 (+990239us) service_pool.cc:224] Handling call
0709 12:26:21.731324 (+ 32825us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.834083 (+102759us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:21.834293 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:21.837073  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:21.840803  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.822275 11680 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.735558 11630 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.840854 11586 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.842298 11593 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.817101 11349 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:21.854604  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.855031  1874 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:21.851661  2416 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:26:21.856295  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:21.862749  1800 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:21.861636  2412 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:21.832147 11583 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:21.831653 11621 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:21.872380  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:21.786698 11698 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.860286  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:21.883127  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 75498us
I20260709 12:26:21.817919 11547 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:21.886150  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:21.886248  1799 raft_consensus.cc:2393] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:21.899820  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:21.909868  1725 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 23: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:21.909704  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:21.900130  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.911077  1879 raft_consensus.cc:2376] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 075704438992440697732181e49ccd09 in term 25. Re-sending same reply.
I20260709 12:26:21.912173  1806 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:21.902864 11586 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.912909  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.913868  1879 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:21.900990  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.915973  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.916589  1803 raft_consensus.cc:2468] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:21.917536  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.918156  1803 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:21.911197  1801 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:21.919241 11554 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.919639  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.825184 11596 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.920778  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:21.921568  1801 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:21.921653 11596 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.923014  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:21.923619 11558 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:21.923772  1801 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:21.926162 11680 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:21.861176 11349 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:21.913789  2416 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:26:21.932022  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:21.927814  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.826565 11568 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:21.875703  1696 connection.cc:256] RPC call timeout handler was delayed by -1.7715854610000861s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3068s for a response
I20260709 12:26:21.933152  1872 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:21.937813  1880 raft_consensus.cc:2393] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:21.938205  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.941380  2416 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 28: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:21.939610  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:21.902588 11007 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:21.831398 11690 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:21.842000 11425 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.148511  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:22.068074  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 303741us
I20260709 12:26:22.037369  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:22.161697  1723 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:26:22.058118  1798 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:26:22.003116  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:22.119179  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:22.164449  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 362441us
I20260709 12:26:21.886511 11447 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:21.902820 11698 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:22.167852  2411 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 25 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:26:21.998085  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:22.059813  1876 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
W20260709 12:26:22.184482  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 92469.6us
I20260709 12:26:22.169137  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:21.948539  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.008711  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:22.188966  1725 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:21.992264 11451 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:21.991978 11440 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:21.931108 11431 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:22.182072 11683 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:22.187013 11488 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:21.991487 11700 raft_consensus.cc:3055] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:26:21.959213 11699 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.200179  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:22.157086 11521 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:22.206128  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.165292 11630 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.120596 11358 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:22.105765 11548 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:22.214370  1696 connection.cc:256] RPC call timeout handler was delayed by -1.3450210129999505s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1564s for a response
I20260709 12:26:22.216763  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:22.221076  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2613210129998151s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16570000000000001s for a response
W20260709 12:26:22.221432  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2451210129997889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1675s for a response
W20260709 12:26:22.221699  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2253210129997569s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16970000000000002s for a response
W20260709 12:26:22.221942  1696 connection.cc:256] RPC call timeout handler was delayed by -1.2073210129997278s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17170000000000002s for a response
W20260709 12:26:22.222167  1696 connection.cc:256] RPC call timeout handler was delayed by -1.1578210130001025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17720000000000002s for a response
W20260709 12:26:22.222404  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0687210129999585s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18710000000000002s for a response
W20260709 12:26:22.222641  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0129210129998683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19330000000000003s for a response
I20260709 12:26:22.224668 11358 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:22.224839 11522 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.244s	user 0.009s	sys 0.009s
I20260709 12:26:22.225241 11358 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:22.225519 11522 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.226033  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:22.225920 11548 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.226715  2411 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:22.226995 11440 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.228335 11690 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:22.229308 11440 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.229991  1803 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:21.992002 11568 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.232137  2417 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:22.241420  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.243561  1805 raft_consensus.cc:3060] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:22.231258  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:22.237203  2411 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
W20260709 12:26:22.209781 11645 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.799s	user 0.014s	sys 0.001s
I20260709 12:26:22.229230 11548 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.259819  1806 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:22.260829  1725 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:22.273140  1875 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:22.273870  1723 raft_consensus.cc:2410] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 28 because replica has last-logged OpId of term: 26 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:26:22.213948 11700 raft_consensus.cc:740] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:22.230743 11440 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:22.028407 11583 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:22.301378  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:22.000000 11349 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.222885 11699 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:22.291085  1801 raft_consensus.cc:2468] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:22.314229  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.264700 11431 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:22.222556  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet def0036446324543ba07e132e885133a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:22.336725  1615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 114252us
I20260709 12:26:22.257408 11447 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.340824  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.269033 11645 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.275454 11451 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:22.305063 11171 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:22.280870 11488 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.356299  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:22.359990  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 151985us
I20260709 12:26:22.360139  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:22.363638  1806 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:22.364122 11700 consensus_queue.cc:260] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:22.364743  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:22.365502 11700 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:22.365801 11451 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:22.366276  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:22.287676 11007 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:22.301266 11440 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:22.368499 11645 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.369859  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:22.389657  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.376260 11547 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.376420  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4167) took 1785 ms. Trace:
I20260709 12:26:22.276410 11683 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.390821  1878 rpcz_store.cc:276] 0709 12:26:20.590761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.592622 (+  1861us) service_pool.cc:224] Handling call
0709 12:26:20.639566 (+ 46944us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.097230 (+457664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.375980 (+1278750us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b5400222754. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:22.376331 (+   351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1033791232}
I20260709 12:26:22.378664  1806 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:22.394629  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.375612 11425 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:22.392059  1727 consensus_meta.cc:287] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.531s	user 0.009s	sys 0.001s
I20260709 12:26:22.396234  1727 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:22.399770 11645 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:22.389462  1724 pending_rounds.cc:85] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Aborting all ops after (but not including) 17
I20260709 12:26:22.400134  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.404844  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.412784  1723 raft_consensus.cc:2410] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 29 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:26:22.412984  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.394469 11593 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.416296  1803 raft_consensus.cc:2393] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 24: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:22.400563  1724 pending_rounds.cc:107] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Aborting uncommitted NO_OP operation due to leader change: 30.18
I20260709 12:26:22.411230  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:22.383867 11554 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.417560  1724 raft_consensus.cc:2889] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:22.420383  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:22.423247 11547 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:22.428016  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:22.428987  1878 raft_consensus.cc:2368] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23.
I20260709 12:26:22.430433  1726 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:22.435374  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:22.440913  1805 raft_consensus.cc:2468] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:22.407848 11425 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:22.449245  1872 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:22.445632  1697 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:22.459120  1803 raft_consensus.cc:1275] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:26:22.461012  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.463608  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.464258  1731 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 27: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:22.464375  1878 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:22.464972  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
W20260709 12:26:22.467839  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 195557us
I20260709 12:26:22.468282  1806 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:26:22.470734  1726 raft_consensus.cc:2393] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:22.472999  1798 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
W20260709 12:26:22.470085  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52227.6us
I20260709 12:26:22.468940  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.454025 11522 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:22.476945  1806 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:22.478835  1872 raft_consensus.cc:2468] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:22.485621  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.487037  1732 raft_consensus.cc:2393] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 22: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:22.480578  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:22.500240  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:22.504693  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 313149us
I20260709 12:26:22.464617 11547 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.472748 11554 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.484489 11593 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:26:22.521093 11554 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:22.525720 11672 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.651s	user 0.011s	sys 0.000s
I20260709 12:26:22.507136 11596 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:22.535112  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.513447  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:22.481864 11425 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.513711 11568 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.542376  1846 leader_election.cc:400] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 30. Current term is 31.
I20260709 12:26:22.540359  1878 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 32.
I20260709 12:26:22.552493  1872 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:22.527935 11431 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:22.526424 11672 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:22.539974 11438 consensus_meta.cc:287] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.716s	user 0.011s	sys 0.003s
I20260709 12:26:22.566641  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.529935 11447 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.537617 11593 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:22.571491  1805 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:26:22.579442  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:22.575325  2415 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:22.580269  2411 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:22.578644  1878 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:22.561671 11596 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:22.592955  1879 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:22.582813  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:22.593911  1872 raft_consensus.cc:3060] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:26:22.595667 11703 log.cc:927] Time spent T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.790s	user 0.004s	sys 0.000s
W20260709 12:26:22.595456  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68385.9us
I20260709 12:26:22.571105 11699 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:22.558817 11719 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:26:22.603470 11398 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.546960 11171 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.037s
I20260709 12:26:22.609751  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.604095 11398 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.584532 11488 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.592238 11440 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:22.630628  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.637372  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:26:22.643339  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 183201us
I20260709 12:26:22.643572  2411 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:26:22.647387  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60934 (request call id 1702) took 1327 ms. Trace:
I20260709 12:26:22.647541 11425 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.647898  2413 rpcz_store.cc:276] 0709 12:26:21.320150 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.342474 (+ 22324us) service_pool.cc:224] Handling call
0709 12:26:21.366850 (+ 24376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:21.598106 (+231256us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.647102 (+1048996us) spinlock_profiling.cc:243] Waited 230 ms on lock 0x7b54005f05d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:22.647295 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":529914240}
I20260709 12:26:22.651335  1732 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:22.654106 11547 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.663481  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:22.669790  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:22.673185  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48213.1us
I20260709 12:26:22.676350 11596 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:22.678030  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:22.679469  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:22.679682  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:22.679665 11720 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:26:22.687294 11358 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:22.687775  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1621) took 1311 ms. Trace:
I20260709 12:26:22.688462  1727 rpcz_store.cc:276] 0709 12:26:21.376676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.383081 (+  6405us) service_pool.cc:224] Handling call
0709 12:26:22.687404 (+1304323us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54000d2ad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:22.687606 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1088190720}
I20260709 12:26:22.692090 11680 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:22.692754 11680 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:22.693234 11680 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:22.694552  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:22.697896 11469 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:22.699195 11593 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:22.702486  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:22.702836 11710 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.377s	user 0.002s	sys 0.002s
I20260709 12:26:22.697729  2413 raft_consensus.cc:2393] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:26:22.705896  1800 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.836s	user 0.007s	sys 0.006s
W20260709 12:26:22.713117  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:22.712191  1800 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:22.716490  2419 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:22.725239  1802 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.855s	user 0.014s	sys 0.000s
I20260709 12:26:22.726244  1802 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:22.730991  1872 raft_consensus.cc:2468] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 33.
I20260709 12:26:22.712321 11469 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:22.710242 11358 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:22.750016  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.747282  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:22.729714 11516 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:22.742705  1697 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.140.68:37665 timed out after 1.884s (SENT) [suppressed 19 similar messages]
I20260709 12:26:22.756238  1872 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:22.618842 11668 consensus_queue.cc:260] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:22.761722  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84827.8us
I20260709 12:26:22.746165 11558 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:26:22.764190  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 117512us
W20260709 12:26:22.767385 11681 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.289s	user 0.014s	sys 0.014s
I20260709 12:26:22.758404 11700 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:22.770876  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:22.772634  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:22.773829  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:22.774856 11699 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:22.775753 11699 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:22.776183 11699 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:22.776595 11699 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:22.778466 11690 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:22.765951 11558 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:22.787808 11558 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:22.793430  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:22.797089  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:22.768213 11681 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.290s	user 0.015s	sys 0.014s
I20260709 12:26:22.800402  1771 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:26:22.802443  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 165174us
I20260709 12:26:22.801383 11668 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:22.811786 11349 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:22.802213 11283 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:22.809424  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:22.821614  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.825268  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3322) took 1064 ms. Trace:
I20260709 12:26:22.825387  1872 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:22.822988  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3315) took 2050 ms. Trace:
I20260709 12:26:22.825850  1879 rpcz_store.cc:276] 0709 12:26:21.761280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.216699 (+455419us) service_pool.cc:224] Handling call
0709 12:26:22.825011 (+608312us) spinlock_profiling.cc:243] Waited 324 ms on lock 0x7b5400116f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:22.825183 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":745153280}
I20260709 12:26:22.828900  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1437) took 3767 ms. Trace:
I20260709 12:26:22.829331  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.830382  1730 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:48886 (request call id 1623) took 1326 ms. Trace:
I20260709 12:26:22.830736  1730 rpcz_store.cc:276] 0709 12:26:21.509004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.827254 (+1318250us) service_pool.cc:224] Handling call
0709 12:26:22.830285 (+  3031us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:22.833863  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:22.834961 11672 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:22.836170  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:22.829418  1807 rpcz_store.cc:276] 0709 12:26:18.998497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:19.630994 (+632497us) service_pool.cc:224] Handling call
0709 12:26:19.631406 (+   412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:19.787324 (+155918us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:26:20.167069 (+379745us) log.cc:844] Serialized 14 byte log entry
0709 12:26:21.089303 (+922234us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:21.089446 (+   143us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:21.089620 (+   174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.762132 (+1672512us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b5400066654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:22.762208 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:22.765621 (+  3413us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":573605,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":83039,"raft.run_cpu_time_us":551,"raft.run_wall_time_us":550,"spinlock_wait_cycles":357893376,"thread_start_us":1297874,"threads_started":2,"wal-append.queue_time_us":1094545,"wal-append.run_cpu_time_us":2673,"wal-append.run_wall_time_us":1166278}
I20260709 12:26:22.839215  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:22.842851 11440 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:22.843554 11698 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.844288 11698 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:22.845213 11727 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:22.845788 11727 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:22.846164 11727 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:22.846629 11735 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:22.848909  1770 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:22.854588  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55305.2us
I20260709 12:26:22.826061  1873 rpcz_store.cc:276] 0709 12:26:20.772732 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.753941 (+981209us) service_pool.cc:224] Handling call
0709 12:26:21.754820 (+   879us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:21.756147 (+  1327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:21.756174 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:21.756862 (+   688us) log.cc:844] Serialized 52 byte log entry
0709 12:26:22.090680 (+333818us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:22.429653 (+338973us) log.cc:844] Serialized 14 byte log entry
0709 12:26:22.430284 (+   631us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.430401 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:22.670710 (+240309us) spinlock_profiling.cc:243] Waited 6.87 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:22.822580 (+151870us) raft_consensus.cc:1685] finished
0709 12:26:22.822726 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:22.822923 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":448479,"prepare.run_cpu_time_us":323,"prepare.run_wall_time_us":3647,"raft.queue_time_us":3495,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":7645,"spinlock_wait_cycles":15797120,"thread_start_us":568197,"threads_started":2,"wal-append.queue_time_us":508530}
I20260709 12:26:22.856449  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.857250  1873 raft_consensus.cc:2393] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 24: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:22.857892  1880 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3303) took 2242 ms (client timeout 2174 ms). Trace:
I20260709 12:26:22.858112  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:22.858456  1880 rpcz_store.cc:269] 0709 12:26:20.615467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.652090 (+ 36623us) service_pool.cc:224] Handling call
0709 12:26:22.857600 (+2205510us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54002df0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:22.857785 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2382015872}
I20260709 12:26:22.859447 11526 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:22.859808  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.860457  1878 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:26:22.861267  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.861792  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1791) took 1237 ms. Trace:
I20260709 12:26:22.862123  1880 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:22.859892 11526 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:22.864611 11526 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:22.872803  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:22.866063  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:22.899967  1730 raft_consensus.cc:1240] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:22.862206  2417 rpcz_store.cc:276] 0709 12:26:21.624921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.625094 (+   173us) service_pool.cc:224] Handling call
0709 12:26:21.667614 (+ 42520us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:22.268182 (+600568us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:22.861502 (+593320us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54005f28d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:22.861726 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1297661184}
I20260709 12:26:22.867892  1728 raft_consensus.cc:1240] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:22.901428  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.901793  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.902611  1872 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 25: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
I20260709 12:26:22.866328 11526 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:22.871778  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:42636 (request call id 1448) took 1440 ms. Trace:
I20260709 12:26:22.904464  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:22.905416  1724 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:22.906356  1732 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:22.907124  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.885268  1873 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:22.873401  1804 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:22.872362  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:22.902542  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.900069  1727 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:22.909789  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3307) took 2212 ms (client timeout 2160 ms). Trace:
I20260709 12:26:22.919485  2419 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:22.908773  1799 rpcz_store.cc:276] 0709 12:26:21.431016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.439808 (+  8792us) service_pool.cc:224] Handling call
0709 12:26:22.871490 (+1431682us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b5400166cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:22.871678 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":627555712}
I20260709 12:26:22.921257  1729 raft_consensus.cc:2393] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 075704438992440697732181e49ccd09 in this term.
W20260709 12:26:22.915409  1876 rpcz_store.cc:269] 0709 12:26:20.697093 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.172662 (+475569us) service_pool.cc:224] Handling call
0709 12:26:22.909530 (+1736868us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b5400074254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:22.909692 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1303806336}
I20260709 12:26:22.925545  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.939253  1880 raft_consensus.cc:2393] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:22.824036 11283 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:22.941040  1732 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:22.941807  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.943718  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.942670  1728 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:22.928316 11672 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.949515  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 27 index: 12 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.953002  1725 raft_consensus.cc:2410] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 33 because replica has last-logged OpId of term: 32 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 12.
I20260709 12:26:22.954747  1731 raft_consensus.cc:2410] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
W20260709 12:26:22.955001  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 212345us
W20260709 12:26:22.955387  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98446.6us
I20260709 12:26:22.959971  1727 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:22.962577  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:22.963753  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:22.962689  1879 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
I20260709 12:26:22.964453  1873 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:22.921171 11349 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:22.968523  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.966213  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.947633 11283 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:22.973091  1730 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:26:22.948423 11736 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:22.977453  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:22.979561  1876 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
W20260709 12:26:22.983775  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 117827us
I20260709 12:26:22.967005 11735 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:22.961333 11007 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:22.985616  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:22.992087  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 143254us
I20260709 12:26:22.989061  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:22.992311  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:22.990626  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:22.994935  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:22.961112 11440 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:26:22.997764  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:22.963043 11672 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:22.974649 11516 consensus_queue.cc:1059] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
I20260709 12:26:23.008055 11630 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:23.008491 11349 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.014830  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:23.026260  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.031958  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 4183) took 2321 ms. Trace:
I20260709 12:26:23.035535  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:23.037396  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:23.057772  1874 rpcz_store.cc:276] 0709 12:26:20.710218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.343811 (+633593us) service_pool.cc:224] Handling call
0709 12:26:23.031710 (+1687899us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b54000f0854. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:23.031865 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1023992192}
I20260709 12:26:23.053136  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.026124 11719 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.087512 11719 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.088215 11555 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.089712 11719 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:22.999545 11736 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.092314 11736 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:23.092576  2419 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1788) took 1569 ms. Trace:
I20260709 12:26:23.092942  2419 rpcz_store.cc:276] 0709 12:26:21.523541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.529538 (+  5997us) service_pool.cc:224] Handling call
0709 12:26:23.092360 (+1562822us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540046a354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:23.092491 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2642194176}
I20260709 12:26:23.096221  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.097474  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.097285 11469 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.099159  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.099519 11349 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.099828  1729 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:23.100087  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:23.100929  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.084008  1804 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3284) took 1216 ms. Trace:
I20260709 12:26:23.082530  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:23.101528  1724 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:26:23.099184  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.065675  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.082530  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.098696  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.104535  1732 raft_consensus.cc:2393] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 33: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:23.107204  1725 raft_consensus.cc:2393] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:23.102327  1804 rpcz_store.cc:276] 0709 12:26:21.867837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:22.531308 (+663471us) service_pool.cc:224] Handling call
0709 12:26:22.873182 (+341874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:22.928782 (+ 55600us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.083737 (+154955us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b5400166cd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:23.083929 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":755689728}
W20260709 12:26:23.055491  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.108393  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.052387  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.033378  1731 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:23.079773  1879 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:23.111168  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.104396  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:23.114342  1874 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:23.115913  1728 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:23.114555  1724 raft_consensus.cc:2410] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
W20260709 12:26:23.037053  2410 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1631) took 2511 ms (client timeout 1884 ms). Trace:
W20260709 12:26:23.118206  2410 rpcz_store.cc:269] 0709 12:26:20.520632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.522908 (+  2276us) service_pool.cc:224] Handling call
0709 12:26:22.991869 (+2468961us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b5400487454. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:23.031412 (+ 39543us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1112521984}
I20260709 12:26:23.121011  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:23.038682 11554 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:23.122926  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.050717 11630 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.082826 11007 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:23.134151  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:23.134904  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 125540us
I20260709 12:26:23.032250 11735 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:23.041066 11583 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:23.144232  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 108581us
I20260709 12:26:23.112920 11593 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:23.152683  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:23.118700 11596 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:23.164976  2419 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:23.156190  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:23.158272  1844 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:23.141608 11719 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.018s
I20260709 12:26:23.159492 11469 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.160074  2383 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:23.160807 11738 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.170174  1872 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 24.
I20260709 12:26:23.154109  1876 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:23.174078  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:23.166016  1873 raft_consensus.cc:2468] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:23.166399  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.045974 11558 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.133651 11736 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.135484 11630 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:23.042068 11271 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.155s
I20260709 12:26:23.112325 11488 consensus_queue.cc:1059] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:26:23.154886 11593 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.169668  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:23.124641 11349 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:26:23.191529  1731 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
W20260709 12:26:23.165500  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 167834us
I20260709 12:26:23.189165 11438 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:23.194876  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:23.177512 11171 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:23.199934  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:23.200929  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.174466 11727 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.189488 11700 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:26:23.187947 11593 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:23.214309  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:23.193910 11719 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:23.218634  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.198108 11438 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.230890 11698 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.235546 11672 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:23.183214 11736 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.239148 11736 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:23.240176 11735 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:23.245566 11741 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.103s	user 0.001s	sys 0.000s
I20260709 12:26:23.203923 11668 raft_consensus.cc:2804] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:23.199285 11521 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.247946  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.279778  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:23.282500  1873 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:26:23.284382  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3314) took 2548 ms. Trace:
I20260709 12:26:23.294247  1877 rpcz_store.cc:276] 0709 12:26:20.736270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:21.739240 (+1002970us) service_pool.cc:224] Handling call
0709 12:26:21.832013 (+ 92773us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:22.205236 (+373223us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:22.205272 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:22.280056 (+ 74784us) log.cc:844] Serialized 26 byte log entry
0709 12:26:22.527314 (+247258us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:22.527366 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.283806 (+756440us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b540021acd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:23.283878 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:23.283950 (+    72us) raft_consensus.cc:1685] finished
0709 12:26:23.283980 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:23.284297 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":194293,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":142,"spinlock_wait_cycles":839602432,"thread_start_us":224108,"threads_started":1,"wal-append.queue_time_us":306385}
I20260709 12:26:23.253067  1728 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:23.177198 11738 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.225494 11719 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:23.217037 11700 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:23.248238 11683 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.207101 11171 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:23.274940 11724 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:23.279831 11681 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.280521 11668 raft_consensus.cc:697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 33 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:26:23.309074  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 90525.4us
W20260709 12:26:23.285835 11740 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
I20260709 12:26:23.215438 11007 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.097s
W20260709 12:26:23.249024 11680 consensus_meta.cc:287] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.555s	user 0.007s	sys 0.009s
W20260709 12:26:23.312508  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 211547us
I20260709 12:26:23.315271 11719 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:23.316461  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 152242us
I20260709 12:26:23.318298  1876 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:23.318554  1874 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 26.
I20260709 12:26:23.282478 11438 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:23.274282 11558 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.369755  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:23.342842 11171 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:23.342401 11683 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:23.311932 11699 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.535s	user 0.007s	sys 0.007s
I20260709 12:26:23.351004  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:23.278153 11737 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.229s	user 0.001s	sys 0.000s
I20260709 12:26:23.314334  1881 raft_consensus.cc:1240] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:23.391836  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3321) took 1631 ms. Trace:
I20260709 12:26:23.392354  1881 rpcz_store.cc:276] 0709 12:26:21.760932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.300941 (+1540009us) service_pool.cc:224] Handling call
0709 12:26:23.301300 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.391469 (+ 90169us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.391763 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:23.393857  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.395546  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.358771  1772 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:23.362017  1845 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:23.357286  1844 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:23.304900  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3320) took 1532 ms. Trace:
I20260709 12:26:23.397981 11522 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:23.254712 11744 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:23.399173 11744 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.401623 11744 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.364019 11680 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.403692 11680 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.407159  1876 raft_consensus.cc:1275] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:26:23.286214 11521 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.417075  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4169) took 2802 ms. Trace:
I20260709 12:26:23.417603  1875 rpcz_store.cc:276] 0709 12:26:20.614357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:20.614612 (+   255us) service_pool.cc:224] Handling call
0709 12:26:20.615036 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:22.303141 (+1688105us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.416647 (+1113506us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540021c0d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:23.416949 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2867595648}
I20260709 12:26:23.427636 11690 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:23.381508 11681 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.383651 11748 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:26:23.384603  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 245281us
I20260709 12:26:23.342653 11724 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:23.397725  1873 rpcz_store.cc:276] 0709 12:26:21.760549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.281587 (+1521038us) service_pool.cc:224] Handling call
0709 12:26:23.282023 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.292872 (+ 10849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.293195 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:23.371157 11271 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:23.430856  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 73669.9us
I20260709 12:26:23.380582 11440 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:23.432178 11349 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:23.433280 11668 consensus_queue.cc:237] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.435055 11746 raft_consensus.cc:3055] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:26:23.435640  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54308.3us
I20260709 12:26:23.438997 11700 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:23.440464  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89659.1us
W20260709 12:26:23.441522  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 82855.9us
I20260709 12:26:23.442260 11742 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:23.443809 11738 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:23.357743 11584 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:23.444450 11747 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:23.445700 11683 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:23.446008 11727 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.397195  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:23.414404 11521 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.521010 11521 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:26:23.521704 11724 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.488176 11283 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.490990  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.488032  1771 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:23.493705 11349 raft_consensus.cc:697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:23.491967  1772 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:23.496415  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 139596us
I20260709 12:26:23.526263  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.486649 11746 raft_consensus.cc:740] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:26:23.499047  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 137152us
I20260709 12:26:23.452057 11690 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:23.531032  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:23.429929 11699 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.482717 11698 raft_consensus.cc:2804] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:23.511173 11584 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.519083  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:23.516733 11700 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:26:23.519217  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.431169 11398 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:23.529327  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:23.554661  2411 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:23.493278 11744 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:26:23.539594  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48684.6us
I20260709 12:26:23.558094  1726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.556456  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:23.559083  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.563249 11630 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.565923 11630 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:23.395538 11736 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:23.527346 11554 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:23.553602  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:23.547701 11526 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.626s	user 0.009s	sys 0.004s
W20260709 12:26:23.571077  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 79169.4us
I20260709 12:26:23.576041  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:23.574193  2419 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:26:23.579042 11521 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:23.579972 11698 raft_consensus.cc:697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:23.580559  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 124888us
I20260709 12:26:23.581420  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.581130 11698 consensus_queue.cc:237] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.582693  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.590688  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.592182  1845 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:23.593988  1728 raft_consensus.cc:1240] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:23.575115  1726 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:23.585371  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:23.597149  1732 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:23.598251  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.599014  1725 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:23.582613 11747 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:23.599103 11746 consensus_queue.cc:260] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.600446 11596 raft_consensus.cc:3055] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Stepping down as leader of term 28
W20260709 12:26:23.600795  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.552983 11672 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:23.603104 11672 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.603499 11672 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:23.611888  1730 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:23.583217 11690 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.614601 11690 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:23.579072 11271 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:23.616622 11271 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:23.617695 11271 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:23.621932  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64728us
I20260709 12:26:23.622735  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:23.624183 11516 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:23.624656 11516 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.624982 11516 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:23.625445 11283 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:23.627059 11349 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.573212 11742 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:23.551313 11583 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:23.632072  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.632481 11521 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:23.588344 11681 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:23.634467  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.589067 11683 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:26:23.586608 11593 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.635399 11683 raft_consensus.cc:697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:23.636574 11683 consensus_queue.cc:237] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.637233 11593 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.584977 11584 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:23.639109  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.639310  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 82935.1us
I20260709 12:26:23.641098 11753 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:23.641728 11753 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.600914 11596 raft_consensus.cc:740] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:23.642179 11753 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:23.643976  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 246892us
I20260709 12:26:23.644956  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:26:23.650097  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 74176.7us
I20260709 12:26:23.643592 11596 consensus_queue.cc:260] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.631263 11699 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:23.652386  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62585.2us
W20260709 12:26:23.652849  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.659511  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 74227.9us
I20260709 12:26:23.660598  1772 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:23.600852 11746 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:23.684923  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:23.686132  1727 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.687444  2419 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:23.689471  1727 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:23.689663  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 97577.4us
W20260709 12:26:23.687430  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 104803us
I20260709 12:26:23.690593  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:23.691284  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.692689  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:23.694494  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.664229 11738 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:23.713142  1696 leader_election.cc:400] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23.
I20260709 12:26:23.716369  1804 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:23.718127  1728 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.708118 11398 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.021s
I20260709 12:26:23.685493 11358 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:23.741842  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:23.742172 11358 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.742576  1798 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:23.742709 11358 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:23.743882  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.741725 11526 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:23.739990 11742 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:23.747499 11526 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.747632 11742 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.750207 11742 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:23.739778 11558 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:23.718750 11584 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:23.751304 11558 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.753688 11558 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:23.751722  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4194) took 2138 ms. Trace:
I20260709 12:26:23.750029 11526 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:23.754290  1879 rpcz_store.cc:276] 0709 12:26:21.578380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.243534 (+1665154us) service_pool.cc:224] Handling call
0709 12:26:23.243988 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.244270 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:26:23.490102 (+245832us) log.cc:844] Serialized 14 byte log entry
0709 12:26:23.715885 (+225783us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:23.715969 (+    84us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:23.716051 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:23.716166 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:23.716351 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28695,"prepare.queue_time_us":250350,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":28859,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":127,"thread_start_us":389174,"threads_started":2}
I20260709 12:26:23.755264 11398 consensus_queue.cc:1059] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:26:23.739127  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.742256 11699 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:23.763563 11699 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:23.693872 11554 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:23.764700 11554 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:23.765797 11699 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:23.762820  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.767822  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.768306  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.765738 11554 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.775499  2419 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:23.741434  1875 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:23.775676  1798 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:23.659682 11621 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:23.685909 11748 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:23.664436 11563 raft_consensus.cc:2804] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:23.719050  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:23.714613  1724 raft_consensus.cc:2368] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29.
I20260709 12:26:23.777807  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:23.777534  1697 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:23.718815  1728 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
W20260709 12:26:23.785764  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 100122us
I20260709 12:26:23.788166  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:23.786711  2382 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:26:23.798131  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 85113.1us
I20260709 12:26:23.738467 11596 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:23.694173 11752 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:26:23.798120  1771 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:23.715485 11744 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:23.713722 11738 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:23.693490 11593 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.101s
I20260709 12:26:23.839951  1770 leader_election.cc:400] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29.
I20260709 12:26:23.842628  1770 leader_election.cc:403] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:26:23.824955  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:23.843353  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:23.844609 11668 raft_consensus.cc:3055] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:23.845486 11668 raft_consensus.cc:740] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:23.782125 11621 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:23.785663 11748 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:23.852165  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:23.688783 11681 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.107s
I20260709 12:26:23.853466  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:23.793330 11747 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:23.857386  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:23.716576 11438 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.847585 11668 consensus_queue.cc:260] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:23.799326 11743 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:26:23.902977  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 217069us
I20260709 12:26:23.842677 11738 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:23.906603  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 62561.4us
I20260709 12:26:23.915925  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:23.857635 11742 raft_consensus.cc:2804] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:26:23.937500  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.855979 11440 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:23.916930  1801 raft_consensus.cc:2393] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:23.945438  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:23.947834  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:23.875526 11748 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:23.876183   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:26:23.959136  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:23.959904  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.871973 11756 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:23.906038 11736 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:23.917250 11007 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:26:23.978953  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 58979.6us
I20260709 12:26:23.952018 11440 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:23.872037 11621 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:23.909960 11735 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:23.870568 11558 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:23.952527 11668 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:23.996601  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:23.910226 11469 consensus_queue.cc:1048] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.081s
I20260709 12:26:23.985160  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:24.006433  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 70863.5us
I20260709 12:26:24.022716  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:23.917377 11743 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 29 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:26:24.046643  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 259977us
I20260709 12:26:24.051326  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:24.048429  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:23.993156 11440 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:23.989311 11742 raft_consensus.cc:697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:23.999966 11007 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:24.063380  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:24.064229  1801 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:24.064599  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:24.065305  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:24.060127  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:23.962970 11358 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.094395  1879 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:24.097631  1874 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:24.113371  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:23.987489 11747 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.086s
I20260709 12:26:24.058315 11488 raft_consensus.cc:2804] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:24.052374 11558 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:24.116284  1804 raft_consensus.cc:1240] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:24.123201  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.145251  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:24.146087  1880 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 22.
W20260709 12:26:24.146874  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 239236us
W20260709 12:26:24.150185  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 98933.7us
I20260709 12:26:24.167711  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:24.168678  1875 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:24.177171  1880 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
W20260709 12:26:24.189244  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.169754 11516 consensus_meta.cc:287] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.544s	user 0.006s	sys 0.011s
I20260709 12:26:24.141511 11736 raft_consensus.cc:697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:26:24.212702 11719 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.821s	user 0.010s	sys 0.004s
I20260709 12:26:24.207487  1879 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:24.215643 11719 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.217013  1877 raft_consensus.cc:1240] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:24.202354 11764 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:24.190511 11765 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.274s	user 0.002s	sys 0.000s
I20260709 12:26:24.223425  1801 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: []
W20260709 12:26:24.212805  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 149702us
W20260709 12:26:24.237639  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89080us
I20260709 12:26:24.218413 11719 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:24.244313  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:24.250140  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:24.149860 11777 consensus_queue.cc:1048] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.084s
I20260709 12:26:24.246018 11283 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.219542 11516 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.267647  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:24.275228  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.275612 11724 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:24.277595 11621 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.278057 11777 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:24.278437 11488 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:24.278970 11488 raft_consensus.cc:3060] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:24.285392 11763 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
W20260709 12:26:24.285455 11767 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20260709 12:26:24.286500  2419 raft_consensus.cc:1240] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:24.243690 11007 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.291332  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.262674  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:24.292243  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:24.289630 11760 log.cc:927] Time spent T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:26:24.253981 11438 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:24.297389  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.299609  1798 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
W20260709 12:26:24.268539 11753 consensus_meta.cc:287] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.625s	user 0.012s	sys 0.000s
W20260709 12:26:24.302243  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 52207.7us
I20260709 12:26:24.200932 11743 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:24.306656  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44064us
W20260709 12:26:24.299095 11759 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.299s	user 0.000s	sys 0.002s
I20260709 12:26:24.311388  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:24.313067 11779 raft_consensus.cc:2764] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
W20260709 12:26:24.313560 11766 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:26:24.242223 11742 consensus_queue.cc:237] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.322381  2383 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:24.317641 11700 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.758s	user 0.010s	sys 0.000s
I20260709 12:26:24.305498 11283 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.318372 11621 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:24.319384  1806 raft_consensus.cc:2410] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:26:24.348470  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.350140  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.351313  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:24.352037  1800 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:24.352744 11745 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:24.353199 11745 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:24.353590  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1475) took 1713 ms. Trace:
W20260709 12:26:24.335003 11768 log.cc:927] Time spent T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
I20260709 12:26:24.264014 11558 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.354902 11683 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:24.355456 11683 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:24.344847 11516 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:24.346318 11171 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.900s	user 0.006s	sys 0.006s
I20260709 12:26:24.347843  1799 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 21. Current term is 22. Ops: []
I20260709 12:26:24.328949 11751 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:24.357308 11751 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:24.357714 11751 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:24.358981 11171 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.360225 11583 raft_consensus.cc:3055] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:24.355093  1802 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:24.356446 11683 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.361637 11683 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:24.363004  1872 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:24.353986  1877 rpcz_store.cc:276] 0709 12:26:22.640631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.333185 (+692554us) service_pool.cc:224] Handling call
0709 12:26:23.368903 (+ 35718us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.217640 (+848737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.353277 (+135637us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b5400223654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:24.353473 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1949677312}
W20260709 12:26:24.367008  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44466us
I20260709 12:26:24.360790 11583 raft_consensus.cc:740] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:24.352283  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.309500 11753 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.373135  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:24.363917  1804 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:24.343137  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.307439 11736 consensus_queue.cc:237] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.376021  1801 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:26:24.413920  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 632210us
I20260709 12:26:24.414480  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1476) took 1714 ms. Trace:
I20260709 12:26:24.376492  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.404659  2411 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 27: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:24.405544  1807 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
W20260709 12:26:24.417204  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49473.7us
W20260709 12:26:24.408077  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 236519us
W20260709 12:26:24.408561  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 97310.8us
I20260709 12:26:24.376698  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.411763  1729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:24.339354 11358 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:24.412700  1798 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:24.415223  1732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:24.422935  1732 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:24.414983  1881 rpcz_store.cc:276] 0709 12:26:22.700465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.395769 (+695304us) service_pool.cc:224] Handling call
0709 12:26:23.396077 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.396476 (+   399us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:23.501822 (+105346us) log.cc:844] Serialized 14 byte log entry
0709 12:26:23.778118 (+276296us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.288959 (+510841us) spinlock_profiling.cc:243] Waited 9.68 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:24.413632 (+124673us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b1800369400. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:26:24.414039 (+   407us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.414197 (+   158us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:24.414234 (+    37us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:24.414281 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.414341 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.414425 (+    84us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2547,"prepare.queue_time_us":98364,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":3066,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":11203,"spinlock_wait_cycles":33255552,"thread_start_us":322944,"threads_started":2,"wal-append.queue_time_us":621269}
I20260709 12:26:24.412456  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:24.425634  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:24.350077 11283 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:24.376916 11171 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:24.389096 11786 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:24.429379 11787 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:24.442152  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:24.429620  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.432873  1730 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:24.434233  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09"
I20260709 12:26:24.435681  1723 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:24.411931 11672 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.808s	user 0.010s	sys 0.003s
I20260709 12:26:24.437518  2417 raft_consensus.cc:2368] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29.
I20260709 12:26:24.436748  1724 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:24.439016  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.443532  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.449554  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.444056  1723 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 30.
I20260709 12:26:24.450372  2382 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:24.426512 11756 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer 075704438992440697732181e49ccd09 (127.0.140.65:35067) with higher term. Message: Invalid argument: T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29.
I20260709 12:26:24.420850 11563 raft_consensus.cc:697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:24.436681 11593 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:24.383600 11440 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:24.448604 11690 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.833s	user 0.010s	sys 0.003s
I20260709 12:26:24.468899  1798 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:24.469288  1844 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:24.446074 11555 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.441740 11425 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.470978  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:24.377035 11717 consensus_queue.cc:1059] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.162s
I20260709 12:26:24.445734 11584 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.475622 11621 raft_consensus.cc:2804] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:24.482828 11584 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.483480 11621 raft_consensus.cc:697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:24.483523 11425 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.484753 11621 consensus_queue.cc:237] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.485884 11584 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:24.486958 11782 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 12:26:24.487102 11522 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:24.476781  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.377480 11583 consensus_queue.cc:260] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.488566 11522 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:26:24.488930  1800 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:24.489182 11555 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.489475 11563 consensus_queue.cc:237] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.354192 11745 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.491370 11745 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:24.458103 11672 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.476126  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:24.459096 11526 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:24.494581  1802 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:24.535171  1798 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:26:24.539727  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:24.545944  1726 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:24.535391  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:24.477730 11746 consensus_meta.cc:287] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.788s	user 0.012s	sys 0.000s
I20260709 12:26:24.469921 11764 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:24.548916  1725 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:24.552462  1725 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:26:24.549937  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:24.502954  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76701.2us
I20260709 12:26:24.553790  1800 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:24.548916  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
W20260709 12:26:24.553983  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 88013.7us
I20260709 12:26:24.470180 11786 raft_consensus.cc:697] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:24.557806  1874 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:24.555626  1807 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 31: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:24.493629 11699 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:24.501484  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet def0036446324543ba07e132e885133a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:24.488991 11522 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:24.498251 11690 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:24.545682 11773 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:24.536090 11727 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:24.577958  1615 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 76555.2us
I20260709 12:26:24.583855  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:24.586733  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:24.468109 11787 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.546057 11521 consensus_queue.cc:1059] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 12:26:24.565673 11584 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:24.574169 11776 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.246s	user 0.000s	sys 0.002s
I20260709 12:26:24.586196 11425 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:24.640407  1731 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "075704438992440697732181e49ccd09" is_pre_election: true
I20260709 12:26:24.633064  1800 raft_consensus.cc:2468] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:24.548954 11526 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:26:24.638149  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 51443us
W20260709 12:26:24.644937  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 77495.1us
I20260709 12:26:24.645784  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:24.575300 11690 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:24.579473 11773 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:24.646979  1731 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:24.654388  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.661594  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:24.568921 11699 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:24.647418 11748 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.600s	user 0.010s	sys 0.003s
I20260709 12:26:24.646108 11668 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:26:24.682565  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 98783.1us
I20260709 12:26:24.670495  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.640231 11563 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:24.672467  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:24.664605 11398 consensus_queue.cc:1059] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.016s
I20260709 12:26:24.673455 11796 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:24.674250   561 tablet_server.cc:179] TabletServer@127.0.140.65:0 shutting down...
I20260709 12:26:24.674841 11440 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:24.596977 11751 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.675051 11800 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.083s
I20260709 12:26:24.682652 11668 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:24.681661 11775 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.366s	user 0.000s	sys 0.002s
I20260709 12:26:24.669263  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:24.654717 11773 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:24.570461 11792 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.098s
I20260709 12:26:24.588598 11736 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.071s
W20260709 12:26:24.686373  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.590008 11727 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:24.687556 11699 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:24.749351  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63117.6us
I20260709 12:26:24.687493 11764 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.052s
I20260709 12:26:24.656590 11425 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:24.728329  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:24.649786 11690 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:24.642745 11584 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.685767 11555 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:24.741515  2413 raft_consensus.cc:1240] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:26:24.686555  1732 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
W20260709 12:26:24.753592  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.736950 11668 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:24.758173  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 87129.6us
I20260709 12:26:24.764055  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.774087  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:24.765254 11698 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.017s	sys 0.006s
I20260709 12:26:24.771276  1770 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:24.686726 11748 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.781265 11787 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:24.782047 11738 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 0.835s	user 0.013s	sys 0.000s
I20260709 12:26:24.783006 11440 consensus_queue.cc:1059] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:24.783994 11440 consensus_queue.cc:1243] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [LEADER]: Peer a20e99e125b84513a44e88860667ed61 log is divergent from this leader: its last log entry 26.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:26:24.784695 11440 raft_consensus.cc:3055] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:24.785095 11440 raft_consensus.cc:740] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:24.787318  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:24.787588  1881 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:26:24.788655  1878 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:26:24.789031  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:24.789763  1877 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:24.697002 11786 consensus_queue.cc:237] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.796823 11727 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:26:24.798712  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4193) took 3223 ms. Trace:
I20260709 12:26:24.801245 11584 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:24.801486  1872 rpcz_store.cc:276] 0709 12:26:21.577910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.192325 (+1614415us) service_pool.cc:224] Handling call
0709 12:26:23.192742 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.363444 (+1170702us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.798380 (+434936us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540001efd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:24.798629 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2690857216}
I20260709 12:26:24.802240  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.800861  2413 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:24.803267  2415 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 31: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:24.804265  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.804308  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:24.805259  2413 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:24.805917  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.806661  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:24.806986 11283 raft_consensus.cc:2804] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:24.807771 11283 raft_consensus.cc:697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:24.782611 11738 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.809377  2414 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:24.808929 11283 consensus_queue.cc:237] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:24.786141 11440 consensus_queue.cc:260] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.811857 11440 raft_consensus.cc:3060] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:24.812141 11724 consensus_meta.cc:287] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.536s	user 0.012s	sys 0.000s
I20260709 12:26:24.811079  1805 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
W20260709 12:26:24.817006  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.783762 11787 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:26:24.777062 11698 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.018s	sys 0.006s
I20260709 12:26:24.832715  1877 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:26:24.834575  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:24.836951  1877 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:24.838908  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:24.840665  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.841499  2419 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 31.
I20260709 12:26:24.841361  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:24.824980 11751 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:24.826929 11803 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 12:26:24.813585 11488 consensus_meta.cc:287] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.534s	user 0.012s	sys 0.000s
I20260709 12:26:24.829290  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:24.825744  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:24.831467 11736 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:26:24.823063  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:24.777262 11672 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:24.847023  1728 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:24.848766  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:24.857380  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.859318  1872 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:24.830188  1875 raft_consensus.cc:1275] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:26:24.860656  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:24.863034  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:24.864199  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:24.920687  1800 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:24.917150  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:24.925190  1696 connection.cc:256] RPC call timeout handler was delayed by -1.9064567199998237s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
W20260709 12:26:24.925712  1696 connection.cc:256] RPC call timeout handler was delayed by -1.8929567199998019s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15980000000000003s for a response
W20260709 12:26:24.926275  1696 connection.cc:256] RPC call timeout handler was delayed by -1.8146567199996753s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1685s for a response
W20260709 12:26:24.926765  1696 connection.cc:256] RPC call timeout handler was delayed by -1.7678567200000543s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17370000000000002s for a response
I20260709 12:26:24.920476  2415 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
W20260709 12:26:24.927325  1696 connection.cc:256] RPC call timeout handler was delayed by -1.7012567199999467s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
W20260709 12:26:24.927673  1696 connection.cc:256] RPC call timeout handler was delayed by -1.6355567199998404s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
W20260709 12:26:24.927951  1696 connection.cc:256] RPC call timeout handler was delayed by -1.5266567199996643s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2005s for a response
W20260709 12:26:24.928141  1696 connection.cc:256] RPC call timeout handler was delayed by -1.5095567199996367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20240000000000002s for a response
W20260709 12:26:24.928330  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4933567200000653s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2042s for a response
W20260709 12:26:24.928504  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4825567200000478s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2054s for a response
W20260709 12:26:24.928689  1696 connection.cc:256] RPC call timeout handler was delayed by -1.4555567200000041s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20840000000000003s for a response
W20260709 12:26:24.928854  1696 connection.cc:256] RPC call timeout handler was delayed by -1.3187567199997829s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22360000000000002s for a response
W20260709 12:26:24.930142  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69559.5us
I20260709 12:26:24.930925  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:24.931740  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:24.844439 11727 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:24.862581 11596 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:24.919922 11738 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:24.921841 11526 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:24.946604  1771 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:24.861896 11736 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:24.937238 11683 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:24.957050 11690 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.958927 11797 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:24.959981 11748 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:24.962795  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:24.964597  1772 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:24.971693  1875 raft_consensus.cc:1240] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:24.969678  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:24.867637 11488 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:24.979771  1872 raft_consensus.cc:2393] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:24.982553  1772 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:24.982471  1727 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4281) took 1221 ms. Trace:
I20260709 12:26:24.985203  1727 rpcz_store.cc:276] 0709 12:26:23.760889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.763421 (+  2532us) service_pool.cc:224] Handling call
0709 12:26:23.763983 (+   562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:23.764212 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:26:24.779369 (+1015157us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b180026cd40. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16af7d03c7 00007f16abde29a7 00007f16abdf20bf
0709 12:26:24.779488 (+   119us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.982104 (+202616us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:24.982160 (+    56us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:24.982210 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:24.982279 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:24.982399 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29460,"prepare.queue_time_us":152433,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":8742,"spinlock_wait_cycles":5026560,"thread_start_us":408206,"threads_started":2}
I20260709 12:26:24.986660  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:24.987402  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 171180us
I20260709 12:26:24.996695  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:24.976819 11727 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:24.999084 11736 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.001230 11630 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:25.002938 11805 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.178s	user 0.002s	sys 0.000s
I20260709 12:26:25.003188  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:24.975684 11683 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.013613  1879 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:25.016029  1875 raft_consensus.cc:1240] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [23.15-25.16]
I20260709 12:26:24.949441 11756 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
W20260709 12:26:25.031605  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46757.4us
I20260709 12:26:25.032769  1879 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:24.989331 11742 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:25.036100  1872 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
W20260709 12:26:25.037268  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 54795.1us
W20260709 12:26:25.040269  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 110992us
I20260709 12:26:25.040761  1772 leader_election.cc:400] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29.
I20260709 12:26:25.005461 11751 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:25.027235 11683 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:25.027866 11727 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:25.049715  1803 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:25.056049 11756 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:25.056710 11756 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:25.057659 11756 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.048477 11751 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.059520 11781 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:25.060709 11804 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.263s	user 0.000s	sys 0.002s
I20260709 12:26:25.061149 11751 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:25.039666 11630 consensus_queue.cc:1059] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 12:26:25.076771  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:25.077708  2419 raft_consensus.cc:3060] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:25.087975 11756 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:25.097273 11811 log.cc:927] Time spent T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:26:25.101609 11773 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.102944 11736 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:25.105352  2416 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:25.105742  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:25.107074 11727 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:25.093034  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48905.4us
I20260709 12:26:25.109583  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.091267 11822 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.123541  2419 raft_consensus.cc:2468] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:25.125443  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.126879 11751 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:25.128384  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:25.130086 11488 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:25.130115 11751 raft_consensus.cc:2804] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:25.131219 11751 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.131567 11751 raft_consensus.cc:3060] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:25.132112 11822 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.133245 11735 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:25.136089  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.139452 11488 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:25.140950 11690 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:25.134580 11822 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:25.153924  2413 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:26:25.156447  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:25.157490  2419 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:25.171015  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.176276  1772 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:25.177928  2419 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:25.179904  1772 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:25.183768  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.185715  2419 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:26:25.182514 11745 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:25.186501 11516 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:25.191272  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:25.180975 11630 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.197505  1806 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:25.219574 11630 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.217280 11717 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:26:25.221225 11717 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:25.222313 11717 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.223520  2411 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29. Ops: []
I20260709 12:26:25.212568 11773 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:25.175910 11797 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.214183  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.247347 11797 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.215049 11488 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:25.206130 11683 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.222308 11630 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:25.225804 11516 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:25.249744 11488 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.228611 11668 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:25.231648  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47953.6us
I20260709 12:26:25.234454  2419 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260709 12:26:25.235810 11789 log.cc:927] Time spent T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.919s	user 0.003s	sys 0.001s
I20260709 12:26:25.237829 11583 raft_consensus.cc:3060] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:25.256036  2383 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:25.251436 11683 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:25.260440 11787 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:25.260946 11738 raft_consensus.cc:2804] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:25.261674 11738 raft_consensus.cc:697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:25.262681 11738 consensus_queue.cc:237] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.252079  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3302) took 1004 ms. Trace:
I20260709 12:26:25.258291  2418 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:26:25.250576 11797 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:25.272763 11488 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:25.273654  2414 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:25.273901  2410 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:26:25.274260 11781 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:25.274833 11781 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:25.279814 11788 log.cc:927] Time spent T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.964s	user 0.002s	sys 0.008s
I20260709 12:26:25.250702 11516 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.283529 11522 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.285216 11522 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:25.287499  2415 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 28: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:25.289949 11668 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:25.265802  1801 rpcz_store.cc:276] 0709 12:26:24.248084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.248312 (+   228us) service_pool.cc:224] Handling call
0709 12:26:24.248626 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.415836 (+167210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.251729 (+835893us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54001676d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:25.251994 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":273962240}
I20260709 12:26:25.264413 11007 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:25.293473 11007 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:25.294042  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 4222) took 1281 ms. Trace:
I20260709 12:26:25.294548  1874 rpcz_store.cc:276] 0709 12:26:24.012307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.113014 (+100707us) service_pool.cc:224] Handling call
0709 12:26:25.293777 (+1180763us) spinlock_profiling.cc:243] Waited 443 ms on lock 0x7b5400260a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:25.293962 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1019696384}
I20260709 12:26:25.294586 11007 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.296837 11007 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:25.289757 11752 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:26:25.258558 11831 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:26:25.306843  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.309057  1801 raft_consensus.cc:3060] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:25.302561 11743 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.975s	user 0.010s	sys 0.021s
I20260709 12:26:25.315380  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.307811 11596 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.013s
I20260709 12:26:25.321381 11746 raft_consensus.cc:3055] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:25.322031 11746 raft_consensus.cc:740] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:25.304598 11554 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.022s
I20260709 12:26:25.326957 11746 consensus_queue.cc:260] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 27.21, Last appended by leader: 21, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.328876 11746 raft_consensus.cc:3060] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:25.313091 11488 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:26:25.329497  1881 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 32: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:25.311681 11787 consensus_queue.cc:1059] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.020s
W20260709 12:26:25.315366 11743 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s	user 0.011s	sys 0.021s
I20260709 12:26:25.352650 11797 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:25.353304 11797 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:25.354254 11797 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.358716 11751 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.368530 11753 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:25.339843 11668 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.370258 11668 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.370419 11751 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:25.372845 11668 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:25.383984  2416 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:26:25.386468  1801 raft_consensus.cc:2468] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:25.388080  2383 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:25.392023  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:25.392905  1805 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 23.
W20260709 12:26:25.394281  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 78969.2us
I20260709 12:26:25.395242  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.399401 11764 raft_consensus.cc:2804] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:25.400209 11764 raft_consensus.cc:697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:25.401432 11764 consensus_queue.cc:237] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.403676 11787 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:25.409576  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3300) took 1255 ms. Trace:
I20260709 12:26:25.389298 11752 consensus_queue.cc:1059] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 27.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:26:25.412621 11752 raft_consensus.cc:3055] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:26:25.413419  1770 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:25.413707 11752 raft_consensus.cc:740] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:26:25.416610  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:25.417145  1844 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.420208 11700 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:25.420821 11700 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.423205  1875 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:26:25.423833 11584 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.423092 11672 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.424398 11584 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.425019 11672 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.427299 11672 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:25.428491 11425 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:25.429267 11425 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:25.423375  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.431747  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:25.435765 11785 raft_consensus.cc:3055] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:25.436316 11785 raft_consensus.cc:740] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:25.412730  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4219) took 1401 ms. Trace:
I20260709 12:26:25.410115  1799 rpcz_store.cc:276] 0709 12:26:24.154064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.154272 (+   208us) service_pool.cc:224] Handling call
0709 12:26:24.190009 (+ 35737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.356956 (+166947us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.409309 (+1052353us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54003dbdd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:25.409510 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":346328832}
W20260709 12:26:25.440032  1770 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:25.438997  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.442629  1877 raft_consensus.cc:2468] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:25.437915  1844 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:25.439651  1876 rpcz_store.cc:276] 0709 12:26:24.011022 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.025240 (+ 14218us) service_pool.cc:224] Handling call
0709 12:26:24.061489 (+ 36249us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:26:24.539570 (+478081us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:24.539604 (+    34us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:26:24.540459 (+   855us) log.cc:844] Serialized 78 byte log entry
0709 12:26:24.641558 (+101099us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:24.670359 (+ 28801us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.684048 (+ 13689us) log.cc:844] Serialized 14 byte log entry
0709 12:26:24.695891 (+ 11843us) log.cc:844] Serialized 14 byte log entry
0709 12:26:25.012423 (+316532us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:25.292622 (+280199us) spinlock_profiling.cc:243] Waited 3.5 ms on lock 0x7b1800197380. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb3f3 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:26:25.412084 (+119462us) spinlock_profiling.cc:243] Waited 3.3 ms on lock 0x7b1800197380. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:26:25.412296 (+   212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.412401 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:25.412469 (+    68us) raft_consensus.cc:1685] finished
0709 12:26:25.412484 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:25.412639 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":537851,"prepare.queue_time_us":79778,"prepare.run_cpu_time_us":154,"prepare.run_wall_time_us":23814,"raft.queue_time_us":1136833,"raft.run_cpu_time_us":482,"raft.run_wall_time_us":5005,"spinlock_wait_cycles":57191296,"thread_start_us":200707,"threads_started":2,"wal-append.queue_time_us":228744}
W20260709 12:26:25.449285  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:25.472991  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 54223.1us
I20260709 12:26:25.414175 11563 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:25.390270 11488 raft_consensus.cc:3055] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:25.468416  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:25.474566 11786 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.020s	sys 0.067s
I20260709 12:26:25.484825 11584 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:25.489552 11786 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.020s	sys 0.067s
I20260709 12:26:25.489835  1624 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 20 to 22, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 22 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:25.443167 11738 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.440896  1875 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:25.493001  1876 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
W20260709 12:26:25.494133  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 98976us
I20260709 12:26:25.464244 11751 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:26:25.507507  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:25.510025  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:25.516381  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:25.520514  1728 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57234 (request call id 4286) took 1573 ms. Trace:
I20260709 12:26:25.521139  1728 rpcz_store.cc:276] 0709 12:26:23.947613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:23.947806 (+   193us) service_pool.cc:224] Handling call
0709 12:26:23.986133 (+ 38327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:24.919997 (+933864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:25.520146 (+600149us) spinlock_profiling.cc:243] Waited 860 ms on lock 0x7b540044aa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:25.520408 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1979091840}
I20260709 12:26:25.522576  1876 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 16. (term mismatch)
I20260709 12:26:25.523104  1876 pending_rounds.cc:85] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 15
I20260709 12:26:25.514935 11738 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.523345  1876 pending_rounds.cc:107] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 26.16
I20260709 12:26:25.523598  1876 raft_consensus.cc:2889] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:25.524959  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.525723  1875 raft_consensus.cc:1275] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:25.484086 11752 consensus_queue.cc:260] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.526921 11785 consensus_queue.cc:260] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.528294 11785 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:25.533399 11752 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:25.535827 11700 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:25.536441  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet def0036446324543ba07e132e885133a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:25.537210  2417 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:25.485137 11488 raft_consensus.cc:740] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:25.541399 11488 consensus_queue.cc:260] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 14, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.554469 11488 raft_consensus.cc:3060] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:25.515403  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 842947us
I20260709 12:26:25.493724 11724 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:25.559264 11724 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:25.561503  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.562515  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:25.525010 11751 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:25.560542  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.525676  1876 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:25.573506  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.575525  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:25.576588  1806 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:25.577966  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.506253 11736 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:26:25.583274  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:25.597846  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.601080  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:25.585212  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.572480 11440 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.021s
I20260709 12:26:25.585716  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:25.609596  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 244537us
I20260709 12:26:25.563824 11724 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:25.585961  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:25.586597  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.586961 11717 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:26:25.587862  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.623548  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:25.623917  1800 raft_consensus.cc:2393] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 28: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:25.590975 11751 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.595106 11753 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:25.595582 11516 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:25.595474  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:25.627665  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.603058  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:25.604502  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:25.607836  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:25.614005  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.618140  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:25.625389  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.630879  1801 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 23: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:25.613122  1804 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 32.
W20260709 12:26:25.631163  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 149124us
I20260709 12:26:25.633440  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.634629  1879 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:26:25.635144 11779 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.599076 11736 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.637696  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:25.638232 11736 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:25.638120 11779 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.618217 11583 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:25.605913 11563 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:25.594918 11800 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:25.607300 11584 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:25.639801 11800 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.640383 11779 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:25.640980 11816 consensus_queue.cc:1048] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:25.642331 11800 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:25.642727  1804 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:25.609869 11593 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.033s
I20260709 12:26:25.594926 11783 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:26:25.636991 11787 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:25.564755 11700 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.071s
I20260709 12:26:25.608618 11683 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.040s
W20260709 12:26:25.648860  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:25.651351  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 48443.6us
I20260709 12:26:25.651998  1805 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:26:25.652318  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:25.653436  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:25.653851 11799 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:25.654114  1876 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:25.654302  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:25.654651 11799 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:26:25.655185  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47452.6us
I20260709 12:26:25.669687  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:25.655997 11584 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:25.655869 11171 consensus_queue.cc:1059] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:26:25.655267  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:25.671222  1879 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:25.656597 11563 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:25.670333  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.657613  1806 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:26:25.659125  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:25.663120  1803 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:25.678053  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:25.680083  1876 raft_consensus.cc:2468] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 23.
I20260709 12:26:25.681721  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.682343  1805 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:26:25.682101  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:25.673365  1696 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:25.685824  1879 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:25.689317  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:25.719861  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:25.669879 11283 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:25.726817  1772 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.713716  2417 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60934 (request call id 1710) took 1076 ms. Trace:
I20260709 12:26:25.690945 11800 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:25.696065 11683 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:25.695708 11596 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.707475 11717 consensus_queue.cc:1059] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 12:26:25.727895  2417 rpcz_store.cc:276] 0709 12:26:24.637782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.637924 (+   142us) service_pool.cc:224] Handling call
0709 12:26:25.713479 (+1075555us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b54007ce554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:25.713641 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2028230528}
W20260709 12:26:25.729790  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:25.731457  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49479.6us
I20260709 12:26:25.732563  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:25.755125  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.760022  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.819383  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:25.732973  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:25.739114  1800 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:25.764643  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:25.749749  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:25.819794  1881 raft_consensus.cc:2468] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:25.767652  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.766652  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:25.826934  1803 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:25.831033  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.832724  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:25.706970 11526 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:25.836839  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 104387us
I20260709 12:26:25.822392  1806 raft_consensus.cc:2468] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:25.839581  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:25.865895  1798 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:25.675767 11630 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.059s
I20260709 12:26:25.885677  2411 raft_consensus.cc:1275] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:26:25.675607 11756 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.099s
I20260709 12:26:25.891196  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:25.892598  2415 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:26:25.694555 11753 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:26:25.910900  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 71413.9us
I20260709 12:26:25.892421  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3342) took 1044 ms. Trace:
I20260709 12:26:25.892395  1880 raft_consensus.cc:1275] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:26:25.914345  1877 rpcz_store.cc:276] 0709 12:26:24.848132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:24.863792 (+ 15660us) service_pool.cc:224] Handling call
0709 12:26:25.892231 (+1028439us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b54002dfad4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:25.892350 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1186317184}
I20260709 12:26:25.723867 11283 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:25.918757  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 185851us
W20260709 12:26:25.839031  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:25.889703  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:25.920079  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:25.729592 11799 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:25.744715 11700 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:25.915987  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.839843  2410 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:26:25.743474 11821 raft_consensus.cc:3055] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:25.735242 11516 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:25.933884 11821 raft_consensus.cc:740] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:26:25.922484  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 102127us
I20260709 12:26:25.887035 11583 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:25.893270 11171 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:25.745024 11844 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:25.838302 11752 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.744081 11699 raft_consensus.cc:3055] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:25.926311  2410 pending_rounds.cc:85] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Aborting all ops after (but not including) 16
I20260709 12:26:25.961755  2410 pending_rounds.cc:107] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:26:25.962033  2410 raft_consensus.cc:2889] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:25.921756  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:25.922813 11672 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:25.926317 11283 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:25.936208 11799 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.973497 11743 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.973691 11799 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:25.974080 11743 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:25.737614 11806 log.cc:927] Time spent T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.935s	user 0.001s	sys 0.007s
I20260709 12:26:25.843292 11596 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.912480 11555 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.050s
I20260709 12:26:25.976567 11743 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:25.734400 11593 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.196s
I20260709 12:26:25.977258 11596 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:25.930944 11748 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:25.952231 11171 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.978994 11171 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:25.978180 11596 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:25.980310 11596 consensus_queue.cc:1243] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer 075704438992440697732181e49ccd09 log is divergent from this leader: its last log entry 22.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:26:25.936012 11777 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:25.931448  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:25.831661 11800 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.925185 11779 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:25.925400 11440 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.884219 11554 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.935024 11425 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:25.942641 11837 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.560s	user 0.002s	sys 0.000s
I20260709 12:26:25.943012 11738 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:25.944535  1844 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:26.004007  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59535.4us
I20260709 12:26:25.923018  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:25.704031 11781 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.891783 11742 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:25.734151 11783 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:26.011984 11742 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.925156 11007 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:25.928979  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:25.840193 11786 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:25.971912  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50257.1us
I20260709 12:26:25.969483  1873 raft_consensus.cc:1240] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:26.027958  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:25.843025 11683 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.948791 11583 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:25.961504 11699 raft_consensus.cc:740] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:25.706596 11563 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:26.006613  1844 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.012100 11783 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:25.927750  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:26.046198  1879 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:25.955066 11753 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:26.057513 11753 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:26.015568  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 103353us
I20260709 12:26:25.970589 11672 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.063081 11672 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:26.065145 11630 consensus_queue.cc:1048] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:25.978233 11748 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:26.067080 11748 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:26.067705 11781 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.069335 11742 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.029199  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:26.071719  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:26.072855  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:26.073850  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:25.986256 11488 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:26.031234  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.003760 11779 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:26.075330  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:26.075912  2382 leader_election.cc:336] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.076449 11779 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:26.076788  2382 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:26.003939 11800 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:26.077340  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:26.077560  1872 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:26.077874  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:26.078502  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:25.992654 11777 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.037415  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:26.045328  1844 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.003557 11554 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:25.935127 11821 consensus_queue.cc:260] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.048659  1624 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 19 to 22, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 22 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:26.049086 11699 consensus_queue.cc:260] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:26.053294  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.003839 11440 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.029803 11798 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:26.037101 11583 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:26.056458 11783 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:26.103257  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.133744  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:26.096588  1873 raft_consensus.cc:1240] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:26.084420  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:26.119335  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:26.121874  1880 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:25.972240 11752 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.024914 11007 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:26.120079  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74809.2us
I20260709 12:26:26.078715 11800 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.100309 11683 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.082373 11488 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:26.125712  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.125537  1624 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 17 to 20, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 20 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:26.137516  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:26.153553  1876 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:26.135720  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:26.032594 11786 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.132506 11844 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.099681 11719 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:26.167057 11719 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:26.166270  1624 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:26.082798 11717 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:26.167589 11554 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.167907 11717 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:26.168466 11717 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:26.170766  1872 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:26.126852 11555 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:26.175877 11555 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.079351 11843 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.086s
I20260709 12:26:26.082573 11593 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:26.082049 11816 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:26.083369 11787 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:26.157577  1879 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:26.181228  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.167066 11522 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.191442 11522 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.193921 11522 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.122855 11700 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.194885 11700 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:26.161952  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:26.101442 11736 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.141831 11584 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:26.166213 11844 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.200001 11736 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:26.168046  1624 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 15 to 18, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 18 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:26.149942  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65628.8us
W20260709 12:26:26.200811  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46012us
I20260709 12:26:26.201539  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:26.201733  1844 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.132882 11171 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.202523  1624 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 5 to 22, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 22 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:26.204674  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:26.152069 11007 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:26.097340 11746 raft_consensus.cc:3055] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:26.176113 11752 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.181541 11717 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:26.158329  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.214768 11816 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:26.159883 11783 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:26:26.123921 11777 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:26.197938  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:26.224197  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:26.224355  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 319036us
I20260709 12:26:26.149350 11773 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:26.225382  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.128458 11425 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.076s
I20260709 12:26:26.172922 11786 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:26.164057 11738 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:26.238850  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.241312  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.242574  2419 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:26:26.247061  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.211495  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.253161  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.142972 11821 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:26.216086 11488 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:26.214130  1624 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 19 to 20, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 20 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:26.216708 11743 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:26.217347 11787 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:26.216101 11699 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:26.270085  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:26.168373 11554 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:26:26.166487  1630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 255) took 75512 ms (client timeout 15000 ms). Trace:
W20260709 12:26:26.213102  1844 leader_election.cc:336] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.230482  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.232038  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.277449  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:26.234330  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 349350us
I20260709 12:26:26.277448  1798 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
W20260709 12:26:26.279943  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66890.4us
I20260709 12:26:26.201045 11440 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.255031  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:26.256986  2382 leader_election.cc:336] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.258137  1877 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 32: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:26.218827 11746 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:26.259590  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:26.302124  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:26.304752  2417 raft_consensus.cc:2393] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 31: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:26.305142  1844 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.260828  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.305580  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.255295  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:26.263864  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.306478  1879 raft_consensus.cc:2393] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 25: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:26.214038 11843 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.265017  1800 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:26:26.211236 11800 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:26.151510 11724 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.271111  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.272233  1804 raft_consensus.cc:1275] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:26.275895  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.280930  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:26.281565  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.281565  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.323927  1801 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 26: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:26.284288  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.327136  1798 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 22.
I20260709 12:26:26.285347  1802 raft_consensus.cc:1275] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:26.282022  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:26.275466 11554 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:26.288735 11779 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.290045  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:26.290918  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.293633  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.293883 11171 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.250504 11816 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:26.092172  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.250713 11752 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:26.302069  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45164.6us
W20260709 12:26:26.369294  2382 leader_election.cc:336] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:26.370390  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:26.305974  1844 leader_election.cc:336] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.362828  1873 raft_consensus.cc:2468] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
W20260709 12:26:26.371408  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 65524.7us
W20260709 12:26:26.372139  1844 leader_election.cc:336] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.372704  1844 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:26.306895  2411 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:26.307435  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52226.4us
I20260709 12:26:26.323186  2416 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:26.323355  1805 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:26.374136  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:26.298399 11555 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:26.353384 11843 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.377462  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:26.378229  1805 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:26.378921 11779 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.382035 11779 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.384042 11724 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:26.385581 11844 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:26.385828  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.387682  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.388217  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.389283  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.320582 11683 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:26.390471 11683 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.390960 11683 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:26.393270  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.318813 11799 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.396369 11719 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.397540 11799 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.319986 11756 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.100s
I20260709 12:26:26.422075 11748 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.397863 11719 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:26.398398 11554 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.398814  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.362650 11800 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:26.456874  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82851.4us
I20260709 12:26:26.399768  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.400796  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.401378  1880 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:26.403613  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:26.404716  1878 raft_consensus.cc:2393] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 26: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:26.405347 11700 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:26.389444  1805 raft_consensus.cc:2468] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 24.
I20260709 12:26:26.412067  1876 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:26.414342  1806 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:26.320425 11820 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.095s
I20260709 12:26:26.419274 11672 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.354839 11717 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.418022  1804 raft_consensus.cc:1275] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:26:26.419482  1799 raft_consensus.cc:1275] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 21. Preceding OpId from leader: term: 28 index: 22. (index mismatch)
I20260709 12:26:26.422765  1801 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:26.434742  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:26.431923  1879 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
W20260709 12:26:26.461126  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 88878.4us
W20260709 12:26:26.371021  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:26.508183  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 137194us
I20260709 12:26:26.464360  2410 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:26.423887 11843 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.481809 11800 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.483793  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.529525  1873 raft_consensus.cc:2468] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:26:26.530893  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.532235  1873 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:26:26.458307  1771 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:26.390478 11798 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:26.490931  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:26.533912  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75771.4us
I20260709 12:26:26.534161  1877 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:26.424547 11746 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.431538 11554 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:26.542506 11700 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:26.556840 11746 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:26.431143 11752 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.558899 11007 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.418710 11777 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.562294 11777 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.497038  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.575582  2412 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:26:26.498085  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 94590.8us
I20260709 12:26:26.576898  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:26.499362  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55680 (request call id 4239) took 1075 ms. Trace:
I20260709 12:26:26.484707 11799 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:26.501447  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:26.516225  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.517036 11738 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:26:26.518913  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 84305.4us
I20260709 12:26:26.524971  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:26.612704  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:26.537268  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:26.544977  1877 raft_consensus.cc:1240] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:26:26.492242 11748 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.563021  1803 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:26.617782  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.494932 11283 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:26.546908 11820 raft_consensus.cc:2804] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:26.624372  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:26.616935  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 79729.9us
I20260709 12:26:26.583304  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:26.585081  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:26.586066  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:26.586544 11672 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.591051 11488 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.592952  1880 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:26.629081  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:26.629240  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4243) took 1080 ms. Trace:
I20260709 12:26:26.629870  1880 rpcz_store.cc:276] 0709 12:26:25.548527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.592288 (+1043761us) service_pool.cc:224] Handling call
0709 12:26:26.592724 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.628874 (+ 36150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.629163 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:26.594581 11752 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.634424 11843 consensus_queue.cc:1048] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:26.635113 11171 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:26.635619 11171 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.635975 11171 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:26.636211 11820 raft_consensus.cc:697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:26.637250 11820 consensus_queue.cc:237] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.639458  2417 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:26:26.640964  2419 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.597084  1873 raft_consensus.cc:1240] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:26.579125  1878 rpcz_store.cc:276] 0709 12:26:25.424301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.427575 (+  3274us) service_pool.cc:224] Handling call
0709 12:26:25.767419 (+339844us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:26:26.499121 (+731702us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54001f7554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:26.499292 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1360767744}
I20260709 12:26:26.643724  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3358) took 1132 ms. Trace:
I20260709 12:26:26.644429  1878 raft_consensus.cc:1240] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
W20260709 12:26:26.645041  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 61858.2us
I20260709 12:26:26.600939  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.603817  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.647727  1806 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:26.605105  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:26.648984  1805 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:26.606359  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:26.653627  1798 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:26.548245  1844 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.618670  2413 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:26.618901  1879 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
W20260709 12:26:26.655174  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 110272us
I20260709 12:26:26.655570  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3357) took 1144 ms. Trace:
I20260709 12:26:26.644088  1873 rpcz_store.cc:276] 0709 12:26:25.511565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.596460 (+1084895us) service_pool.cc:224] Handling call
0709 12:26:26.596860 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.643457 (+ 46597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.643667 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:26.655894  1879 rpcz_store.cc:276] 0709 12:26:25.511314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.598212 (+1086898us) service_pool.cc:224] Handling call
0709 12:26:26.618546 (+ 20334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.655301 (+ 36755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.655513 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:26.619007 11717 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.602424  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.503530 11756 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:26:26.620589  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 95745.1us
I20260709 12:26:26.738842  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:26.542425 11743 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.622879  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3353) took 1112 ms. Trace:
I20260709 12:26:26.743938  1877 rpcz_store.cc:276] 0709 12:26:25.510272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.535624 (+1025352us) service_pool.cc:224] Handling call
0709 12:26:26.536097 (+   473us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.615055 (+ 78958us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.622544 (+  7489us) spinlock_profiling.cc:243] Waited 8.59 ms on lock 0x7b5400260a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:26.622818 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19751808}
I20260709 12:26:26.625527  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.544189 11798 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:26.631537 11488 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.657859  1873 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:26.731015 11849 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:26.708647  1773 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.033s	sys 0.029s
W20260709 12:26:26.759780  1773 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.033s	sys 0.029s
I20260709 12:26:26.637856 11800 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:26.623554 11700 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:26.564713 11596 consensus_queue.cc:1048] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.081s
I20260709 12:26:26.623889 11807 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:26.720321 11850 consensus_queue.cc:1059] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:26:26.721166  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 73320.3us
I20260709 12:26:26.722257 11843 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:26.722333  1874 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.019s	sys 0.065s
I20260709 12:26:26.724651  1879 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 31. Current term is 32. Ops: [31.22-31.22]
I20260709 12:26:26.746291 11007 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:26.725596  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63440.7us
I20260709 12:26:26.620832 11719 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.075s
I20260709 12:26:26.616988 11583 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.729269  1807 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
I20260709 12:26:26.736887  1801 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
W20260709 12:26:26.742913  1881 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.044s	sys 0.025s
W20260709 12:26:26.792738  1881 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.045s	sys 0.026s
W20260709 12:26:26.744601  2418 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.013s	sys 0.031s
I20260709 12:26:26.744833  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:26.630707  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:26.745645  2412 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
W20260709 12:26:26.795567  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 164938us
I20260709 12:26:26.741242 11683 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.748778  1873 raft_consensus.cc:1275] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:26:26.796577  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:26.657950  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:26.749176  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:26.799417  2418 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.705s	user 0.014s	sys 0.031s
W20260709 12:26:26.804157  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 146303us
W20260709 12:26:26.807482  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 58399.2us
I20260709 12:26:26.808704  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:26.814103 11584 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:26:26.815788  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 3220) took 1158 ms. Trace:
I20260709 12:26:26.777213 11807 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:26.735450 11742 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.018s	sys 0.025s
W20260709 12:26:26.763548  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:26.831899 11742 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.762s	user 0.018s	sys 0.026s
W20260709 12:26:26.764075  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:26.833125  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69120us
W20260709 12:26:26.833808  2382 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:26.734089 11829 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.041s	sys 0.069s
W20260709 12:26:26.275842  1630 rpcz_store.cc:269] 0709 12:25:10.650770 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.703459 (+ 52689us) service_pool.cc:224] Handling call
0709 12:26:26.150936 (+75447477us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:19.204720 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:19.437376 (+232656us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:19.437435 (+    59us) write_op.cc:435] Acquired schema lock
0709 12:26:19.437455 (+    20us) tablet.cc:662] Decoding operations
0709 12:26:19.438413 (+   958us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:19.438517 (+   104us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:19.438556 (+    39us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:26:19.439295 (+   739us) tablet.cc:701] Row locks acquired
0709 12:26:19.439347 (+    52us) write_op.cc:260] PREPARE: finished
0709 12:26:19.439561 (+   214us) write_op.cc:270] Start()
0709 12:26:19.439722 (+   161us) write_op.cc:276] Timestamp: P: 1783599979439488 usec, L: 0
0709 12:26:19.439747 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:26:19.490240 (+ 50493us) log.cc:844] Serialized 2882 byte log entry
0709 12:26:20.737537 (+1247297us) op_driver.cc:464] REPLICATION: finished
0709 12:26:21.277809 (+540272us) write_op.cc:301] APPLY: starting
0709 12:26:21.278113 (+   304us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:21.427798 (+149685us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:21.427934 (+   136us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:21.737507 (+309573us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:21.764978 (+ 27471us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 12:26:21.765062 (+    84us) write_op.cc:312] APPLY: finished
0709 12:26:21.765586 (+   524us) log.cc:844] Serialized 73 byte log entry
0709 12:26:22.198450 (+432864us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:22.216781 (+ 18331us) write_op.cc:454] Released schema lock
0709 12:26:22.224494 (+  7713us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:22.225199 (+   705us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:22.225233 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":272578,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":500307,"apply.run_cpu_time_us":60501,"apply.run_wall_time_us":951935,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33415,"num_ops":7,"prepare.queue_time_us":672613,"prepare.run_cpu_time_us":22149,"prepare.run_wall_time_us":460208,"raft.queue_time_us":622459,"raft.run_cpu_time_us":12555,"raft.run_wall_time_us":301624,"replication_time_us":1226556,"thread_start_us":1580278,"threads_started":5,"wal-append.queue_time_us":1000082,"wal-append.run_cpu_time_us":38545,"wal-append.run_wall_time_us":2743261}]]}
I20260709 12:26:26.834467  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:26.834483 11829 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.041s	sys 0.070s
W20260709 12:26:26.835714  1630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 178) took 15217 ms (client timeout 15000 ms). Trace:
I20260709 12:26:26.743813 11756 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:26.747545 11748 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:26.836328  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:26.778036  2383 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:26.836727 11748 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.837287 11829 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:26.837491  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 59556.4us
I20260709 12:26:26.784337  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3366) took 1075 ms. Trace:
I20260709 12:26:26.838892 11522 raft_consensus.cc:2804] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:26:26.839205  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.839063  1879 rpcz_store.cc:276] 0709 12:26:25.709368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.656740 (+947372us) service_pool.cc:224] Handling call
0709 12:26:26.724403 (+ 67663us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.783931 (+ 59528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.784255 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:26:26.757002 11672 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:26.750204 11851 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:26.840785  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:26.633306 11752 consensus_queue.cc:1048] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:26.788832 11843 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.722951 11717 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.843689 11777 consensus_queue.cc:1048] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:26:26.835784 11488 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.845038 11843 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:26.845196 11777 consensus_queue.cc:1048] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:26:26.845821 11717 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:26.758332 11743 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.846760 11583 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:26.846930 11752 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:26.836336  1630 rpcz_store.cc:269] 0709 12:26:11.618825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.835483 (+15216658us) service_pool.cc:211] Skipping call since client already timed out
0709 12:26:26.835626 (+   143us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:26:26.847972  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.821660 11724 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:26.816375  1807 rpcz_store.cc:276] 0709 12:26:25.656928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.659647 (+  2719us) service_pool.cc:224] Handling call
0709 12:26:25.677274 (+ 17627us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.795249 (+1117975us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.815410 (+ 20161us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54001d4554. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:26.815668 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2258418560}
I20260709 12:26:26.850131 11752 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.851420  1880 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:26:26.765245 11849 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.852404 11743 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.822103 11440 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:26.852702  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3364) took 1143 ms. Trace:
I20260709 12:26:26.853576 11756 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:26.854499 11849 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.827564 11283 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:26:26.834255  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:26.802409 11738 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:26.853210 11440 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:26.856140  1873 raft_consensus.cc:1275] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:26:26.835608  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:26.857623 11440 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.830575 11798 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:26.858865 11798 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:26.859632  1873 raft_consensus.cc:1275] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:26:26.860937  1873 raft_consensus.cc:1240] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:26.798749  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:26.833649  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:26.878961  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.842986 11736 consensus_queue.cc:1059] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:26:26.834203 11699 consensus_meta.cc:287] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.553s	user 0.007s	sys 0.003s
W20260709 12:26:26.784816  1874 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.020s	sys 0.065s
I20260709 12:26:26.853168  1880 rpcz_store.cc:276] 0709 12:26:25.708872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.645280 (+936408us) service_pool.cc:224] Handling call
0709 12:26:26.719698 (+ 74418us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.851869 (+132171us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.852418 (+   549us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b54001f8e54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:26.852643 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":302296448}
I20260709 12:26:26.815333 11700 consensus_queue.cc:1048] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.047s
W20260709 12:26:26.873723  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.801438 11799 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:26.814832 11844 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:26.852407 11752 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.734822  1805 raft_consensus.cc:2468] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:26.935350  1806 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
W20260709 12:26:26.882406  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.939366  1873 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 24. Current term is 25. Ops: [21.14-24.15]
W20260709 12:26:26.898725  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet def0036446324543ba07e132e885133a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:26.940922  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:26.901289  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:26.903139  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:26.876873  1803 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 32.
I20260709 12:26:26.820943  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:26.861089 11555 consensus_queue.cc:1175] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "123095127f334445889c21a42ba1d41e" responder_term: 25 status { last_received { term: 22 index: 16 } last_committed_idx: 15 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: [24.17-24.17]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:26:26.913338  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4228) took 1492 ms. Trace:
I20260709 12:26:26.913385  1875 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
W20260709 12:26:26.915398  2382 leader_election.cc:336] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:26.918393  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:26.912758 11748 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:26.947528  2413 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:26.862926 11807 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:26.893196  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:27.011235  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:26.907176  1844 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:27.011134  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 108123us
W20260709 12:26:26.943451  1872 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.673s	user 0.015s	sys 0.018s
I20260709 12:26:26.896302 11719 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:26.946943  1875 pending_rounds.cc:85] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 17
W20260709 12:26:26.880431  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 81791.9us
I20260709 12:26:27.013962  1875 pending_rounds.cc:107] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260709 12:26:27.014278  1875 raft_consensus.cc:2889] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:27.014595  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:26.904129 11558 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.969s	user 0.005s	sys 0.025s
W20260709 12:26:26.948359  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:27.023846  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 75560.2us
I20260709 12:26:26.946761  1874 rpcz_store.cc:276] 0709 12:26:25.421169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.424682 (+  3513us) service_pool.cc:224] Handling call
0709 12:26:25.740365 (+315683us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5800136b98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:26:25.740587 (+   222us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:25.750656 (+ 10069us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:25.750686 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:25.751244 (+   558us) log.cc:844] Serialized 26 byte log entry
0709 12:26:26.127138 (+375894us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:26:26.896985 (+769847us) log.cc:844] Serialized 14 byte log entry
0709 12:26:26.897270 (+   285us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.897360 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:26.897413 (+    53us) raft_consensus.cc:1685] finished
0709 12:26:26.913083 (+ 15670us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:26.913264 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9572,"raft.queue_time_us":18459,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":10090,"spinlock_wait_cycles":33638144,"thread_start_us":1103192,"threads_started":2,"wal-append.queue_time_us":728189}
W20260709 12:26:27.013140  1872 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.015s	sys 0.019s
W20260709 12:26:27.018373  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 197557us
I20260709 12:26:27.019469  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:26.930874 11844 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:26.948711 11516 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.014s	user 0.025s	sys 0.009s
I20260709 12:26:27.035989  1845 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:27.012486  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 105348us
W20260709 12:26:27.011660 11563 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.920s	user 0.040s	sys 0.087s
W20260709 12:26:27.052539  1844 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:27.025388  1772 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:27.023106 11440 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:27.023744 11558 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.089s	user 0.006s	sys 0.025s
W20260709 12:26:27.060876  1844 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:27.016554 11554 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.023877 11283 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.031813 11849 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:27.043390 11799 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:27.047313 11516 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.111s	user 0.025s	sys 0.010s
W20260709 12:26:27.083103  1696 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.807s	user 0.004s	sys 0.024s
W20260709 12:26:27.083600  1696 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.004s	sys 0.024s
I20260709 12:26:27.084278 11748 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.084496  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:27.085055 11748 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:27.085798 11717 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:27.074028  1873 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:26:27.057385 11719 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:27.034180 11777 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:27.044546 11844 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:27.055191  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:26:27.108206  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53143.5us
W20260709 12:26:27.080570  1624 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.007s	sys 0.032s
W20260709 12:26:27.050558 11563 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.959s	user 0.040s	sys 0.087s
W20260709 12:26:27.082067  1770 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.803s	user 0.018s	sys 0.010s
W20260709 12:26:27.076653  2414 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.903s	user 0.006s	sys 0.022s
I20260709 12:26:27.045377 11752 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
W20260709 12:26:27.075799  1844 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:27.078596 11849 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:27.109946  2414 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.936s	user 0.006s	sys 0.022s
W20260709 12:26:27.110285  1624 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.008s	sys 0.032s
I20260709 12:26:27.109382  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:27.111919  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:27.114691  1770 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.831s	user 0.018s	sys 0.010s
I20260709 12:26:27.083391 11516 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:27.065205 11440 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.141366  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:27.123032 11753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.954s	user 0.006s	sys 0.023s
I20260709 12:26:27.121698  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4240) took 1668 ms. Trace:
I20260709 12:26:27.127188  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:27.143813  1874 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:27.143430  1875 rpcz_store.cc:276] 0709 12:26:25.452909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.033593 (+580684us) service_pool.cc:224] Handling call
0709 12:26:26.033912 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:27.045341 (+1011429us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.121425 (+ 76084us) spinlock_profiling.cc:243] Waited 879 ms on lock 0x7b5400222254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:27.121633 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2021655424}
I20260709 12:26:27.146252  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:27.146375  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:27.147087  1875 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:26:27.149906  1877 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 23. Current term is 24. Ops: []
I20260709 12:26:27.115195 11800 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:27.086334  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:27.163447 11440 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:27.103644 11777 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.164392 11777 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:27.164784 11855 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:27.165401 11855 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:27.164417 11683 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:27.166613 11855 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.168547 11855 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:27.153578  1874 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:26:27.179869 11440 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:27.180756 11440 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:26:27.183394  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 97134us
I20260709 12:26:27.185598  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:27.079890 11554 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.188664 11554 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:27.155773  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 80021.2us
W20260709 12:26:27.190136  1844 leader_election.cc:336] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:27.152756 11753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.006s	sys 0.024s
W20260709 12:26:27.119124  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 840622us
I20260709 12:26:27.194383  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:27.199704 11743 raft_consensus.cc:2804] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:27.200268 11743 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.200680 11743 raft_consensus.cc:3060] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:27.201833 11829 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:27.207551 11516 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:27.212584 11821 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.949s	user 0.007s	sys 0.008s
I20260709 12:26:27.114722 11854 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:27.213929 11854 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.214411 11854 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:27.159940 11555 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:27.181989 11440 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.121320 11764 raft_consensus.cc:3055] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:27.141027 11672 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:27.118983 11849 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:27.222252  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:27.223824 11746 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.666s	user 0.011s	sys 0.001s
I20260709 12:26:27.082856 11283 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:27.224702 11799 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:27.260314  1874 raft_consensus.cc:1275] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:26:27.261520  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:27.262217  1874 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:27.262499  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.229117  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1493) took 1269 ms. Trace:
I20260709 12:26:27.263331  1877 rpcz_store.cc:276] 0709 12:26:25.959909 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.766566 (+806657us) service_pool.cc:224] Handling call
0709 12:26:26.789005 (+ 22439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:27.150542 (+361537us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.228782 (+ 78240us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b5400116f54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:27.229040 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":829310720}
I20260709 12:26:27.157814 11563 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:27.163695 11800 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.195226 11753 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:27.221232  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:27.235334  1799 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:27.218205 11743 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:27.245481  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:27.306200  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 60837.4us
W20260709 12:26:27.117038 11787 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.009s	sys 0.019s
W20260709 12:26:27.239027  1844 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:27.320010  1844 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:27.257004  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 110760us
I20260709 12:26:27.240355 11764 raft_consensus.cc:740] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:27.285105  1879 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:27.285997  1873 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:26:27.252631 11821 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:27.328016  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4255) took 1055 ms. Trace:
W20260709 12:26:27.289232  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68111.7us
W20260709 12:26:27.328681  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 89696.2us
I20260709 12:26:27.329387  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:27.321394  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:27.328521  1873 rpcz_store.cc:276] 0709 12:26:26.272224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.241748 (+969524us) service_pool.cc:224] Handling call
0709 12:26:27.282046 (+ 40298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.327613 (+ 45567us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.327941 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:26:27.281052  1802 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.878s	user 0.063s	sys 0.093s
W20260709 12:26:27.325022  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:27.329753  1844 leader_election.cc:336] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:27.333657  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:27.334502  1802 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.063s	sys 0.093s
I20260709 12:26:27.324000  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:27.337848  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:27.341324  1845 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:27.349154  1844 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:27.335510 11851 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:27.358887 11554 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:27.359512 11781 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:27.350450 11799 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:27.360821 11683 consensus_queue.cc:1059] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:27.363520  2418 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1667) took 1272 ms. Trace:
I20260709 12:26:27.288079 11800 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:27.366868  1806 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:27.368577  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:27.368786  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:26:27.319746 11787 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.031s	user 0.009s	sys 0.020s
W20260709 12:26:27.372566  2416 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.820s	user 0.008s	sys 0.043s
I20260709 12:26:27.305418 11488 raft_consensus.cc:2764] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:26:27.375809  1844 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:27.417255  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68129us
I20260709 12:26:27.374812 11555 raft_consensus.cc:2804] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:27.377023  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:27.417743  1878 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.694s	user 0.010s	sys 0.035s
W20260709 12:26:27.398325  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.225294 11777 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:27.384562 11851 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.365175 11672 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:27.373713 11753 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:26:27.414191 11171 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.778s	user 0.014s	sys 0.000s
W20260709 12:26:27.414870  2415 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.020s	sys 0.032s
I20260709 12:26:27.417155  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:27.352059 11563 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:26:27.379200 11630 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.989s	user 0.016s	sys 0.040s
I20260709 12:26:27.350853 11558 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.058s
I20260709 12:26:27.408114  2418 rpcz_store.cc:276] 0709 12:26:26.091244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.092669 (+  1425us) service_pool.cc:224] Handling call
0709 12:26:26.093135 (+   466us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.093591 (+   456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.093631 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.094170 (+   539us) log.cc:844] Serialized 26 byte log entry
0709 12:26:26.800084 (+705914us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:26:26.800146 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:26.800249 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.363308 (+563059us) raft_consensus.cc:1685] finished
0709 12:26:27.363328 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.363460 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2381,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"thread_start_us":705432,"threads_started":1,"wal-append.queue_time_us":1066747}
I20260709 12:26:27.392587 11554 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:27.418753  1844 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:27.419994  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51293.4us
W20260709 12:26:27.426941  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:27.419395  1878 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.011s	sys 0.035s
I20260709 12:26:27.430169  1877 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260709 12:26:27.428146  2410 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.785s	user 0.011s	sys 0.035s
W20260709 12:26:27.463570  2410 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.011s	sys 0.036s
W20260709 12:26:27.435277  2411 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.838s	user 0.027s	sys 0.034s
W20260709 12:26:27.464327  2411 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.027s	sys 0.034s
I20260709 12:26:27.392171  1874 raft_consensus.cc:2468] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
W20260709 12:26:27.414428  2416 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.862s	user 0.008s	sys 0.043s
W20260709 12:26:27.442836  1876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.012s	sys 0.033s
I20260709 12:26:27.364785 11821 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:27.466107  1876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.012s	sys 0.033s
W20260709 12:26:27.445767  2415 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.890s	user 0.021s	sys 0.032s
I20260709 12:26:27.431568 11717 raft_consensus.cc:2804] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:27.430924 11799 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.460845  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:27.429332 11583 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:27.443255 11700 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.475172  1771 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:27.476676  1844 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:27.450996 11630 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.070s	user 0.016s	sys 0.041s
I20260709 12:26:27.442847 11672 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:27.455190 11756 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.600s	user 0.013s	sys 0.000s
I20260709 12:26:27.443176 11851 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:27.478235 11753 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:27.445771 11555 raft_consensus.cc:697] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:27.473834 11717 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:27.461442 11554 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:27.514513  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:27.593374  1877 raft_consensus.cc:2393] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:27.549815 11858 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:26:27.495107  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:27.557845 11864 raft_consensus.cc:2804] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:26:27.453752 11171 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.483053 11853 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.069s
W20260709 12:26:27.563072  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 86451us
W20260709 12:26:27.563831  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 88765.7us
I20260709 12:26:27.548141 11488 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.486826 11799 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:27.471354 11764 consensus_queue.cc:260] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:27.561623 11746 consensus_queue.cc:1059] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.040s
I20260709 12:26:27.527351 11866 consensus_queue.cc:1059] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
W20260709 12:26:27.539839  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.540920 11862 consensus_queue.cc:1048] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.058s
I20260709 12:26:27.579051 11672 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:27.578707 11683 raft_consensus.cc:2804] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:26:27.600144  1881 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.019s	sys 0.006s
W20260709 12:26:27.639034  1881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.020s	sys 0.006s
W20260709 12:26:27.563400 11820 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s	user 0.017s	sys 0.006s
I20260709 12:26:27.641625  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4238) took 2217 ms. Trace:
W20260709 12:26:27.600744  1844 leader_election.cc:336] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:27.645864  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:27.615929  2417 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.675s	user 0.009s	sys 0.009s
W20260709 12:26:27.647488  2417 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.009s	sys 0.009s
I20260709 12:26:27.553778 11700 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:27.593503 11522 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.004s	sys 0.031s
I20260709 12:26:27.644088  1880 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:27.592617 11717 raft_consensus.cc:3060] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:27.635061  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:27.628993  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.665153  2418 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:27.646200  1881 rpcz_store.cc:276] 0709 12:26:25.424033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.033016 (+608983us) service_pool.cc:224] Handling call
0709 12:26:26.037041 (+  4025us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.037290 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:26:26.793342 (+756052us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.640055 (+846713us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:27.640132 (+    77us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:27.640211 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.641362 (+  1151us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.641553 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1102955,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"thread_start_us":1594605,"threads_started":2}
W20260709 12:26:27.553208 11850 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.745s	user 0.021s	sys 0.049s
W20260709 12:26:27.645922  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45203.6us
W20260709 12:26:27.620558  1872 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.017s	sys 0.012s
W20260709 12:26:27.678488  1872 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.017s	sys 0.013s
W20260709 12:26:27.658510 11522 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.004s	sys 0.031s
W20260709 12:26:27.597538 11742 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.007s	sys 0.030s
I20260709 12:26:27.582974 11756 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.646922  1877 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
W20260709 12:26:27.644979 11820 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s	user 0.017s	sys 0.007s
W20260709 12:26:27.604909  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 224645us
I20260709 12:26:27.600090 11864 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:27.689601 11868 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:26:27.606066 11283 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:27.717038  2410 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1823) took 1152 ms. Trace:
I20260709 12:26:27.633320 11798 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.697104  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:27.659202 11786 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:27.660821 11852 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:27.675377 11850 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.874s	user 0.021s	sys 0.050s
I20260709 12:26:27.710988 11864 raft_consensus.cc:3060] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:27.682848 11743 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:27.713172 11700 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:27.615811 11488 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.685843 11555 consensus_queue.cc:237] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:27.715274 11844 consensus_meta.cc:287] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.598s	user 0.010s	sys 0.002s
W20260709 12:26:27.699697 11742 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.007s	sys 0.031s
W20260709 12:26:27.698817  1875 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.015s	sys 0.010s
I20260709 12:26:27.717510  2410 rpcz_store.cc:276] 0709 12:26:26.564391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.565500 (+  1109us) service_pool.cc:224] Handling call
0709 12:26:26.566006 (+   506us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.566510 (+   504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.566551 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.567222 (+   671us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.463934 (+896712us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:27.463987 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.464049 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.716767 (+252718us) raft_consensus.cc:1685] finished
0709 12:26:27.716791 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.716975 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13315,"raft.queue_time_us":4742,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":13841,"thread_start_us":820706,"threads_started":1,"wal-append.queue_time_us":1098358}
W20260709 12:26:27.751186  1875 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.015s	sys 0.010s
W20260709 12:26:27.720471  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 286644us
I20260709 12:26:27.669104 11683 raft_consensus.cc:697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:26:27.658406 11785 consensus_queue.cc:1175] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 30 status { last_received { term: 26 index: 17 } last_committed_idx: 17 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:26:27.734963  2382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.009s	sys 0.021s
I20260709 12:26:27.697218 11777 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:27.699152 11522 raft_consensus.cc:697] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:27.740978  1798 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:27.670022 11866 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:26:27.752242  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:27.734107 11283 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:26:27.774032  2382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.010s	sys 0.021s
I20260709 12:26:27.734324 11798 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.739362 11821 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:27.741663 11862 consensus_queue.cc:1048] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:26:27.795275 11522 consensus_queue.cc:237] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:27.705297 11753 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.799001 11777 consensus_queue.cc:1048] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:27.799060  1876 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3362) took 2090 ms. Trace:
I20260709 12:26:27.800594 11798 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:27.801805 11798 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:27.800464  1876 rpcz_store.cc:276] 0709 12:26:25.708322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.616424 (+908102us) service_pool.cc:224] Handling call
0709 12:26:26.619327 (+  2903us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:26.619810 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.619860 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:26.630406 (+ 10546us) log.cc:844] Serialized 52 byte log entry
0709 12:26:27.466667 (+836261us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:26:27.497805 (+ 31138us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.577816 (+ 80011us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.578044 (+   228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.578119 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.798809 (+220690us) raft_consensus.cc:1685] finished
0709 12:26:27.798833 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.798994 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3263,"prepare.queue_time_us":40049,"prepare.run_cpu_time_us":1189,"prepare.run_wall_time_us":20708,"raft.queue_time_us":2998,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":14905,"spinlock_wait_cycles":44581760,"thread_start_us":729413,"threads_started":1,"wal-append.queue_time_us":993350}
I20260709 12:26:27.804965 11171 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:27.806493  2411 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1820) took 1262 ms. Trace:
I20260709 12:26:27.807085  2411 rpcz_store.cc:276] 0709 12:26:26.544175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.546954 (+  2779us) service_pool.cc:224] Handling call
0709 12:26:26.586944 (+ 39990us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.596693 (+  9749us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.596744 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.597376 (+   632us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.464911 (+867535us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:27.464947 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.465015 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.806057 (+341042us) raft_consensus.cc:1685] finished
0709 12:26:27.806083 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.806411 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10252,"raft.queue_time_us":10612,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":9477,"thread_start_us":867137,"threads_started":1,"wal-append.queue_time_us":1107671}
W20260709 12:26:27.812527  2382 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 74 similar messages]
I20260709 12:26:27.786700 11283 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:27.814946 11883 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:26:27.815376 11861 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.237s	user 0.001s	sys 0.001s
I20260709 12:26:27.802356 11798 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.816608  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33790 (request call id 3329) took 1524 ms. Trace:
I20260709 12:26:27.816861  2416 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1822) took 1269 ms. Trace:
I20260709 12:26:27.817963  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3365) took 2108 ms. Trace:
I20260709 12:26:27.817212  1802 rpcz_store.cc:276] 0709 12:26:26.291997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.396860 (+104863us) service_pool.cc:224] Handling call
0709 12:26:26.400466 (+  3606us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:26.401362 (+   896us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.401411 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:26.402193 (+   782us) log.cc:844] Serialized 52 byte log entry
0709 12:26:27.335036 (+932843us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:27.372744 (+ 37708us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.378123 (+  5379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.378243 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.816361 (+438118us) raft_consensus.cc:1685] finished
0709 12:26:27.816382 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.816528 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19963,"prepare.queue_time_us":37476,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":11163,"raft.run_cpu_time_us":322,"raft.run_wall_time_us":17440,"thread_start_us":932298,"threads_started":1,"wal-append.queue_time_us":1176783}
I20260709 12:26:27.819193  1878 rpcz_store.cc:276] 0709 12:26:25.709112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.645406 (+936294us) service_pool.cc:224] Handling call
0709 12:26:26.645704 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.646090 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.646120 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.654866 (+  8746us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.485055 (+830189us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:26:27.485103 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.485173 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.817784 (+332611us) raft_consensus.cc:1685] finished
0709 12:26:27.817804 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.817913 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6778,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":42784,"thread_start_us":739626,"threads_started":1,"wal-append.queue_time_us":1024041}
I20260709 12:26:27.814869 11283 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:27.817358  2416 rpcz_store.cc:276] 0709 12:26:26.547482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.547722 (+   240us) service_pool.cc:224] Handling call
0709 12:26:26.550981 (+  3259us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:26.551453 (+   472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:26.551504 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:26.552115 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.465755 (+913640us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:27.465791 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.465851 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.816527 (+350676us) raft_consensus.cc:1685] finished
0709 12:26:27.816562 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.816780 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":126,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"thread_start_us":913215,"threads_started":1,"wal-append.queue_time_us":1148163}
I20260709 12:26:27.754058 11844 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.743567 11852 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.839900 11756 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:27.851954 11700 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:27.886468  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:27.921373  2415 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1821) took 1376 ms. Trace:
W20260709 12:26:27.928174  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 372374us
I20260709 12:26:27.880118 11171 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:27.922106  2415 rpcz_store.cc:276] 0709 12:26:26.544588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.546602 (+  2014us) service_pool.cc:224] Handling call
0709 12:26:26.555128 (+  8526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.555397 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:26:27.466824 (+911427us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.920845 (+454021us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:27.920943 (+    98us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:26:27.921014 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.921108 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.921278 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36122,"prepare.queue_time_us":1162462,"prepare.run_cpu_time_us":141,"prepare.run_wall_time_us":2496,"thread_start_us":1328348,"threads_started":2}
I20260709 12:26:27.854712 11890 consensus_queue.cc:1048] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 12:26:27.949559  1872 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3351) took 2439 ms. Trace:
I20260709 12:26:27.928471 11488 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:27.952575 11171 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.875813 11683 consensus_queue.cc:237] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:27.960995 11855 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.730s	user 0.009s	sys 0.003s
I20260709 12:26:27.961997 11171 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:27.959967  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4257) took 1138 ms. Trace:
I20260709 12:26:27.962864  1873 rpcz_store.cc:276] 0709 12:26:26.821137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.331378 (+510241us) service_pool.cc:224] Handling call
0709 12:26:27.331922 (+   544us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:27.361971 (+ 30049us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.362019 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:27.362927 (+   908us) log.cc:844] Serialized 52 byte log entry
0709 12:26:27.787305 (+424378us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:27.787352 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.787444 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:27.959645 (+172201us) raft_consensus.cc:1685] finished
0709 12:26:27.959673 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.959885 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19041,"raft.queue_time_us":31348,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":121,"thread_start_us":423843,"threads_started":1,"wal-append.queue_time_us":594100}
W20260709 12:26:27.965870 11849 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.684s	user 0.004s	sys 0.008s
I20260709 12:26:27.884379 11800 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:27.967867  1874 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:27.973922 11748 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:27.986258 11593 raft_consensus.cc:3055] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:27.986821 11593 raft_consensus.cc:740] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:27.987754 11593 consensus_queue.cc:260] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.989125 11593 raft_consensus.cc:3060] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:27.990574 11799 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:27.993840 11583 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:27.994341 11583 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:27.997617  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3367) took 1768 ms. Trace:
I20260709 12:26:27.998085  1880 rpcz_store.cc:276] 0709 12:26:26.229247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.936764 (+707517us) service_pool.cc:224] Handling call
0709 12:26:26.937164 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:27.664482 (+727318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.997369 (+332887us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:27.997553 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1609576832}
W20260709 12:26:27.998888 11877 log.cc:927] Time spent T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:26:28.015303  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:28.016181  1807 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:28.017953  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.018729  1802 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:27.953401  1872 rpcz_store.cc:276] 0709 12:26:25.509692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.213355 (+703663us) service_pool.cc:224] Handling call
0709 12:26:26.236753 (+ 23398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:26.237010 (+   257us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:27.027463 (+790453us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.949077 (+921614us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:27.949157 (+    80us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:27.949223 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.949319 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:27.949491 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":978206,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":122229,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":97,"thread_start_us":1677570,"threads_started":3,"wal-append.queue_time_us":926319}
I20260709 12:26:28.021493  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:28.023029  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 293488us
I20260709 12:26:28.026043  1802 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:27.930922 11700 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:27.966337 11849 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:28.064852  1845 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.068615  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:28.071812  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.076221  1804 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 33 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:26:28.006141 11799 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:28.056916 11807 consensus_meta.cc:287] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Time spent flushing consensus metadata: real 1.022s	user 0.011s	sys 0.004s
I20260709 12:26:28.079063  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:28.081301  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.072656  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:28.081189  1800 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3328) took 1778 ms (client timeout 1580 ms). Trace:
W20260709 12:26:28.088088  1800 rpcz_store.cc:269] 0709 12:26:26.291750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.387904 (+ 96154us) service_pool.cc:224] Handling call
0709 12:26:28.069931 (+1682027us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b5400132254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:28.070155 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3226990976}
W20260709 12:26:28.091418  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:28.055632 11563 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.065088 11748 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:28.127851  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:28.078650 11746 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.066565 11866 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.080327 11900 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.098188 11752 raft_consensus.cc:3055] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:28.078151 11798 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:28.085759 11892 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:28.074318 11488 consensus_queue.cc:1059] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:26:28.097411 11878 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:26:28.144471  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 331974us
I20260709 12:26:28.092206  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:28.131671  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 202073us
W20260709 12:26:28.155176  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 63814.4us
W20260709 12:26:28.158411  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.126246 11895 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:28.176872  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.176261  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:28.151893 11854 consensus_meta.cc:287] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.897s	user 0.014s	sys 0.000s
I20260709 12:26:28.183334  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.185151  1877 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:28.099309 11799 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.097467 11807 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.145610 11752 raft_consensus.cc:740] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:28.210330 11583 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:28.210662 11904 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:28.196964 11880 log.cc:927] Time spent T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.332s	user 0.000s	sys 0.001s
I20260709 12:26:28.139780 11748 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.133059 11563 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:28.211835 11888 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:26:28.202663 11522 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.143096 11746 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.148129 11866 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.230989  2413 raft_consensus.cc:1240] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:28.153050 11717 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.150825 11900 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.216377 11878 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:28.218621 11898 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.218155 11488 raft_consensus.cc:3055] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:28.215835 11909 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:28.198685 11854 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.224086 11852 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.228688 11844 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:28.161854 11892 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.245621 11878 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.232640 11753 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.067s
I20260709 12:26:28.184191 11895 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:28.276319  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.307237  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:28.319429  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:28.326592  1877 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:28.373281  1807 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 23.
I20260709 12:26:28.392728  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.390595  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:28.393453  1878 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:28.235896 11522 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.246713 11488 raft_consensus.cc:740] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:26:28.337208 11897 log.cc:927] Time spent T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
W20260709 12:26:28.336555 11886 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.280s	user 0.000s	sys 0.001s
W20260709 12:26:28.419414  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 180956us
I20260709 12:26:28.246466 11909 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:28.318161 11903 consensus_queue.cc:1059] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 29.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.187s
I20260709 12:26:28.228269 11904 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.318141 11925 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:28.316032 11807 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.317400 11854 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.318131 11736 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:28.336464 11799 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.331521 11700 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.244535 11440 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.157s
I20260709 12:26:28.336375 11866 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.315997 11900 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:28.407721 11849 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.438495 11736 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.445701 11488 consensus_queue.cc:260] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.447427 11488 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:28.447665 11736 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.448941 11736 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:28.449420 11736 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 30 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:28.452634 11522 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.456202  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4248) took 2224 ms. Trace:
I20260709 12:26:28.251612 11898 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.472149  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4260) took 1431 ms. Trace:
I20260709 12:26:28.473593  1879 rpcz_store.cc:276] 0709 12:26:27.040632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.466774 (+426142us) service_pool.cc:224] Handling call
0709 12:26:27.517539 (+ 50765us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.555441 (+ 37902us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.555467 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:27.555984 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.848635 (+292651us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:27.848707 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.848828 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.471834 (+623006us) raft_consensus.cc:1685] finished
0709 12:26:28.471863 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.472058 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1785,"raft.queue_time_us":2815,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":2326,"thread_start_us":258494,"threads_started":1,"wal-append.queue_time_us":498055}
I20260709 12:26:28.476249  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.477137  1879 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:28.483551  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.456744  1875 rpcz_store.cc:276] 0709 12:26:26.231573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.147974 (+916401us) service_pool.cc:224] Handling call
0709 12:26:27.149585 (+  1611us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.176823 (+ 27238us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.176880 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:27.177801 (+   921us) log.cc:844] Serialized 26 byte log entry
0709 12:26:27.751617 (+573816us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:27.751663 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.751745 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.455934 (+704189us) raft_consensus.cc:1685] finished
0709 12:26:28.455958 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.456113 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7325,"raft.queue_time_us":9642,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"thread_start_us":573274,"threads_started":1,"wal-append.queue_time_us":751188}
I20260709 12:26:28.425558 11909 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:28.252460 11752 consensus_queue.cc:260] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.450336 11736 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.19, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:28.528039  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.509989 11906 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:28.495299 11583 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:28.436076 11925 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.545246  1878 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 31.
I20260709 12:26:28.522408 11440 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:28.531543 11849 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.531961 11903 raft_consensus.cc:3055] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:28.533756 11746 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:28.576404 11554 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.973s	user 0.008s	sys 0.008s
W20260709 12:26:28.592211 11899 log.cc:927] Time spent T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.293s	user 0.001s	sys 0.000s
W20260709 12:26:28.593204 11905 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:26:28.594983 11932 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.567605 11854 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:28.599319  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.573731 11906 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:26:28.592074 11916 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:26:28.564875 11928 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:26:28.575857 11903 raft_consensus.cc:740] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:28.651927  1875 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:28.578097 11717 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.551303 11583 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:26:28.601536 11672 consensus_meta.cc:287] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.934s	user 0.015s	sys 0.003s
I20260709 12:26:28.600752 11516 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.679692  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.686192  1875 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:28.578984 11440 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.687192  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.690462  1875 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 30: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:26:28.668946 11864 consensus_meta.cc:287] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.922s	user 0.015s	sys 0.002s
W20260709 12:26:28.692541  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.697645  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.698671  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:28.698077  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 275483us
I20260709 12:26:28.625914 11850 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:28.580685 11849 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:28.700686  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.702046  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.692310  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.601538 11554 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.681738 11892 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:28.681160 11901 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:28.718923  2413 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1678) took 1605 ms. Trace:
W20260709 12:26:28.702129 11917 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
I20260709 12:26:28.692838 11516 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.689625 11672 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.702270 11864 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.655004 11932 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.716493 11849 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.716980 11878 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:28.718161  1801 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:28.703889 11935 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.719341  2413 rpcz_store.cc:276] 0709 12:26:27.113187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.113397 (+   210us) service_pool.cc:224] Handling call
0709 12:26:27.123529 (+ 10132us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:28.236453 (+1112924us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.718593 (+482140us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540049eb54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:28.718851 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2546379520}
I20260709 12:26:28.664011 11854 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.734037 11800 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.752907  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:28.753125  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:28.754249  2418 raft_consensus.cc:2410] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 31 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:26:28.753996  2415 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:28.754628  1875 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:28.768853  1880 raft_consensus.cc:1240] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:26:28.707768 11850 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.762696  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:28.770700  1696 connection.cc:256] RPC call timeout handler was delayed by -1.9500122899999042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
I20260709 12:26:28.770202  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:28.739390 11898 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.783087 11854 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.784432 11931 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:28.785310 11931 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.747818 11878 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:28.786574  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0847517239999434s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
W20260709 12:26:28.787228  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0534032339996884s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1575s for a response
W20260709 12:26:28.787683  1696 connection.cc:256] RPC call timeout handler was delayed by -1.0183032339996316s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
I20260709 12:26:28.761556 11904 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:28.756609 11924 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:28.790894  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:28.726593 11901 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.761051 11895 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.788496 11898 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:28.806439  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.748965 11935 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.718837 11903 consensus_queue.cc:260] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.821126  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3342) took 1439 ms. Trace:
I20260709 12:26:28.821477 11864 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.821647  1798 rpcz_store.cc:276] 0709 12:26:27.382206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.382390 (+   184us) service_pool.cc:224] Handling call
0709 12:26:28.820845 (+1438455us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b540035fa54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:28.821050 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":678811904}
I20260709 12:26:28.823355 11932 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.824887 11672 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:28.793111 11878 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.826330 11596 raft_consensus.cc:3055] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:28.826864 11596 raft_consensus.cc:740] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:28.828114 11583 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:28.827909 11596 consensus_queue.cc:260] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.829219 11878 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.830008 11440 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.831398 11901 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.805557 11925 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:28.833037 11851 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.240s	user 0.014s	sys 0.003s
I20260709 12:26:28.833545 11851 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.748566 11906 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.836393 11563 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.837193 11554 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.837555 11935 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:28.814440 11930 log.cc:927] Time spent T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.234s	user 0.000s	sys 0.002s
W20260709 12:26:28.838222 11884 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.909s	user 0.001s	sys 0.002s
I20260709 12:26:28.802745 11924 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.840255  2414 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60950 (request call id 1665) took 3048 ms. Trace:
I20260709 12:26:28.841279 11924 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:28.842278 11593 consensus_meta.cc:287] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.853s	user 0.012s	sys 0.003s
I20260709 12:26:28.829319 11516 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.843089  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4269) took 1238 ms. Trace:
I20260709 12:26:28.819031 11898 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.843549  1881 rpcz_store.cc:276] 0709 12:26:27.604940 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.733548 (+128608us) service_pool.cc:224] Handling call
0709 12:26:27.746204 (+ 12656us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:27.752037 (+  5833us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.752072 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:27.782759 (+ 30687us) log.cc:844] Serialized 26 byte log entry
0709 12:26:28.092824 (+310065us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:28.092866 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:28.092947 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.842857 (+749910us) raft_consensus.cc:1685] finished
0709 12:26:28.842882 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.843026 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":54928,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":58,"thread_start_us":249720,"threads_started":1,"wal-append.queue_time_us":987952}
I20260709 12:26:28.819039 11913 raft_consensus.cc:3055] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:28.845726 11913 raft_consensus.cc:740] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:28.846158 11898 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.846679 11913 consensus_queue.cc:260] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.848094 11851 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:28.853760 11926 log.cc:927] Time spent T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.227s	user 0.000s	sys 0.002s
I20260709 12:26:28.854997  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.854596 11672 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:28.855789  1880 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:28.855873 11672 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.858305  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3341) took 1481 ms. Trace:
I20260709 12:26:28.858732  1801 rpcz_store.cc:276] 0709 12:26:27.376454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.376659 (+   205us) service_pool.cc:224] Handling call
0709 12:26:28.858077 (+1481418us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540013a954. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:28.858224 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2986516736}
I20260709 12:26:28.859189  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.860258  1873 raft_consensus.cc:2368] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30.
I20260709 12:26:28.858680 11672 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:28.861821  2418 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:28.863082  1770 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:28.863511  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 1.67031e+06us
W20260709 12:26:28.864969  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet def0036446324543ba07e132e885133a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:28.865417  1874 raft_consensus.cc:1240] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:28.868335  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:28.848158 11913 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:28.855182  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:28.864156  1770 leader_election.cc:336] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:28.874504  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.877137  1770 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:28.878259  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.877151  2413 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:28.866686  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:28.883481  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:28.884172  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:28.886773  1880 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:28.883268  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:28.888049  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:28.889016  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 271104us
I20260709 12:26:28.891968  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.894317  2416 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:28.840675  2414 rpcz_store.cc:276] 0709 12:26:25.791425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:25.791598 (+   173us) service_pool.cc:224] Handling call
0709 12:26:25.909129 (+117531us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:27.115811 (+1206682us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.115870 (+    59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:27.171608 (+ 55738us) log.cc:844] Serialized 52 byte log entry
0709 12:26:27.658177 (+486569us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:27.658226 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.968034 (+309808us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b54001c3ed4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:27.968092 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.837405 (+869313us) spinlock_profiling.cc:243] Waited 5.18 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:28.839974 (+  2569us) raft_consensus.cc:1685] finished
0709 12:26:28.839995 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:28.840161 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7191,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":619398912,"thread_start_us":1428069,"threads_started":2,"wal-append.queue_time_us":648110}
I20260709 12:26:28.885890  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.896567  1875 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 32: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:28.807876 11743 raft_consensus.cc:2804] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:28.903712  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.902220  1801 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: []
I20260709 12:26:28.905055  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:28.900929  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:28.905978  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.906668  1806 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 30.
I20260709 12:26:28.882331  2383 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:28.907801  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:28.908737  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.901175  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:28.891964  1770 consensus_peers.cc:597] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:28.901728  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.912678  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.914104  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.914973  1805 raft_consensus.cc:2410] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 29 because replica has last-logged OpId of term: 28 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 21.
I20260709 12:26:28.915570  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.916229  1880 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:28.808329 11945 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:28.918718  1881 raft_consensus.cc:1240] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:28.918901  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:28.846257 11787 raft_consensus.cc:3055] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:26:28.910368  1770 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:28.920972  1770 consensus_peers.cc:597] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:28.922258  1770 leader_election.cc:336] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:28.923480  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:28.919287  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.925745  1801 raft_consensus.cc:2410] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
W20260709 12:26:28.925326  1844 leader_election.cc:336] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:28.954098  1799 raft_consensus.cc:1240] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:26:28.920938  1806 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:28.930346  1872 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:28.956306  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 139916us
I20260709 12:26:28.890559 11786 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:28.958622  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.961176  1798 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:28.959255  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:28.965410  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.902539 11743 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:28.900104 11854 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:28.963577  1800 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:28.908437 11904 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.921211 11945 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:28.978139  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.980562  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.965099  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:28.925695 11787 raft_consensus.cc:740] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:28.951016 11936 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:28.987327 11936 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.900317 11554 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:28.989965 11554 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:28.910734 11855 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.953914  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.993418 11855 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.953410  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.919775 11800 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:28.920066 11555 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:28.886056 11909 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:28.999763  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.000684  1798 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
W20260709 12:26:28.977687  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65957.1us
I20260709 12:26:28.960814  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:28.920039 11516 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:28.969959  1873 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
W20260709 12:26:28.975461 11887 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.046s	user 0.006s	sys 0.000s
I20260709 12:26:28.958894 11923 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:28.995699  1874 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:26:28.954985 11892 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:29.003923 11892 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:29.004398 11892 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:28.997017  1805 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
I20260709 12:26:28.998543 11855 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.968515 11786 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:29.005004  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:28.975806  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.003207 11753 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:29.005251 11898 consensus_queue.cc:1059] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:26:29.002895  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 77597.4us
I20260709 12:26:29.005900  1800 raft_consensus.cc:2368] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29.
I20260709 12:26:29.005053  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1493) took 1980 ms. Trace:
I20260709 12:26:29.006232 11786 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:29.008626 11753 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:29.008749 11898 raft_consensus.cc:3055] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:29.009274 11753 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 32 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:29.009371 11898 raft_consensus.cc:740] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:29.010427 11753 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.010710 11898 consensus_queue.cc:260] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.007582  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:28.977514 11743 raft_consensus.cc:3060] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:29.002307  1845 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:29.016085  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.017081  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:29.007385  1844 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:29.006937  1803 rpcz_store.cc:276] 0709 12:26:27.024462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:27.025021 (+   559us) service_pool.cc:224] Handling call
0709 12:26:27.134261 (+109240us) spinlock_profiling.cc:243] Waited 1.59 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3eee 000000000036eb39 00007f16b0d271e0 00007f16b0df00c4 00007f16b0df0131 00007f16b0db1fc0 00007f16b0d9c929 00007f16b0d9c3d2 00007f16aa313bd4
0709 12:26:27.134484 (+   223us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:27.211738 (+ 77254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:27.211787 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:27.212697 (+   910us) log.cc:844] Serialized 52 byte log entry
0709 12:26:27.644194 (+431497us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:27.688267 (+ 44073us) log.cc:844] Serialized 14 byte log entry
0709 12:26:27.688527 (+   260us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:27.688632 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:28.782717 (+1094085us) spinlock_profiling.cc:243] Waited 5.27 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:29.004770 (+222053us) raft_consensus.cc:1685] finished
0709 12:26:29.004804 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:29.004984 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":76628,"prepare.queue_time_us":43858,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":89314,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":9914,"spinlock_wait_cycles":15796480,"thread_start_us":430930,"threads_started":1,"wal-append.queue_time_us":606843}
I20260709 12:26:29.007920  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:29.021158  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.012933 11898 raft_consensus.cc:3060] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:29.019706  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:29.025795  1805 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:26:28.974289 11928 consensus_queue.cc:1059] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:26:29.024923  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:28.981365 11945 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:29.030835  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.030712  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:29.034121  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 82155.3us
I20260709 12:26:29.029227  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:29.035059  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.016094 11583 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:29.078899 11283 consensus_meta.cc:287] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 1.239s	user 0.017s	sys 0.000s
I20260709 12:26:29.066527  1799 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:29.075748  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.073936  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:29.077037 11866 raft_consensus.cc:3055] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:29.079906 11746 raft_consensus.cc:3055] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:26:29.025631 11516 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.093107 11746 raft_consensus.cc:740] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:29.095687  1878 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:26:29.065569  1805 pending_rounds.cc:85] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting all ops after (but not including) 13
I20260709 12:26:28.983976 11904 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.082994 11944 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:29.027424 11923 raft_consensus.cc:697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 31 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:29.038971 11931 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:28.993019 11554 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:29.066434  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59075.9us
W20260709 12:26:29.152356  1844 leader_election.cc:336] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:29.078563 11941 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 12:26:29.073135  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43989.1us
W20260709 12:26:29.157464  1770 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.089830  1798 raft_consensus.cc:2393] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:29.041920 11909 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:29.091198  1803 raft_consensus.cc:2393] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:29.028219 11563 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:26:29.037250 11850 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:29.043720 11910 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:26:29.029824 11939 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:29.097980  1801 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:29.099222  1799 raft_consensus.cc:1275] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:29.097334  1805 pending_rounds.cc:107] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:26:29.047089 11929 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:29.091710 11866 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:29.052265 11748 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:29.096269 11786 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.184660 11719 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.083894 11800 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:29.151296  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 49086us
I20260709 12:26:29.185496 11719 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.019402 11555 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.030308 11851 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:29.163825  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:29.165134  1844 leader_election.cc:336] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:29.047804 11488 consensus_queue.cc:1059] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.098s
I20260709 12:26:29.206219 11923 consensus_queue.cc:237] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.163903  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:29.173066  1805 raft_consensus.cc:2889] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:26:29.210537  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46770.3us
I20260709 12:26:29.211850  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:29.212324  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.213028  1696 leader_election.cc:400] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29.
I20260709 12:26:29.213442  1696 leader_election.cc:403] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:29.214144 11593 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:29.137838 11516 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:29.215405 11944 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:29.215819 11944 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:29.216127 11944 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:29.179899  1878 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:26:29.221019 11596 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:29.169914 11933 log.cc:927] Time spent T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:26:29.097966 11283 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.226971 11866 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:29.172482 11939 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:29.079103 11787 consensus_queue.cc:260] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.232995 11939 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:29.234040 11787 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:29.234293 11924 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:26:29.238651 11752 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:29.214547 11593 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:29.192669 11563 raft_consensus.cc:697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:29.241757 11563 consensus_queue.cc:237] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.249259  1803 raft_consensus.cc:1240] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:29.240572 11593 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:29.249624  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:29.167161 11953 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:29.254837  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 337320us
I20260709 12:26:29.260136  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:29.260962  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:29.263268  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.264468  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:29.265118  1806 raft_consensus.cc:2393] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:29.276571  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 111470us
W20260709 12:26:29.276697  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:29.278577  1844 leader_election.cc:336] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:29.278705  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:29.280058  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.214532  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:29.178736 11928 consensus_queue.cc:1059] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.084s
I20260709 12:26:29.176743 11950 consensus_queue.cc:1059] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.091s
W20260709 12:26:29.280227  1844 leader_election.cc:336] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:29.296687  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:29.300025  1844 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:29.285382 11937 log.cc:927] Time spent T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:26:29.308302  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:29.308691  1801 raft_consensus.cc:1275] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:26:29.199604 11892 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.276823 11929 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:29.303821  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:29.309198  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94725.7us
I20260709 12:26:29.315930 11931 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:29.324047  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:29.204849 11851 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.320240  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:29.322844  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.277251 11488 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.325572  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:29.331100  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.355901  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:29.371397  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.372607  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:29.380023  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.383400  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:29.384227  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.373001  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.389039  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:29.392716  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.266165 11953 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.385190  1880 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:29.394552  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.396377  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:29.417244  1875 raft_consensus.cc:2468] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:29.331302 11864 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.075s
I20260709 12:26:29.420946  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:29.332517 11555 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:29.404641  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:29.426591  2415 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:29.427071 11171 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:29.408241  1878 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
W20260709 12:26:29.432564  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 128808us
I20260709 12:26:29.433614  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:29.409941  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.436254  1880 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 32.
I20260709 12:26:29.412415  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.437913  1877 raft_consensus.cc:2393] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 31: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:29.418704  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:29.367916 11928 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:29.327872 11283 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:29.397179 11929 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:29.436316  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.465839  1801 raft_consensus.cc:2393] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 24: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:29.400194  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.425675  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:29.368451 11488 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.467677  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:29.474349  1845 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:29.421113 11962 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.482185  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:29.483295  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.449970 11850 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:29.449652 11554 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:29.499763  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:29.492467  1845 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:29.504067  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.481782 11583 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:29.477501 11672 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:29.527107 11171 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:29.485965 11922 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:29.490934 11850 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:29.533763 11743 consensus_meta.cc:287] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.003s
I20260709 12:26:29.519968 11904 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:29.538371 11913 consensus_meta.cc:287] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.640s	user 0.007s	sys 0.007s
I20260709 12:26:29.530721 11583 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:29.572038 11975 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:26:29.593704  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.502095 11555 consensus_queue.cc:1048] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.077s
I20260709 12:26:29.486116 11962 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.573441 11672 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:29.581732 11850 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:29.604401  1696 leader_election.cc:400] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30.
I20260709 12:26:29.622510  1798 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:29.684626  1696 leader_election.cc:403] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:26:29.572692 11953 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:29.636291  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:29.693336 11988 log.cc:927] Time spent T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.187s	user 0.001s	sys 0.000s
W20260709 12:26:29.694080 11974 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
W20260709 12:26:29.703328 11963 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.229s	user 0.000s	sys 0.000s
I20260709 12:26:29.703850  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.705327  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.706854  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:29.693418 11966 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
W20260709 12:26:29.719461 11967 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.328s	user 0.001s	sys 0.000s
I20260709 12:26:29.720603 11488 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:29.721109 11593 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:29.730222 11596 consensus_meta.cc:287] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.509s	user 0.008s	sys 0.005s
I20260709 12:26:29.605770 11753 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.732771 11753 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:29.735500 11787 consensus_meta.cc:287] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.501s	user 0.013s	sys 0.000s
I20260709 12:26:29.600811 11583 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:29.676326 11892 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:29.710309 11954 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.297s	user 0.002s	sys 0.000s
I20260709 12:26:29.732113  2415 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
W20260709 12:26:29.760038  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 136445us
I20260709 12:26:29.733384  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:29.582468 11923 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:29.750180  1801 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:29.763885  1798 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:29.756237  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:29.756505 11672 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:29.758683  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:29.759027  2382 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:29.759553  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:29.769016  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:29.760958 11752 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.522s	user 0.013s	sys 0.000s
I20260709 12:26:29.675298 11907 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:29.594672 11743 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:29.763278  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 159093us
W20260709 12:26:29.682758  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 190387us
I20260709 12:26:29.766629  2413 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:29.741578 11901 raft_consensus.cc:3055] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:29.765043  1697 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:29.761826  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 322358us
I20260709 12:26:29.776031  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:29.759797  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:29.763715 11898 consensus_meta.cc:287] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.737s	user 0.011s	sys 0.003s
I20260709 12:26:29.761296 11864 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.792065  1807 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 28 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:26:29.792891  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:29.767323 11913 raft_consensus.cc:3060] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:29.800820 11787 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:29.805871  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:29.808581 11991 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.809137 11991 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.809790 11719 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:29.796808  1772 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:29.772627 11753 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:29.812474  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:29.812705  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.762006 11931 consensus_queue.cc:1059] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.036s
W20260709 12:26:29.815395  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:29.800916 11944 consensus_meta.cc:287] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.582s	user 0.011s	sys 0.000s
I20260709 12:26:29.815192 11935 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:29.823892  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:29.814643 11953 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:29.835952 11953 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:29.837085 11953 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.833631 11753 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:29.840540 11939 consensus_meta.cc:287] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.607s	user 0.009s	sys 0.004s
I20260709 12:26:29.841158 11939 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.841958 11913 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30.
I20260709 12:26:29.845999  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:29.847128 11990 raft_consensus.cc:3055] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:29.847631 11990 raft_consensus.cc:740] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:29.848536 11990 consensus_queue.cc:260] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.828665  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:29.794087 11901 raft_consensus.cc:740] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:29.834668  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.816745 11593 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:29.777222 11672 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.812870 11851 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:29.772873 11923 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.813012 11864 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:29.858018 11984 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:26:29.846724  1806 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:26:29.871327  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:29.875737  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.794484 11907 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.892937  1804 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:29.827924  1798 raft_consensus.cc:2468] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
W20260709 12:26:29.877627  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 53873.7us
W20260709 12:26:29.867952 11924 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.629s	user 0.018s	sys 0.008s
I20260709 12:26:29.895287  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:29.895705  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:29.896471  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:29.902937  1875 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:29.896987  1798 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:26:29.901587  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.905339  1876 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:29.896360  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.906386  1873 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:29.908320  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:29.908356  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:29.870509 11990 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:29.864775 11555 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:29.916513  1804 raft_consensus.cc:1240] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
I20260709 12:26:29.909456  1873 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:29.943679 11901 consensus_queue.cc:260] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:29.895565 11945 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.813s	user 0.007s	sys 0.008s
I20260709 12:26:29.951160  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.834206 11944 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:29.895123 11939 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:29.895224 11931 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:29.956079  1806 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
I20260709 12:26:29.907582 11924 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29.
W20260709 12:26:29.969897  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:29.912125 11787 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:29.984004  1878 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:26:29.945291 11935 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:29.914139 11596 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:29.994192  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:29.988543  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:29.994863  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 175492us
I20260709 12:26:29.995872  1771 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:30.003583 11851 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:29.945082 11593 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.077s
W20260709 12:26:29.953083 11913 consensus_queue.cc:1175] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a20e99e125b84513a44e88860667ed61" responder_term: 28 status { last_received { term: 26 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: [27.15-27.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:26:30.014899  1873 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:29.986271 11962 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:29.970844 11753 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:26:29.956840 11555 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:29.993779 11929 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:30.007212 11752 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:30.023712  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:29.961170 11945 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.019508 11991 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:30.048406 11743 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.050125 11907 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.050882 11907 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:30.051424 11907 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.051841 11890 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:30.052466 11890 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.053522 11945 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:30.000424 11596 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:30.061932  1881 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:30.045002 11752 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:30.069674  1805 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.008s	sys 0.014s
W20260709 12:26:30.072643  1805 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.009s	sys 0.015s
I20260709 12:26:30.074198  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:30.079253  1873 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:30.062372 11929 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:30.082454  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:30.091837  1875 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:30.096529  1771 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:30.109529  1879 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:30.070554 11752 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:30.206522  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.232547  1877 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:26:30.161428  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 55117us
I20260709 12:26:30.247435  1872 raft_consensus.cc:1240] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 31. Current term is 32. Ops: [31.22-31.22]
W20260709 12:26:30.254125  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 307966us
I20260709 12:26:30.234192  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:30.260727  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:30.260988  1872 raft_consensus.cc:1275] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:30.242348  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:30.228843  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:30.226410  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 3230) took 1525 ms. Trace:
W20260709 12:26:30.157737  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.265646  1807 rpcz_store.cc:276] 0709 12:26:28.701099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.701560 (+   461us) service_pool.cc:224] Handling call
0709 12:26:30.226156 (+1524596us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54003b79d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:30.226338 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2497651456}
I20260709 12:26:30.185808  2413 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 26.
W20260709 12:26:30.264729  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:30.328413  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.263913  1845 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:30.329631  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65832us
W20260709 12:26:30.193761 11850 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.506s	user 0.004s	sys 0.008s
I20260709 12:26:30.100719 11563 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:30.275139  2412 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:30.147313 11864 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.339939  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60950 (request call id 1684) took 1091 ms. Trace:
I20260709 12:26:30.117506 11986 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:30.341545 11986 raft_consensus.cc:697] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:30.342736 11986 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:30.172077 11907 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.268913 11964 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:26:30.278431  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:30.362264  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 83939.6us
I20260709 12:26:30.192797 11890 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.156224 11672 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:30.135737 11923 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.284914 11990 raft_consensus.cc:2804] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:30.286274  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:30.287545 11522 consensus_queue.cc:1048] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:30.291450 11764 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:30.117734 11992 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:30.265280 11991 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:30.219369 11944 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:30.259296 11555 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:30.312969 11746 consensus_queue.cc:260] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:30.311117 11866 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:30.330538  1845 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:30.400252  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 69832.8us
I20260709 12:26:30.186359 11945 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:30.340420  2412 rpcz_store.cc:276] 0709 12:26:29.248655 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.251486 (+  2831us) service_pool.cc:224] Handling call
0709 12:26:30.242137 (+990651us) spinlock_profiling.cc:243] Waited 1.88 ms on lock 0x7b5800003f98. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:26:30.339855 (+ 97718us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4330368}
I20260709 12:26:30.370262  1806 raft_consensus.cc:1240] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:26:30.407562  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:30.111536 11929 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.340286 11909 raft_consensus.cc:2804] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:30.386799  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:30.395430  1804 raft_consensus.cc:1240] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
W20260709 12:26:30.397112  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 110908us
I20260709 12:26:30.383082 11990 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:30.372072 11907 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:30.467092  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:30.361231 11563 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.469353  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:30.372087 11864 raft_consensus.cc:2804] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:30.392580 11992 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:30.392937 11991 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:30.503258  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:30.359858 11850 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:30.467378 11719 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:30.512913  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:30.473685 11746 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:30.518710  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:30.482152 11990 raft_consensus.cc:3060] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:30.526762  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.521845  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:30.387171 12002 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.128s
I20260709 12:26:30.524667  1799 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: []
I20260709 12:26:30.530596  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:30.534176 11898 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.535077 11898 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:30.508280 11944 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.537392 11898 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.537551 11944 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:30.509014 11991 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:30.503917 11992 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:30.547021 11929 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:30.547369 11583 consensus_meta.cc:287] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.782s	user 0.009s	sys 0.004s
I20260709 12:26:30.548112 11583 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:30.549458 12006 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.151s	user 0.000s	sys 0.000s
I20260709 12:26:30.508708 11928 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.551349 11928 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.554697 11928 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:30.559777 11850 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.511938 11683 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:30.517336 11878 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.047s
I20260709 12:26:30.403187 11923 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:30.512056 11890 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:30.498862 11909 raft_consensus.cc:697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:30.506589 11807 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.975s	user 0.025s	sys 0.028s
I20260709 12:26:30.563962 11683 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.502678 11864 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:30.588274 11787 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:30.589274 11583 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:30.469007 11945 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.522536 11743 raft_consensus.cc:3060] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:30.596701  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:30.636291 11683 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:30.647377  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 116874us
I20260709 12:26:30.636966 11945 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:30.622494 11864 raft_consensus.cc:3060] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:30.527993 11934 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
W20260709 12:26:30.562706 12005 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.165s	user 0.000s	sys 0.003s
I20260709 12:26:30.640793  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:30.525745 11719 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:30.502799 11907 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.645380  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:30.648337  1804 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:30.590415 11807 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.061s	user 0.025s	sys 0.029s
W20260709 12:26:30.582434  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.556195 11913 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:30.566040 11786 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.676980  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:30.646903 11563 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:30.639389 11787 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:30.651299 11928 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:30.688653  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:30.666119  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:30.659938 11945 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:30.661674 11944 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.704563  1876 raft_consensus.cc:2393] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 32: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:30.719770  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.739398  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:30.739219  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:30.741631  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:30.742841  2417 raft_consensus.cc:2393] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:30.743105  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:30.740494  2415 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:26:30.740518  1876 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:30.695591 11683 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.693780 11719 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.744446  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:30.711434 12002 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:30.754869  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.756433  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:30.757475  2414 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:30.762591  2410 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:30.763339 11777 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:30.764165  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.771924  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:30.773166  1798 raft_consensus.cc:2368] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26.
I20260709 12:26:30.775491  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33802 (request call id 3258) took 1378 ms. Trace:
I20260709 12:26:30.778365  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:30.776140  1801 rpcz_store.cc:276] 0709 12:26:29.397318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.503775 (+106457us) service_pool.cc:224] Handling call
0709 12:26:30.775252 (+1271477us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b5400133b54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:30.775417 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":563766144}
I20260709 12:26:30.782927  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:30.746145 11996 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:30.783880  1801 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:30.786960  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:30.729302 11945 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.767473 11777 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:30.805742  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d891ed30ca204d09b7b083088037b63d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 30 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:30.700977 11909 consensus_queue.cc:237] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:30.837926  1876 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:30.766899 11683 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.839128  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:30.793509 11996 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:30.853653  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:30.861147  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:30.862571  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3403) took 1379 ms. Trace:
I20260709 12:26:30.865643  1878 rpcz_store.cc:276] 0709 12:26:29.482915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.483047 (+   132us) service_pool.cc:224] Handling call
0709 12:26:29.507694 (+ 24647us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:29.984635 (+476941us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.850979 (+866344us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b5400223154. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:30.862486 (+ 11507us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1094523264}
W20260709 12:26:30.852027  1874 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3383) took 1973 ms (client timeout 1923 ms). Trace:
W20260709 12:26:30.867795  1874 rpcz_store.cc:269] 0709 12:26:28.881765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.900924 (+ 19159us) service_pool.cc:224] Handling call
0709 12:26:30.851651 (+1950727us) spinlock_profiling.cc:243] Waited 85.8 ms on lock 0x7b540001ead4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:30.851906 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":197419392}
I20260709 12:26:30.869532  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:30.870473  1874 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:30.871666  1798 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:26:30.865682  1879 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:30.873962  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:30.835207 11953 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:30.838564 11936 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:30.871728  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:30.875737  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:30.899399  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.885936  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:30.905267  1881 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3388) took 1919 ms (client timeout 2123 ms). Trace:
W20260709 12:26:30.912961  1881 rpcz_store.cc:269] 0709 12:26:28.985831 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.372736 (+386905us) service_pool.cc:224] Handling call
0709 12:26:30.904983 (+1532247us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b5400223654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:30.905195 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1535255296}
I20260709 12:26:30.914211  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:30.883947  1874 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:30.887799  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:30.917800  1880 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 28: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:30.889626  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1498) took 1688 ms. Trace:
I20260709 12:26:30.918623  1803 rpcz_store.cc:276] 0709 12:26:29.201358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.204368 (+  3010us) service_pool.cc:224] Handling call
0709 12:26:29.248988 (+ 44620us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:29.256239 (+  7251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:30.889369 (+1633130us) spinlock_profiling.cc:243] Waited 44.3 ms on lock 0x7b540009fbd0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd78ad 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:30.889548 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":101805312}
I20260709 12:26:30.919517  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:30.920276  1879 raft_consensus.cc:2410] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 34 because replica has last-logged OpId of term: 33 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 20.
I20260709 12:26:30.928233  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:30.944229  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:30.914211  1802 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33790 (request call id 3367) took 1534 ms. Trace:
I20260709 12:26:30.958949  1802 rpcz_store.cc:276] 0709 12:26:29.379514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.721189 (+341675us) service_pool.cc:224] Handling call
0709 12:26:30.885756 (+1164567us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b5800154598. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb3a22 00007f16b1bcc8d9 00007f16b0d26ee0 00007f16b0df00ab 00007f16b0df0131 00007f16b0d9e0a0 00007f16b0d9d984 00007f16aa3186d4 00007f16aa318662
0709 12:26:30.914147 (+ 28391us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25702784}
W20260709 12:26:31.002772  1614 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 171390us
I20260709 12:26:31.006299  1879 raft_consensus.cc:1240] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [22.14-25.15]
I20260709 12:26:31.009879  1880 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:26:31.013031  1879 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
W20260709 12:26:31.020000  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57796.9us
I20260709 12:26:31.025924  1880 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:26:31.041577 11901 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:31.042063 12017 log.cc:927] Time spent T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260709 12:26:30.883220 11907 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:30.845655 11992 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:30.907634 11953 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:31.060550 11752 consensus_meta.cc:287] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.796s	user 0.013s	sys 0.001s
I20260709 12:26:31.091683  1804 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:31.096069  1874 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:31.098481  1804 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:26:31.104851  1806 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:31.112228  1880 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
W20260709 12:26:31.117347  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 375428us
W20260709 12:26:31.127071  2382 leader_election.cc:336] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:31.128261  2382 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:31.128641  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.129863 11787 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.131071 11787 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:26:31.132700 11746 consensus_meta.cc:287] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.605s	user 0.010s	sys 0.001s
I20260709 12:26:31.135113 11906 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:31.129860  2382 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:31.138047  2382 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:31.138512  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:31.140209  2382 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:31.146656  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:31.151010  2382 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:31.151790  2382 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.152949 11925 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:31.095229 11752 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.121917 11945 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:31.157150  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:31.166311  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 263407us
I20260709 12:26:31.137041 11748 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.140647 11906 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:31.174335 12019 log.cc:927] Time spent T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 12:26:31.174469 11992 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.177879  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:31.185129  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.186851  1807 raft_consensus.cc:1240] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:26:31.186245  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:31.192374 11939 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:31.192858 11939 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:31.193220 11939 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:31.188103 11996 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:31.196552 11990 consensus_meta.cc:287] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.641s	user 0.016s	sys 0.001s
I20260709 12:26:31.197118 11990 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:31.199496 11931 consensus_meta.cc:287] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 1.213s	user 0.017s	sys 0.000s
I20260709 12:26:31.200053 11931 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:31.203361  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:31.204248 11743 consensus_meta.cc:287] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.558s	user 0.007s	sys 0.004s
I20260709 12:26:31.204649 11743 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:31.204803  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:31.205863  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:31.206398  1881 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:31.207671 11953 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:31.209633 11948 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:31.209885 11939 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.210382 11948 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:31.216013 11752 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.218487 11939 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.220003  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.221578  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.172830 11748 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.225240  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:31.226533  2382 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:31.227711  2382 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:31.241122  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:31.173468 11683 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:31.248422  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.253257  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:31.231588 11948 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.247046 11939 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:31.269695  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:31.235574 11743 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:31.274523  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:31.264986 11928 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:26:31.277774  1614 catalog_manager.cc:4755] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): ChangeConfig:REMOVE_PEER RPC failed for tablet 8bb813e4fab14a61b810a24b56899586: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:31.263432 11931 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:31.296233  2382 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:31.292100 11928 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.269299 11746 consensus_queue.cc:1059] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:26:31.318661 11928 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:26:31.324841 11864 consensus_meta.cc:287] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.672s	user 0.011s	sys 0.003s
W20260709 12:26:31.322338 11991 consensus_meta.cc:287] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.782s	user 0.016s	sys 0.000s
W20260709 12:26:31.323000 11866 consensus_meta.cc:287] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.857s	user 0.011s	sys 0.004s
I20260709 12:26:31.370386 11925 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:31.384837 11903 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:31.371595 11993 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 1.217s	user 0.005s	sys 0.003s
W20260709 12:26:31.387393 11979 log.cc:927] Time spent T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 1.582s	user 0.011s	sys 0.005s
I20260709 12:26:31.392122 11991 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.395924 11864 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:31.407444 12032 log.cc:927] Time spent T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:26:31.396359 11913 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.409046 11913 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:31.410665 11777 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.408556 11864 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.412518 11913 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:31.398485 11925 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:31.417668 11925 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:31.420742 11717 raft_consensus.cc:3055] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:31.421300 11717 raft_consensus.cc:740] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:31.429388 11991 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:31.431433 12030 log.cc:927] Time spent T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:26:31.422302 11717 consensus_queue.cc:260] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.436508 11717 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:31.464841 11928 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:31.515995  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.472402 11753 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.547999  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:31.550011  1801 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:31.532349  1876 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 27: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:31.549484 11753 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.564432 11913 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.565979 11913 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.577419 11913 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.583540  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:31.585492  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:31.585877  1798 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1511) took 1061 ms. Trace:
I20260709 12:26:31.587952  1799 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 3257) took 2192 ms. Trace:
I20260709 12:26:31.588415  1799 rpcz_store.cc:276] 0709 12:26:29.397018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:29.800980 (+403962us) service_pool.cc:224] Handling call
0709 12:26:29.801295 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.556371 (+755076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.587373 (+1031002us) spinlock_profiling.cc:243] Waited 719 ms on lock 0x7b54003b83d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:31.587728 (+   355us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1654546816}
I20260709 12:26:31.589977  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:31.596931 11928 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:31.596961  1807 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 3263) took 1229 ms. Trace:
I20260709 12:26:31.598654  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:31.600395  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:31.596751 11903 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.601121  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 23 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:31.602566 11903 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:31.604362 11876 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.606092 11964 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:31.605675 11876 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.607138 11964 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:31.611580  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:31.597867  1807 rpcz_store.cc:276] 0709 12:26:30.367257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.367456 (+   199us) service_pool.cc:224] Handling call
0709 12:26:30.368014 (+   558us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:31.187242 (+819228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.595939 (+408697us) spinlock_profiling.cc:243] Waited 818 ms on lock 0x7b54001d4a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:31.596641 (+   702us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1882493696}
I20260709 12:26:31.615976  1872 raft_consensus.cc:2393] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 23: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:31.620415  2410 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:60928 (request call id 1836) took 2751 ms (client timeout 2910 ms). Trace:
W20260709 12:26:31.621091  2410 rpcz_store.cc:269] 0709 12:26:28.869194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:28.869374 (+   180us) service_pool.cc:224] Handling call
0709 12:26:31.619801 (+2750427us) spinlock_profiling.cc:243] Waited 1.86 s on lock 0x7b5400783054. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:31.620111 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4268584320}
I20260709 12:26:31.624162  2412 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:60928 (request call id 1848) took 1033 ms. Trace:
I20260709 12:26:31.624687  2412 rpcz_store.cc:276] 0709 12:26:30.591106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.591381 (+   275us) service_pool.cc:224] Handling call
0709 12:26:30.637519 (+ 46138us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.694864 (+ 57345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:30.694916 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:30.694980 (+    64us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:30.695026 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.623466 (+928440us) spinlock_profiling.cc:243] Waited 57 ms on lock 0x7b54005b0254. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:31.623581 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.623926 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":131172736}
I20260709 12:26:31.628033  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.586365  1798 rpcz_store.cc:276] 0709 12:26:30.523966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.796690 (+272724us) service_pool.cc:224] Handling call
0709 12:26:30.813092 (+ 16402us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.872293 (+ 59201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.585257 (+712964us) spinlock_profiling.cc:243] Waited 58 ms on lock 0x7b54003b74d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:31.585660 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":133435776}
I20260709 12:26:31.632720 11488 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:31.635551  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.636456  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.639051 11913 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:26:31.642110 11596 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.638341  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.646265  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:31.660791 11890 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:31.662503  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:31.666033 11748 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.675676 11934 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:31.676731  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:31.678602 11938 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:31.679255 11938 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:31.679675 11938 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:31.685963 11753 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.687813 11876 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:31.688099  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:31.691872 11903 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:31.668910 11964 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:31.699090  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:31.717857 12044 log.cc:927] Time spent T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.141s	user 0.001s	sys 0.004s
W20260709 12:26:31.717689  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 79686.1us
I20260709 12:26:31.713701  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:31.741168 11862 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:31.765327 11890 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:31.773895  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 75220.3us
I20260709 12:26:31.788363  1696 leader_election.cc:400] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26.
I20260709 12:26:31.794185  1696 leader_election.cc:403] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
W20260709 12:26:31.792851  1876 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:55690 (request call id 3410) took 1393 ms (client timeout 1746 ms). Trace:
I20260709 12:26:31.812880  1877 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4294) took 1676 ms. Trace:
I20260709 12:26:31.819371 11555 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.821986 11563 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:31.823132 11990 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:31.800390  1876 rpcz_store.cc:269] 0709 12:26:30.397517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.838863 (+441346us) service_pool.cc:224] Handling call
0709 12:26:31.790223 (+951360us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b5400260a54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abddc2fa 00007f16b0d9db17 00007f16aa3186d4 00007f16aa318662 00007f16aa318591 00007f16aa31850c 00007f16aa3177c2 00007f16aa11348c 00007f16aa1128c6 00007f16aa11226f
0709 12:26:31.790476 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1589786496}
W20260709 12:26:31.846211 12047 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.070s	user 0.002s	sys 0.000s
I20260709 12:26:31.852188  1800 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42636 (request call id 1521) took 1171 ms. Trace:
I20260709 12:26:31.852696  1800 rpcz_store.cc:276] 0709 12:26:30.680694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.044004 (+363310us) service_pool.cc:224] Handling call
0709 12:26:31.044453 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:31.115570 (+ 71117us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:31.115621 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.115689 (+    68us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:31.115738 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.851708 (+735970us) spinlock_profiling.cc:243] Waited 70.1 ms on lock 0x7b54001eb9d4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:31.851789 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.852003 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":161208064}
I20260709 12:26:31.856707 11753 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.823179 11672 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.857649 11753 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:31.857990 11672 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:31.860847 11672 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:31.862062 11753 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:31.864595 11938 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:31.871254 11555 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.874191 11555 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:31.894461  1872 raft_consensus.cc:1275] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:26:31.897351  1880 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4299) took 1002 ms. Trace:
I20260709 12:26:31.829283  1877 rpcz_store.cc:276] 0709 12:26:30.136835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.269020 (+132185us) service_pool.cc:224] Handling call
0709 12:26:30.269332 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.636032 (+366700us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:26:30.636389 (+   357us) log.cc:844] Serialized 14 byte log entry
0709 12:26:30.746711 (+110322us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.204398 (+457687us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b18004e7900. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb522 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:26:31.810561 (+606163us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b18004e7900. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16a8ceda40 00007f16a8ce84e4 00007f16a8ce8455 00007f16abdd0060 00007f16abde2716 00007f16abdf3287 00007f16abdf3208
0709 12:26:31.811032 (+   471us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.811118 (+    86us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:26:31.811201 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.812289 (+  1088us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b5400206cd4. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:31.812334 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.812625 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1063023,"prepare.queue_time_us":16059,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":92,"raft.queue_time_us":1071193,"raft.run_cpu_time_us":648,"raft.run_wall_time_us":16379,"spinlock_wait_cycles":890738176,"thread_start_us":94659,"threads_started":1,"wal-append.queue_time_us":193845}
I20260709 12:26:31.899993  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:31.888989 11903 raft_consensus.cc:3060] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:31.902639  1878 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4296) took 1655 ms. Trace:
I20260709 12:26:31.903192  1878 rpcz_store.cc:276] 0709 12:26:30.247014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.929990 (+682976us) service_pool.cc:224] Handling call
0709 12:26:31.038345 (+108355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:31.038654 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:31.316523 (+277869us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.901861 (+585338us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.902011 (+   150us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:31.902108 (+    97us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.902239 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.902414 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":185709,"prepare.queue_time_us":103862,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":79,"raft.queue_time_us":176103,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":199,"thread_start_us":598721,"threads_started":2,"wal-append.queue_time_us":408908}
I20260709 12:26:31.904950 11936 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:31.906015 12046 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
W20260709 12:26:31.908821 12045 log.cc:927] Time spent T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:26:31.910815  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:31.911731  1876 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:31.914745  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.897850  1880 rpcz_store.cc:276] 0709 12:26:30.895333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.117818 (+222485us) service_pool.cc:224] Handling call
0709 12:26:31.119059 (+  1241us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:31.119389 (+   330us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:26:31.312736 (+193347us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.716364 (+403628us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:31.896294 (+179930us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.896535 (+   241us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.896597 (+    62us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:26:31.896762 (+   165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.896901 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.897116 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":212606,"prepare.queue_time_us":50398,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":2754,"raft.queue_time_us":220845,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"spinlock_wait_cycles":54764416,"thread_start_us":520639,"threads_started":2,"wal-append.queue_time_us":262585}
I20260709 12:26:31.955610 11938 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:31.956506  2382 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:31.957197  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71053.6us
W20260709 12:26:31.957816  2382 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:31.960176 11890 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:31.964540 11903 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 25. Current term is 26.
W20260709 12:26:31.966815  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 170872us
I20260709 12:26:31.923233 11555 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:31.968705 11555 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:31.970021  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 208020us
I20260709 12:26:31.969805 11938 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:26:31.967221  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 44342.1us
I20260709 12:26:31.975324 11736 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:31.974895  1803 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 3276) took 1068 ms. Trace:
I20260709 12:26:31.977458  1803 rpcz_store.cc:276] 0709 12:26:30.824748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.185179 (+360431us) service_pool.cc:224] Handling call
0709 12:26:31.185483 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:31.185709 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:26:31.616685 (+430976us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.763783 (+147098us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.778543 (+ 14760us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.783770 (+  5227us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.785352 (+  1582us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:26:31.829631 (+ 44279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.837041 (+  7410us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.860570 (+ 23529us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b180034cba0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8ceb557 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd 0000000000388634 0000000000388479 00007f16a8cce83a
0709 12:26:31.892356 (+ 31786us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72696,"prepare.queue_time_us":214084,"prepare.run_cpu_time_us":188,"prepare.run_wall_time_us":3086,"raft.queue_time_us":5582,"raft.run_cpu_time_us":540,"raft.run_wall_time_us":9945,"spinlock_wait_cycles":4955008,"thread_start_us":367205,"threads_started":2,"wal-append.queue_time_us":78652}
I20260709 12:26:31.979377  1772 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:31.979912 11903 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.980901  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.980998 11903 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:31.983119  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 825913us
I20260709 12:26:31.983645 11936 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.984725 11903 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:31.984520 11936 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.987725  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.987908 11991 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.988886 11991 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:31.989763  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:31.991132 11945 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:31.991386  1844 connection.cc:256] RPC call timeout handler was delayed by -1.0373342570001114s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1779s for a response
I20260709 12:26:31.992123 11945 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:31.992486 11583 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:31.993695 11717 consensus_meta.cc:287] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.556s	user 0.008s	sys 0.015s
I20260709 12:26:31.993355 11583 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:31.992139 11785 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:31.995239 11945 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:31.996304 11736 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:31.998230 11990 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:31.998760 11583 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:32.001266  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:32.003005 11903 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:32.005110  1771 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:32.008579  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:32.009320  2410 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:31.998539  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4287) took 2045 ms. Trace:
W20260709 12:26:32.025074  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 46064.2us
I20260709 12:26:32.011191 11903 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:32.049799  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.037101  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.028860 11876 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:32.010497 11990 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:26:32.071229  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.016867  1875 rpcz_store.cc:276] 0709 12:26:29.952945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.095185 (+142240us) service_pool.cc:224] Handling call
0709 12:26:30.095554 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.473311 (+377757us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:26:30.505971 (+ 32660us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.980851 (+1474880us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.981033 (+   182us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:26:31.981179 (+   146us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.997911 (+ 16732us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b5400073d54. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd8e01 00007f16abdd783e 00007f16b0d9c419 00007f16aa313bd4 00007f16aa313b62 00007f16aa313a91 00007f16aa313a0c 00007f16aa312cc2 00007f16aa11348c 00007f16aa1128c6
0709 12:26:31.997999 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:31.998264 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1001795,"prepare.queue_time_us":38801,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":1009075,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":202,"spinlock_wait_cycles":868083968,"thread_start_us":423328,"threads_started":2,"wal-append.queue_time_us":631303}
I20260709 12:26:32.076599  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:32.078574  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89023.9us
I20260709 12:26:32.054939 11876 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.082589  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.086808  2414 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 22.
I20260709 12:26:32.087376 11683 consensus_queue.cc:1048] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:26:32.088150  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.091292 11753 consensus_queue.cc:1059] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:32.091727 11852 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:26:32.091509 11903 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.092501 11852 raft_consensus.cc:697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:32.092794 11934 raft_consensus.cc:3055] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:26:32.093331 11934 raft_consensus.cc:740] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:32.093729 12049 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.093837 11852 consensus_queue.cc:237] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.094560 11934 consensus_queue.cc:260] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.095183  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.094574 12049 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.096957  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:32.098244  1630 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 reported cstate change: term changed from 20 to 25, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:32.101831  1798 raft_consensus.cc:1275] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:26:32.104130  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.085124 11876 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.118090  1875 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:32.105087  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:32.105643  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.106010  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.106528 11986 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.106935  1879 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58100 (request call id 1531) took 1331 ms. Trace:
I20260709 12:26:32.107856  1878 raft_consensus.cc:1240] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:32.108857  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.108600 11717 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.134912  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.138346  2410 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:26:32.148670  1803 raft_consensus.cc:2410] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 33 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:26:32.113348 11913 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.151105  1876 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:32.153822  1801 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:32.155449 12055 consensus_queue.cc:1048] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:26:32.158316  1624 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 176) took 73965 ms (client timeout 15000 ms). Trace:
I20260709 12:26:32.159005  1872 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:32.160346  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.161907 11986 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.166419 11991 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:32.167521  2414 raft_consensus.cc:1240] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 24. Current term is 25. Ops: []
I20260709 12:26:32.168756  1630 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 17 to 21, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 21 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:32.178014  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:32.178728  1880 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:32.179759  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:32.175386  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:32.178030  1873 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4286) took 2164 ms. Trace:
I20260709 12:26:32.175992  1799 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 31.
W20260709 12:26:32.158869  1624 rpcz_store.cc:269] 0709 12:25:18.193199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.193561 (+   362us) service_pool.cc:224] Handling call
0709 12:26:32.157997 (+73964436us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:27.545174 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:27.757821 (+212647us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:27.783177 (+ 25356us) write_op.cc:435] Acquired schema lock
0709 12:26:27.787156 (+  3979us) tablet.cc:662] Decoding operations
0709 12:26:27.801587 (+ 14431us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:27.801698 (+   111us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:27.801726 (+    28us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:26:27.802518 (+   792us) tablet.cc:701] Row locks acquired
0709 12:26:27.802575 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:26:27.802851 (+   276us) write_op.cc:270] Start()
0709 12:26:27.803172 (+   321us) write_op.cc:276] Timestamp: P: 1783599987802702 usec, L: 0
0709 12:26:27.803216 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 12:26:27.955833 (+152617us) log.cc:844] Serialized 2814 byte log entry
0709 12:26:28.598512 (+642679us) spinlock_profiling.cc:243] Waited 32.6 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd3e28 00007f16af80e581 00007f16af80dd58 00007f16af8129f1 00007f16af8129a9 00007f16af812939 00007f16af812901 00007f16af811b2d 0000000000388634 0000000000388479
0709 12:26:28.789880 (+191368us) op_driver.cc:464] REPLICATION: finished
0709 12:26:29.162622 (+372742us) write_op.cc:301] APPLY: starting
0709 12:26:29.170113 (+  7491us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:29.221150 (+ 51037us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:29.221181 (+    31us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:30.217194 (+996013us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b18004696c0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16abb170ec 00007f16abb19af7 00007f16abb1afb0 00007f16af82f1bb 00007f16af8765d4 00007f16af7228da 00007f16af7253cb 00007f16af724fd3 00007f16af823e34
0709 12:26:30.217918 (+   724us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:30.218202 (+   284us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 12:26:30.218260 (+    58us) write_op.cc:312] APPLY: finished
0709 12:26:30.218578 (+   318us) log.cc:844] Serialized 73 byte log entry
0709 12:26:30.533223 (+314645us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:30.534001 (+   778us) write_op.cc:454] Released schema lock
0709 12:26:30.534072 (+    71us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:30.534592 (+   520us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:30.534628 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2740536,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":306011,"apply.run_cpu_time_us":46785,"apply.run_wall_time_us":1389563,"cfile_cache_hit":30,"cfile_cache_hit_bytes":33415,"num_ops":7,"prepare.queue_time_us":975520,"prepare.run_cpu_time_us":16084,"prepare.run_wall_time_us":1113181,"raft.queue_time_us":219153,"raft.run_cpu_time_us":15372,"raft.run_wall_time_us":225508,"replication_time_us":957152,"spinlock_wait_cycles":81781376,"thread_start_us":1315088,"threads_started":5,"wal-append.queue_time_us":979282,"wal-append.run_cpu_time_us":19002,"wal-append.run_wall_time_us":2220280}]]}
I20260709 12:26:32.195909  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.196244 11890 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:32.199229 12049 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:32.200066 12050 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:32.200747  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 293382us
I20260709 12:26:32.203341  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.203698 11913 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.200802 12050 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.121886  1879 rpcz_store.cc:276] 0709 12:26:30.686187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.015989 (+329802us) service_pool.cc:224] Handling call
0709 12:26:31.016365 (+   376us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:31.016660 (+   295us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:26:31.064707 (+ 48047us) log.cc:844] Serialized 14 byte log entry
0709 12:26:31.967021 (+902314us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:31.967147 (+   126us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:31.967274 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:31.970945 (+  3671us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:32.017969 (+ 47024us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":545217,"prepare.queue_time_us":65001,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":64,"raft.queue_time_us":536566,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":84,"thread_start_us":336738,"threads_started":2,"wal-append.queue_time_us":247673}
I20260709 12:26:32.206264 11913 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:32.206486  1881 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4300) took 1290 ms. Trace:
I20260709 12:26:32.175175 11555 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:32.207327  2414 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:26:32.204151  1876 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:32.170704  1630 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 23 to 30, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 30 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.209370 12050 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:32.143594 11488 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.209373  1630 catalog_manager.cc:5697] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 20 to 27. New cstate: current_term: 27 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.180732  1873 rpcz_store.cc:276] 0709 12:26:29.913525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:30.084027 (+170502us) service_pool.cc:224] Handling call
0709 12:26:30.084465 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:26:30.084750 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:26:30.385623 (+300873us) log.cc:844] Serialized 14 byte log entry
0709 12:26:32.030227 (+1644604us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:26:32.046606 (+ 16379us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:26:32.046751 (+   145us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.058506 (+ 11755us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:32.077947 (+ 19441us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":990144,"prepare.queue_time_us":153046,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":997464,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":273,"thread_start_us":928094,"threads_started":3,"wal-append.queue_time_us":650062,"wal-append.run_cpu_time_us":1483,"wal-append.run_wall_time_us":1006317}
I20260709 12:26:32.219146  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.191417 11717 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.223560  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.225265  1875 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 33.
I20260709 12:26:32.225332 11717 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:32.226425  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.227152  1875 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 32.
I20260709 12:26:32.218642  1630 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 22 to 27, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 27 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:32.227317  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:32.227977  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.228401  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.228935  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.228502  1630 catalog_manager.cc:5697] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 20 to 25, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 25 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.230415 11876 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.227238  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.232398  1876 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
W20260709 12:26:32.233548  1770 leader_election.cc:336] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:32.234239  1770 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:32.213826  1802 raft_consensus.cc:2468] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 23.
I20260709 12:26:32.234371 11876 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:32.229354  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.234885  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.180881 11990 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:32.212189  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:32.246971  1798 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:32.176959 11986 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:32.235016  1630 catalog_manager.cc:5697] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 19 to 27, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:32.259739  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.210729 11488 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.256711 11862 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:26:32.261042  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.266543  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:32.207016  1881 rpcz_store.cc:276] 0709 12:26:30.916189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.215030 (+298841us) service_pool.cc:224] Handling call
0709 12:26:31.215384 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:32.052733 (+837349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.057208 (+  4475us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:32.091105 (+ 33897us) log.cc:844] Serialized 26 byte log entry
0709 12:26:32.133701 (+ 42596us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:32.133748 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.144247 (+ 10499us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:32.206156 (+ 61909us) raft_consensus.cc:1685] finished
0709 12:26:32.206175 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:32.206328 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":602325,"raft.queue_time_us":609068,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":193,"thread_start_us":242794,"threads_started":2,"wal-append.queue_time_us":89483}
I20260709 12:26:32.269044 11890 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:32.269263  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:32.269642  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.270118  1875 raft_consensus.cc:2393] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 27: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:32.269816 11890 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.270620  1798 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:32.271148  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.271797  1875 raft_consensus.cc:2468] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:32.272266  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.272850  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:32.273309  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.273550  1875 raft_consensus.cc:2376] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29. Re-sending same reply.
I20260709 12:26:32.274073  1802 raft_consensus.cc:2393] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 31: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:32.274921  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.275682  1805 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:32.276299  1873 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260709 12:26:32.278282  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:32.245038 11964 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.281260  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
W20260709 12:26:32.286362  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 293910us
I20260709 12:26:32.287132 11890 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:32.287407  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:32.289978  1872 raft_consensus.cc:2393] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 27: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:32.283532 11876 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:32.298187  1630 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 reported cstate change: term changed from 22 to 29, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 29 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:32.294099  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:32.284969 11717 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.314606  2414 raft_consensus.cc:1240] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 28. Current term is 29. Ops: []
W20260709 12:26:32.314904  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 188618us
I20260709 12:26:32.319444  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:32.288827 11990 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.320614 11986 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:32.332218 11683 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.006s
I20260709 12:26:32.336535  1630 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 28, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 28 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:32.307884 11964 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.344977 11717 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.348831  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:32.349076 11736 consensus_queue.cc:1048] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:26:32.352872  1876 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:26:32.356259 11936 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:32.344255  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82768.6us
I20260709 12:26:32.365509  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.369915  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.365875  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:32.321610 11903 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:32.374998  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 54877.3us
I20260709 12:26:32.376573  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:32.369678  1630 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 22 to 27, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:32.380015 12050 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:32.382014  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:32.382856  2413 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.380049  1630 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 20 to 25, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 25 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.388577 11964 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:32.391343 11934 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:32.397426 11903 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.398484  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.400593 11903 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:32.400612  1879 raft_consensus.cc:1240] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 32. Current term is 33. Ops: []
I20260709 12:26:32.389964  1630 catalog_manager.cc:5697] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 23 to 29. New cstate: current_term: 29 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:32.362692 11986 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:32.371345  1844 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:32.412739  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46952.8us
W20260709 12:26:32.425869  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 192417us
I20260709 12:26:32.413993 11986 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:32.422727 12050 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:32.432976  1880 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:32.441216 12050 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.447606  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.376492 11683 consensus_queue.cc:1059] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.061s
I20260709 12:26:32.448752 11683 raft_consensus.cc:3055] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:32.449189 11683 raft_consensus.cc:740] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:32.447613 12050 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:32.451540 11488 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:32.435734 11748 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.017s
I20260709 12:26:32.433687  1630 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 15 to 25, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 25 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:32.459156  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 156216us
I20260709 12:26:32.470736  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.473551  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:32.474640  2413 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:32.482594  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:32.468750 11964 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:32.494724 12062 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
W20260709 12:26:32.495265  1770 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:32.495870  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 147079us
I20260709 12:26:32.450456 11683 consensus_queue.cc:260] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.515372  1697 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:32.516615 11683 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:32.517904 11748 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:32.480517 11785 raft_consensus.cc:2764] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:26:32.519685 12055 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:32.519984 11936 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:32.520511 12055 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.521253  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:32.522017 11990 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:32.524053  1630 catalog_manager.cc:5697] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 reported cstate change: term changed from 14 to 25. New cstate: current_term: 25 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:32.525873  1630 catalog_manager.cc:5697] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 25 to 31. New cstate: current_term: 31 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:26:32.542404  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 210582us
I20260709 12:26:32.546510  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.547715  2414 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 26.
I20260709 12:26:32.495901 11717 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:32.551445  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.553099  1804 raft_consensus.cc:1275] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:26:32.555632 12055 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:32.556460  1844 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:32.565122  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 104148us
I20260709 12:26:32.573431  1630 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 reported cstate change: term changed from 26 to 31, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 31 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:32.567665  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:32.575222  1630 catalog_manager.cc:5697] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 17 to 22, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 22 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.579208  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:32.578703  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 60652.6us
I20260709 12:26:32.582484  1806 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:32.511680 11964 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:32.586726  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65590.8us
W20260709 12:26:32.592139  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 109690us
I20260709 12:26:32.593173  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:32.570046 11938 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:32.589186 11876 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:26:32.598380  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 103195us
I20260709 12:26:32.603645  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:32.593410 11936 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.605283 11945 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:32.609833  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:32.605309 11583 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:32.605484 12067 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:32.607733 12051 consensus_queue.cc:1048] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:32.606001  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:32.605875  1630 catalog_manager.cc:5697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 22 to 30, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 30 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.602269 11876 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:32.621570  1770 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:32.622738  1874 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55690 (request call id 3419) took 1852 ms. Trace:
W20260709 12:26:32.624639  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68278.5us
I20260709 12:26:32.623099 11555 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:32.630075 11555 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.630519 11555 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:32.608635 11991 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:32.635317  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 45778.8us
I20260709 12:26:32.622156 11876 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:32.602979 11785 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.027s
I20260709 12:26:32.641639 11753 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:32.642247 11717 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:32.623221  1874 rpcz_store.cc:276] 0709 12:26:30.768165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:31.097334 (+329169us) service_pool.cc:224] Handling call
0709 12:26:31.097742 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:32.066005 (+968263us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.069923 (+  3918us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:32.099234 (+ 29311us) log.cc:844] Serialized 26 byte log entry
0709 12:26:32.235293 (+136059us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:26:32.235334 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.243924 (+  8590us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:32.615637 (+371713us) raft_consensus.cc:1685] finished
0709 12:26:32.615662 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:32.620890 (+  5228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":718987,"raft.queue_time_us":726092,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":111,"thread_start_us":333971,"threads_started":2,"wal-append.queue_time_us":236760}
I20260709 12:26:32.646088  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.647116  1630 catalog_manager.cc:5697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 21 to 26, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 26 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.626389 11862 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:32.654668  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:32.660171  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:32.616123 11936 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:32.657891  1844 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:32.664529  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.665511  2415 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 32.
W20260709 12:26:32.667219  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 69714.3us
I20260709 12:26:32.656744 11717 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.668609  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:32.669787 11717 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:32.670257 11488 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:32.671027 11488 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.671490 11488 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.670264  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:32.682332  2411 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:32.678735  1630 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 reported cstate change: term changed from 20 to 28, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 28 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.687633  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.692836 11753 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:32.694727  1696 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:32.700807 11990 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:32.701479 11990 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 33 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:32.648325 11583 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.702541 11990 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.15, Last appended by leader: 15, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.705705 11583 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:32.706642  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 91145.1us
W20260709 12:26:32.707604  1770 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:32.709235 12064 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.182s	user 0.000s	sys 0.004s
W20260709 12:26:32.710773  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72461.5us
I20260709 12:26:32.708143  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:32.712793  1844 leader_election.cc:336] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:32.713290  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:32.714165  1770 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:32.714735 12072 log.cc:927] Time spent T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 12:26:32.715463  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 109545us
I20260709 12:26:32.716288 12051 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:32.717375  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 64159.5us
I20260709 12:26:32.708979  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:32.721083  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.716208 12050 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.740355  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:32.736408  1770 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:32.742913 11903 raft_consensus.cc:2804] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:32.743394 11903 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.743788 11903 raft_consensus.cc:3060] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:32.744678 11854 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:32.744913  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.670454 11717 consensus_queue.cc:1059] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.065s
I20260709 12:26:32.748437 11913 raft_consensus.cc:3055] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:32.748874 11913 raft_consensus.cc:740] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:32.749859 11913 consensus_queue.cc:260] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:32.753041 11913 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:32.758939 12071 log.cc:927] Time spent T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:26:32.761092 12054 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:32.763656  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.764449  1806 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:32.739368 11583 consensus_queue.cc:1059] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.025s
I20260709 12:26:32.768654  2411 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 24.
I20260709 12:26:32.771694  1696 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:32.778632  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.779932  2410 raft_consensus.cc:2410] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 33 because replica has last-logged OpId of term: 31 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 21.
I20260709 12:26:32.774967  1630 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 20 to 28, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 28 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:32.785494  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:32.742908 11748 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:32.788888  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:32.795763  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.699585 11986 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:32.803139  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 90382.1us
I20260709 12:26:32.765689 11583 raft_consensus.cc:3055] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:26:32.798840  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.808032  2415 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:26:32.806715  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:32.826258  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 82269.3us
I20260709 12:26:32.795666 11748 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.795509 11991 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.828208  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:32.832921  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:32.845412  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 56623.4us
I20260709 12:26:32.846064  1799 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 29. Current term is 30. Ops: [29.20-29.20]
I20260709 12:26:32.806375 11717 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:26:32.856861  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.807447 11583 raft_consensus.cc:740] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:32.832720  1630 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 24 to 27, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 27 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.863574 12075 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:32.864971  1844 leader_election.cc:336] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:32.851671 11934 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:32.873653  1844 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:32.841361 11748 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:32.887233  1872 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 31.
I20260709 12:26:32.891621  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.892944  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:32.894299  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 100438us
I20260709 12:26:32.883769  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.889099  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:32.896637  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 182531us
I20260709 12:26:32.897929  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.898236  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:32.899068  1872 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:32.893818  1803 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:32.898926  1874 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 32.
I20260709 12:26:32.878487 11934 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.907370  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.918133  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:32.914711  1630 catalog_manager.cc:5697] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 24 to 31, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 31 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:32.895705 12054 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:26:32.931002  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 68536.5us
I20260709 12:26:32.903270 12067 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:32.895645 11717 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
W20260709 12:26:32.933781  1770 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:32.944926  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:32.935663  1630 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 22 to 26, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 26 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:32.940179  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 51219.1us
I20260709 12:26:32.947458 12067 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:32.949654  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.950593  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.914471 11934 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:32.957158  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:32.949126  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:32.948238 12070 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.315s	user 0.000s	sys 0.003s
I20260709 12:26:32.901165 11991 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:32.959609  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:32.948608  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:32.948163 12067 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:32.973544  2382 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 123 similar messages]
I20260709 12:26:32.981096  1802 raft_consensus.cc:2468] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:32.981200  1879 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:32.933562 11488 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:32.973784 11936 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:32.977236  1630 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 reported cstate change: term changed from 22 to 27, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 27 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:32.987370  1881 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:33.011004  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.016980 11903 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.022189 11903 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.025445 11583 consensus_queue.cc:260] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.978922 11522 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.027091 12087 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:33.027225 11522 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.030408 11522 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:32.975178  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:33.020153 11991 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:33.019731 12088 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:33.032076 12067 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.032964 11717 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 22.18, Last appended by leader: 18, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.017292 11876 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:32.988268  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:33.083096  1630 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 15 to 21, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 21 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:32.987929 11990 raft_consensus.cc:493] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.972992 12054 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:33.108937  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 120760us
W20260709 12:26:33.110965  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 191890us
I20260709 12:26:33.122321  1630 catalog_manager.cc:5697] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 22 to 25. New cstate: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:33.107239 11986 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.129835  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:33.140239  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:33.132021  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 215893us
W20260709 12:26:33.132850  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 176021us
I20260709 12:26:33.135402  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:33.189308  1806 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:33.136538  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 232632us
I20260709 12:26:33.138592  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:33.111522 11852 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:33.194914  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70235.8us
I20260709 12:26:33.192822  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:33.203542 11583 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:33.208096 11753 consensus_meta.cc:287] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.515s	user 0.011s	sys 0.000s
I20260709 12:26:33.208693 11753 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:33.209862  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:33.211019 12101 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:33.212122 12101 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:33.212738 12101 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:33.217665 11748 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.219115  1806 raft_consensus.cc:2468] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 32.
I20260709 12:26:33.220095  1630 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 27 to 31, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 31 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:33.220459  1615 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet d891ed30ca204d09b7b083088037b63d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 6084fbc6918e4e0b95a9bd0206a620d8 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 31 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:33.210659  1807 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:33.257243  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:33.258193  1799 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: []
I20260709 12:26:33.260020  1630 catalog_manager.cc:5697] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 23, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 23 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:33.261802 12050 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:33.262120  1630 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 23, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 23 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:33.263897  1630 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 23 to 27, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 27 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:33.264816  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.265683  1630 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 reported cstate change: term changed from 17 to 24, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 24 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:33.224561  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:33.196215 12070 log.cc:927] Time spent T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:26:33.254841  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 280040us
I20260709 12:26:33.140799 12067 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:33.117827 11990 raft_consensus.cc:515] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:33.292019  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.273741  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:26:33.268098  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 64880.7us
I20260709 12:26:33.234324 11934 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.324314 11934 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.252449 11913 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.273501  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:33.327371 11913 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.329797  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:33.234678 11991 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:33.344074 12091 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.332491 11913 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:33.333204 11683 consensus_meta.cc:287] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.816s	user 0.010s	sys 0.003s
I20260709 12:26:33.344797 12091 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.329726  1630 catalog_manager.cc:5697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 reported cstate change: term changed from 23 to 26, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 26 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:33.347471  1630 catalog_manager.cc:5697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 reported cstate change: term changed from 18 to 23, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 23 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:33.333844  1800 raft_consensus.cc:2410] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:26:33.331833 12086 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:33.335354 11488 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:33.339592  1875 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:55680 (request call id 4310) took 1114 ms. Trace:
W20260709 12:26:33.333133  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 77737.1us
W20260709 12:26:33.340262 12074 log.cc:927] Time spent T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.376s	user 0.001s	sys 0.000s
I20260709 12:26:33.340817  1879 raft_consensus.cc:1275] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:26:33.341342 12090 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:26:33.342394 12085 log.cc:927] Time spent T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20260709 12:26:33.342211  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 68574.9us
I20260709 12:26:33.343037 12067 raft_consensus.cc:2804] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:26:33.343421  1847 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.375818 11683 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.008s
I20260709 12:26:33.388336 12054 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.389312  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:33.389997  2412 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:33.333519 11852 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:33.290040 12077 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:33.333469 12063 log.cc:927] Time spent T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.818s	user 0.003s	sys 0.001s
I20260709 12:26:33.267452 11945 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:33.354259  1875 rpcz_store.cc:276] 0709 12:26:32.225608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.274266 (+ 48658us) service_pool.cc:224] Handling call
0709 12:26:32.274604 (+   338us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:26:32.283800 (+  9196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:32.283836 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:26:32.303325 (+ 19489us) log.cc:844] Serialized 52 byte log entry
0709 12:26:32.589771 (+286446us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:32.603952 (+ 14181us) log.cc:844] Serialized 14 byte log entry
0709 12:26:32.610482 (+  6530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:32.610566 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:32.713733 (+103167us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b18004a9020. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8c65c0a 00007f16b23a5219 00007f16afcac58f 00007f16a8ceb571 00007f16a8ced7c1 00007f16a8ced779 00007f16a8ced709 00007f16a8ced6d1 00007f16a8cec9cd
0709 12:26:32.852433 (+138700us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b4400022b4c. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8c11c85 00007f16a8c126a2 00007f16ab9841db 00007f16ab97faaf 00007f16ab97386e 00007f16ab972bb6 00007f16ab9727d8 00007f16ab97c221 00007f16ab97c1d9 00007f16ab97c169
0709 12:26:33.339344 (+486911us) raft_consensus.cc:1685] finished
0709 12:26:33.339369 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:33.339526 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12547,"prepare.queue_time_us":10546,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":6236,"raft.run_cpu_time_us":254,"raft.run_wall_time_us":257,"spinlock_wait_cycles":7694720,"thread_start_us":267872,"threads_started":1,"wal-append.queue_time_us":318636}
I20260709 12:26:33.339830 11876 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.368690 11990 leader_election.cc:290] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.376628  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
I20260709 12:26:33.271235 12100 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:33.343961 11991 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.389263  1872 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
W20260709 12:26:33.378623  1770 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.396273  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:33.377317 12067 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.409502  1798 raft_consensus.cc:2468] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
W20260709 12:26:33.399349 11555 consensus_meta.cc:287] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.768s	user 0.011s	sys 0.003s
I20260709 12:26:33.402889  1845 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:33.400820 11852 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:33.416410 11991 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.416021 11852 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.419142 12078 consensus_queue.cc:1059] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 32.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:33.420576 12078 raft_consensus.cc:3055] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:26:33.421018 12078 raft_consensus.cc:740] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:33.421689 11748 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:33.423138  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 592809us
I20260709 12:26:33.409060 12100 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:33.428056  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 51527.2us
I20260709 12:26:33.427942 11555 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.429096  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:33.429239 11876 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:33.429595 12100 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.407442 12067 raft_consensus.cc:3060] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:26:33.430374 11555 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:33.412426 12077 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.430967 12054 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:33.414141  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:33.421972 12078 consensus_queue.cc:260] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.15, Last appended by leader: 15, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.407377 11717 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.440001 12100 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:33.442613 12096 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.447072 12054 raft_consensus.cc:697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 32 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:33.448263 11488 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.448796  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:33.448864 11748 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:33.450271 12078 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:33.437288 11990 raft_consensus.cc:2804] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:33.456985 12101 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:33.464964  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 92194.7us
W20260709 12:26:33.469169  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55138.3us
I20260709 12:26:33.486696  2412 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
W20260709 12:26:33.487445  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63432.6us
W20260709 12:26:33.493424  1844 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.489980  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:33.491763 11990 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:33.493963  1844 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.494692 11990 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:33.496549 12101 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:33.493708 11913 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:33.433373 12091 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:33.499027  2417 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 31. Current term is 32. Ops: []
I20260709 12:26:33.488269 12100 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.501987  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:33.502882  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.503006  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:33.506243 11748 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:33.506609  1844 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.523590  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.523541 11934 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:33.525861 11753 raft_consensus.cc:3060] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:33.526270 12092 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:33.530997 12107 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.532253  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.532004 12107 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.533588  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:33.492306 11488 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.537408 11903 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:33.537814 11488 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.538221 11903 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:33.538726 11903 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.539137 11903 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:33.507982 12088 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:26:33.541159 12088 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:26:33.541677 12088 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:33.544322  2412 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:33.529094 11964 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:33.563024  1844 leader_election.cc:336] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:33.564014  1770 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.524659 12100 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:33.569278 11748 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.569662 11913 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:33.593273 11913 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.594659 11522 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:33.596495 12101 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:33.596808 11913 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.597116 12101 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:33.570766  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:33.571388  1770 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.598318  1806 raft_consensus.cc:3060] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:33.599380  1770 leader_election.cc:336] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.572664  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:33.548592 11936 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:33.542817 11753 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:33.598485 12101 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.577252 11934 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.545444 12091 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.590260  2412 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 33.
W20260709 12:26:33.591082  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 89669.7us
I20260709 12:26:33.599867  1770 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:33.600493  1874 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:26:33.548770 12107 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:33.492489 12096 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.631969  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:33.615377 12108 log.cc:927] Time spent T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.149s	user 0.002s	sys 0.000s
I20260709 12:26:33.635238  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:33.635536  1806 raft_consensus.cc:2468] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 30.
W20260709 12:26:33.641145  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 75323us
I20260709 12:26:33.502933 11717 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.610288 11934 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.643957 11964 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:33.651589 12111 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:26:33.636799 12054 consensus_queue.cc:237] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.660372 11717 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:33.662643 11934 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:33.666915  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 132412us
I20260709 12:26:33.668704  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.542760 12088 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.672462  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.665926 12077 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.622512 12091 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.712604  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:33.689646  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:33.713465  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 608048us
W20260709 12:26:33.690364  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 91068us
I20260709 12:26:33.692143 12096 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.695643 11488 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.694567 12092 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:33.654588 12067 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:33.701484 12051 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:33.702224 12055 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:33.711126  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:33.708647  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
W20260709 12:26:33.688970   561 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 1757)
I20260709 12:26:33.722330  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:33.736162  2413 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
W20260709 12:26:33.723905  1695 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.686811 11934 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:33.725426  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:33.735219  1771 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:33.725590  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 53280us
I20260709 12:26:33.674520  2383 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: a20e99e125b84513a44e88860667ed61
W20260709 12:26:33.729964  1846 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.713881  1697 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e; no voters: 
I20260709 12:26:33.736588 12088 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:26:33.743325  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:33.817812  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 74548.8us
W20260709 12:26:33.790238  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46247.3us
W20260709 12:26:33.811012  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52993.3us
I20260709 12:26:33.813019  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:33.814080  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:33.733913 11748 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:33.825050  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62531.7us
I20260709 12:26:33.824182  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e
I20260709 12:26:33.738703 12096 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:33.819362  1844 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.840530  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.745543 11876 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:33.736178 11736 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.745949 12075 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:33.842931 12119 raft_consensus.cc:2804] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:33.788690 12098 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:33.861783  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:33.862421  1770 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.725440 11903 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:33.866401  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 130519us
I20260709 12:26:33.737653 11852 consensus_queue.cc:1048] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.097s
I20260709 12:26:33.867237  1770 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:33.842810  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 168656us
I20260709 12:26:33.871755  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:33.868283 12092 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:33.872956 12098 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 33 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:33.873236 12092 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.874094 12098 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.877559  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:33.878472  1877 raft_consensus.cc:2410] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:26:33.878985  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:33.859701 11748 raft_consensus.cc:2804] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:26:33.879561  1879 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:33.880371  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:33.743193 11488 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:33.885677  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50086.8us
I20260709 12:26:33.881412  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:33.883839  1875 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:33.887519  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:33.757329 12055 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:33.888674  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69342.7us
W20260709 12:26:33.892299  2383 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.748067 12051 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.816059 11986 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.063s
I20260709 12:26:33.893977  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:33.894558  1844 leader_election.cc:336] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.870280 12086 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:33.909569  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:33.910251  1798 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:33.910446  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b550688b5e84ddda5fcf049010ac1e8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:33.875839 12092 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.914208 11986 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:33.914556 11903 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:33.899447 12096 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:33.879455 12119 raft_consensus.cc:697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:33.882725 11964 raft_consensus.cc:2804] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:33.922820  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:33.925385  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:33.941160  1877 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 32 index: 15. Preceding OpId from leader: term: 33 index: 16. (index mismatch)
I20260709 12:26:33.927443  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:33.928673  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:33.931124  2417 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 26. Current term is 27. Ops: [24.17-26.18]
I20260709 12:26:33.935634 12075 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:33.938169 12101 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:33.949882 12122 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:26:33.950425 12122 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:33.949664 12075 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:33.951368 12122 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.952749 12122 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:26:33.955265  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 123172us
I20260709 12:26:33.955514 11990 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:33.956148  1770 leader_election.cc:336] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.956554  1770 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:33.957378  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68085us
W20260709 12:26:33.958755  1844 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.959223  1844 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.959533 12051 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:33.960126 12051 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:33.960465  1844 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.963424 12119 consensus_queue.cc:237] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.970312 12091 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:33.883472 11748 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:33.948175  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:33.971203 11748 raft_consensus.cc:3060] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:26:33.974032  1801 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33802 (request call id 3289) took 1112 ms. Trace:
W20260709 12:26:33.949833  2384 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:33.977574  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:33.865839 11736 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:33.979025  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.980369  1845 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:33.995647  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:33.985741 11748 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:33.985512  2382 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.985587  1798 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:34.001299  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:34.005354  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:33.974514  1801 rpcz_store.cc:276] 0709 12:26:32.862141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.878332 (+ 16191us) service_pool.cc:224] Handling call
0709 12:26:32.903729 (+ 25397us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:26:33.610557 (+706828us) spinlock_profiling.cc:243] Waited 6.34 ms on lock 0x7b18003e0b00. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16ab9804bc 00007f16a8cce48b 00007f16b1b289a9 00007f16a8cea69b 00007f16a8ce893f 00007f16a8ce8455 00007f16abdd0060 00007f16abdd0bfb 00007f16abddb64b
0709 12:26:33.617030 (+  6473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:26:33.617062 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:26:33.641865 (+ 24803us) log.cc:844] Serialized 26 byte log entry
0709 12:26:33.740029 (+ 98164us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:26:33.740077 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:26:33.740156 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:26:33.973712 (+233556us) raft_consensus.cc:1685] finished
0709 12:26:33.973734 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:26:33.973967 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6429,"raft.queue_time_us":64774,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":8236,"spinlock_wait_cycles":14571776,"thread_start_us":112978,"threads_started":2,"wal-append.queue_time_us":180432}
I20260709 12:26:33.911650 11934 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.087s
I20260709 12:26:34.011767  1878 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:34.013650  1872 raft_consensus.cc:2468] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
W20260709 12:26:34.013649  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 131611us
W20260709 12:26:34.010911  1844 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.012437  2382 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:34.019544  1844 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.019946 12082 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:34.020125  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.021193 11945 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:26:34.023000  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:34.023343  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 98969.9us
W20260709 12:26:34.023722  1844 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:33.976490 11964 raft_consensus.cc:697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:34.025043  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.026921  1844 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:33.988411 12091 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.008879 12094 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:26:33.988224 11876 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:34.002403 12086 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:34.029371  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:34.037086  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.042273  2413 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 22 index: 18. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:26:34.051908  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.040488  1771 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.055943  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.056978  1798 raft_consensus.cc:2410] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 30 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 18.
I20260709 12:26:34.031697 12054 raft_consensus.cc:2804] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:34.059423  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:34.061224  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 105161us
I20260709 12:26:34.067443  1806 raft_consensus.cc:2393] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:34.051404 11938 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:26:34.074570  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 50895.2us
W20260709 12:26:34.062933  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:34.077188  1770 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:34.078562  1770 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.078984  1770 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.058626 12101 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:26:34.080416  1844 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.080974  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.090929 11938 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.091794 12055 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:34.093097 11748 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:34.093361 11938 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.095017 11488 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.095237  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.058370 11990 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.037818 12091 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.097448  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.099788  1875 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 33.
I20260709 12:26:34.099992  1803 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:26:34.097834  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:34.060696 12126 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:34.102365  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.103327  1801 raft_consensus.cc:2410] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 16.
I20260709 12:26:34.103130 11876 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.104132 12055 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:34.104935  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:34.105471  1804 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:26:34.101645  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:34.109407 12092 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.058902 11945 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:34.102494  1805 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:26:34.095994  1798 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:34.068485 12054 raft_consensus.cc:697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 30 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:34.119342  1807 raft_consensus.cc:2368] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 31. Current term is 32.
W20260709 12:26:34.098310  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:34.120836  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.127353 12092 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.131197  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.133000  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.134063  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.130609 12092 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:34.139837  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 63136us
I20260709 12:26:34.060741 12077 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.053s
W20260709 12:26:34.151245  1844 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.153862  1844 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.152484  1696 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:34.147855  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:34.158965  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.156327  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:34.159611  1876 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:34.108891 11964 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.174340  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:34.174942  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.171384  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:34.145555 11852 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:34.179584  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:34.183550  2411 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:26:34.179312  1696 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:34.194203 12098 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:34.200239 12098 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.199844 12054 consensus_queue.cc:237] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.200903 12098 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.203104 12098 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.203387 11736 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.212780  1804 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
W20260709 12:26:34.213802 12100 consensus_meta.cc:287] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.598s	user 0.008s	sys 0.000s
I20260709 12:26:34.216487  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.217569  1875 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:34.232733  1879 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:34.234246  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.220645 12091 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.229000  1874 raft_consensus.cc:2410] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
W20260709 12:26:34.230203 12078 consensus_meta.cc:287] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: Time spent flushing consensus metadata: real 0.738s	user 0.009s	sys 0.003s
I20260709 12:26:34.169358 12096 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:34.214320 12100 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.216931 11990 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:34.250447 11990 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 31 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:34.251616 11990 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:34.231199 12075 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.255545 12075 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.257460  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:34.259172 11736 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:34.259601  1872 raft_consensus.cc:2393] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:34.259604 11736 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.260690 11736 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:34.237126  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:34.263197  2382 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.263684 11522 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.179116 11945 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:26:34.243094  1876 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:34.244271  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.248636  1802 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 32 index: 15. Preceding OpId from leader: term: 33 index: 16. (index mismatch)
I20260709 12:26:34.249508  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.260639 12100 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:34.240085  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 99497.9us
W20260709 12:26:34.275864  1844 leader_election.cc:336] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.276414  1844 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.277532 11936 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:34.278276  1844 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.280090  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.281566 12077 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:34.288538  1800 raft_consensus.cc:2393] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:34.275743  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.237583 12097 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:34.279084  1844 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.298336 12091 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:34.302672  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.303984  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:34.305087  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.274355 11522 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.308874 11522 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.316335  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.318126  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:34.324891  1875 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:34.321902  2413 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
W20260709 12:26:34.323707  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 245167us
I20260709 12:26:34.326232  1873 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
W20260709 12:26:34.332060  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:34.321833 12097 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.294660  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:34.337268  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:34.338550  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44013.1us
I20260709 12:26:34.343351  1875 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
W20260709 12:26:34.344494  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 66286.8us
W20260709 12:26:34.350670  1844 leader_election.cc:336] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.351243  1844 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.291270 12077 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.346817  1696 leader_election.cc:400] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 31. Current term is 32.
I20260709 12:26:34.352171  1696 leader_election.cc:403] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
W20260709 12:26:34.353222  2382 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.352427  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:34.355242  1844 leader_election.cc:336] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:34.358429  1770 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.336529 12097 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.361138  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.338972 11945 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:34.346490 12091 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:34.344684 12100 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:34.369927  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.376554  1807 raft_consensus.cc:2410] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:26:34.353227 12098 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:26:34.363472 11488 raft_consensus.cc:3060] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:34.361310 12086 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:34.337060 12096 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:26:34.390645  1696 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.373618 12091 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.407498  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:34.352262 12077 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.469143  1800 raft_consensus.cc:1275] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:26:34.375036 12113 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:26:34.468127 11748 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:34.466271  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:34.467412  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 118991us
I20260709 12:26:34.479298  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.442090 12091 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:34.483258  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.444000 11522 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.484449  1877 raft_consensus.cc:1240] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 29. Current term is 30. Ops: []
I20260709 12:26:34.389791 12100 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.484248 11748 raft_consensus.cc:2804] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:34.497721 11488 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for earlier term 31. Current term is 32.
I20260709 12:26:34.475389 11753 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.013s
I20260709 12:26:34.501361 12096 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:34.509225 12098 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:34.509584  1770 leader_election.cc:336] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.517664  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.514390 11964 consensus_queue.cc:1059] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:34.489035 12113 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:34.501770 11748 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.521507 11488 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.488724 11522 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.527940 11748 raft_consensus.cc:3060] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:34.527546 12076 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:34.553973 12140 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.180s	user 0.001s	sys 0.000s
W20260709 12:26:34.559723 12146 log.cc:927] Time spent T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:26:34.561084 11522 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.567741 12100 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:34.567497 11748 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.570453 11748 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.573057 12077 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.529982 12096 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:34.583321 12143 log.cc:927] Time spent T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
I20260709 12:26:34.585541  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:34.535686 11990 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:34.585827  1844 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.602228  1844 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:34.591830  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 101462us
I20260709 12:26:34.578449 12096 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:34.609995  1844 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.610575  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.611774  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.612687  1806 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:34.615585  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.616308  1802 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:34.566406 11736 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.540141 11488 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:34.619146  1770 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.629946  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.608155 12086 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.617373 12100 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:34.664716  1845 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.633303 12054 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:34.682541  1770 leader_election.cc:336] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.683125  1770 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.620841 11990 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:34.701876  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 91928.5us
W20260709 12:26:34.706447  1844 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.707015  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.665186 12086 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:34.718736  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50664.1us
I20260709 12:26:34.691633 12092 raft_consensus.cc:3060] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:34.689803 12121 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:34.722301  1844 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.722858  1844 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.698834 12076 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:34.731880  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 59764.3us
W20260709 12:26:34.736179   561 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 1757)
I20260709 12:26:34.735854  1845 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:34.731582 12141 log.cc:927] Time spent T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:26:34.723752 12100 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:34.743556  1770 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:34.741786  1844 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.744242 12157 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.745400  2417 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:26:34.744345  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:34.745193  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 261950us
W20260709 12:26:34.753273  1844 leader_election.cc:336] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.743408 12054 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:34.736589 12076 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:34.755851  1844 leader_election.cc:336] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.752472 11945 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:34.759027  1844 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.758291 11488 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:34.761255  1844 leader_election.cc:336] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.760726 12055 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:34.762039 12153 consensus_queue.cc:1048] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.008s
I20260709 12:26:34.763845 12086 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:34.766923 12154 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:34.741118 12091 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:34.770495  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:34.771695 11990 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:34.774557 12148 log.cc:927] Time spent T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:26:34.780195 12124 raft_consensus.cc:3055] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:26:34.761529 12121 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:26:34.783000 12076 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.827020  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:34.785089 12150 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.059s	user 0.002s	sys 0.000s
I20260709 12:26:34.788594 12092 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:34.808743  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.836539  1879 raft_consensus.cc:2468] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 31.
I20260709 12:26:34.794063  1844 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.801972  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:34.814637  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:34.810256  1807 raft_consensus.cc:1275] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:34.838299  1800 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:34.838351  2417 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 30.
W20260709 12:26:34.825393  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 63730.4us
I20260709 12:26:34.813665 12124 raft_consensus.cc:740] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:34.830030  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:34.830062  1697 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61681.4us
W20260709 12:26:34.842872  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 81652us
I20260709 12:26:34.794009 12091 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:34.814041  1770 leader_election.cc:336] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.835700  1880 raft_consensus.cc:2393] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 31: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:34.841451  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:34.843990  1844 leader_election.cc:336] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:34.847373  1844 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.847661  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:34.848340  1874 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:34.848950 12153 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:34.848373  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:34.850330  1804 raft_consensus.cc:1275] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:26:34.855340  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.845155  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.859108  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:34.863452  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:34.867084  2382 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:34.873759  2382 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.871016 12153 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:34.880817 12153 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:34.881335 12153 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:34.885831 12076 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:34.850675  1844 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.891505  1844 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.850296 12122 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:34.892102 11748 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:34.892478 12122 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.894438 12067 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.895431 12092 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:34.897226  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46584.5us
W20260709 12:26:34.898382  1844 leader_election.cc:336] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.901156  1844 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.863260  1696 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:26:34.907394  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44298us
W20260709 12:26:34.910516  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 51505.3us
I20260709 12:26:34.911731  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.899240 12091 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.919281 12091 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.921732 12091 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.754837 12157 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.924310 12157 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.913481 12092 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:34.914913 12154 consensus_queue.cc:1048] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:34.926446 11736 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:34.915776 12051 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:34.929630 12122 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:34.911562 12067 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:34.930900 12157 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:34.932068  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 118060us
I20260709 12:26:34.932864  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:34.902846  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 74482.2us
I20260709 12:26:34.934918 12081 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:34.936107  1874 raft_consensus.cc:2468] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:26:34.936302  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:34.875277 12124 consensus_queue.cc:260] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:34.947162 12124 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:34.947597  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:34.949108 12114 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:34.949705 12114 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.940646  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:34.941725  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:34.950747  2411 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
W20260709 12:26:34.942981 12138 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.571s	user 0.003s	sys 0.000s
W20260709 12:26:34.943997  1844 leader_election.cc:336] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.949216  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:34.952827 11736 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:34.926506 12092 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.950044 12114 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:34.953809 12092 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:34.961457 12114 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.963784 12081 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:34.964588 12081 consensus_queue.cc:1048] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:26:34.965636 12153 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:34.972023  2383 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:34.952910  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.980230  1880 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260709 12:26:34.980726  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46649us
I20260709 12:26:34.981546  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:34.986090  1844 leader_election.cc:336] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:34.988284  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:34.995110  1874 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 24 index: 19. (index mismatch)
I20260709 12:26:35.001992  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.002846  1876 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:35.001992  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:34.980232 12158 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.031878  1879 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
W20260709 12:26:35.020841  1844 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.041255  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:35.041638  1844 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:35.042840 12120 raft_consensus.cc:2804] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:26:35.043687  1844 leader_election.cc:336] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.044224  1844 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:34.996719 12051 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:35.045428 12051 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:35.046123 12088 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:34.996713 11717 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:26:35.010809  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.051900  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:35.052857  1807 raft_consensus.cc:2393] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:35.057168  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:35.057976  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.058769  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.004372 12157 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:35.059168  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:35.047549  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 114642us
I20260709 12:26:35.068691  1696 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.070448  1798 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 26.
I20260709 12:26:35.023684  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.075275  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.075491  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:35.077061  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:35.077034  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.038985 11990 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.078214  1802 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:35.081100  2413 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:26:35.081488  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.081967  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.082306  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:35.061249  1771 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:35.083194  1881 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:35.078248  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.079797  1805 raft_consensus.cc:1240] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 32. Current term is 33. Ops: []
I20260709 12:26:35.073244  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.073820  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.084293  1876 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 31.
I20260709 12:26:35.077144  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:35.085172  1872 raft_consensus.cc:2410] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:26:35.083134  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:35.090378  1844 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.085290  1798 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:35.063580 12120 raft_consensus.cc:697] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 30 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:35.107343 12120 consensus_queue.cc:237] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.110113 11990 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.080487 12067 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.086912 12114 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:35.070246 11717 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:35.111431 12067 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.112728 11990 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.112936 11717 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:35.114166 12067 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.100190 12091 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:26:35.038264 11986 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:26:35.122656  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.140659  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:35.126541  1770 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.117583  1874 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:35.160524  1874 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:26:35.161818  2411 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 30.
I20260709 12:26:35.119302  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.120461  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:35.167119  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:35.168265  1846 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:35.103670  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:35.049029 12158 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.126586 12137 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:35.178710  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:35.143158 12154 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.122655 12096 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:26:35.149076  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:35.149706  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:35.148531 11994 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.003s
I20260709 12:26:35.151757 12067 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.137014 12160 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.157013  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:35.199579  1873 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:35.183630  1799 raft_consensus.cc:3060] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:35.182307  1807 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:26:35.190152 12154 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.199098 12067 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:35.219205 12067 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:35.220549 12092 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:35.222534 12158 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.187094 12051 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:35.224074  1878 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:26:35.179008 12137 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:35.245889 12051 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 30 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:35.247228 12051 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.267387  1881 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:35.203989 11748 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:26:35.272001  1798 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
W20260709 12:26:35.208014  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 104400us
W20260709 12:26:35.274058  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 184775us
W20260709 12:26:35.274950  1770 leader_election.cc:336] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.274302  1696 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:35.275385  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:35.236212 12092 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:35.283663  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:35.251078 12137 raft_consensus.cc:3060] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:26:35.287084  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:35.275476 11748 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:35.297755  2382 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.234346 12160 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:35.300637  1696 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:35.289127 12091 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:35.295271 12067 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:35.300617 11748 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:35.300446 11876 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:35.321055  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 46139.8us
W20260709 12:26:35.326452  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:35.325757  1770 leader_election.cc:336] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.315397 11903 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:35.298489 11717 consensus_queue.cc:1059] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 32.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.019s
I20260709 12:26:35.315904 12067 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.343677 11717 raft_consensus.cc:3055] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:26:35.344209 11717 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:35.345099 11717 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:35.345944  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.346349 11717 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:35.351264  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:35.325712 12091 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.027s
I20260709 12:26:35.354621 12153 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.338704 12067 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:35.363041 12160 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:35.373420  1873 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:35.375725  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:35.399791  2417 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:26:35.377202 12051 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:35.400832 12051 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.402880 12067 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:35.407836 12067 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:35.394068  1799 raft_consensus.cc:2468] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 29.
I20260709 12:26:35.403534 12051 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.382547  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.405014 11748 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:35.407711  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.416757  1802 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 32.
I20260709 12:26:35.386013 12157 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:35.417704 11748 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:35.419173  2383 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:35.390172 12137 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.421098  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:35.430315 12169 log.cc:927] Time spent T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.157s	user 0.003s	sys 0.000s
I20260709 12:26:35.444747 12157 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.513334 12157 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:35.446929  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 314990us
W20260709 12:26:35.515576  1844 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.516135  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:35.516923 12088 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:26:35.517491  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:35.518450  1844 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.448544 12153 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:35.519294 12153 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.521833 12153 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.461356 12154 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.464686  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:35.523850  2413 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 32: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:35.469115  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:35.475068  1873 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 18. (term mismatch)
I20260709 12:26:35.525185  1873 pending_rounds.cc:85] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Aborting all ops after (but not including) 17
I20260709 12:26:35.525465  1873 pending_rounds.cc:107] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Aborting uncommitted NO_OP operation due to leader change: 26.18
I20260709 12:26:35.525766  1873 raft_consensus.cc:2889] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:35.475178  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:35.492301  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 185805us
W20260709 12:26:35.529387  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:35.530637  1696 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.530718 12067 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.532922 12067 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.535175  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:35.535597  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:35.535982  1804 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:26:35.536237  1877 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:35.537489 12067 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.539227  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:35.539930  1875 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:26:35.535261  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.540853  1881 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:26:35.542598  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.543066 12160 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:35.544090  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.544519  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:35.547091  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.548022  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:35.548588  1798 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:26:35.544765  1803 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
W20260709 12:26:35.551359  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 130332us
I20260709 12:26:35.554739  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.556707  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:35.557170  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:35.557458  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 138405us
I20260709 12:26:35.557922  1880 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:35.562944  1807 raft_consensus.cc:2410] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:26:35.563120  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.563252  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.509876 12051 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:35.575294 12086 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.578331 12086 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.580931 12086 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.582990 11748 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:35.586725  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:35.590714 12096 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:26:35.592947  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 267190us
I20260709 12:26:35.596122 12051 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:35.585968  1630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52630 (request call id 261) took 12538 ms (client timeout 15000 ms). Trace:
W20260709 12:26:35.603747  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:35.585194 12137 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.573031  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.602205  1875 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:35.567008 12088 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:26:35.620652 12088 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:35.621304 12088 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:35.603664  1798 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 32.
W20260709 12:26:35.595878  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 48522.8us
I20260709 12:26:35.624897  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:35.628659  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 55753.7us
I20260709 12:26:35.570150 12097 raft_consensus.cc:2804] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:26:35.604244  1630 rpcz_store.cc:269] 0709 12:26:23.054221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:26.848196 (+3793975us) service_pool.cc:224] Handling call
0709 12:26:35.585871 (+8737675us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:33.876418 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:33.877155 (+   737us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:33.877210 (+    55us) write_op.cc:435] Acquired schema lock
0709 12:26:33.877236 (+    26us) tablet.cc:662] Decoding operations
0709 12:26:33.982529 (+105293us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:33.982640 (+   111us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:33.982670 (+    30us) tablet.cc:685] Acquiring locks for 34 operations
0709 12:26:33.985101 (+  2431us) tablet.cc:701] Row locks acquired
0709 12:26:33.985159 (+    58us) write_op.cc:260] PREPARE: finished
0709 12:26:33.985421 (+   262us) write_op.cc:270] Start()
0709 12:26:33.985604 (+   183us) write_op.cc:276] Timestamp: P: 1783599993985338 usec, L: 0
0709 12:26:33.985649 (+    45us) op_driver.cc:348] REPLICATION: starting
0709 12:26:33.986745 (+  1096us) log.cc:844] Serialized 13279 byte log entry
0709 12:26:34.063845 (+ 77100us) op_driver.cc:464] REPLICATION: finished
0709 12:26:34.065285 (+  1440us) write_op.cc:301] APPLY: starting
0709 12:26:34.072054 (+  6769us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:34.557947 (+485893us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:34.558117 (+   170us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:35.228879 (+670762us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:35.229470 (+   591us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=68,key_file_lookups=68,delta_file_lookups=0,mrs_lookups=0
0709 12:26:35.229553 (+    83us) write_op.cc:312] APPLY: finished
0709 12:26:35.230336 (+   783us) log.cc:844] Serialized 292 byte log entry
0709 12:26:35.296873 (+ 66537us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:35.314105 (+ 17232us) write_op.cc:454] Released schema lock
0709 12:26:35.328496 (+ 14391us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:35.356025 (+ 27529us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:35.356058 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":1109,"apply.run_cpu_time_us":61161,"apply.run_wall_time_us":1290706,"cfile_cache_hit":138,"cfile_cache_hit_bytes":146383,"num_ops":34,"prepare.queue_time_us":407784,"prepare.run_cpu_time_us":20807,"prepare.run_wall_time_us":142859,"raft.queue_time_us":35138,"raft.run_cpu_time_us":1258,"raft.run_wall_time_us":1259,"replication_time_us":78028,"thread_start_us":343821,"threads_started":5,"wal-append.queue_time_us":112643,"wal-append.run_cpu_time_us":6992,"wal-append.run_wall_time_us":1050730}]]}
I20260709 12:26:35.632899  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.631969  1877 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 32.
I20260709 12:26:35.653599  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:35.596071 12157 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:35.675362  1696 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.681882  1875 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
I20260709 12:26:35.685029  1804 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:35.657349 12091 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:35.696882  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.665469 12097 raft_consensus.cc:697] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:35.702708  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:35.704008  1844 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.695331 11717 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:35.697480 12096 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.014s
W20260709 12:26:35.746440  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 49629.6us
I20260709 12:26:35.707166 12174 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:35.732718  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:35.736719   561 thread.cc:527] Waited for 3000ms trying to join with heartbeat (tid 1757)
W20260709 12:26:35.722419  1770 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.743484  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:35.749294 11748 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:35.737232 12055 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:35.755544 12154 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:35.759997 11936 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:26:35.760164  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 57718us
W20260709 12:26:35.761207  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:35.767406 12097 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.771441 12051 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:35.793826  2417 consensus_queue.cc:237] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:35.795078  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47779.9us
I20260709 12:26:35.796001  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.797747  1696 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.798539 12174 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:35.799379 12174 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:35.803052  1625 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 20 to 22. New cstate: current_term: 22 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:35.828351  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:35.837657 12121 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.838353 12121 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:35.838874  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:35.841181 12121 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:26:35.844820  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.846485  1873 raft_consensus.cc:2410] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for term 32 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
W20260709 12:26:35.875218  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 315454us
I20260709 12:26:35.879129  1770 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:35.886950  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 84609.8us
I20260709 12:26:35.888209  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.742000  2410 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 25.
W20260709 12:26:35.889732  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 48363.6us
W20260709 12:26:35.899902  1844 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.903576 12091 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:26:35.902545  1844 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:35.909628  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:35.912352 12092 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:35.916261 12174 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.917042 12174 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.921799  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:35.924307 12114 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.925120 12114 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:35.929564 12114 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:35.935230  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:35.954743   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:35.957643   561 tablet_replica.cc:333] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09: stopping tablet replica
W20260709 12:26:35.958639  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 70521us
I20260709 12:26:35.959980  1696 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:35.949211 12158 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:35.962584  1696 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:35.988682  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:35.944799 12174 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:35.967209 12121 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:35.951522 12092 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:35.968351 11717 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:36.007707  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:36.011438   561 raft_consensus.cc:2243] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.001085 11748 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:36.018832  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 296436us
I20260709 12:26:36.022228  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.023721  1799 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 32.
I20260709 12:26:35.976174 12157 consensus_queue.cc:1059] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:26:36.006567 11990 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.032325   561 raft_consensus.cc:2272] T 5e573c99734f4d91bb30d1fe72b56cf7 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.019588  1696 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:26:36.037978  1770 leader_election.cc:336] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.038623  1770 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:36.038712  1772 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:36.021080 12174 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:36.040323 12160 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.040504 12174 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.041060 12174 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.041029 12160 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.048089 11936 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:36.048774 11936 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.015159 12092 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.051862  1798 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:26:36.060422  1696 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:36.061393  1696 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:36.021104 11717 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 29 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:36.066951  1696 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:36.028510 12188 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:36.061601 12126 raft_consensus.cc:2804] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:36.064585 12160 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:36.064971 12092 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:36.106767  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 68828.5us
I20260709 12:26:36.065344 12067 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:36.055878 11990 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.085985 12157 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:26:36.109895  1770 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.088164 11488 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:36.127166  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:36.105489 11986 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:36.099109 12091 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:36.158813  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.159776  1873 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:36.124524 12126 raft_consensus.cc:697] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:36.178575  1873 raft_consensus.cc:1275] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
W20260709 12:26:36.181102  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 93638.3us
W20260709 12:26:36.182077  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 72250.6us
W20260709 12:26:36.182860  1770 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:36.168874 12088 consensus_meta.cc:287] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: Time spent flushing consensus metadata: real 0.547s	user 0.016s	sys 0.000s
I20260709 12:26:36.189983  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.194569  1877 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:26:36.160836 12157 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:36.195760  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.167160 12128 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:36.201006  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.201797  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.202598  1873 raft_consensus.cc:2393] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 30: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:36.203998  1799 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:26:36.216850  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.186070 11936 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:36.240530  2410 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:26:36.208407 12128 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.242568  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:36.186487 12051 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.053s
W20260709 12:26:36.257351  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.178274 11717 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.204892 12120 consensus_queue.cc:1048] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.045s
I20260709 12:26:36.205937 12088 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.260279  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:36.245848 12128 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:36.258587 11488 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 32 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:36.283587  1872 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 32.
W20260709 12:26:36.289309  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 88085.8us
I20260709 12:26:36.295075  1696 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:36.274307 12092 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.024s
I20260709 12:26:36.279731 12198 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 12:26:36.273379 12157 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:36.314124 12193 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.144s	user 0.000s	sys 0.002s
I20260709 12:26:36.274698 12126 consensus_queue.cc:237] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.334293  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.313862 12205 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:36.337399  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:36.336509  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 76348.1us
I20260709 12:26:36.339073   561 tablet_replica.cc:333] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:36.321626 11990 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:36.345453 12192 log.cc:927] Time spent T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.138s	user 0.001s	sys 0.001s
I20260709 12:26:36.340934 12205 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.321131 11488 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:36.394419  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 48029us
I20260709 12:26:36.399272  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.401105  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:36.394609  1770 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.419157   561 raft_consensus.cc:2243] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.405886 11854 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.419194 12206 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.070s
I20260709 12:26:36.423496 11854 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.405597 12205 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.431672 12128 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.436884 12128 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:36.432621  1770 leader_election.cc:336] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.438098  1770 leader_election.cc:304] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:36.437446 12211 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:36.443328 11854 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:36.414276 12198 raft_consensus.cc:493] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.446099 12198 raft_consensus.cc:515] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.447028 12174 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.448570 12198 leader_election.cc:290] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:36.450649 12174 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:36.454222 12088 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:36.454674  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:36.442610   561 raft_consensus.cc:2272] T 2b797b0728dc4e01976ab61ef164fa90 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.458408  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.459033  1877 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:36.405791 11990 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
W20260709 12:26:36.465021  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53277.1us
I20260709 12:26:36.473858  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:36.464485 12200 raft_consensus.cc:2749] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:36.495474 12092 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:36.496214 12092 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.498483 12092 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:36.502645 11748 raft_consensus.cc:3055] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:26:36.503273 11748 raft_consensus.cc:740] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:36.504113 12091 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.504428 11748 consensus_queue.cc:260] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.512228 12088 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.512892 12088 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.505604 11748 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:36.489048 12199 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:26:36.561051  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 128472us
W20260709 12:26:36.533586  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 287741us
W20260709 12:26:36.534608  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 50889.2us
I20260709 12:26:36.504693 12091 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.532447 12158 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.571759 12088 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:36.574942 12199 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.475137 11990 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:36.588021  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 51892us
W20260709 12:26:36.594034  1770 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.617401  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:36.628988  1844 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.629812  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:36.624270   561 tablet_replica.cc:333] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:36.633553 12202 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:26:36.634478   561 raft_consensus.cc:2243] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.635314   561 pending_rounds.cc:70] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Trying to abort 1 pending ops.
I20260709 12:26:36.602413 12174 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.635656   561 pending_rounds.cc:77] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 28 index: 17 } timestamp: 7305625524108214272 op_type: NO_OP noop_request { }
I20260709 12:26:36.636215   561 raft_consensus.cc:2889] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:36.635984 12174 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.636646   561 raft_consensus.cc:2272] T ae12891834034d968900107cc58fa821 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.602087 12199 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:36.635092  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:36.646132  1881 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:36.639873  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:36.647270  2410 raft_consensus.cc:2368] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 32. Current term is 33.
I20260709 12:26:36.641706  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:36.599545 12051 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:36.656919  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62931.9us
I20260709 12:26:36.668175  1875 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
W20260709 12:26:36.667750  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:36.668373  1770 leader_election.cc:336] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.673012  1770 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:36.638759 12174 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:36.639070 12091 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:36.601102 12158 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.654285 12088 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:26:36.685493  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.680917  1696 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:36.691133  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:36.683485   561 tablet_replica.cc:333] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09: stopping tablet replica
W20260709 12:26:36.690481  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 216704us
I20260709 12:26:36.694062  1696 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:26:36.699365  1773 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.699648  1881 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 33.
W20260709 12:26:36.698141  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 211093us
I20260709 12:26:36.684170 12174 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.683856 12137 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:36.668463 12051 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.707453  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:36.710876   561 raft_consensus.cc:2243] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:36.711993 11488 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:36.715039 12174 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.716964 12051 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:36.717254 12174 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:36.700944 12091 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:36.717597  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:36.718559 12051 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:36.718775  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.719748  1846 leader_election.cc:400] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 32. Current term is 33.
I20260709 12:26:36.720067  1846 leader_election.cc:403] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
W20260709 12:26:36.720192  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.721163  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:36.721504 12153 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 32. Current term is 33.
W20260709 12:26:36.722656  1697 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:36.709905 12224 raft_consensus.cc:3060] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:36.728927  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 71132.3us
W20260709 12:26:36.729715  1770 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.730139  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:36.686201 12205 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:36.733515  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:36.738198  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:36.738857  1876 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.738693  1878 raft_consensus.cc:1240] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:26:36.736483  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "79f5c05e64b945cbacc02914f31c5762" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.744113  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:36.745975  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.746407  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.747923  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.749890  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.751132  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.751322 12214 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.751811 12214 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.748528  1880 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 32.
I20260709 12:26:36.747164  1802 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 33.
I20260709 12:26:36.757956 12158 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:36.757622 12128 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:36.741204 11990 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:36.761734  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:36.762388  1798 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:36.760903  1800 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:26:36.763123   561 raft_consensus.cc:2272] T 23caea311a774ba3b29b7749bd3f1cd6 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:36.756057  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:36.764056  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:36.764897  1800 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:36.767006  1801 raft_consensus.cc:1240] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Rejecting Update request from peer 075704438992440697732181e49ccd09 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
I20260709 12:26:36.774080 12081 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:36.776541  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:36.769631 12214 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:36.781736  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52048.7us
W20260709 12:26:36.781999  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46263.9us
W20260709 12:26:36.783628  1770 leader_election.cc:336] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.784096  1770 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:36.787336 11854 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:26:36.769397 12114 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:26:36.796617  1770 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:36.798365  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:36.799332  1770 leader_election.cc:336] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.799865  1770 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:36.803676  1770 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.826376  1798 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:36.827889  1876 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:36.790602 12091 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:36.808743 12126 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:36.808959 11854 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:36.851620 11990 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.852272 11990 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.819289 12114 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 33 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:36.854578 11990 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:36.856434 12114 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:36.858762  1800 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 33.
I20260709 12:26:36.861353  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.862205  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.867867  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.868597  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:36.869262  1804 raft_consensus.cc:2368] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 32. Current term is 33.
I20260709 12:26:36.869175  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:36.863260  1874 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:26:36.864301  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:36.869942  1800 raft_consensus.cc:2393] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 33: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
W20260709 12:26:36.868268  1770 leader_election.cc:336] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.885074  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:36.886847  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:36.887969  1770 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.888370  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:36.865922  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:36.866644  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:36.889554  1770 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.889781 11748 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:36.894773 11748 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.895277 11748 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:36.902298  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:36.889616  1696 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:36.905058  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:36.890556 12126 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:26:36.894363  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:36.907708 12174 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:36.908327 12174 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 28 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:36.909289 12174 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:36.911165 12224 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:36.912168 12224 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:26:36.911254   561 tablet_replica.cc:333] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:36.915342 11854 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:36.916193   561 raft_consensus.cc:2243] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:36.916589  1844 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.916920   561 raft_consensus.cc:2272] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:36.917670  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:36.928769  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:36.973574  1801 raft_consensus.cc:1275] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:26:36.974045   561 tablet_replica.cc:333] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09: stopping tablet replica
W20260709 12:26:36.932480  1770 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.844113 12091 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:36.976456  1770 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:36.976788 12091 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:36.978363 12224 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:36.981314  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48565.7us
I20260709 12:26:36.990334  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:36.990901  1696 leader_election.cc:304] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:36.977070  1770 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:36.996344  1844 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.001730  1844 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:37.007683  1770 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:37.008060  1844 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.011754 12126 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:37.015033  1801 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:26:37.016641  2410 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:26:37.017006  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.019654 11854 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:26:37.020929  1807 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:26:37.016597  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.022322  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.028633  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:37.027089 12154 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:37.032187 12203 consensus_queue.cc:1059] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:26:37.033386 12203 raft_consensus.cc:3055] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:37.033591 12204 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:37.033879 12203 raft_consensus.cc:740] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:37.034293 12204 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:37.036361 11990 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:37.036492  1696 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:37.037801 11990 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:37.038038 12204 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:36.988029 12158 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:37.035431 12203 consensus_queue.cc:260] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.048197  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:36.995621 12216 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:37.050747   561 raft_consensus.cc:2243] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.055294  1696 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:36.982561 11748 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:37.062685  1696 raft_consensus.cc:2686] T 34263595c9ed4e0a9eba4bfe41c0702e P 075704438992440697732181e49ccd09: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:26:37.023901 12067 raft_consensus.cc:2749] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:37.066668  1696 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:37.067466  1770 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.074923  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:37.077292   561 raft_consensus.cc:2272] T 7b550688b5e84ddda5fcf049010ac1e8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:37.077445  1770 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.081106  1696 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:37.078954  1770 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.025703 11785 consensus_queue.cc:1048] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.048s
I20260709 12:26:37.080338 12203 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:37.053233 12199 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.099354  1878 raft_consensus.cc:1275] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:26:37.084297 12158 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.019s
I20260709 12:26:37.102124   561 tablet_replica.cc:333] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:37.104027 11748 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 34 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:37.082985 12091 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:37.082839 12122 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:37.110687  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:37.112232  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 29 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:37.112964  1877 raft_consensus.cc:3060] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:37.083101 11990 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:37.118480  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:37.119467  1803 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:37.064687 11488 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.033s
W20260709 12:26:37.137982 12229 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.301s	user 0.004s	sys 0.000s
I20260709 12:26:37.137593 12137 consensus_queue.cc:1048] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:37.125236 12128 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:37.139328 12128 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:37.094985 12199 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:37.149853  1771 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:37.163357   561 raft_consensus.cc:2243] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.145004 12203 raft_consensus.cc:493] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.186764   561 raft_consensus.cc:2272] T 82c4d9259f1a44119e802a40207d64b5 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.200404  1806 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:26:37.177451 12122 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:37.208225  1877 raft_consensus.cc:2468] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:37.211602  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 34 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:37.215000  1875 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Advancing to term 34
I20260709 12:26:37.212268  1803 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 34.
I20260709 12:26:37.219841  1771 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:37.218350  1696 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:37.223373 12158 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:26:37.223920 12158 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 34 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:37.224753 12158 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.227571 11876 raft_consensus.cc:2804] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:37.228168 11876 raft_consensus.cc:697] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:37.229007 11876 consensus_queue.cc:237] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.15, Last appended by leader: 15, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.188133 12203 raft_consensus.cc:515] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:37.240633   561 tablet_replica.cc:333] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:37.241431 12203 leader_election.cc:290] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:37.242076  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.242096 12128 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:37.207609 12091 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.050s
I20260709 12:26:37.241600   561 raft_consensus.cc:2243] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:37.266435  1875 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 34.
I20260709 12:26:37.286633  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a57b66dec794db09a7f9048a3d0de10" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:37.297441  1697 leader_election.cc:304] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:37.255263 12203 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.288168 12067 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.293175 11990 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.300572 12206 raft_consensus.cc:2749] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:26:37.324101 12235 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:26:37.344982 12205 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:37.316938 12067 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.350922 12067 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:37.352510  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:37.365779  1770 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.366290  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:37.336517 11990 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:37.382193  1770 leader_election.cc:336] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.384037 12202 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:37.391533 11990 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:37.393893  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:37.313676 12203 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:37.404943  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:37.406562 12239 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.406693 12092 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:37.407179 12239 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.409622 12239 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:37.414124  1845 leader_election.cc:400] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Vote denied by peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361) with higher term. Message: Invalid argument: T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for earlier term 32. Current term is 33.
I20260709 12:26:37.434082  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:37.432776  1798 consensus_queue.cc:237] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:37.447610  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45289.8us
I20260709 12:26:37.458616   561 pending_rounds.cc:70] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Trying to abort 2 pending ops.
I20260709 12:26:37.459172   561 pending_rounds.cc:77] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 29 index: 16 } timestamp: 7305625585469411328 op_type: NO_OP noop_request { }
I20260709 12:26:37.459627   561 raft_consensus.cc:2889] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 29 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:37.460034   561 pending_rounds.cc:77] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 27 index: 15 } timestamp: 7305625540020981760 op_type: NO_OP noop_request { }
I20260709 12:26:37.460449   561 raft_consensus.cc:2889] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 29 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:37.460855   561 raft_consensus.cc:2272] T 84a31d3e62ae470d9c01e48ea29fbaee P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.454490 12096 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:37.471829  1770 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.472680  1802 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:26:37.477075  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.479076  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.444206 12154 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:37.452359 12092 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:37.495690  1696 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:37.517002  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:37.518417  1872 raft_consensus.cc:2393] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 075704438992440697732181e49ccd09 in current term 29: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
I20260709 12:26:37.528271  1874 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:26:37.531154  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:37.482033 11990 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.534812  1875 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
W20260709 12:26:37.557549  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 94215.3us
I20260709 12:26:37.527881 11748 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:37.537511 12242 log.cc:927] Time spent T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:26:37.466684 12096 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:37.500058 12206 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.068s
W20260709 12:26:37.608004  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:37.519440 12154 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:37.612587  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:37.611940  1844 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:37.562206 11990 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.604668 11748 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:37.668973 12245 log.cc:927] Time spent T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:26:37.669675 11748 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.662539 11990 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:37.687947   561 tablet_replica.cc:333] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:37.712491   561 raft_consensus.cc:2243] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:37.715827  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 193135us
W20260709 12:26:37.730894  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 217381us
I20260709 12:26:37.736249  1798 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.763914  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:37.793239   561 raft_consensus.cc:2272] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:37.805214  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 87841.4us
I20260709 12:26:37.807480  1799 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:26:37.807966  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:37.741276 12203 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:37.811074  2410 raft_consensus.cc:1275] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
W20260709 12:26:37.811581  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 47782.4us
I20260709 12:26:37.818658  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.819186 12067 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:37.819900  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.821048 11990 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:37.821074 12250 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:26:37.831686 12249 log.cc:927] Time spent T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:26:37.831948 12096 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:37.833434 12246 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
W20260709 12:26:37.836259 12233 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.306s	user 0.001s	sys 0.000s
I20260709 12:26:37.836803 11748 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:37.843950  1698 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.855708  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:37.856439  1800 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:26:37.860688  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:37.867808  1772 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:37.878104   561 tablet_replica.cc:333] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:37.878998 12101 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:26:37.879143 12081 raft_consensus.cc:2955] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Committing config change with OpId 25.17: config changed from index 4 to 17, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:37.853966 11748 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:37.883572 11748 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:37.859246 11990 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.884805 11990 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:37.886471  1770 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:37.843585 12203 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:37.890084 12203 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:37.892807 12203 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:37.856803 12221 consensus_queue.cc:1048] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.037s
I20260709 12:26:37.893951   561 raft_consensus.cc:2243] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:37.894996   561 pending_rounds.cc:70] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Trying to abort 2 pending ops.
I20260709 12:26:37.895395   561 pending_rounds.cc:77] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 25 index: 16 } timestamp: 7305625503449829376 op_type: NO_OP noop_request { }
I20260709 12:26:37.895957   561 raft_consensus.cc:2889] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:37.896318   561 pending_rounds.cc:77] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 23 index: 15 } timestamp: 7305625444483981312 op_type: NO_OP noop_request { }
I20260709 12:26:37.896746   561 raft_consensus.cc:2889] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:37.898054   561 raft_consensus.cc:2272] T a534e4794d9548638a2781e4dc451bc6 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:37.915300  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:37.917898  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.919693  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.921558  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.922240  1802 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:37.926622  1802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:37.927244  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:37.922361  1806 raft_consensus.cc:2955] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Committing config change with OpId 25.17: config changed from index 4 to 17, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:37.965663 12088 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.966347 12088 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.969434 12088 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:37.971233  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:37.971864  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:37.927417 11990 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:37.981344 12224 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.982237 12224 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.985283  1876 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:26:37.985584 12224 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:37.987910  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:37.991935  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:37.952323 12096 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.993472 12096 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:37.995462  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:37.996279  1876 raft_consensus.cc:2468] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 25.
I20260709 12:26:37.996495 12096 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:37.951593 12137 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:37.998040 12137 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:38.001188 12137 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:38.002362   561 tablet_replica.cc:333] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:37.915269 11748 leader_election.cc:290] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:38.008759  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 61494.4us
W20260709 12:26:38.009639  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 56269.9us
W20260709 12:26:38.005533  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:38.010401  1696 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:38.011250  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 111971us
I20260709 12:26:38.014374   561 raft_consensus.cc:2243] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:38.023027  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:38.028232  1801 raft_consensus.cc:2410] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
W20260709 12:26:38.019676  1844 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 206 similar messages]
W20260709 12:26:38.032337  1696 raft_consensus.cc:2686] T 84ce9896327e4c9d9888e80e8dbeeb0d P 075704438992440697732181e49ccd09: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260709 12:26:38.039567  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:38.044935  1696 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:38.046638  1697 leader_election.cc:304] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:38.059384   561 raft_consensus.cc:2272] T 6388b1a5243d463589750794e8ac5511 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:38.051915 12224 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:38.068035  1876 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:26:38.073027  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:38.074343  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:38.074728  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:38.075979  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:38.056164 12203 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:38.079305  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:38.064139 12174 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.068754 12224 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.089198 11748 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:38.059873 12260 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:38.047763 12096 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:26:38.099565 12174 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:38.075968  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:38.093778  1770 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.096295  1876 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
W20260709 12:26:38.108426  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:38.108788 12224 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:38.155409  1770 consensus_peers.cc:597] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:38.156522 12154 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:38.184341  1625 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 175) took 99784 ms (client timeout 15000 ms). Trace:
W20260709 12:26:38.184870  1625 rpcz_store.cc:269] 0709 12:24:58.386493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.388075 (+  1582us) service_pool.cc:224] Handling call
0709 12:26:38.165129 (+99777054us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:36.107147 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:36.215768 (+108621us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:36.215818 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:26:36.215836 (+    18us) tablet.cc:662] Decoding operations
0709 12:26:36.261693 (+ 45857us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:36.264642 (+  2949us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:36.266600 (+  1958us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:26:36.285075 (+ 18475us) tablet.cc:701] Row locks acquired
0709 12:26:36.287799 (+  2724us) write_op.cc:260] PREPARE: finished
0709 12:26:36.296953 (+  9154us) write_op.cc:270] Start()
0709 12:26:36.302065 (+  5112us) write_op.cc:276] Timestamp: P: 1783599996293303 usec, L: 0
0709 12:26:36.303720 (+  1655us) op_driver.cc:348] REPLICATION: starting
0709 12:26:36.346445 (+ 42725us) log.cc:844] Serialized 533 byte log entry
0709 12:26:36.718232 (+371787us) op_driver.cc:464] REPLICATION: finished
0709 12:26:36.848441 (+130209us) write_op.cc:301] APPLY: starting
0709 12:26:36.859535 (+ 11094us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:36.906870 (+ 47335us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:36.906968 (+    98us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:36.907725 (+   757us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:36.907953 (+   228us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 12:26:36.908004 (+    51us) write_op.cc:312] APPLY: finished
0709 12:26:36.908212 (+   208us) log.cc:844] Serialized 25 byte log entry
0709 12:26:36.908398 (+   186us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:36.908494 (+    96us) write_op.cc:454] Released schema lock
0709 12:26:36.908537 (+    43us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:36.909033 (+   496us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:36.909061 (+    28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":546012,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":103869,"apply.run_cpu_time_us":3909,"apply.run_wall_time_us":68604,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8311,"num_ops":1,"prepare.queue_time_us":299847,"prepare.run_cpu_time_us":7627,"prepare.run_wall_time_us":307298,"raft.queue_time_us":233277,"raft.run_cpu_time_us":24651,"raft.run_wall_time_us":99414,"replication_time_us":397470,"spinlock_wait_cycles":3328,"thread_start_us":319186,"threads_started":3,"wal-append.queue_time_us":56218,"wal-append.run_cpu_time_us":13010,"wal-append.run_wall_time_us":1471414}]]}
I20260709 12:26:38.207247  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:38.221870  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 119316us
I20260709 12:26:38.225129  1696 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
W20260709 12:26:38.307195  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 97191.8us
W20260709 12:26:38.317706  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 92769.4us
I20260709 12:26:38.319301  1696 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:38.306855 12153 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:38.354570  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:38.355825  1879 raft_consensus.cc:2393] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 in current term 29: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:38.358652  1873 raft_consensus.cc:1275] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 31 index: 16. (index mismatch)
W20260709 12:26:38.366863  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 47787.3us
I20260709 12:26:38.306476 11717 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:26:38.373936 12205 raft_consensus.cc:2955] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Committing config change with OpId 31.20: config changed from index 5 to 20, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:38.383769  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:38.337131 12158 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:38.401057 12262 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260709 12:26:38.421177  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 52472.6us
I20260709 12:26:38.407447 12214 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:38.428591  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 44906.7us
I20260709 12:26:38.437261   561 tablet_replica.cc:333] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:38.369036 12153 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:38.431459 12174 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:38.444720 12202 consensus_queue.cc:1048] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.008s
I20260709 12:26:38.438293 12271 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:38.426087 12203 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.026s
I20260709 12:26:38.464183  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:38.471987   561 raft_consensus.cc:2243] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:38.472123  1696 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:38.487470  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:38.491005 12096 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:38.492133 12267 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.492895 12267 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:38.496297 12267 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:38.470697 11717 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:38.499361 12277 log.cc:927] Time spent T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:26:38.501328 12276 log.cc:927] Time spent T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:26:38.459021  1621 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:38.530161  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 489408us
W20260709 12:26:38.539516  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 52132us
I20260709 12:26:38.539592   561 pending_rounds.cc:70] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Trying to abort 2 pending ops.
I20260709 12:26:38.541348   561 pending_rounds.cc:77] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 25 index: 15 } timestamp: 7305625549242257408 op_type: NO_OP noop_request { }
I20260709 12:26:38.541760   561 raft_consensus.cc:2889] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:38.542110   561 pending_rounds.cc:77] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 22 index: 14 } timestamp: 7305625490904637440 op_type: NO_OP noop_request { }
I20260709 12:26:38.542466   561 raft_consensus.cc:2889] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:38.542862   561 raft_consensus.cc:2272] T 7295d6494bad4378b63658f8274dfde8 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:38.544006  2382 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.475832 12081 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:47186 (request call id 0) took 2666 ms. Trace:
I20260709 12:26:38.546413  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:38.568091   561 tablet_replica.cc:333] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:38.568235 12153 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:38.583999   561 raft_consensus.cc:2243] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:38.575088  2410 consensus_queue.cc:237] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:38.592286  2385 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:26:38.597502  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53534.7us
W20260709 12:26:38.598974 12233 log.cc:927] Time spent T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:26:38.499348 11717 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:38.549696 12158 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:38.601747  1621 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 22 to 26, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 26 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:38.618889   561 raft_consensus.cc:2272] T 8bb813e4fab14a61b810a24b56899586 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:38.632920  2415 raft_consensus.cc:2955] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Committing config change with OpId 31.20: config changed from index 5 to 20, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:38.653015  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:38.632906 12205 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:49336 (request call id 18) took 1217 ms. Trace:
I20260709 12:26:38.647677  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:38.649271  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:38.669744  1873 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:26:38.679018 12051 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:38.679662 12051 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:38.681900 12051 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:38.682091 12224 consensus_meta.cc:287] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: Time spent flushing consensus metadata: real 0.522s	user 0.008s	sys 0.002s
I20260709 12:26:38.686394   561 tablet_replica.cc:333] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:38.690526 12284 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:38.698648  1846 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:38.699251  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:38.701718   561 raft_consensus.cc:2243] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:38.712363  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:38.725477  1696 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:38.727108 11717 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:38.727541 12251 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:38.728291 12251 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:38.728816 11717 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:38.659322 12205 rpcz_store.cc:276] 0709 12:26:37.415645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:37.429644 (+ 13999us) service_pool.cc:224] Handling call
0709 12:26:37.446657 (+ 17013us) log.cc:844] Serialized 494 byte log entry
0709 12:26:37.733721 (+287064us) spinlock_profiling.cc:243] Waited 8.23 ms on lock 0x7b5400401120. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd51142 00007f16abd31eaa 00007f16abd38d53 00007f16abd38cd9 00007f16abd38c69 00007f16abd38c31 00007f16abd37dfd 0000000000388634 0000000000388479 00007f16a8ceb475
0709 12:26:37.743620 (+  9899us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b18004b4780. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd58b26 00007f16abd5249a 00007f16abd51714
0709 12:26:37.744940 (+  1320us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b5400401120. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd58b26 00007f16abd5249a 00007f16abd51714 00007f16abd31eaa 00007f16abd38d53 00007f16abd38cd9 00007f16abd38c69 00007f16abd38c31 00007f16abd37dfd 0000000000388634
0709 12:26:38.632566 (+887626us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7786,"raft.run_cpu_time_us":7682,"raft.run_wall_time_us":322456,"spinlock_wait_cycles":34637696,"thread_start_us":124790,"threads_started":1,"wal-append.queue_time_us":255309}
I20260709 12:26:38.731292 12250 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:38.549265 12081 rpcz_store.cc:276] 0709 12:26:35.790673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.791250 (+   577us) service_pool.cc:224] Handling call
0709 12:26:35.854683 (+ 63433us) log.cc:844] Serialized 494 byte log entry
0709 12:26:36.503123 (+648440us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b180041d580. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16b1bb3ab1 00007f16b1bc0281 00007f16a8cb92a6 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd58b26 00007f16abd5249a 00007f16abd51714
0709 12:26:36.531265 (+ 28142us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b5400840920. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd51142 00007f16abd31eaa 00007f16abd38d53 00007f16abd38cd9 00007f16abd38c69 00007f16abd38c31 00007f16abd37dfd 0000000000388634 0000000000388479 00007f16a8ceb475
0709 12:26:36.758308 (+227043us) spinlock_profiling.cc:243] Waited 5.95 ms on lock 0x7b5400840920. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abd58b26 00007f16abd5249a 00007f16abd51714 00007f16abd31eaa 00007f16abd38d53 00007f16abd38cd9 00007f16abd38c69 00007f16abd38c31 00007f16abd37dfd 0000000000388634
0709 12:26:38.457334 (+1699026us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30427,"raft.queue_time_us":33368,"raft.run_cpu_time_us":8889,"raft.run_wall_time_us":1113847,"spinlock_wait_cycles":48137216,"thread_start_us":201215,"threads_started":1,"wal-append.queue_time_us":308988,"wal-append.run_cpu_time_us":8593,"wal-append.run_wall_time_us":1259333}
I20260709 12:26:38.735175   561 raft_consensus.cc:2272] T aca584e2f1114809bcadeaf6faad34f8 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:38.744315  1844 leader_election.cc:336] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.745437   561 tablet_replica.cc:333] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09: stopping tablet replica
W20260709 12:26:38.745896  1844 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.745358  1621 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 20 to 23, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 23 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:38.746367  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:38.746575   561 raft_consensus.cc:2243] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:38.747314   561 raft_consensus.cc:2272] T 79f5c05e64b945cbacc02914f31c5762 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:38.750100 12271 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:38.682521 12224 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:38.753554 12224 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:38.755189 12128 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:26:38.759074  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:38.764585  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d891ed30ca204d09b7b083088037b63d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 10)
W20260709 12:26:38.765679  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:38.766880  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:38.767997  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:38.768807  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45383us
I20260709 12:26:38.736370  1616 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet def0036446324543ba07e132e885133a with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 10)
W20260709 12:26:38.783382  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:38.759845  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:38.783763  1876 raft_consensus.cc:1275] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
W20260709 12:26:38.789330  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:38.794181   561 tablet_replica.cc:333] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:38.795083  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:38.795888  1799 raft_consensus.cc:2393] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 28: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:38.801756 12251 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:38.808899  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:38.813099  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:38.813963  1803 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 28.
I20260709 12:26:38.814636  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:38.815323  1876 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 27.
I20260709 12:26:38.815333  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:38.817204  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:38.817775  1803 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:38.817800  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:38.782899 12051 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:38.820175   561 raft_consensus.cc:2243] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:38.824285 12224 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.824890 12224 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:38.828297  1879 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:38.830302  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:38.787736 12154 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:26:38.832213  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:38.814198  1616 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 72785.4us
W20260709 12:26:38.838898  1844 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.842484  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:38.853552  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:38.855911  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:38.856528  1879 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:38.859628  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 31 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:38.860430  1845 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:38.856220  1844 leader_election.cc:336] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:38.862545  1844 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:38.868536  1803 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 30.
I20260709 12:26:38.852499 12067 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:38.831920 12081 consensus_queue.cc:1048] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:26:38.861986 12224 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:38.873925 12067 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.866657 12088 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:38.875953   561 raft_consensus.cc:2272] T ea5d8f652fc448d6af846ff6850004b8 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:38.863521 12154 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:38.875506  1696 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:38.894407 12051 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:38.929174  1696 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 54442.9us
I20260709 12:26:38.932677 12224 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:38.944730 12224 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:38.945659 12224 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:38.929509  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:38.933550  1696 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:38.932814  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:38.913200 12067 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:38.970984 12253 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:38.991278  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:39.005376   561 tablet_replica.cc:333] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: stopping tablet replica
W20260709 12:26:39.005431  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:39.006747   561 raft_consensus.cc:2243] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:38.932883  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 79085.6us
W20260709 12:26:39.019546  1844 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:39.024962 12271 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:39.025794 12271 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:39.038159   561 pending_rounds.cc:70] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Trying to abort 1 pending ops.
I20260709 12:26:39.039162   561 pending_rounds.cc:77] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 22 index: 14 } timestamp: 7305625518491705344 op_type: NO_OP noop_request { }
I20260709 12:26:39.039836   561 raft_consensus.cc:2889] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:39.040620   561 raft_consensus.cc:2272] T 36bc2f66a73947468d61113c226ed318 P 075704438992440697732181e49ccd09 [term 23 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:39.097911  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 44719.1us
W20260709 12:26:39.114907  1770 consensus_peers.cc:597] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:39.143491 12271 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:39.151208   561 tablet_replica.cc:333] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:39.152530   561 raft_consensus.cc:2243] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:39.153523   561 raft_consensus.cc:2272] T def0036446324543ba07e132e885133a P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:39.166807  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:39.167610  1872 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:39.175995  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:39.161258  1874 raft_consensus.cc:2955] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Committing config change with OpId 29.18: config changed from index 6 to 18, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:39.186719  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:39.194094  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 52097.3us
I20260709 12:26:39.194183 12290 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:26:39.223483  1770 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:39.220921 12290 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:39.234658 12154 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:39.244639   561 tablet_replica.cc:333] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:39.245988   561 raft_consensus.cc:2243] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:39.235618 12290 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:39.263300 12067 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:39.268525  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:39.270010  1800 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:39.289768 12114 raft_consensus.cc:2955] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Committing config change with OpId 29.18: config changed from index 6 to 18, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:39.293205 12306 log.cc:927] Time spent T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.115s	user 0.002s	sys 0.000s
W20260709 12:26:39.300124  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:39.308007  1800 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:39.309818  1696 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:39.317039 12067 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:39.320026  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:39.320695  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:39.320847  1876 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:39.322443 11717 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:39.323052 11717 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 29 LEADER]: Becoming Leader. State: Replica: 075704438992440697732181e49ccd09, State: Running, Role: LEADER
I20260709 12:26:39.325357 11717 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:39.346340 12290 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:39.349874 12290 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:39.362321   561 raft_consensus.cc:2272] T 9585ba97eaf14e069dcd2dcaf726a09b P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:39.362277  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:39.383697  1872 raft_consensus.cc:3060] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:39.372243  1770 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:39.387636  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:39.395371   561 tablet_replica.cc:333] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: stopping tablet replica
W20260709 12:26:39.395596  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:39.399936   561 raft_consensus.cc:2243] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:39.405787  1872 raft_consensus.cc:2468] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:39.409263  1771 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:39.417058  1876 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 075704438992440697732181e49ccd09 in term 29.
I20260709 12:26:39.420662 11986 raft_consensus.cc:2804] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:39.438884 11986 raft_consensus.cc:697] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:39.443384 12114 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:47186 (request call id 1) took 1278 ms. Trace:
I20260709 12:26:39.455137   561 pending_rounds.cc:70] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Trying to abort 1 pending ops.
I20260709 12:26:39.457019   561 pending_rounds.cc:77] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 22 index: 17 } timestamp: 7305625504285835264 op_type: NO_OP noop_request { }
I20260709 12:26:39.457576   561 raft_consensus.cc:2889] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:39.458057   561 raft_consensus.cc:2272] T 66d3405528d44edb8c0be3b296ef72be P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:39.456564 12114 rpcz_store.cc:276] 0709 12:26:38.163603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.163790 (+   187us) service_pool.cc:224] Handling call
0709 12:26:38.573564 (+409774us) spinlock_profiling.cc:243] Waited 5.08 ms on lock 0x7b180047e1a0. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abe1373d 00007f16abddef3f 00007f16abdde6b3 00007f16abdd69fa 00007f16b0d9e4c9 00007f16aa31ac54 00007f16aa31abe2 00007f16aa31ab11 00007f16aa31aa8c 00007f16aa319d42
0709 12:26:38.663619 (+ 90055us) log.cc:844] Serialized 494 byte log entry
0709 12:26:38.735415 (+ 71796us) spinlock_profiling.cc:243] Waited 64.7 ms on lock 0x7b54008c0354. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdde735 00007f16abdd69fa 00007f16b0d9e4c9 00007f16aa31ac54 00007f16aa31abe2 00007f16aa31ab11 00007f16aa31aa8c 00007f16aa319d42 00007f16aa11348c 00007f16aa1128c6
0709 12:26:38.927469 (+192054us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b4c00270500. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16aa0a4a20 00007f16aa0b5a29 00007f16aa0b613f 00007f16aa07a2ab 00007f16aa0a7f3c 00007f16aa0a83fe 00007f16aa33550b 00007f16abd34381 00007f16abd3226b 00007f16abd38d53 00007f16abd38cd9
0709 12:26:39.440435 (+512966us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":11696,"mutex_wait_us":15253,"raft.queue_time_us":30914,"raft.run_cpu_time_us":4479,"raft.run_wall_time_us":149658,"spinlock_wait_cycles":201955328}
I20260709 12:26:39.467823  1616 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet ea5d8f652fc448d6af846ff6850004b8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:26:39.485859 12214 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.486650 12214 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:39.488397 11986 consensus_queue.cc:237] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:39.491080 12214 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:39.491947  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:39.520530  1621 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 177) took 51174 ms (client timeout 15000 ms). Trace:
I20260709 12:26:39.547308  1874 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260709 12:26:39.559278 12310 log.cc:927] Time spent T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:26:39.570938 11990 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.014s
W20260709 12:26:39.585366  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:39.589854  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:39.569962 11717 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.597574  1803 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:26:39.604764 12224 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.002s
I20260709 12:26:39.603008 11717 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:39.614478 11717 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:39.616089  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:39.521153  1621 rpcz_store.cc:269] 0709 12:25:48.346578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:53.134278 (+4787700us) service_pool.cc:224] Handling call
0709 12:25:53.567850 (+433572us) spinlock_profiling.cc:243] Waited 3.79 ms on lock 0x7b5400002490. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb40b2 00007f16b1c2d759 00007f16b1bc5a76 00007f16b1b96d58 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc
0709 12:26:39.520409 (+45952559us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:38.918008 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:38.926141 (+  8133us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:38.926194 (+    53us) write_op.cc:435] Acquired schema lock
0709 12:26:38.926217 (+    23us) tablet.cc:662] Decoding operations
0709 12:26:38.926917 (+   700us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:38.926991 (+    74us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:38.929697 (+  2706us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:26:38.930220 (+   523us) tablet.cc:701] Row locks acquired
0709 12:26:38.930245 (+    25us) write_op.cc:260] PREPARE: finished
0709 12:26:38.932268 (+  2023us) write_op.cc:270] Start()
0709 12:26:38.933343 (+  1075us) write_op.cc:276] Timestamp: P: 1783599998930400 usec, L: 0
0709 12:26:38.933377 (+    34us) op_driver.cc:348] REPLICATION: starting
0709 12:26:38.934983 (+  1606us) log.cc:844] Serialized 1355 byte log entry
0709 12:26:39.232807 (+297824us) op_driver.cc:464] REPLICATION: finished
0709 12:26:39.324224 (+ 91417us) write_op.cc:301] APPLY: starting
0709 12:26:39.324327 (+   103us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:39.326708 (+  2381us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:39.362049 (+ 35341us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:39.424540 (+ 62491us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:39.424800 (+   260us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=0,mrs_lookups=0
0709 12:26:39.424867 (+    67us) write_op.cc:312] APPLY: finished
0709 12:26:39.425125 (+   258us) log.cc:844] Serialized 41 byte log entry
0709 12:26:39.436737 (+ 11612us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:39.439541 (+  2804us) write_op.cc:454] Released schema lock
0709 12:26:39.440956 (+  1415us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:39.477299 (+ 36343us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:39.477351 (+    52us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":586386,"reported_tablets":32,"spinlock_wait_cycles":8706944,"child_traces":[["op",{"apply.queue_time_us":90739,"apply.run_cpu_time_us":7103,"apply.run_wall_time_us":132665,"cfile_cache_hit":14,"cfile_cache_hit_bytes":16679,"num_ops":3,"prepare.queue_time_us":82582,"prepare.run_cpu_time_us":36476,"prepare.run_wall_time_us":135978,"raft.queue_time_us":92435,"raft.run_cpu_time_us":28092,"raft.run_wall_time_us":77325,"replication_time_us":299229,"thread_start_us":276982,"threads_started":4,"wal-append.queue_time_us":129224}]]}
I20260709 12:26:39.619174  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:39.621028  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:39.634523  1696 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:39.637742 12260 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:39.646622   561 tablet_replica.cc:333] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:39.658608   561 raft_consensus.cc:2243] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:39.666303  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:39.663488  1623 catalog_manager.cc:5697] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 22 to 24. New cstate: current_term: 24 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:39.671258  1623 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 21 to 24, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 24 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:39.675379   561 raft_consensus.cc:2272] T bb48ef23a7434834929994987a36f360 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:39.691010   561 tablet_replica.cc:333] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:39.688688  1623 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 21 to 26, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 26 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:39.691987   561 raft_consensus.cc:2243] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:39.692957   561 raft_consensus.cc:2272] T 2e89c188ae524548ae35a90336c0c92c P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:39.694476  1623 catalog_manager.cc:5697] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 17 to 19, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 19 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:26:39.698100  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:39.709569  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 62449.2us
I20260709 12:26:39.703713  1623 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 23 to 25, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 25 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:39.715643  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:39.721709  1696 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:39.726750 12260 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:39.740024   561 tablet_replica.cc:333] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:39.752677   561 raft_consensus.cc:2243] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:39.752470  1623 catalog_manager.cc:5697] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 22 to 27, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:39.757910  1623 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 19 to 24. New cstate: current_term: 24 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:39.768570  1806 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:26:39.770637   561 pending_rounds.cc:70] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Trying to abort 1 pending ops.
I20260709 12:26:39.771040   561 pending_rounds.cc:77] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 27 index: 16 } timestamp: 7305625510778114048 op_type: NO_OP noop_request { }
I20260709 12:26:39.771427   561 raft_consensus.cc:2889] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:39.771785   561 raft_consensus.cc:2272] T f15e08b3d9804a00b9836e5223042266 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:39.773252  1623 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 19 to 20, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 20 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:39.778735 12091 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.009s
I20260709 12:26:39.816182  1623 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 17 to 21, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 21 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:39.819341 12312 log.cc:927] Time spent T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:26:39.825350   561 tablet_replica.cc:333] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:39.826479   561 raft_consensus.cc:2243] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:39.827090  1872 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 075704438992440697732181e49ccd09: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:26:39.827327   561 raft_consensus.cc:2272] T 4a57b66dec794db09a7f9048a3d0de10 P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:39.830873   561 tablet_replica.cc:333] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09: stopping tablet replica
W20260709 12:26:39.818534  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:39.833595 12224 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:26:39.835597   561 raft_consensus.cc:2243] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:39.841953  1623 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 22 to 27, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:39.843631   561 raft_consensus.cc:2272] T 80ed6fcb806a4023b94ecd7b266b421e P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:39.872643   561 tablet_replica.cc:333] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:39.875458   561 raft_consensus.cc:2243] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:39.873551  1623 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 21 to 27. New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:39.881752  1623 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 23 to 26, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 26 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:39.895357   561 raft_consensus.cc:2272] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:39.968529 12286 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:40.026623   561 tablet_replica.cc:333] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.028123   561 raft_consensus.cc:2243] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:40.051136   561 raft_consensus.cc:2272] T b1cb2a27e8444930b93f1e21c2deb19c P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.054760   561 tablet_replica.cc:333] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.070739 12067 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.112358   561 raft_consensus.cc:2243] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:40.012192 12286 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:40.118106   561 raft_consensus.cc:2272] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 075704438992440697732181e49ccd09 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.117615 12067 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:40.171919 12067 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:40.174538   561 tablet_replica.cc:333] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.179507   561 raft_consensus.cc:2243] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 24 LEADER]: Raft consensus shutting down.
I20260709 12:26:40.187333  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:40.203436  1879 raft_consensus.cc:1275] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:40.186621 12214 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.239614   561 raft_consensus.cc:2272] T 65799458c3fb44e7a162099338c0bb62 P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.209380 12067 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:40.231318  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:40.236327 12260 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.251977 12214 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:40.305580 12286 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:40.260689 12067 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:40.319226  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:40.279170 12260 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:40.318997 12224 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:40.341691  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 84870.2us
I20260709 12:26:40.363504  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:40.330643 12338 consensus_queue.cc:1048] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.039s
W20260709 12:26:40.378948  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:40.382252   561 tablet_replica.cc:333] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.383374   561 raft_consensus.cc:2243] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:40.383245  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:40.405375  1771 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:40.405368 12214 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:40.415014 11876 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:40.417404 12260 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:40.356976 12224 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:40.422003  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:40.422889  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:40.425201 12067 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:40.427878  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 45036us
W20260709 12:26:40.429087  1770 consensus_peers.cc:597] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:40.420500 12224 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:40.432205  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 64770.8us
I20260709 12:26:40.438864   561 raft_consensus.cc:2272] T 8f285ce2457a47ebbd05920b42b35fba P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.448004 12067 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:40.448657 12067 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:40.454658  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:40.480224 12067 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:40.483150   561 tablet_replica.cc:333] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.484388   561 raft_consensus.cc:2243] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:40.503289  1873 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:40.503880  1696 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:40.505085  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:40.506768  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:40.508401  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:40.509298  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:40.512316  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:40.519907  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:40.529037  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f86194bda894372ab64d2c6a9937a5b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:40.533177  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:40.545953   561 raft_consensus.cc:2272] T 513580259d0a4e6d96fb16084d5a90dc P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.548893   561 tablet_replica.cc:333] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.551440   561 raft_consensus.cc:2243] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:40.513870 12260 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:40.570431  1770 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.593291  1696 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:40.595055 12328 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:40.595693 12328 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:40.582273 12323 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:40.597002 12323 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:40.600630 12328 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:40.604070  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:40.606381   561 raft_consensus.cc:2272] T d891ed30ca204d09b7b083088037b63d P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.610095   561 tablet_replica.cc:333] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.613021 12214 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:40.617470   561 raft_consensus.cc:2243] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:40.614199  1696 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:40.620270  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:40.626246  1696 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:40.624982   561 raft_consensus.cc:2272] T 1b33fb206f534e1e890a4546d3d43e84 P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:40.633890  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:40.641490 12332 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:40.642225 12332 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:40.627736 12224 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:40.643043 12214 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:40.644829 12332 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:26:40.644783  1696 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:40.648532 12214 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:40.648742  1696 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:40.650866 12214 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:40.644865 12323 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:40.655138  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:40.668592   561 tablet_replica.cc:333] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.669369  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:40.669668   561 raft_consensus.cc:2243] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:40.670449   561 raft_consensus.cc:2272] T 36f1f37b7d544bb29cd9cace84f6e946 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:40.648741  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:40.691704  1770 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:40.691838  1846 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:40.692852 12323 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:40.694658 12342 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.695477 12342 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:40.696555  1844 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.699285  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:40.703159  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:40.706228  1844 leader_election.cc:336] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.706723  1844 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:40.708379 12323 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:40.711642  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "075704438992440697732181e49ccd09" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:40.708377 12342 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:40.716454  1696 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 075704438992440697732181e49ccd09; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:40.717994 12224 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:40.735703   561 tablet_replica.cc:333] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.736842   561 raft_consensus.cc:2243] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:40.737550   561 raft_consensus.cc:2272] T 908089442ca44109875257bfbc9d06c2 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:40.704953  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:40.757097  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:40.758021  1770 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.758515  1770 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:40.759442 12332 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:40.764351   561 tablet_replica.cc:333] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.765312   561 raft_consensus.cc:2243] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:40.765918   561 raft_consensus.cc:2272] T 9b8cae5d30dd4a68a4fba5846e83873e P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.769572   561 tablet_replica.cc:333] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.774628   561 raft_consensus.cc:2243] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:40.775472   561 raft_consensus.cc:2272] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.780380   561 tablet_replica.cc:333] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.781339   561 raft_consensus.cc:2243] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:40.782338   561 raft_consensus.cc:2272] T 82eacf0c2a8c4d8db4052327304a8793 P 075704438992440697732181e49ccd09 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.787897   561 tablet_replica.cc:333] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.788873   561 raft_consensus.cc:2243] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:40.792272   561 raft_consensus.cc:2272] T c22ef2bc934c4b7ea57aad5853ab6122 P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.796880   561 tablet_replica.cc:333] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:40.797780   561 raft_consensus.cc:2243] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:40.798556   561 raft_consensus.cc:2272] T 6f86194bda894372ab64d2c6a9937a5b P 075704438992440697732181e49ccd09 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.802412 12344 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:40.807049 12344 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
W20260709 12:26:40.878916  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 71346.2us
W20260709 12:26:40.883599  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 75563.2us
I20260709 12:26:40.886644 12344 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:40.887614  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 79098.5us
I20260709 12:26:40.890578  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:40.891418  1800 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 31.
I20260709 12:26:40.892926  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:40.904189  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:40.904728   561 tablet_replica.cc:333] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09: stopping tablet replica
W20260709 12:26:40.912465  1844 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:40.901722 12344 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:26:40.929914  1770 consensus_peers.cc:597] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:40.934113   561 raft_consensus.cc:2243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:26:40.941897   561 raft_consensus.cc:2272] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:40.933403 12344 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:40.955613 12344 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:40.997709   561 tablet_replica.cc:333] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:41.011987   561 raft_consensus.cc:2243] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:41.012826   561 raft_consensus.cc:2272] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 075704438992440697732181e49ccd09 [term 27 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:41.049950  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:41.057142  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:41.061385   561 tablet_replica.cc:333] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:41.067639   561 raft_consensus.cc:2243] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:41.072315   561 raft_consensus.cc:2272] T efbb8c187a4e410aa09e81ceb9ebecc3 P 075704438992440697732181e49ccd09 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:41.086823   561 tablet_replica.cc:333] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:41.096088   561 raft_consensus.cc:2243] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:41.093022 11876 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.090267 12344 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:41.104090   561 raft_consensus.cc:2272] T 5280e62b42a141359d53bd4f241659ef P 075704438992440697732181e49ccd09 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:41.120564 12337 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.123178 12337 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:41.129766  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:41.121372 11876 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:41.134438 11876 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:41.136224  1629 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 273) took 136028 ms (client timeout 15000 ms). Trace:
I20260709 12:26:41.136837 12337 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:41.138019  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:41.139029  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:41.150801  2410 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 33.
I20260709 12:26:41.152120  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:41.146106  1629 rpcz_store.cc:269] 0709 12:24:25.107988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.111062 (+  3074us) service_pool.cc:224] Handling call
0709 12:26:41.136008 (+136024946us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38280,"reported_tablets":50}
W20260709 12:26:41.154340  1844 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.155514 12351 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:41.155974 12351 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.156343 12351 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:41.160956 12344 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:41.162874  1623 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 257) took 36105 ms (client timeout 15000 ms). Trace:
W20260709 12:26:41.163426  1623 rpcz_store.cc:269] 0709 12:26:05.054423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:17.731741 (+12677318us) service_pool.cc:224] Handling call
0709 12:26:41.159225 (+23427484us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:40.036944 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:40.037672 (+   728us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:40.037711 (+    39us) write_op.cc:435] Acquired schema lock
0709 12:26:40.037729 (+    18us) tablet.cc:662] Decoding operations
0709 12:26:40.104847 (+ 67118us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:40.112757 (+  7910us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:40.114944 (+  2187us) tablet.cc:685] Acquiring locks for 12 operations
0709 12:26:40.317714 (+202770us) tablet.cc:701] Row locks acquired
0709 12:26:40.321053 (+  3339us) write_op.cc:260] PREPARE: finished
0709 12:26:40.338106 (+ 17053us) write_op.cc:270] Start()
0709 12:26:40.345005 (+  6899us) write_op.cc:276] Timestamp: P: 1783600000333765 usec, L: 0
0709 12:26:40.346980 (+  1975us) op_driver.cc:348] REPLICATION: starting
0709 12:26:40.375263 (+ 28283us) log.cc:844] Serialized 4806 byte log entry
0709 12:26:40.402630 (+ 27367us) spinlock_profiling.cc:243] Waited 6.21 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd41571 00007f16abdd3e28 00007f16af80e581 00007f16af80dd58 00007f16af8129f1 00007f16af8129a9 00007f16af812939 00007f16af812901 00007f16af811b2d 0000000000388634 0000000000388479
0709 12:26:40.589137 (+186507us) op_driver.cc:464] REPLICATION: finished
0709 12:26:40.656126 (+ 66989us) write_op.cc:301] APPLY: starting
0709 12:26:40.656240 (+   114us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:40.716508 (+ 60268us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:40.716533 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:40.763783 (+ 47250us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:40.771508 (+  7725us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=24,key_file_lookups=24,delta_file_lookups=0,mrs_lookups=0
0709 12:26:40.771575 (+    67us) write_op.cc:312] APPLY: finished
0709 12:26:40.772007 (+   432us) log.cc:844] Serialized 113 byte log entry
0709 12:26:40.772236 (+   229us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:40.772466 (+   230us) write_op.cc:454] Released schema lock
0709 12:26:40.772533 (+    67us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:40.773048 (+   515us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:40.773083 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3473,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":66701,"apply.run_cpu_time_us":16884,"apply.run_wall_time_us":116905,"cfile_cache_hit":50,"cfile_cache_hit_bytes":54335,"num_ops":12,"prepare.queue_time_us":49551,"prepare.run_cpu_time_us":10121,"prepare.run_wall_time_us":365902,"replication_time_us":241921,"spinlock_wait_cycles":14291072,"thread_start_us":70355,"threads_started":2}]]}
I20260709 12:26:41.162936  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:41.175917  1800 raft_consensus.cc:3060] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:26:41.191167  1800 raft_consensus.cc:2468] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 32.
I20260709 12:26:41.192988  1845 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:41.193951 12344 raft_consensus.cc:2804] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:26:41.195267 12344 raft_consensus.cc:697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 32 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:41.196209 12344 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } }
I20260709 12:26:41.198266   561 tablet_replica.cc:333] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: stopping tablet replica
W20260709 12:26:41.203982  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:41.206954  1844 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.208835   561 raft_consensus.cc:2243] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:41.222108  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:41.226512  1770 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.227200   561 pending_rounds.cc:70] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Trying to abort 1 pending ops.
I20260709 12:26:41.227097  1770 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:41.227524   561 pending_rounds.cc:77] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 25 index: 16 } timestamp: 7305625541193043968 op_type: NO_OP noop_request { }
I20260709 12:26:41.227903 11876 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:41.227927   561 raft_consensus.cc:2889] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:26:41.229086   561 raft_consensus.cc:2272] T b3034677490f4f0fac2a4ed3e2a8142b P 075704438992440697732181e49ccd09 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:41.232990   561 tablet_replica.cc:333] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:41.233860   561 raft_consensus.cc:2243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:41.234517   561 raft_consensus.cc:2272] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 075704438992440697732181e49ccd09 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:41.245872  1622 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 22 to 28, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 28 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:41.237766 12351 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:41.254057  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:41.254703  2415 raft_consensus.cc:3060] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:26:41.259991  1844 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.265789  2415 raft_consensus.cc:2468] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 34.
I20260709 12:26:41.266983  1846 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:41.268239 12344 raft_consensus.cc:2804] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:26:41.268788 12344 raft_consensus.cc:697] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 34 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:41.269752   561 tablet_replica.cc:333] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:41.269649 12344 consensus_queue.cc:237] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:41.270563   561 raft_consensus.cc:2243] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:41.267925 12351 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:41.309363   561 raft_consensus.cc:2272] T bd91edf20ef54a30a2f5af0c0b682350 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:41.314213 12300 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.315057 12300 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:41.318168 12300 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:41.321800  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:41.327155  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:41.327980  1803 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:41.331108  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:41.332249 12351 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:41.332799 12351 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.333272 12351 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:41.333352   561 tablet_replica.cc:333] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:41.341625 12300 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.342420 12300 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:41.334690   561 raft_consensus.cc:2243] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 34 LEADER]: Raft consensus shutting down.
W20260709 12:26:41.348893  1844 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.349058   561 raft_consensus.cc:2272] T c5f3d785e05841fbb0efa210c3d9675c P 075704438992440697732181e49ccd09 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:41.355718  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:41.358283 12300 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:41.346531 12351 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:41.371542  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:41.372247  1803 raft_consensus.cc:3060] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:26:41.378394  1844 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.379088  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:41.379849 12351 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:41.382388 12344 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:41.382512  1803 raft_consensus.cc:2468] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:41.384289  1845 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:41.385260 12344 raft_consensus.cc:2804] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:41.385919 12344 raft_consensus.cc:697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:41.387022 12344 consensus_queue.cc:237] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:41.389645   561 tablet_replica.cc:333] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:41.391249   561 raft_consensus.cc:2243] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:41.392419   561 pending_rounds.cc:70] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Trying to abort 1 pending ops.
I20260709 12:26:41.393661   561 pending_rounds.cc:77] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09: Aborting op as it isn't in flight: id { term: 22 index: 16 } timestamp: 7305625539480879104 op_type: NO_OP noop_request { }
W20260709 12:26:41.393045  1844 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.394218   561 raft_consensus.cc:2889] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
W20260709 12:26:41.396757  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:41.396807   561 raft_consensus.cc:2272] T bbbcc16cebc244769e7f1a1b6e29594c P 075704438992440697732181e49ccd09 [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:41.413731 11876 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.415045 11876 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:41.419286 11876 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:41.419480  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:41.420361  1876 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:41.421312   561 tablet_replica.cc:333] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09: stopping tablet replica
I20260709 12:26:41.422206   561 raft_consensus.cc:2243] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:41.422895  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:41.423764  1770 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.424292 12340 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:41.424795 12340 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.425181 12340 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:41.430567   561 raft_consensus.cc:2272] T 76f6bc3c4cce45d7855e6b3429341da2 P 075704438992440697732181e49ccd09 [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:41.463493  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:41.474071 12340 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:41.483697  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:41.484609  1876 raft_consensus.cc:3060] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:41.488889  1770 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:41.505270  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:41.483470 12340 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:41.516505  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:41.529991  1876 raft_consensus.cc:2468] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:41.531502  1771 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:41.533140 12326 raft_consensus.cc:2804] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:41.533753 12326 raft_consensus.cc:697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:41.534993 12326 consensus_queue.cc:237] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:41.542974  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:41.598730 12336 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.599411 12336 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:41.603152  1622 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 276) took 40131 ms (client timeout 15000 ms). Trace:
W20260709 12:26:41.605405  1844 consensus_peers.cc:597] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:41.608376  1770 leader_election.cc:336] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:41.610069  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:41.611478 12336 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:41.613559  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:41.610575  1622 rpcz_store.cc:269] 0709 12:26:01.472079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:08.730350 (+7258271us) service_pool.cc:224] Handling call
0709 12:26:41.603031 (+32872681us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:41.322189 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:41.322710 (+   521us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:41.322747 (+    37us) write_op.cc:435] Acquired schema lock
0709 12:26:41.322764 (+    17us) tablet.cc:662] Decoding operations
0709 12:26:41.323133 (+   369us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:41.323205 (+    72us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:41.323232 (+    27us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:26:41.323415 (+   183us) tablet.cc:701] Row locks acquired
0709 12:26:41.323436 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:26:41.323568 (+   132us) write_op.cc:270] Start()
0709 12:26:41.323669 (+   101us) write_op.cc:276] Timestamp: P: 1783600001323529 usec, L: 0
0709 12:26:41.323683 (+    14us) op_driver.cc:348] REPLICATION: starting
0709 12:26:41.324354 (+   671us) log.cc:844] Serialized 533 byte log entry
0709 12:26:41.352414 (+ 28060us) op_driver.cc:464] REPLICATION: finished
0709 12:26:41.406464 (+ 54050us) write_op.cc:301] APPLY: starting
0709 12:26:41.406576 (+   112us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:41.407928 (+  1352us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:41.408058 (+   130us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:41.408838 (+   780us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:41.409064 (+   226us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=2,key_file_lookups=2,delta_file_lookups=0,mrs_lookups=0
0709 12:26:41.409120 (+    56us) write_op.cc:312] APPLY: finished
0709 12:26:41.412725 (+  3605us) log.cc:844] Serialized 25 byte log entry
0709 12:26:41.413733 (+  1008us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:41.413816 (+    83us) write_op.cc:454] Released schema lock
0709 12:26:41.413860 (+    44us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:41.414310 (+   450us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:41.414344 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":16829,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":47665,"apply.run_cpu_time_us":3691,"apply.run_wall_time_us":7842,"cfile_cache_hit":6,"cfile_cache_hit_bytes":8311,"num_ops":1,"prepare.queue_time_us":70023,"prepare.run_cpu_time_us":2471,"prepare.run_wall_time_us":2471,"replication_time_us":28405,"thread_start_us":110612,"threads_started":2}]]}
I20260709 12:26:41.619048  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:41.624612 11876 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:41.630342 12308 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.631130 12308 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:41.637645  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:41.638944 12334 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.639616 12334 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:41.642706 12334 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:41.643868  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:41.651002  2382 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.651801  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:41.658712 12335 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:41.661815  2382 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.662456  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:41.649827 12308 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:41.664952 12335 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:26:41.671110  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:41.757788  1626 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 174) took 124008 ms (client timeout 15000 ms). Trace:
W20260709 12:26:41.774044  1626 rpcz_store.cc:269] 0709 12:24:37.748737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.750938 (+  2201us) service_pool.cc:224] Handling call
0709 12:26:41.757259 (+124006321us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62153,"reported_tablets":32}
I20260709 12:26:41.778069 12300 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.787846 12300 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:41.784816  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:41.797250  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:41.800696  2415 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 24.
I20260709 12:26:41.802716  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:41.805167 12344 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:26:41.805637 12344 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.806089 12344 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:41.812887 12337 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.814137 12337 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:41.820348 12337 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:41.821125  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:41.822664  2410 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:41.829480  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:41.832284  1844 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.794426 12300 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:41.836434  1844 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:41.836176 12300 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:41.837455 12300 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:41.838122 12300 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:41.854302  1800 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:26:41.887985 12351 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:41.909469  2410 raft_consensus.cc:1275] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 34 index: 24. (index mismatch)
I20260709 12:26:41.916060 12351 consensus_queue.cc:1048] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:26:41.959167  1804 raft_consensus.cc:1275] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
W20260709 12:26:41.974426  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:42.023819 12300 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:42.028061  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:42.033891 12337 consensus_queue.cc:1048] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:42.034371 12344 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:42.046433 12344 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:42.049731  1627 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 253) took 146671 ms (client timeout 15000 ms). Trace:
W20260709 12:26:42.052788  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.050462  1627 rpcz_store.cc:269] 0709 12:24:15.378336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.397586 (+ 19250us) service_pool.cc:224] Handling call
0709 12:24:15.823807 (+426221us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b5400002490. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16a8b73db1 00007f16a8cb3b71 00007f16a8cb40b2 00007f16b1c2d759 00007f16b1bc5a76 00007f16b1b96d58 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc
0709 12:26:42.049397 (+146225590us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":239172,"reported_tablets":49,"spinlock_wait_cycles":23709440}
W20260709 12:26:42.065867  1844 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.070545  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 25 candidate_status { last_received { term: 24 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:42.071456  2415 raft_consensus.cc:3060] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:26:42.100006  2415 raft_consensus.cc:2468] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:26:42.108618 12326 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.109614 12326 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:42.125054 12326 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:42.135844  1846 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:42.138741 12351 raft_consensus.cc:2804] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:26:42.141499 12351 raft_consensus.cc:697] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:42.142928 12351 consensus_queue.cc:237] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 24.19, Last appended by leader: 19, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:42.146076  1876 raft_consensus.cc:1275] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:26:42.149657 12300 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:42.160792  1844 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.161193 12308 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
W20260709 12:26:42.161612  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:42.162638  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:42.162204 12308 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:42.164795  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:42.165618  2415 raft_consensus.cc:3060] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:42.165814 12308 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:42.165814 12326 consensus_queue.cc:1048] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:42.181370 12337 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.182181 12337 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.184785 12337 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:42.188987  1770 consensus_peers.cc:597] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.190735  2382 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.202718  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:42.203372  2415 raft_consensus.cc:2468] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:42.211337  1846 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:42.212915  1770 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:42.213855  1770 consensus_peers.cc:597] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:42.215792 12344 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.216410 12344 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:42.214128  1844 leader_election.cc:336] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.204257  2410 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:42.219800 12344 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:42.228015  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:42.236797 12337 raft_consensus.cc:2804] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:42.219689 12326 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.237962 12326 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:42.234444  1844 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.240644 12326 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:42.244045  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:42.245365 12336 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:42.245828 12336 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.246227 12336 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:42.252979 12337 raft_consensus.cc:697] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
W20260709 12:26:42.253461  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:42.254359 12337 consensus_queue.cc:237] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:42.267735  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:42.277058  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:42.278021  1800 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
I20260709 12:26:42.325541  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:42.326253  1770 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.280251 12336 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:42.327775  1770 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:42.335047 12326 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:26:42.336673  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:42.336947  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:42.339082  1806 raft_consensus.cc:2393] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 30: Already voted for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in this term.
I20260709 12:26:42.348438 12336 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:42.354765  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:42.356757  1770 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.356931  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
W20260709 12:26:42.375964  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:42.359894 12334 raft_consensus.cc:3060] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:42.352550  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:42.378400 12351 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:42.378919 12351 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.379343 12351 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:42.385231 12337 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.386039 12337 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.387552 12324 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.388360 12324 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.391804 12324 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:42.393419  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:42.397877  1844 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.399263 12337 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:42.384382  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:42.409009 12337 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:42.418620  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:42.419456  1806 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
W20260709 12:26:42.421852  1770 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.451629  1625 catalog_manager.cc:5697] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 30 to 31, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 31 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.458859  1625 catalog_manager.cc:5697] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 27 to 28, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 28 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.460994  1625 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 27 to 30. New cstate: current_term: 30 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:42.468312 12351 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.473229 12334 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:42.486390 12324 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:42.484614 12351 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:42.487203 12324 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:42.488042  1625 catalog_manager.cc:5697] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 20 to 26, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 26 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:42.489954  1625 catalog_manager.cc:5697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 24 to 27. New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:42.492480  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:42.492488 12337 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.492344  1625 catalog_manager.cc:5697] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 26 to 28, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 28 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.493518 12337 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.496654 12337 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:42.498622 12332 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.499460 12332 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.501847 12332 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:42.515187  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:42.517391  2410 raft_consensus.cc:2468] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:42.523838  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:42.524374  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:42.503149  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:42.505784  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:42.490744 12324 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:42.510219  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69288.9us
I20260709 12:26:42.528664  1872 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 33.
I20260709 12:26:42.512336  1770 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:42.524530  1799 raft_consensus.cc:3060] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:42.539876  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:42.540953 12324 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:42.541381 12324 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.541781 12324 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:42.546525  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:42.547304  1806 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:42.549289 12337 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:42.549782 12337 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.550195 12337 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:42.503845  1628 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 275) took 79011 ms (client timeout 15000 ms). Trace:
W20260709 12:26:42.554978  1628 rpcz_store.cc:269] 0709 12:25:23.492568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.504237 (+ 11669us) service_pool.cc:224] Handling call
0709 12:26:42.503585 (+78999348us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":50}
W20260709 12:26:42.560983  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 44576.7us
I20260709 12:26:42.561801  1625 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:42.577209 12337 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.579650  2410 raft_consensus.cc:1275] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 24 index: 19. Preceding OpId from leader: term: 25 index: 20. (index mismatch)
I20260709 12:26:42.581048 12332 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:42.581663 12361 consensus_queue.cc:1048] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:26:42.582075  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 162112us
W20260709 12:26:42.588809  1844 leader_election.cc:336] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.589500  1625 catalog_manager.cc:5697] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 27 to 28, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 28 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.591987 12332 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.618414 12337 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:42.593962  1799 raft_consensus.cc:2468] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:26:42.571658  1772 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:42.618278 12332 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.619654 12326 raft_consensus.cc:2804] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:42.620267 12326 raft_consensus.cc:697] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
W20260709 12:26:42.622520  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 50970.6us
I20260709 12:26:42.621426 12326 consensus_queue.cc:237] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:42.623986  1625 catalog_manager.cc:5697] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.627172  1772 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:42.628166 12336 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:42.628305  1625 catalog_manager.cc:5697] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 18 to 24. New cstate: current_term: 24 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.630414 12332 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:42.600616  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:42.622656  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:42.634182 12361 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:26:42.634619 12361 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.635057 12361 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:26:42.639643  1844 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.630059  1625 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 23 to 30, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 30 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.649895  1625 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 24 to 27. New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:42.656755 12351 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.657449 12351 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.659914 12351 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:42.663754  1845 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:42.665088  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:42.665709  1799 raft_consensus.cc:3060] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:42.672676 12324 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.677804 12351 raft_consensus.cc:2804] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:42.678743 12351 raft_consensus.cc:697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:42.679992 12351 consensus_queue.cc:237] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.683785 12336 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.684577 12336 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.660755  1625 catalog_manager.cc:5697] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 28 to 29, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 29 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.690541 12336 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:42.690474  1625 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 25 to 31, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 31 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:42.692232  1625 catalog_manager.cc:5697] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 24 to 25, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.694255  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:42.723552  1625 catalog_manager.cc:5697] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 25 to 26, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 26 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.726123  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:42.729368  1844 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.731329  1803 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 34.
W20260709 12:26:42.731983  1844 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:42.732677  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.733253  1844 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:42.739722  1625 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:42.741653  1625 catalog_manager.cc:5697] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 28 to 33, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 33 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.746172  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:42.747812 12373 log.cc:927] Time spent T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:26:42.722685  1799 raft_consensus.cc:2468] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
W20260709 12:26:42.749630  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:42.751006 12337 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:26:42.751183  1625 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 23 to 29. New cstate: current_term: 29 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:42.754164 12337 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.758307 12337 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:42.763295 12370 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.763530 12324 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:42.764097 12370 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.764911 12361 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:42.765952  1770 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.767170 12370 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:42.767984 12361 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:42.772475  1770 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:42.775676  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:42.775964  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 34 candidate_status { last_received { term: 33 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:42.778126  1876 raft_consensus.cc:3060] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:26:42.778492  1844 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:26:42.778752  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:42.779966  1845 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
W20260709 12:26:42.779228  1844 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:26:42.783756  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:42.784520  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:42.784550  1803 raft_consensus.cc:3060] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:26:42.785266  1799 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 32.
I20260709 12:26:42.788295  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:42.789870 12370 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:26:42.781932 12361 raft_consensus.cc:2804] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:42.791712 12370 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 075704438992440697732181e49ccd09)
I20260709 12:26:42.794214 12370 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:42.797289 12361 raft_consensus.cc:697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:42.814556  1803 raft_consensus.cc:2468] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 30.
W20260709 12:26:42.800918  1770 connection.cc:769] client connection to 127.0.140.65:35067 send error: Network error: sendmsg error: Broken pipe (error 32)
W20260709 12:26:42.871435  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52480.3us
W20260709 12:26:42.872287  1770 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: sendmsg error: Broken pipe (error 32)
I20260709 12:26:42.872735  1770 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:42.873862  1770 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: sendmsg error: Broken pipe (error 32)
I20260709 12:26:42.874293  1770 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:42.874467 12376 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:42.876562  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 57510.2us
I20260709 12:26:42.878348 12337 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.879876 12361 consensus_queue.cc:237] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.886019 12370 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.887707 12337 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:42.878023  1845 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:42.885216  1876 raft_consensus.cc:2468] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 34.
I20260709 12:26:42.895169 12324 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:42.943888  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 65969.6us
I20260709 12:26:42.944438 12370 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 election: Requested vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:42.945218 12370 raft_consensus.cc:2804] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:26:42.945326  1771 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:42.945808 12370 raft_consensus.cc:697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 30 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:42.946355 12324 raft_consensus.cc:2804] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:26:42.946947 12324 raft_consensus.cc:697] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:42.948042 12324 consensus_queue.cc:237] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 33.16, Last appended by leader: 16, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:42.951638  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 35 candidate_status { last_received { term: 34 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:42.952317  1799 raft_consensus.cc:3060] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:42.954474  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 33 candidate_status { last_received { term: 32 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:42.955121  1804 raft_consensus.cc:3060] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:26:42.961539  1799 raft_consensus.cc:2468] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 35.
I20260709 12:26:42.964397  2410 raft_consensus.cc:1275] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
W20260709 12:26:42.967767  1770 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:42.971377  1845 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:42.942126  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:42.976130  1844 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:42.977416  1804 raft_consensus.cc:2468] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 33.
I20260709 12:26:42.978318 12351 consensus_queue.cc:1048] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:26:42.978927 12378 raft_consensus.cc:2804] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:26:42.979794 12378 raft_consensus.cc:697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 35 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:42.980852 12378 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 34.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:42.998435  1844 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:42.986935 12370 consensus_queue.cc:237] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:42.999497  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:42.986909  1770 consensus_peers.cc:597] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:43.009940  1845 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09
I20260709 12:26:43.012995 12361 raft_consensus.cc:2804] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:26:43.013571 12361 raft_consensus.cc:697] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 33 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:43.014521 12361 consensus_queue.cc:237] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:26:43.006882  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:43.024865  2410 raft_consensus.cc:1275] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
W20260709 12:26:43.031725  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:43.035751 12326 consensus_queue.cc:1048] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.013s
W20260709 12:26:43.033195  1770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111) [suppressed 387 similar messages]
W20260709 12:26:43.105543  1770 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:43.108011  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:43.152391  1844 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:43.157090  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:43.182723  1799 raft_consensus.cc:1275] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:26:43.185204 12378 consensus_queue.cc:1048] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:26:43.193464  1770 consensus_peers.cc:597] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:43.198107  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:43.345150  1625 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 179) took 9632 ms. Trace:
W20260709 12:26:43.367398  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:43.376292  1624 catalog_manager.cc:5697] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 25 to 27. New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:26:43.386574  1844 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:43.345702  1625 rpcz_store.cc:276] 0709 12:26:33.712681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:38.310134 (+4597453us) service_pool.cc:224] Handling call
0709 12:26:43.345044 (+5034910us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:42.907988 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:42.919464 (+ 11476us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:42.919559 (+    95us) write_op.cc:435] Acquired schema lock
0709 12:26:42.919611 (+    52us) tablet.cc:662] Decoding operations
0709 12:26:42.922352 (+  2741us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:42.922538 (+   186us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:42.922584 (+    46us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:26:42.972695 (+ 50111us) tablet.cc:701] Row locks acquired
0709 12:26:42.972751 (+    56us) write_op.cc:260] PREPARE: finished
0709 12:26:42.972997 (+   246us) write_op.cc:270] Start()
0709 12:26:42.973210 (+   213us) write_op.cc:276] Timestamp: P: 1783600002972906 usec, L: 0
0709 12:26:42.973262 (+    52us) op_driver.cc:348] REPLICATION: starting
0709 12:26:43.034671 (+ 61409us) log.cc:844] Serialized 7374 byte log entry
0709 12:26:43.166184 (+131513us) op_driver.cc:464] REPLICATION: finished
0709 12:26:43.209276 (+ 43092us) write_op.cc:301] APPLY: starting
0709 12:26:43.209418 (+   142us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:43.246970 (+ 37552us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:43.247107 (+   137us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:43.261837 (+ 14730us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:43.262261 (+   424us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=0,mrs_lookups=0
0709 12:26:43.262361 (+   100us) write_op.cc:312] APPLY: finished
0709 12:26:43.279693 (+ 17332us) log.cc:844] Serialized 172 byte log entry
0709 12:26:43.279972 (+   279us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:43.281255 (+  1283us) write_op.cc:454] Released schema lock
0709 12:26:43.281909 (+   654us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:43.309050 (+ 27141us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:43.309087 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":490,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":42748,"apply.run_cpu_time_us":27528,"apply.run_wall_time_us":99741,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83623,"num_ops":19,"prepare.queue_time_us":95115,"prepare.run_cpu_time_us":23169,"prepare.run_wall_time_us":184893,"raft.queue_time_us":44042,"raft.run_cpu_time_us":5719,"raft.run_wall_time_us":32687,"replication_time_us":192682,"thread_start_us":187482,"threads_started":4,"wal-append.queue_time_us":19930}]]}
W20260709 12:26:43.403868  1844 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:43.379907  1624 catalog_manager.cc:5697] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 27 to 28, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 28 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:43.418093  1804 raft_consensus.cc:1275] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:26:43.425793 12344 consensus_queue.cc:1048] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:26:43.468525  1624 catalog_manager.cc:5697] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 26 to 29, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 29 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:43.473652  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:43.477710  1624 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 31 to 32, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 32 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:43.481556  1624 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 24 to 26. New cstate: current_term: 26 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:43.488884  1624 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 19 to 27, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 27 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:43.494154  1624 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 21 to 24, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 24 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:43.506474  1799 raft_consensus.cc:1275] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:26:43.509584 12378 consensus_queue.cc:1048] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:26:43.517817  1844 consensus_peers.cc:597] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:43.519785  1799 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:26:43.526484 12344 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:26:43.535049  1872 raft_consensus.cc:1275] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 33 index: 16. Preceding OpId from leader: term: 34 index: 17. (index mismatch)
I20260709 12:26:43.536535 12326 consensus_queue.cc:1048] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:26:43.555214  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:43.557991 12361 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:43.558990 12361 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:43.562139 12361 leader_election.cc:290] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:43.564069  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:43.565814  1770 consensus_peers.cc:597] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:43.570060  1804 raft_consensus.cc:1275] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
W20260709 12:26:43.573674  1770 consensus_peers.cc:597] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:43.576320 12378 consensus_queue.cc:1048] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:26:43.577668  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:43.581653  1844 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:43.582875  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:43.588613  1846 leader_election.cc:304] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:43.594112 12300 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:43.603412  1844 leader_election.cc:336] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:43.640929  1625 catalog_manager.cc:5697] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 4 to 17, VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 25 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:43.730639  1625 catalog_manager.cc:5697] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 27 to 30. New cstate: current_term: 30 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:43.756853  1625 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 24 to 27, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:43.758291  1625 catalog_manager.cc:5697] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 29 to 30, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 30 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:43.759850  1625 catalog_manager.cc:5697] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 5 to 20, term changed from 26 to 31, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66), VOTER a20e99e125b84513a44e88860667ed61 (127.0.140.67) evicted. New cstate: current_term: 31 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:43.761585  1625 catalog_manager.cc:5697] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:43.828611  1625 catalog_manager.cc:5697] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 21 to 25, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 25 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:43.830359  1625 catalog_manager.cc:5697] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 23 to 25, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 25 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:43.832219  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:43.832157  1625 catalog_manager.cc:5697] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 6 to 18, term changed from 22 to 29, VOTER 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) evicted. New cstate: current_term: 29 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:43.840020  1625 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 31 to 33, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 33 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:43.842643  1625 catalog_manager.cc:5697] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 26 to 34. New cstate: current_term: 34 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:43.844461  1625 catalog_manager.cc:5697] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 28 to 29, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 29 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:43.855914  1625 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 29 to 30, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 30 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:43.861606  1624 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 277) took 15102 ms (client timeout 15000 ms). Trace:
W20260709 12:26:43.874125  1624 rpcz_store.cc:269] 0709 12:26:28.758769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:32.220911 (+3462142us) service_pool.cc:224] Handling call
0709 12:26:43.861489 (+11640578us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:43.519291 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:43.519896 (+   605us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:43.519934 (+    38us) write_op.cc:435] Acquired schema lock
0709 12:26:43.519952 (+    18us) tablet.cc:662] Decoding operations
0709 12:26:43.520812 (+   860us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:43.520874 (+    62us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:43.520914 (+    40us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:26:43.521534 (+   620us) tablet.cc:701] Row locks acquired
0709 12:26:43.521550 (+    16us) write_op.cc:260] PREPARE: finished
0709 12:26:43.521720 (+   170us) write_op.cc:270] Start()
0709 12:26:43.521826 (+   106us) write_op.cc:276] Timestamp: P: 1783600003521664 usec, L: 0
0709 12:26:43.521846 (+    20us) op_driver.cc:348] REPLICATION: starting
0709 12:26:43.522595 (+   749us) log.cc:844] Serialized 2839 byte log entry
0709 12:26:43.527458 (+  4863us) op_driver.cc:464] REPLICATION: finished
0709 12:26:43.527975 (+   517us) write_op.cc:301] APPLY: starting
0709 12:26:43.528062 (+    87us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:43.554072 (+ 26010us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:43.554189 (+   117us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:43.609662 (+ 55473us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:43.610934 (+  1272us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 12:26:43.610987 (+    53us) write_op.cc:312] APPLY: finished
0709 12:26:43.611279 (+   292us) log.cc:844] Serialized 73 byte log entry
0709 12:26:43.612239 (+   960us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:43.612436 (+   197us) write_op.cc:454] Released schema lock
0709 12:26:43.612515 (+    79us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:43.691119 (+ 78604us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:43.691151 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":33649,"reported_tablets":50,"spinlock_wait_cycles":3456,"child_traces":[["op",{"apply.queue_time_us":291,"apply.run_cpu_time_us":10752,"apply.run_wall_time_us":142535,"cfile_cache_hit":28,"cfile_cache_hit_bytes":29288,"num_ops":7,"prepare.queue_time_us":2666,"prepare.run_cpu_time_us":3630,"prepare.run_wall_time_us":13023,"replication_time_us":5459}]]}
W20260709 12:26:43.883343  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:43.904034   561 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:44.000897  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:44.034754 12370 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:44.035667 12370 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:44.040001  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:44.047816  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d891ed30ca204d09b7b083088037b63d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:44.049765  1846 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:44.050870 12300 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:44.051743  1844 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:44.066638  1630 catalog_manager.cc:5697] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 23 to 25, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 25 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:44.077054  1630 catalog_manager.cc:5697] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 21 to 22, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 22 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:44.077874  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:44.079246  1630 catalog_manager.cc:5697] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 25 to 26, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 26 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:44.081797  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:44.049710 12370 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:44.080940  1630 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 27 to 28, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 075704438992440697732181e49ccd09 (127.0.140.65). New cstate: current_term: 28 leader_uuid: "075704438992440697732181e49ccd09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:44.092972  1630 catalog_manager.cc:5697] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 22 to 23, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 23 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:44.103449  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:44.116601  1796 tablet_service.cc:1558] Processing DeleteTablet for tablet ea5d8f652fc448d6af846ff6850004b8 with delete_type TABLET_DATA_TOMBSTONED (TS 6084fbc6918e4e0b95a9bd0206a620d8 not found in new config with opid_index 18) from {username='slave'} at 127.0.0.1:49336
W20260709 12:26:44.121044  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:44.143211  1630 catalog_manager.cc:5697] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 23 to 30. New cstate: current_term: 30 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:44.145812  1630 catalog_manager.cc:5697] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 20 to 26, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 26 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:44.156736 12406 tablet_replica.cc:333] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:26:44.177898 12406 raft_consensus.cc:2243] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:44.179435 12406 raft_consensus.cc:2272] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:44.196743  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet def0036446324543ba07e132e885133a with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:41756
I20260709 12:26:44.198407  1864 tablet_service.cc:1558] Processing DeleteTablet for tablet d891ed30ca204d09b7b083088037b63d with delete_type TABLET_DATA_TOMBSTONED (TS a20e99e125b84513a44e88860667ed61 not found in new config with opid_index 20) from {username='slave'} at 127.0.0.1:41756
I20260709 12:26:44.198939 12406 ts_tablet_manager.cc:1916] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:26:44.235492  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:44.258775 12407 tablet_replica.cc:333] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:26:44.260401 12407 raft_consensus.cc:2243] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:44.260506  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.262019 12407 raft_consensus.cc:2272] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:44.264708  1624 catalog_manager.cc:5697] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 15 to 28, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 28 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.302907 12407 ts_tablet_manager.cc:1916] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:26:44.311136  1624 catalog_manager.cc:5697] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 25 to 29, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 29 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.314232  1624 catalog_manager.cc:5697] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 27 to 32, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 32 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:44.310118  1630 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52636 (request call id 258) took 11548 ms (client timeout 15000 ms). Trace:
W20260709 12:26:44.318176  1630 rpcz_store.cc:269] 0709 12:26:32.761827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:35.797488 (+3035661us) service_pool.cc:224] Handling call
0709 12:26:35.845426 (+ 47938us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b5400156654. stack: 00007f16a8cb9343 00007f16a8cb9240 00007f16a86481d4 00007f16b2386304 00007f16b2386299 00007f16abd42090 00007f16abde0f3a 00007f16b1b96e9c 00007f16b1c9504e 00007f16b01562a4 00007f16b0156232 00007f16b0156161 00007f16b01560dc 00007f16b0155392 00007f16aa11348c 00007f16aa1128c6
0709 12:26:44.310041 (+8464615us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:26:44.163086 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:26:44.163679 (+   593us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:26:44.163765 (+    86us) write_op.cc:435] Acquired schema lock
0709 12:26:44.163781 (+    16us) tablet.cc:662] Decoding operations
0709 12:26:44.164803 (+  1022us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:26:44.164858 (+    55us) write_op.cc:641] Partition lock acquired for write op
0709 12:26:44.164890 (+    32us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:26:44.165595 (+   705us) tablet.cc:701] Row locks acquired
0709 12:26:44.165615 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:26:44.165802 (+   187us) write_op.cc:270] Start()
0709 12:26:44.165917 (+   115us) write_op.cc:276] Timestamp: P: 1783600004165746 usec, L: 0
0709 12:26:44.165941 (+    24us) op_driver.cc:348] REPLICATION: starting
0709 12:26:44.166702 (+   761us) log.cc:844] Serialized 2849 byte log entry
0709 12:26:44.169887 (+  3185us) op_driver.cc:464] REPLICATION: finished
0709 12:26:44.170446 (+   559us) write_op.cc:301] APPLY: starting
0709 12:26:44.170547 (+   101us) tablet.cc:1367] starting BulkCheckPresence
0709 12:26:44.175171 (+  4624us) tablet.cc:1370] finished BulkCheckPresence
0709 12:26:44.175692 (+   521us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:26:44.180557 (+  4865us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:26:44.180810 (+   253us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=14,key_file_lookups=14,delta_file_lookups=0,mrs_lookups=0
0709 12:26:44.180863 (+    53us) write_op.cc:312] APPLY: finished
0709 12:26:44.181231 (+   368us) log.cc:844] Serialized 73 byte log entry
0709 12:26:44.182168 (+   937us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:44.182308 (+   140us) write_op.cc:454] Released schema lock
0709 12:26:44.182374 (+    66us) write_op.cc:341] FINISH: Updating metrics
0709 12:26:44.182850 (+   476us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:26:44.182886 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3370,"reported_tablets":49,"spinlock_wait_cycles":26351872,"child_traces":[["op",{"apply.queue_time_us":321,"apply.run_cpu_time_us":10068,"apply.run_wall_time_us":12333,"cfile_cache_hit":28,"cfile_cache_hit_bytes":29288,"num_ops":7,"prepare.queue_time_us":3658,"prepare.run_cpu_time_us":3936,"prepare.run_wall_time_us":3950,"replication_time_us":3623}]]}
I20260709 12:26:44.317147  1624 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 29 to 32, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 32 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.330657  1624 catalog_manager.cc:5697] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:44.347450  1624 catalog_manager.cc:5697] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 25 to 30, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 30 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.349556  1624 catalog_manager.cc:5697] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 26 to 30, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 30 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:44.357757 12406 ts_tablet_manager.cc:1929] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 29.17
I20260709 12:26:44.358500 12406 log.cc:1199] T ea5d8f652fc448d6af846ff6850004b8 P 6084fbc6918e4e0b95a9bd0206a620d8: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-1-root/wals/ea5d8f652fc448d6af846ff6850004b8
I20260709 12:26:44.354405  1624 catalog_manager.cc:5697] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 23 to 26, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 26 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.362200  1624 catalog_manager.cc:5697] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 20 to 26, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 26 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
W20260709 12:26:44.364328  1844 consensus_peers.cc:597] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.368209  1624 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 25 to 27, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:44.369993  1624 catalog_manager.cc:5697] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 32 to 34. New cstate: current_term: 34 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:44.372154  1624 catalog_manager.cc:5697] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 22 to 30. New cstate: current_term: 30 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.379485  1624 catalog_manager.cc:5697] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 26 to 31, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 31 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } }
I20260709 12:26:44.383574  1624 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 31 to 35, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 35 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.388849  1615 catalog_manager.cc:5028] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): tablet ea5d8f652fc448d6af846ff6850004b8 (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:26:44.395709  1624 catalog_manager.cc:5697] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 19 to 23. New cstate: current_term: 23 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.399067  1624 catalog_manager.cc:5697] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 30 to 33, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 33 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:44.401054  1624 catalog_manager.cc:5697] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 28 to 29, leader changed from 075704438992440697732181e49ccd09 (127.0.140.65) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 29 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } }
W20260709 12:26:44.405287  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.456794  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:44.466701  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:44.476543  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:44.477115 12407 ts_tablet_manager.cc:1929] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 25.16
I20260709 12:26:44.482728 12407 log.cc:1199] T def0036446324543ba07e132e885133a P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/def0036446324543ba07e132e885133a
W20260709 12:26:44.481426  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:44.494360  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet def0036446324543ba07e132e885133a (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:26:44.494448 12407 tablet_replica.cc:333] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:26:44.496031 12407 raft_consensus.cc:2243] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:44.497898 12407 raft_consensus.cc:2272] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:44.525075 12407 ts_tablet_manager.cc:1916] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:26:44.569978  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:44.570585  1863 tablet_service.cc:1558] Processing DeleteTablet for tablet def0036446324543ba07e132e885133a with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 4 (current committed config index is 17)) from {username='slave'} at 127.0.0.1:41756
W20260709 12:26:44.683212  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 85601.4us
W20260709 12:26:44.698479  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 48485.9us
I20260709 12:26:44.714265 12407 ts_tablet_manager.cc:1929] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 31.19
I20260709 12:26:44.715031 12407 log.cc:1199] T d891ed30ca204d09b7b083088037b63d P a20e99e125b84513a44e88860667ed61: Deleting WAL directory at /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599624140507-561-0/minicluster-data/ts-2-root/wals/d891ed30ca204d09b7b083088037b63d
I20260709 12:26:44.717132  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet d891ed30ca204d09b7b083088037b63d (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
I20260709 12:26:44.718540  1615 catalog_manager.cc:5028] TS a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): tablet def0036446324543ba07e132e885133a (table test-workload [id=29763a3d717a49f7ac399e7da9d39ca4]) successfully deleted
W20260709 12:26:44.721383  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 84773.6us
W20260709 12:26:44.726892  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 90330.4us
W20260709 12:26:44.779610  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:44.821787  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:44.822887  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:44.823525  1770 consensus_peers.cc:597] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.845753  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:44.847469  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:44.860562  1770 consensus_peers.cc:597] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:44.886940  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:44.893607  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.904595   561 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:44.987119  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:45.140596  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:45.157684  1770 consensus_peers.cc:597] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:45.411525  1844 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.430742  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:45.435307  1873 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:45.543649  1799 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
W20260709 12:26:45.544406  1846 consensus_peers.cc:597] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9585ba97eaf14e069dcd2dcaf726a09b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:45.547221 12410 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:26:45.549316  1844 consensus_peers.cc:597] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:45.557734  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:45.569042  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:45.580116 12410 raft_consensus.cc:2955] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 32 LEADER]: Committing config change with OpId 32.19: config changed from index -1 to 19, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:26:45.586575  1799 raft_consensus.cc:2955] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Committing config change with OpId 32.19: config changed from index -1 to 19, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:26:45.604570  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:45.621208  1630 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 19, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 32 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:26:45.641911  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:45.633549  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:45.673575  1873 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:26:45.696928  1844 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.720264  1770 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:45.726606  1770 consensus_peers.cc:597] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.735581  1844 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.753669  1804 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:26:45.756088 12411 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:26:45.758876  1846 consensus_peers.cc:597] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5f3d785e05841fbb0efa210c3d9675c. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:45.775054  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:45.777283  1844 consensus_peers.cc:597] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:45.796015  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:45.810631 12410 raft_consensus.cc:2955] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 35 LEADER]: Committing config change with OpId 35.21: config changed from index -1 to 21, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:26:45.818234  1799 raft_consensus.cc:2955] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Committing config change with OpId 35.21: config changed from index -1 to 21, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:26:45.854570  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:45.905984   561 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 1753)
I20260709 12:26:45.916692  1630 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index -1 to 21, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 35 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
W20260709 12:26:45.957716  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:45.984611  1770 consensus_peers.cc:597] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:45.987577 12426 ts_tablet_manager.cc:933] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:45.988967  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:46.000507  1844 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:46.011773  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:46.016664  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:46.035362  1844 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:46.010049  1804 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:26:46.063055  1772 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 908089442ca44109875257bfbc9d06c2. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:46.064625  1874 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:26:46.067211 12332 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:46.071076 12426 tablet_copy_client.cc:323] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
W20260709 12:26:46.073439  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:46.076045  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 9585ba97eaf14e069dcd2dcaf726a09b from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:26:46.076680  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 9585ba97eaf14e069dcd2dcaf726a09b from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-9585ba97eaf14e069dcd2dcaf726a09b
I20260709 12:26:46.127357  1890 tablet_copy_source_session.cc:215] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:26:46.133512 12429 raft_consensus.cc:2955] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Committing config change with OpId 27.17: config changed from index -1 to 17, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:26:46.140578 12426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9585ba97eaf14e069dcd2dcaf726a09b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:46.143033  1876 raft_consensus.cc:2955] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Committing config change with OpId 27.17: config changed from index -1 to 17, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:26:46.189157  1630 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index -1 to 17, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:26:46.197333 12426 tablet_copy_client.cc:806] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:26:46.198470 12426 tablet_copy_client.cc:670] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
W20260709 12:26:46.173666  1770 consensus_peers.cc:597] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:46.205276 12426 tablet_copy_client.cc:538] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 12:26:46.225176  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:46.241684 12432 ts_tablet_manager.cc:933] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:46.246997 12426 tablet_bootstrap.cc:492] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:26:46.267504  1874 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:46.277415 12432 tablet_copy_client.cc:323] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:26:46.279680  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet c5f3d785e05841fbb0efa210c3d9675c from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:26:46.280296  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet c5f3d785e05841fbb0efa210c3d9675c from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-c5f3d785e05841fbb0efa210c3d9675c
I20260709 12:26:46.289131  1890 tablet_copy_source_session.cc:215] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:26:46.304806  1799 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:26:46.312992 12417 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:26:46.307737  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:46.308035  1846 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665): Couldn't send request to peer 123095127f334445889c21a42ba1d41e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12cfc1bb936b41fbbc9cdfafc0cd9470. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:46.314884 12432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5f3d785e05841fbb0efa210c3d9675c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:46.361008 12417 raft_consensus.cc:2955] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Committing config change with OpId 26.16: config changed from index -1 to 16, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:26:46.362316  1804 raft_consensus.cc:2955] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Committing config change with OpId 26.16: config changed from index -1 to 16, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } } }
I20260709 12:26:46.383729 12432 tablet_copy_client.cc:806] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:26:46.394825 12432 tablet_copy_client.cc:670] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:26:46.395915  1624 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index -1 to 16, NON_VOTER 123095127f334445889c21a42ba1d41e (127.0.140.68) added. New cstate: current_term: 26 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:46.402242 12432 tablet_copy_client.cc:538] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:26:46.433817 12432 tablet_bootstrap.cc:492] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
W20260709 12:26:46.439426  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:46.459343  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:46.531107 12426 tablet_bootstrap.cc:492] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.532279 12426 tablet_bootstrap.cc:492] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:26:46.533150 12426 ts_tablet_manager.cc:1403] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.290s	user 0.052s	sys 0.011s
I20260709 12:26:46.549564 12426 raft_consensus.cc:359] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
W20260709 12:26:46.554250  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:46.550545 12426 raft_consensus.cc:740] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
W20260709 12:26:46.557503  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:46.555799 12426 consensus_queue.cc:260] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:46.566480 12426 ts_tablet_manager.cc:1434] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.021s	user 0.002s	sys 0.005s
I20260709 12:26:46.576725  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-9585ba97eaf14e069dcd2dcaf726a09b received from {username='slave'} at 127.0.0.1:58100
I20260709 12:26:46.577309  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-9585ba97eaf14e069dcd2dcaf726a09b on tablet 9585ba97eaf14e069dcd2dcaf726a09b with peer 123095127f334445889c21a42ba1d41e
W20260709 12:26:46.592283  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:46.632555 12426 ts_tablet_manager.cc:933] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:46.634325  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:46.646306  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:46.655859 12432 tablet_bootstrap.cc:492] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=2 applied=21 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.657348 12432 tablet_bootstrap.cc:492] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:26:46.658262 12432 ts_tablet_manager.cc:1403] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.225s	user 0.052s	sys 0.031s
I20260709 12:26:46.661760 12432 raft_consensus.cc:359] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:46.663686 12432 raft_consensus.cc:740] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:26:46.664604 12432 consensus_queue.cc:260] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:46.672924 12426 tablet_copy_client.cc:323] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.66:40361
I20260709 12:26:46.675205  1816 tablet_copy_service.cc:140] P 6084fbc6918e4e0b95a9bd0206a620d8: Received BeginTabletCopySession request for tablet 908089442ca44109875257bfbc9d06c2 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:42636)
I20260709 12:26:46.675895  1816 tablet_copy_service.cc:161] P 6084fbc6918e4e0b95a9bd0206a620d8: Beginning new tablet copy session on tablet 908089442ca44109875257bfbc9d06c2 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:42636: session id = 123095127f334445889c21a42ba1d41e-908089442ca44109875257bfbc9d06c2
I20260709 12:26:46.678104 12432 ts_tablet_manager.cc:1434] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.001s
I20260709 12:26:46.691941  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-c5f3d785e05841fbb0efa210c3d9675c received from {username='slave'} at 127.0.0.1:58100
I20260709 12:26:46.692561  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-c5f3d785e05841fbb0efa210c3d9675c on tablet c5f3d785e05841fbb0efa210c3d9675c with peer 123095127f334445889c21a42ba1d41e
I20260709 12:26:46.695650  1816 tablet_copy_source_session.cc:215] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:26:46.702687 12426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 908089442ca44109875257bfbc9d06c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:46.720304  2413 raft_consensus.cc:1217] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 LEARNER]: Deduplicated request from leader. Original: 35.20->[35.21-35.21]   Dedup: 35.21->[]
I20260709 12:26:46.792587 12426 tablet_copy_client.cc:806] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:26:46.794059 12426 tablet_copy_client.cc:670] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:26:46.828492 12426 tablet_copy_client.cc:538] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 12:26:46.860646  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:46.866463 12426 tablet_bootstrap.cc:492] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:26:46.901465  2413 raft_consensus.cc:1217] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 LEARNER]: Deduplicated request from leader. Original: 32.18->[32.19-32.19]   Dedup: 32.19->[]
W20260709 12:26:46.906559   561 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 1753)
I20260709 12:26:46.921126 12432 ts_tablet_manager.cc:933] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: Initiating tablet copy from peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:46.928064  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:46.937296 12432 tablet_copy_client.cc:323] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.140.67:35761
I20260709 12:26:46.939672  1890 tablet_copy_service.cc:140] P a20e99e125b84513a44e88860667ed61: Received BeginTabletCopySession request for tablet 12cfc1bb936b41fbbc9cdfafc0cd9470 from peer 123095127f334445889c21a42ba1d41e ({username='slave'} at 127.0.0.1:58100)
I20260709 12:26:46.940382  1890 tablet_copy_service.cc:161] P a20e99e125b84513a44e88860667ed61: Beginning new tablet copy session on tablet 12cfc1bb936b41fbbc9cdfafc0cd9470 from peer 123095127f334445889c21a42ba1d41e at {username='slave'} at 127.0.0.1:58100: session id = 123095127f334445889c21a42ba1d41e-12cfc1bb936b41fbbc9cdfafc0cd9470
W20260709 12:26:46.945241  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:46.986907  1890 tablet_copy_source_session.cc:215] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:26:46.991745 12432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12cfc1bb936b41fbbc9cdfafc0cd9470. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:47.010846  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:47.046707  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:47.058606 12432 tablet_copy_client.cc:806] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 0 data blocks...
I20260709 12:26:47.059484 12432 tablet_copy_client.cc:670] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: tablet copy: Starting download of 1 WAL segments...
I20260709 12:26:47.064311 12432 tablet_copy_client.cc:538] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:26:47.071568 12426 tablet_bootstrap.cc:492] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=1 applied=17 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:47.073212 12426 tablet_bootstrap.cc:492] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:26:47.074141 12426 ts_tablet_manager.cc:1403] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.208s	user 0.061s	sys 0.020s
I20260709 12:26:47.078547 12426 raft_consensus.cc:359] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:47.079591 12426 raft_consensus.cc:740] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:26:47.080293 12426 consensus_queue.cc:260] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:47.082809 12426 ts_tablet_manager.cc:1434] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.003s
I20260709 12:26:47.085278  1816 tablet_copy_service.cc:342] P 6084fbc6918e4e0b95a9bd0206a620d8: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-908089442ca44109875257bfbc9d06c2 received from {username='slave'} at 127.0.0.1:42636
I20260709 12:26:47.085821  1816 tablet_copy_service.cc:434] P 6084fbc6918e4e0b95a9bd0206a620d8: ending tablet copy session 123095127f334445889c21a42ba1d41e-908089442ca44109875257bfbc9d06c2 on tablet 908089442ca44109875257bfbc9d06c2 with peer 123095127f334445889c21a42ba1d41e
I20260709 12:26:47.096830 12432 tablet_bootstrap.cc:492] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: Bootstrap starting.
I20260709 12:26:47.202085  2413 raft_consensus.cc:1217] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 LEARNER]: Deduplicated request from leader. Original: 27.16->[27.17-27.17]   Dedup: 27.17->[]
W20260709 12:26:47.214553  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:47.222056  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:47.241235 12432 tablet_bootstrap.cc:492] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: Bootstrap replayed 1/1 log segments. Stats: ops{read=17 overwritten=1 applied=16 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:47.242509 12432 tablet_bootstrap.cc:492] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: Bootstrap complete.
I20260709 12:26:47.243449 12432 ts_tablet_manager.cc:1403] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: Time spent bootstrapping tablet: real 0.147s	user 0.046s	sys 0.008s
I20260709 12:26:47.247325 12432 raft_consensus.cc:359] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:47.248492 12432 raft_consensus.cc:740] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 LEARNER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Initialized, Role: LEARNER
I20260709 12:26:47.249308 12432 consensus_queue.cc:260] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:47.264655 12432 ts_tablet_manager.cc:1434] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.001s
W20260709 12:26:47.292616  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 136506us
I20260709 12:26:47.312274 12417 raft_consensus.cc:1064] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
W20260709 12:26:47.317260  1770 consensus_peers.cc:597] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:47.323701  1890 tablet_copy_service.cc:342] P a20e99e125b84513a44e88860667ed61: Request end of tablet copy session 123095127f334445889c21a42ba1d41e-12cfc1bb936b41fbbc9cdfafc0cd9470 received from {username='slave'} at 127.0.0.1:58100
I20260709 12:26:47.324247  1890 tablet_copy_service.cc:434] P a20e99e125b84513a44e88860667ed61: ending tablet copy session 123095127f334445889c21a42ba1d41e-12cfc1bb936b41fbbc9cdfafc0cd9470 on tablet 12cfc1bb936b41fbbc9cdfafc0cd9470 with peer 123095127f334445889c21a42ba1d41e
W20260709 12:26:47.332865  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:47.339519 12410 raft_consensus.cc:1064] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:26:47.341761 12417 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 17, Current term: 32, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:47.342716  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:47.351570  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:47.353626 12410 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 19, Current term: 35, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:47.436935  1799 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:26:47.449156 12424 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:26:47.463685  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:47.484207  2413 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:26:47.485477  2410 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:26:47.511453 12448 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:26:47.514829  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:47.515669 12424 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:26:47.544016  2413 raft_consensus.cc:1217] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 LEARNER]: Deduplicated request from leader. Original: 26.15->[26.16-26.16]   Dedup: 26.16->[]
I20260709 12:26:47.723838  1803 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:26:48.322117 12456 raft_consensus.cc:493] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.322950 12456 raft_consensus.cc:515] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.325958 12456 leader_election.cc:290] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.577199 12456 raft_consensus.cc:493] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.577947 12456 raft_consensus.cc:515] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.580361 12456 leader_election.cc:290] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.660080 12456 raft_consensus.cc:493] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.661057 12456 raft_consensus.cc:515] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.664057 12456 leader_election.cc:290] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.765652 12456 raft_consensus.cc:493] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.766487 12456 raft_consensus.cc:515] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.769872 12456 leader_election.cc:290] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:48.766839   561 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 1753)
I20260709 12:26:48.808396 12457 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
W20260709 12:26:48.814422  2382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111) [suppressed 455 similar messages]
I20260709 12:26:48.818110 12456 raft_consensus.cc:493] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:48.818331 12332 raft_consensus.cc:493] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:48.819631 12332 raft_consensus.cc:515] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:48.821866 12332 leader_election.cc:290] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:48.822460 12456 raft_consensus.cc:515] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:48.809476 12457 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:48.838572 12457 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 075704438992440697732181e49ccd09 (127.0.140.65:35067)
I20260709 12:26:48.857812 12459 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.858260 12332 raft_consensus.cc:493] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:48.859230 12332 raft_consensus.cc:515] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:48.861442 12332 leader_election.cc:290] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:48.862645 12456 leader_election.cc:290] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:48.870862 12456 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.871877 12456 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:48.858402 12459 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:48.882743 12458 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:48.900758 12458 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:48.903694 12458 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.908344 12464 raft_consensus.cc:493] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.908969 12464 raft_consensus.cc:515] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.911100 12464 leader_election.cc:290] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.916139 12459 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:48.908358 12456 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.935254  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:48.945258 12458 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:48.945740 12469 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:48.949633 12456 raft_consensus.cc:493] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.951280 12456 raft_consensus.cc:515] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.946188 12458 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:48.962244 12332 raft_consensus.cc:493] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.963953 12471 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:48.964855 12332 raft_consensus.cc:515] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.968104  2414 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f285ce2457a47ebbd05920b42b35fba" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:48.968268  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d3405528d44edb8c0be3b296ef72be" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:48.968873  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:48.969601  2410 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a2d4ac0ec9e4adcb3adb743647e0e78" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:48.969668  2411 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 30.
I20260709 12:26:48.964531 12471 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.972338 12471 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:48.973906 12332 leader_election.cc:290] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.974928 12431 raft_consensus.cc:493] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.975620 12431 raft_consensus.cc:515] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.978679 12431 leader_election.cc:290] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.980947 12457 raft_consensus.cc:493] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.981594 12457 raft_consensus.cc:515] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.982342 12431 raft_consensus.cc:493] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.983229 12431 raft_consensus.cc:515] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.984117 12457 leader_election.cc:290] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.984709 12457 raft_consensus.cc:493] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.985160 12457 raft_consensus.cc:515] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.986500 12431 leader_election.cc:290] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.987607 12332 raft_consensus.cc:493] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:48.988306 12471 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:48.988859 12457 leader_election.cc:290] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.991775  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:48.992806 12471 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:48.997028 12471 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:48.998099 12471 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:48.999130 12471 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.003880 12471 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:48.993546 12457 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:49.031296 12457 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting leader election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:49.000758 12332 raft_consensus.cc:515] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:49.005249 12475 raft_consensus.cc:493] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.032585 12475 raft_consensus.cc:515] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.034310 12332 leader_election.cc:290] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:49.031934 12457 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:49.035760 12475 leader_election.cc:290] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.037751 12475 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:26:49.039374  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 90764.1us
I20260709 12:26:49.038260 12475 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:49.041872 12475 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.042832 12467 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.043573 12467 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.046423 12467 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:49.046869 12466 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:48.960580 12456 leader_election.cc:290] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.959152 12468 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.047519 12466 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.048260 12458 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:48.946556 12469 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.057420 12476 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:49.069404 12442 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:49.089326 12442 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.048414 12468 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.097175  1624 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:46106 (request call id 195) took 1519 ms. Trace:
I20260709 12:26:49.102588 12457 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.070348 12476 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.068527 12445 raft_consensus.cc:493] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.106945 12445 raft_consensus.cc:515] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.108144 12476 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:49.109150 12445 leader_election.cc:290] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:49.101173  1770 consensus_peers.cc:597] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:49.102809  1624 rpcz_store.cc:276] 0709 12:26:47.577829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:48.370093 (+792264us) service_pool.cc:224] Handling call
0709 12:26:49.097102 (+727009us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":273,"reported_tablets":1}
I20260709 12:26:49.111794 12429 raft_consensus.cc:1064] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:26:49.115459 12469 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.067607 12448 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.100698 12466 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.065009 12472 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.116526 12448 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:49.116976 12472 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.119331 12472 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:49.103811 12468 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.120771 12442 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:49.121718 12471 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.122954  1630 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52644 (request call id 290) took 1580 ms. Trace:
I20260709 12:26:49.123350  1630 rpcz_store.cc:276] 0709 12:26:47.543045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:26:48.914340 (+1371295us) service_pool.cc:224] Handling call
0709 12:26:49.122884 (+208544us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":179771,"reported_tablets":1,"spinlock_wait_cycles":84096}
I20260709 12:26:49.124406 12448 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.124555 12457 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.125313 12448 raft_consensus.cc:493] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.125938 12448 raft_consensus.cc:515] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.126660  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:49.127298  2417 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:49.128212 12448 leader_election.cc:290] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:49.098970  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 58925.5us
I20260709 12:26:49.122539 12471 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.114621 12429 consensus_queue.cc:237] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 15, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:49.156217  1770 consensus_peers.cc:597] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:49.157500  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:49.150660 12471 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.158280  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.155457 12469 raft_consensus.cc:493] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.180136 12469 raft_consensus.cc:515] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.182420 12469 leader_election.cc:290] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.159973 12448 raft_consensus.cc:493] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.184126 12448 raft_consensus.cc:515] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.186336 12448 leader_election.cc:290] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.164829  2417 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 31.
I20260709 12:26:49.167555 12473 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.188159  1772 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:49.189163 12457 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:49.188586 12473 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:49.173018 12481 raft_consensus.cc:493] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.174242  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.177410  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84ce9896327e4c9d9888e80e8dbeeb0d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.191267 12481 raft_consensus.cc:515] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.190718 12457 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:49.159157 12445 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.193851 12445 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:49.187522  1770 leader_election.cc:336] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.198333  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22ef2bc934c4b7ea57aad5853ab6122" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.200409  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:49.201519  1770 leader_election.cc:336] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.209553  1770 leader_election.cc:336] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.211897  1770 leader_election.cc:336] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.213593  1770 leader_election.cc:336] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.215328  1770 leader_election.cc:304] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:49.229017 12456 raft_consensus.cc:2749] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:26:49.194752 12481 leader_election.cc:290] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.196589 12473 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:26:49.230955  1828 debug-util.cc:398] Leaking SignalData structure 0x7b080030ce80 after lost signal to thread 564
I20260709 12:26:49.196889 12457 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:49.232312  1828 debug-util.cc:398] Leaking SignalData structure 0x7b08001ff220 after lost signal to thread 1664
W20260709 12:26:49.233808  1828 debug-util.cc:398] Leaking SignalData structure 0x7b080020a120 after lost signal to thread 1831
I20260709 12:26:49.196168 12445 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:49.237586  1828 debug-util.cc:398] Leaking SignalData structure 0x7b08002add40 after lost signal to thread 1905
W20260709 12:26:49.237982  1828 debug-util.cc:398] Leaking SignalData structure 0x7b0800345d40 after lost signal to thread 1906
W20260709 12:26:49.239663  1828 debug-util.cc:398] Leaking SignalData structure 0x7b080009aa20 after lost signal to thread 2443
W20260709 12:26:49.240247  1828 debug-util.cc:398] Leaking SignalData structure 0x7b08000a1680 after lost signal to thread 12412
W20260709 12:26:49.240819  1770 leader_election.cc:336] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.241341  1770 leader_election.cc:304] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:49.242461  1770 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.246132  1770 leader_election.cc:336] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.247005 12473 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.254200 12473 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.255241 12431 raft_consensus.cc:493] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.247440 12481 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.248673 12465 raft_consensus.cc:2749] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:49.249171 12470 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.249759 12477 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.227025 12468 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:49.256418  1770 leader_election.cc:304] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:49.260905 12332 raft_consensus.cc:493] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.262681 12473 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.267277  2417 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:49.255858 12431 raft_consensus.cc:515] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.270726  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.272185 12468 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.274037 12332 raft_consensus.cc:515] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.268916 12477 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:49.276162 12332 leader_election.cc:290] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:49.276736  1771 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:26:49.277853 12477 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.285377 12473 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.265913 12470 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:49.286058 12473 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:49.287789 12332 raft_consensus.cc:2749] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:49.295634 12332 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.263819 12481 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: NON_VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: true } }
I20260709 12:26:49.305110 12480 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.305940 12480 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.269264 12472 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:26:49.308290 12472 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.296226 12332 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.332824 12475 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.002s
I20260709 12:26:49.341492  1872 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.342448  1872 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:49.344283 12483 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.344980 12483 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:49.346994 12469 raft_consensus.cc:493] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.347668 12469 raft_consensus.cc:515] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.349658 12445 raft_consensus.cc:493] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.349797 12448 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.350328 12445 raft_consensus.cc:515] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.350759 12448 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:26:49.329123 12471 raft_consensus.cc:493] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.362291 12471 raft_consensus.cc:515] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.362735  1878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76f6bc3c4cce45d7855e6b3429341da2" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.362941 12431 leader_election.cc:290] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.363524  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b1cb2a27e8444930b93f1e21c2deb19c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.364257  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f15e08b3d9804a00b9836e5223042266" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.375936  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.391991 12332 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.365319  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbbcc16cebc244769e7f1a1b6e29594c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.405448  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd91edf20ef54a30a2f5af0c0b682350" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:49.405973  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 116819us
I20260709 12:26:49.375589  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.410490  1875 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
I20260709 12:26:49.391090 12480 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.435833 12445 leader_election.cc:290] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.435954  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.406528 12468 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:49.455145 12473 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.455480 12470 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.365829 12477 raft_consensus.cc:2955] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 35 LEADER]: Committing config change with OpId 35.22: config changed from index 21 to 22, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:49.436866  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "513580259d0a4e6d96fb16084d5a90dc" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.461457 12481 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.436872 12483 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.462978 12448 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:26:49.465819 12472 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:49.438966  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36f1f37b7d544bb29cd9cace84f6e946" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.442286  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "65799458c3fb44e7a162099338c0bb62" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.443634  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82eacf0c2a8c4d8db4052327304a8793" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.446240  2411 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 LEARNER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:26:49.448171  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "def0036446324543ba07e132e885133a" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.449219 12469 leader_election.cc:290] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.449887  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.446307 12471 leader_election.cc:290] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.472918  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9585ba97eaf14e069dcd2dcaf726a09b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.474654  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "12cfc1bb936b41fbbc9cdfafc0cd9470" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.476312  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae12891834034d968900107cc58fa821" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.481035  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.483165  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3034677490f4f0fac2a4ed3e2a8142b" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.483506  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.490358  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.492971  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "80ed6fcb806a4023b94ecd7b266b421e" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 35 candidate_status { last_received { term: 34 index: 24 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.494284  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.496551  1875 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.499480 12490 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:49.500861 12442 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:49.501369 12442 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.501791 12442 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:49.502285 12332 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.505144  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efbb8c187a4e410aa09e81ceb9ebecc3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 35 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.507076  1801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82c4d9259f1a44119e802a40207d64b5" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.507862  1805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "36bc2f66a73947468d61113c226ed318" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.508384  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.506853 12332 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.509016  1806 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 33.
I20260709 12:26:49.510456 12332 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.516341  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.519959  1630 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 30 to 31, leader changed from a20e99e125b84513a44e88860667ed61 (127.0.140.67) to 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66). New cstate: current_term: 31 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:49.524899  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.525947 12467 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:26:49.526355 12467 raft_consensus.cc:493] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:26:49.526736 12467 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:26:49.478628  1770 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 232512us
I20260709 12:26:49.528851  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4c45f93428e4cc68f85f17ea3ec3ad0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.529453  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6388b1a5243d463589750794e8ac5511" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.530189  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.531235  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b797b0728dc4e01976ab61ef164fa90" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.531518  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "23caea311a774ba3b29b7749bd3f1cd6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.534722  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.538605  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "aca584e2f1114809bcadeaf6faad34f8" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.539042  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34263595c9ed4e0a9eba4bfe41c0702e" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.539793 12466 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:49.540472  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e89c188ae524548ae35a90336c0c92c" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.540892  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.541555  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.541975  1805 raft_consensus.cc:1240] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Rejecting Update request from peer a20e99e125b84513a44e88860667ed61 for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:49.542300  1800 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 30.
I20260709 12:26:49.543256  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84a31d3e62ae470d9c01e48ea29fbaee" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.543475  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.535136  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.538400  1879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5280e62b42a141359d53bd4f241659ef" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.545758  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7295d6494bad4378b63658f8274dfde8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.538952  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc103cf2bf8545cf8f5f5f34e38ad0e1" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:49.527426  1770 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.547461  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.547242  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1b33fb206f534e1e890a4546d3d43e84" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.549345  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e3f88cbac3d4fa4bdb3a23367a0e160" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:49.551028  1844 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 104666us
W20260709 12:26:49.547550  1770 leader_election.cc:336] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.571048 12466 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:49.583492  1807 raft_consensus.cc:2955] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Committing config change with OpId 35.22: config changed from index 21 to 22, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:49.594861  1844 leader_election.cc:336] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.597940  1770 leader_election.cc:336] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.537776 12442 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:49.602952  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 69302.9us
W20260709 12:26:49.604132  1770 leader_election.cc:336] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.604215  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:26:49.607266  1770 leader_election.cc:336] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.608706  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:49.608290  1770 leader_election.cc:336] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.609678  1770 leader_election.cc:336] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.610191  1770 leader_election.cc:336] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.610440  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.611660  1874 raft_consensus.cc:1275] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:26:49.614444  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.620954 12467 raft_consensus.cc:515] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:49.621459  1770 leader_election.cc:336] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.643707  1770 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:49.627714  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:49.627942  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:49.644917  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.636487  1806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.636445  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb48ef23a7434834929994987a36f360" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.636847  1800 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e573c99734f4d91bb30d1fe72b56cf7" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.639161  1804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc4fe4ebd84a44b9bee2535ccd62ba22" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:49.639917  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a534e4794d9548638a2781e4dc451bc6" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
W20260709 12:26:49.642241  1844 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.648232  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:49.648751  1844 leader_election.cc:336] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.649309  1844 leader_election.cc:336] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.649778  1844 leader_election.cc:336] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.650223  1844 leader_election.cc:336] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.650676  1844 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.651328  1844 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.644193  2383 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
W20260709 12:26:49.653723  1844 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.654455  1844 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.654341  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
W20260709 12:26:49.655198  1844 leader_election.cc:336] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.662317  1771 leader_election.cc:304] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.661252 12483 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:26:49.660115 12479 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:26:49.669919  1771 leader_election.cc:304] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:49.674407  1844 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:49.681068  1844 leader_election.cc:336] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.681113  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:49.681680  1844 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.682279  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:49.683010  1770 leader_election.cc:336] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.664664  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.683637  1770 leader_election.cc:304] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:49.684512 12332 raft_consensus.cc:2749] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:49.684854  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:49.685451  1770 leader_election.cc:336] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.685813  1770 leader_election.cc:304] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:49.651610 12442 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:49.670063 12492 raft_consensus.cc:2749] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:49.688903 12464 raft_consensus.cc:2749] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:26:49.691300  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:49.692304  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:49.692993  2382 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.693490  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:49.694670  2382 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.695276  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.677989 12483 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.696023 12442 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:49.696413 12483 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:26:49.698841 12429 raft_consensus.cc:2749] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:49.702987 12442 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:49.704411  2382 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.706327  2382 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.709947 12467 leader_election.cc:290] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:49.707111 12471 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:49.711762 12477 consensus_queue.cc:1059] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.002s
I20260709 12:26:49.715705 12477 consensus_queue.cc:1243] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [LEADER]: Peer 6084fbc6918e4e0b95a9bd0206a620d8 log is divergent from this leader: its last log entry 31.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:26:49.716380 12469 raft_consensus.cc:3055] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:49.716804 12469 raft_consensus.cc:740] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:49.717736 12469 consensus_queue.cc:260] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.718232 12483 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.718770  1771 leader_election.cc:304] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:49.720249  1844 leader_election.cc:336] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.720455  1771 leader_election.cc:304] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:49.720847  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:49.721344  1770 consensus_peers.cc:597] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:49.723839 12470 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:49.724656 12429 raft_consensus.cc:2749] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:49.727886  2382 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.728824  2382 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:49.729925  2382 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.721753  1845 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:49.730574 12442 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:49.730588 12431 raft_consensus.cc:2749] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:49.719328 12469 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:49.732990 12483 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:49.735534  1844 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.736862  1771 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.734562  2382 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:49.737370  1844 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:49.737949  1844 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.738492 12332 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:49.738494  1844 consensus_peers.cc:597] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:49.739732  1844 consensus_peers.cc:597] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:49.740792  2412 raft_consensus.cc:1240] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:49.740967  2413 raft_consensus.cc:1240] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Rejecting Update request from peer 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 27. Current term is 28. Ops: []
I20260709 12:26:49.743691 12442 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:49.744963  2382 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.750448  1771 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.752455 12464 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:49.733664 12467 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:49.752095  1771 leader_election.cc:304] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.754115  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.754472  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.755267 12429 raft_consensus.cc:2749] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:49.756367  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "034c167cd4e24138be60353036ce8092" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.757566  1771 leader_election.cc:304] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:49.757786  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:49.758584  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b8cae5d30dd4a68a4fba5846e83873e" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:49.759071  1844 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.760226 12429 raft_consensus.cc:2749] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:49.760648  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea5d8f652fc448d6af846ff6850004b8" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:49.760468  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.767278 12470 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:49.768155  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 46507.3us
I20260709 12:26:49.768762 12483 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:49.769625 12483 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:49.770059 12483 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:49.771209 12431 consensus_queue.cc:1059] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:26:49.772276 12431 raft_consensus.cc:3055] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:26:49.772516   561 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 1753)
I20260709 12:26:49.772635 12431 raft_consensus.cc:740] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:49.773815 12431 consensus_queue.cc:260] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.775234 12332 consensus_queue.cc:1059] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:26:49.775491 12431 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:49.777137 12332 raft_consensus.cc:3055] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:26:49.777493 12332 raft_consensus.cc:740] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:49.778231 12332 consensus_queue.cc:260] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.774646  1624 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 21 to 22, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 35 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:49.781734  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.783596 12442 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:26:49.782606 12332 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:49.788936  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
W20260709 12:26:49.790088  1844 leader_election.cc:336] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:49.794203  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.797075  1771 leader_election.cc:304] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.798092 12431 raft_consensus.cc:2749] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:49.798522  1771 leader_election.cc:304] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
W20260709 12:26:49.799197  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:49.799772  1771 leader_election.cc:304] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.793380 12483 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.802273 12483 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:49.807775  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.809633 12448 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:49.810145  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.810557 12481 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:26:49.811322 12481 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:49.811550  1771 leader_election.cc:304] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.812763  1771 leader_election.cc:304] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.813786 12487 raft_consensus.cc:2749] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:49.813848  1771 leader_election.cc:304] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:26:49.814844 12431 raft_consensus.cc:2749] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:49.814877 12465 raft_consensus.cc:2749] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:49.815797 12431 raft_consensus.cc:2749] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:49.815892 12464 raft_consensus.cc:2749] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:49.817495  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.817823  1807 raft_consensus.cc:2468] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 28.
I20260709 12:26:49.818521 12470 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:49.820688  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.821828  2383 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:49.822710 12442 raft_consensus.cc:2804] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:26:49.823462 12442 raft_consensus.cc:697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:49.824322 12442 consensus_queue.cc:237] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:49.828606 12470 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:26:49.830060  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.831142 12448 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:49.831223  1845 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.833706  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.834656 12429 consensus_queue.cc:1048] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:49.834559 12448 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:49.841295  2415 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:26:49.843010  1845 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:49.843940 12448 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:49.846108 12477 raft_consensus.cc:1064] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: attempting to promote NON_VOTER 123095127f334445889c21a42ba1d41e to VOTER
I20260709 12:26:49.852312 12429 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:26:49.853276 12477 consensus_queue.cc:237] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 14, Current term: 26, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:49.855994  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:49.844710  1846 leader_election.cc:304] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:49.865444 12470 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:49.882860  1625 catalog_manager.cc:5697] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 27 to 28, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 28 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:26:49.895893 12469 raft_consensus.cc:2749] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:49.898729  1846 leader_election.cc:304] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:49.903363  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 43789.4us
W20260709 12:26:49.903955  1844 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:49.905520  1807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:49.907307  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:49.916363 12483 raft_consensus.cc:2955] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 32 LEADER]: Committing config change with OpId 32.20: config changed from index 19 to 20, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:49.918931 12471 raft_consensus.cc:2749] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:49.927224  1881 consensus_queue.cc:237] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:49.933213  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:49.944638  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:49.963024  1846 leader_election.cc:304] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:49.972723  1846 leader_election.cc:304] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:49.975699 12469 raft_consensus.cc:2749] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:49.970551 12477 raft_consensus.cc:2749] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:26:49.976393  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:26:50.000025  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:50.008057  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:50.029722  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 53426.5us
I20260709 12:26:50.031239  1846 leader_election.cc:304] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:50.032793 12448 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:50.041141  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f27ba9b299f46b7a89ef2d660780f8d" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:50.043606  1881 raft_consensus.cc:1240] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 30. Current term is 31. Ops: []
I20260709 12:26:50.053504 12469 raft_consensus.cc:2749] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:50.058219  1803 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:26:50.058979  1801 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:26:50.060696 12442 consensus_queue.cc:1059] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:26:50.062295 12442 raft_consensus.cc:3055] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:26:50.062712 12442 raft_consensus.cc:740] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:50.063771 12442 consensus_queue.cc:260] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:50.065322 12442 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:50.055354  1846 leader_election.cc:304] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
W20260709 12:26:50.081051  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:50.088466  1846 leader_election.cc:304] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:50.089454 12470 raft_consensus.cc:2749] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:50.089818  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:50.092373 12448 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:26:50.088855 12471 raft_consensus.cc:2749] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:50.093957  1770 consensus_peers.cc:597] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:50.107275  1846 leader_election.cc:304] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:50.109418 12483 raft_consensus.cc:2749] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:50.110875  1846 leader_election.cc:304] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:50.112493 12477 raft_consensus.cc:2749] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:26:50.114529 12448 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:26:50.116958  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:50.128178  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:50.129084 12448 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:26:50.138407  1846 leader_election.cc:304] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 123095127f334445889c21a42ba1d41e
I20260709 12:26:50.143733  1803 raft_consensus.cc:2955] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Committing config change with OpId 32.20: config changed from index 19 to 20, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.173008 12448 raft_consensus.cc:2749] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:26:50.175180  1771 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:50.176838 12429 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:26:50.188346  1771 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:26:50.189973 12464 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:50.201113  1844 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:50.214474  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:50.216480  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9c190a9868442e4a7e6d84d752329f3" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:50.226119  2415 raft_consensus.cc:2468] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 31.
I20260709 12:26:50.228277  2412 raft_consensus.cc:2955] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Committing config change with OpId 32.20: config changed from index 19 to 20, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.241204  2413 raft_consensus.cc:1275] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 LEARNER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:26:50.243799  2417 raft_consensus.cc:1275] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:26:50.269272  1799 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:26:50.280754 12477 raft_consensus.cc:2955] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 35 LEADER]: Committing config change with OpId 35.23: config changed from index 22 to 23, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.293506 12472 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:26:50.308593  1625 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: config changed from index 19 to 20, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 32 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.325909  1874 raft_consensus.cc:1240] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Rejecting Update request from peer 123095127f334445889c21a42ba1d41e for earlier term 33. Current term is 34. Ops: []
I20260709 12:26:50.346236  1879 raft_consensus.cc:3060] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:26:50.346906  1880 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:26:50.352011 12479 consensus_queue.cc:1059] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: INVALID_TERM, Last received: 33.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:26:50.353535 12466 raft_consensus.cc:3055] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:26:50.353940 12466 raft_consensus.cc:740] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:26:50.354921 12466 consensus_queue.cc:260] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:50.356294 12466 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:26:50.379444  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:50.416930  1770 consensus_peers.cc:597] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:50.382012  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c5f3d785e05841fbb0efa210c3d9675c with cas_config_opid_index 22: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:26:50.417441  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:50.380129  1803 raft_consensus.cc:2955] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Committing config change with OpId 35.23: config changed from index 22 to 23, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.377775  1879 raft_consensus.cc:1275] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:26:50.426119  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:50.441605  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:50.444690  1846 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:50.445466 12429 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:26:50.448042  1846 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:26:50.451588  1879 consensus_queue.cc:237] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:50.450354 12477 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:26:50.457249 12448 raft_consensus.cc:2804] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:50.484186  2412 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:26:50.488299  1799 raft_consensus.cc:1275] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:26:50.493903 12448 raft_consensus.cc:697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 31 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:50.471203 12483 consensus_queue.cc:1048] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.025s
I20260709 12:26:50.457434 12475 consensus_queue.cc:1048] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.028s
I20260709 12:26:50.497711  1624 catalog_manager.cc:5697] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 22 to 23, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 35 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:50.465049 12472 consensus_queue.cc:1048] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:26:50.502748 12473 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:26:50.505038  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 64924us
I20260709 12:26:50.508592 12477 consensus_queue.cc:1048] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:26:50.452754 12431 raft_consensus.cc:2955] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Committing config change with OpId 27.18: config changed from index 17 to 18, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.537915  2411 raft_consensus.cc:2955] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Committing config change with OpId 35.22: config changed from index 21 to 22, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:50.556078  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 81086.8us
I20260709 12:26:50.521701 12448 consensus_queue.cc:237] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:50.579859  2417 raft_consensus.cc:2955] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Committing config change with OpId 27.18: config changed from index 17 to 18, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.610596  2411 raft_consensus.cc:2955] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Committing config change with OpId 35.23: config changed from index 22 to 23, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.617859  1879 raft_consensus.cc:2955] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Committing config change with OpId 27.18: config changed from index 17 to 18, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.622943 12483 raft_consensus.cc:2955] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Committing config change with OpId 26.17: config changed from index 16 to 17, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.693094  1799 raft_consensus.cc:2955] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Committing config change with OpId 26.17: config changed from index 16 to 17, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:50.728346  1844 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:50.726024 12500 log.cc:927] Time spent T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:26:50.737716 12470 raft_consensus.cc:2955] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 32 LEADER]: Committing config change with OpId 32.21: config changed from index 20 to 21, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.740463  1800 raft_consensus.cc:2955] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Committing config change with OpId 32.21: config changed from index 20 to 21, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.747951  1624 catalog_manager.cc:5697] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 17 to 18, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 27 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:50.753958  1844 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:50.773320   561 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:50.782843  2383 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 155641us
I20260709 12:26:50.811478  1625 catalog_manager.cc:5697] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 30 to 31, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 31 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:50.829973  2412 raft_consensus.cc:2955] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Committing config change with OpId 26.17: config changed from index 16 to 17, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.860381  2411 raft_consensus.cc:2955] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Committing config change with OpId 32.21: config changed from index 20 to 21, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
I20260709 12:26:50.902185  1625 catalog_manager.cc:5697] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e reported cstate change: config changed from index 16 to 17, 123095127f334445889c21a42ba1d41e (127.0.140.68) changed from NON_VOTER to VOTER. New cstate: current_term: 26 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } } }
W20260709 12:26:50.906535  1614 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 185095us
W20260709 12:26:50.913477  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:50.914216  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:50.916563  1615 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 9585ba97eaf14e069dcd2dcaf726a09b with cas_config_opid_index 20: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:26:50.928499  1772 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 93707.5us
W20260709 12:26:50.938686  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:50.916700  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:50.951835  1624 catalog_manager.cc:5697] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 reported cstate change: config changed from index 20 to 21, VOTER 075704438992440697732181e49ccd09 (127.0.140.65) evicted. New cstate: current_term: 32 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:50.962369  1614 catalog_manager.cc:4755] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c5f3d785e05841fbb0efa210c3d9675c: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:50.975679  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 157341us
W20260709 12:26:50.988101  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:51.001623  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:51.017784  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:51.022934  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 46292.4us
W20260709 12:26:51.098598  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:51.115908  1803 raft_consensus.cc:3060] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:26:51.127975  1803 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:26:51.136641 12469 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:26:51.166865  2412 raft_consensus.cc:1275] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:26:51.169555 12521 consensus_queue.cc:1048] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:26:51.232717  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:51.245289  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:51.256176  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:51.260926  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:51.275760  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:51.326476  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:51.402264  1770 consensus_peers.cc:597] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:51.421162  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:51.431767  1770 consensus_peers.cc:597] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:51.467661  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:51.514209 12332 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:51.514770 12332 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:51.516806 12332 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:51.519505  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:51.520212  2413 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 28.
I20260709 12:26:51.521916  1772 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:51.523075 12538 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:26:51.523582 12538 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:51.523952 12538 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:51.531272  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:51.532141  1874 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 27.
I20260709 12:26:51.570257 12538 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:51.580557  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:51.581333  1877 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:26:51.588617  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:51.589375  2413 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:26:51.595923  1877 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:51.598414  1771 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:51.600387 12332 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:26:51.601733 12332 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 LEADER]: Becoming Leader. State: Replica: 6084fbc6918e4e0b95a9bd0206a620d8, State: Running, Role: LEADER
I20260709 12:26:51.602638 12332 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:51.644503  2413 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084fbc6918e4e0b95a9bd0206a620d8 in term 29.
I20260709 12:26:51.645118 12538 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 29 election: Requested vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:26:51.682061  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:51.688077  1625 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:51.773967   561 thread.cc:527] Waited for 8000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:51.853425  1770 consensus_peers.cc:597] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:51.911257  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:52.029531 12448 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:52.031469 12448 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:52.043637 12479 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:52.044761 12479 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:52.049643 12479 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:26:52.050324 12448 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:52.052845  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:52.054447  1803 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 33.
I20260709 12:26:52.058280  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 
I20260709 12:26:52.062631 12479 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:26:52.063156 12479 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:52.042203  2412 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:26:52.063536 12479 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:52.071002 12431 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 123095127f334445889c21a42ba1d41e)
I20260709 12:26:52.070892 12514 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:26:52.072034 12431 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:52.073669  1770 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:52.085721  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" is_pre_election: true
I20260709 12:26:52.086673  1803 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 33.
I20260709 12:26:52.105792  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:52.106949 12448 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:26:52.116016 12448 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:52.117178 12448 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:26:52.129899  1874 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:26:52.152854  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:52.168865  2412 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:26:52.183390  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:52.178825 12431 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:52.174125 12448 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:26:52.185091  1844 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:52.191543  1874 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 35: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
I20260709 12:26:52.201740  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 34 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:26:52.202865  1876 raft_consensus.cc:2368] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 34. Current term is 35.
I20260709 12:26:52.209393 12448 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:26:52.219324  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:26:52.239315  1799 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:52.240242  1799 raft_consensus.cc:3060] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:26:52.177677 12479 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:52.263811  2412 raft_consensus.cc:2368] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 34. Current term is 35.
I20260709 12:26:52.266364  1772 leader_election.cc:400] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 34. Current term is 35.
I20260709 12:26:52.266755  1772 leader_election.cc:403] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:26:52.268186  1803 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "6084fbc6918e4e0b95a9bd0206a620d8"
I20260709 12:26:52.269646 12479 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:26:52.271368  1844 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:52.303161  1876 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:26:52.304237  1876 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 123095127f334445889c21a42ba1d41e in current term 35: Already voted for candidate a20e99e125b84513a44e88860667ed61 in this term.
W20260709 12:26:52.314962 12550 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:26:52.318320  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:26:52.320230 12479 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:26:52.331512  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1bd4f4834ed2497290417394b678edc0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:26:52.332587  2413 raft_consensus.cc:2393] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate a20e99e125b84513a44e88860667ed61 in current term 35: Already voted for candidate 123095127f334445889c21a42ba1d41e in this term.
W20260709 12:26:52.332846  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:52.338196  1799 raft_consensus.cc:2468] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 35.
I20260709 12:26:52.340562 12538 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 123095127f334445889c21a42ba1d41e (127.0.140.68:37665) with higher term. Message: Invalid argument: T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 34. Current term is 35.
I20260709 12:26:52.360366  1771 leader_election.cc:400] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 34 pre-election: Vote denied by peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761) with higher term. Message: Invalid argument: T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6084fbc6918e4e0b95a9bd0206a620d8 for earlier term 34. Current term is 35.
I20260709 12:26:52.366909  1845 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:26:52.368018 12549 raft_consensus.cc:2804] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:26:52.368693 12549 raft_consensus.cc:697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 35 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:26:52.370313 12549 consensus_queue.cc:237] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 33.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:26:52.378077  1880 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Refusing update from remote peer 6084fbc6918e4e0b95a9bd0206a620d8: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
W20260709 12:26:52.407882  1614 catalog_manager.cc:4755] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c5f3d785e05841fbb0efa210c3d9675c: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:26:52.415547 12332 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:26:52.436585  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:52.450073  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:52.450860  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 48881.4us
W20260709 12:26:52.477455  1770 consensus_peers.cc:597] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:52.546851  1625 catalog_manager.cc:5697] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 33 to 35, leader changed from 123095127f334445889c21a42ba1d41e (127.0.140.68) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 35 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:52.555315  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:52.562167  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:52.571977  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:52.597056  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:52.605125  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:52.606383  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:52.670857  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:52.715802  1844 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:52.734452  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:52.741914  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:52.764091  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:52.774809   561 thread.cc:527] Waited for 9000ms trying to join with diag-logger (tid 1753)
I20260709 12:26:52.830569  2417 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:26:52.839408 12448 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:26:52.843989  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:52.852268  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:52.865548  1803 raft_consensus.cc:1275] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 33 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:26:52.867318 12448 consensus_queue.cc:1048] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
W20260709 12:26:52.950631  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:53.025353  1770 consensus_peers.cc:597] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:53.102550  1770 consensus_peers.cc:597] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:53.175810  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:53.192515  1844 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:53.226159  1844 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:53.322916  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:53.407567  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:53.424619  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:53.455170  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:53.534497  1614 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 52102.9us
W20260709 12:26:53.566013  1614 catalog_manager.cc:4755] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9585ba97eaf14e069dcd2dcaf726a09b: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:53.598409  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:53.651161  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.725666  1770 consensus_peers.cc:597] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.728627  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:53.766989  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:53.775225  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:53.775516   561 thread.cc:527] Waited for 10000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:53.835862  2382 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111) [suppressed 499 similar messages]
W20260709 12:26:53.868741  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:53.884115  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:53.924793  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:54.012607  1770 consensus_peers.cc:597] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:54.126204  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:54.143664  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:54.156447  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:54.359934  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:54.474602  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:54.494979  1770 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:54.588656  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:54.594224  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:54.614156  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:54.701007  1844 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:54.778389   561 thread.cc:527] Waited for 11000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:54.785311  1770 consensus_peers.cc:597] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:54.876874  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:54.919864  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:54.942636  1844 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:55.039266  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:55.057101  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:55.131709  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:55.192939  1770 consensus_peers.cc:597] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:55.211860  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:55.227614  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:55.229317  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:55.229918  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:55.249027  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:55.267457  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:55.302416  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:55.319178  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:55.359431  1844 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:55.465021  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:55.494939  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:55.525277  1770 consensus_peers.cc:597] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:55.527134  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:55.554391  1770 consensus_peers.cc:597] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:55.608420  1844 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:55.704214  1614 catalog_manager.cc:4755] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9585ba97eaf14e069dcd2dcaf726a09b: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:55.778869   561 thread.cc:527] Waited for 12000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:55.827862  1844 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:55.829005  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:56.004391  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:56.007572  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:56.024246  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:56.028667  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:56.227437  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:56.253993  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:56.270992  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.288774  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.300809  1770 consensus_peers.cc:597] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:56.366891  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:56.382637  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:56.463682  1770 consensus_peers.cc:597] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:56.485387  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:56.503177  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:56.527047  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:56.724387  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:56.779414   561 thread.cc:527] Waited for 13000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:56.792677  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:57.025858  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:57.058954  1770 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:57.074862  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:57.122239  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:57.158294  1770 consensus_peers.cc:597] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:57.168825  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:57.285841  1844 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:57.293745  1844 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:57.327365  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:57.330473  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:57.342265  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:57.401136  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:57.451959  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:57.488410  1770 consensus_peers.cc:597] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:57.587709  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:57.593169  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:57.608881  1771 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35761, user_credentials={real_user=slave}} blocked reactor thread for 89901.2us
W20260709 12:26:57.640373  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:57.779915   561 thread.cc:527] Waited for 14000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:57.836840  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:57.853823  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:57.895339  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:57.920450  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:57.927451  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:57.940444  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:57.941829  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:58.046748  1844 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:58.069331  1770 consensus_peers.cc:597] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:58.079010  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:58.193398  1770 consensus_peers.cc:597] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:58.226846  1844 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:58.241906  1844 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:58.378724  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:58.428985  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:58.454252  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:58.535302  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:58.567643  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:58.568737  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:58.589612  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.623803  1770 consensus_peers.cc:597] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:58.776280  1770 consensus_peers.cc:597] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.780543   561 thread.cc:527] Waited for 15000ms trying to join with diag-logger (tid 1753)
W20260709 12:26:58.795620  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:58.797683  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:58.823462  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:58.826006  1845 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:40361, user_credentials={real_user=slave}} blocked reactor thread for 55982us
W20260709 12:26:58.846697  1770 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111) [suppressed 481 similar messages]
W20260709 12:26:58.876577  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:58.893270  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:58.965663  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:58.972123  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:59.287683  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:59.376691  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:59.394675  1770 consensus_peers.cc:597] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:59.424389  1770 consensus_peers.cc:597] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:59.658128  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:59.678129  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:59.685529  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:59.686271  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:59.739979  2382 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:35067, user_credentials={real_user=slave}} blocked reactor thread for 53766.7us
I20260709 12:26:59.757234   561 tablet_server.cc:196] TabletServer@127.0.140.65:0 shutdown complete.
W20260709 12:26:59.829449  1770 consensus_peers.cc:597] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:59.860204  1770 consensus_peers.cc:597] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:59.897490  1614 catalog_manager.cc:4755] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9585ba97eaf14e069dcd2dcaf726a09b: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:26:59.948503  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:00.062980  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:00.064296  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:00.110875  1844 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:00.148705  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:00.155230  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:00.174832  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:00.236517  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:00.241998  1770 consensus_peers.cc:597] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:00.261922  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:00.273468  1844 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:00.282536  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:00.362910  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:00.389766  1770 consensus_peers.cc:597] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:00.437928  1770 consensus_peers.cc:597] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:00.484496  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:00.531507  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:00.556095  1844 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:00.584203  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:00.816895  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:00.824090  1844 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:00.853106  1770 consensus_peers.cc:597] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:00.866221  1770 consensus_peers.cc:597] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:00.870217  1844 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:00.923528  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:01.035030   561 tablet_server.cc:179] TabletServer@127.0.140.66:0 shutting down...
W20260709 12:27:01.063262  1770 consensus_peers.cc:597] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:01.072890  1770 consensus_peers.cc:597] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:01.073797  2383 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.081077  1845 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.140589  1770 consensus_peers.cc:597] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:01.153913  1770 consensus_peers.cc:597] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:01.155522  1770 consensus_peers.cc:597] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:01.158759  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.160967  1845 consensus_peers.cc:597] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.165897  2383 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.167395  1845 consensus_peers.cc:597] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.205477  1845 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.206525  1845 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.209693  1845 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.220891  2383 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.223018  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:01.224185  2383 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.234404  1845 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.236933  2383 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.292894  1845 consensus_peers.cc:597] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.315132  1845 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.326107  2383 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.345011   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:27:01.347337  1770 consensus_peers.cc:597] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:01.348555   561 tablet_replica.cc:333] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.351138   561 raft_consensus.cc:2243] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.353655   561 raft_consensus.cc:2272] T 84a31d3e62ae470d9c01e48ea29fbaee P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.372828   561 tablet_replica.cc:333] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.373819   561 raft_consensus.cc:2243] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 LEADER]: Raft consensus shutting down.
W20260709 12:27:01.398633  1845 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.401177   561 raft_consensus.cc:2272] T 5e573c99734f4d91bb30d1fe72b56cf7 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 22 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:01.404613  1845 consensus_peers.cc:597] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.412637  1845 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.409231   561 tablet_replica.cc:333] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
W20260709 12:27:01.411305  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.415817   561 raft_consensus.cc:2243] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.418392   561 raft_consensus.cc:2272] T 4f27ba9b299f46b7a89ef2d660780f8d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:01.417439  1845 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.424824  2383 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.429242   561 tablet_replica.cc:333] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.433086   561 raft_consensus.cc:2243] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.439568   561 raft_consensus.cc:2272] T 84ce9896327e4c9d9888e80e8dbeeb0d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:01.445890  2383 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:01.448041  1845 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.456771   561 tablet_replica.cc:333] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.458105   561 raft_consensus.cc:2243] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.471397   561 raft_consensus.cc:2272] T 34263595c9ed4e0a9eba4bfe41c0702e P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.475838   561 tablet_replica.cc:333] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.477006   561 raft_consensus.cc:2243] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.477820   561 raft_consensus.cc:2272] T 7b550688b5e84ddda5fcf049010ac1e8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.481398   561 tablet_replica.cc:333] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.482362   561 raft_consensus.cc:2243] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.483896   561 raft_consensus.cc:2272] T 82c4d9259f1a44119e802a40207d64b5 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.491122   561 tablet_replica.cc:333] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.499275   561 raft_consensus.cc:2243] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.501297   561 raft_consensus.cc:2272] T ae12891834034d968900107cc58fa821 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:01.503722  1845 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.505188   561 tablet_replica.cc:333] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.506222   561 raft_consensus.cc:2243] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.516407   561 raft_consensus.cc:2272] T 8bb813e4fab14a61b810a24b56899586 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.524307   561 tablet_replica.cc:333] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.525278   561 raft_consensus.cc:2243] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.526805   561 raft_consensus.cc:2272] T 7295d6494bad4378b63658f8274dfde8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.548944   561 tablet_replica.cc:333] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.549918   561 raft_consensus.cc:2243] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.551236   561 raft_consensus.cc:2272] T c072b820694347cabf7868a1b3805f76 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:01.559295  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:01.587251   561 tablet_replica.cc:333] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.595638   561 raft_consensus.cc:2243] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.596284   561 raft_consensus.cc:2272] T aca584e2f1114809bcadeaf6faad34f8 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:01.596479  1845 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.616833   561 tablet_replica.cc:333] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.617775   561 raft_consensus.cc:2243] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.618561   561 raft_consensus.cc:2272] T 79f5c05e64b945cbacc02914f31c5762 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.645262   561 tablet_replica.cc:333] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.646091   561 raft_consensus.cc:2243] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 LEADER]: Raft consensus shutting down.
W20260709 12:27:01.625727  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:01.688376  1845 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:01.722267   561 raft_consensus.cc:2272] T a534e4794d9548638a2781e4dc451bc6 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:01.735235  1846 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:37665, user_credentials={real_user=slave}} blocked reactor thread for 51300.9us
I20260709 12:27:01.772065   561 tablet_replica.cc:333] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.775678   561 raft_consensus.cc:2243] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.777609   561 raft_consensus.cc:2272] T 9585ba97eaf14e069dcd2dcaf726a09b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.801704   561 tablet_replica.cc:333] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.810245   561 raft_consensus.cc:2243] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.815982   561 raft_consensus.cc:2272] T 66d3405528d44edb8c0be3b296ef72be P 6084fbc6918e4e0b95a9bd0206a620d8 [term 24 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:01.819039  1770 consensus_peers.cc:597] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:01.836365   561 tablet_replica.cc:333] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.837185   561 raft_consensus.cc:2243] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.838310   561 raft_consensus.cc:2272] T bb48ef23a7434834929994987a36f360 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.841668   561 tablet_replica.cc:333] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.842617   561 raft_consensus.cc:2243] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.843617   561 raft_consensus.cc:2272] T def0036446324543ba07e132e885133a P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.856614   561 tablet_replica.cc:333] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.857722   561 raft_consensus.cc:2243] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.858623   561 raft_consensus.cc:2272] T b1cb2a27e8444930b93f1e21c2deb19c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.874814   561 tablet_replica.cc:333] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.877180   561 raft_consensus.cc:2243] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.878149   561 raft_consensus.cc:2272] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.884835   561 tablet_replica.cc:333] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.885825   561 raft_consensus.cc:2243] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.887171   561 raft_consensus.cc:2272] T 2e89c188ae524548ae35a90336c0c92c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.897374   561 tablet_replica.cc:333] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.898598   561 raft_consensus.cc:2243] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.899852   561 raft_consensus.cc:2272] T 8c92163822f64868a647b688599f49bd P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.902675   561 tablet_replica.cc:333] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.910738   561 raft_consensus.cc:2243] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.914052   561 raft_consensus.cc:2272] T 1bd4f4834ed2497290417394b678edc0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.918990   561 tablet_replica.cc:333] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.919998   561 raft_consensus.cc:2243] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.927253   561 raft_consensus.cc:2272] T bc4fe4ebd84a44b9bee2535ccd62ba22 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.930438   561 tablet_replica.cc:333] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.931414   561 raft_consensus.cc:2243] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.932142   561 raft_consensus.cc:2272] T f15e08b3d9804a00b9836e5223042266 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.936249   561 tablet_replica.cc:333] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.937158   561 raft_consensus.cc:2243] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.938462   561 raft_consensus.cc:2272] T 5280e62b42a141359d53bd4f241659ef P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.941555   561 tablet_replica.cc:333] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.942471   561 raft_consensus.cc:2243] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.943511   561 raft_consensus.cc:2272] T b9c190a9868442e4a7e6d84d752329f3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.946341   561 tablet_replica.cc:333] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.947189   561 raft_consensus.cc:2243] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.947938   561 raft_consensus.cc:2272] T 4a57b66dec794db09a7f9048a3d0de10 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.957986   561 tablet_replica.cc:333] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.959066   561 raft_consensus.cc:2243] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.960115   561 raft_consensus.cc:2272] T 034c167cd4e24138be60353036ce8092 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.969097   561 tablet_replica.cc:333] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.972016   561 raft_consensus.cc:2243] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:27:01.975056   561 raft_consensus.cc:2272] T 62f288182b21420f8ef28e25fc10cd70 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.986471   561 tablet_replica.cc:333] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.989406   561 raft_consensus.cc:2243] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:01.990417   561 raft_consensus.cc:2272] T 8f285ce2457a47ebbd05920b42b35fba P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:01.995339   561 tablet_replica.cc:333] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:01.996268   561 raft_consensus.cc:2243] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:02.004215  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:02.007704   561 raft_consensus.cc:2272] T 82eacf0c2a8c4d8db4052327304a8793 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.013659   561 tablet_replica.cc:333] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.015043   561 raft_consensus.cc:2243] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:02.016402   561 raft_consensus.cc:2272] T c22ef2bc934c4b7ea57aad5853ab6122 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.019881   561 tablet_replica.cc:333] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.020741   561 raft_consensus.cc:2243] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:27:02.022251   561 raft_consensus.cc:2272] T 6f86194bda894372ab64d2c6a9937a5b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.026062   561 tablet_replica.cc:333] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.027060   561 raft_consensus.cc:2243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.027894   561 raft_consensus.cc:2272] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.030997   561 tablet_replica.cc:333] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.031865   561 raft_consensus.cc:2243] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:02.033461   561 raft_consensus.cc:2272] T d891ed30ca204d09b7b083088037b63d P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:02.036468  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:02.036794   561 tablet_replica.cc:333] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.037765   561 raft_consensus.cc:2243] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.038475   561 raft_consensus.cc:2272] T 1b33fb206f534e1e890a4546d3d43e84 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:02.040565  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:27:02.041281  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:02.041746   561 tablet_replica.cc:333] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.042827   561 raft_consensus.cc:2243] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:02.044095   561 raft_consensus.cc:2272] T 9875eb35aaed48d7b2670b5ea57e3948 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.047570   561 tablet_replica.cc:333] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.048470   561 raft_consensus.cc:2243] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.049247   561 raft_consensus.cc:2272] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.052590   561 tablet_replica.cc:333] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.053562   561 raft_consensus.cc:2243] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.055510   561 raft_consensus.cc:2272] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.058746   561 tablet_replica.cc:333] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.060721   561 raft_consensus.cc:2243] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:27:02.062094   561 raft_consensus.cc:2272] T efbb8c187a4e410aa09e81ceb9ebecc3 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.065392   561 tablet_replica.cc:333] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.066313   561 raft_consensus.cc:2243] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:02.067744   561 raft_consensus.cc:2272] T ae23688e76624153b5a7089586669bb0 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.076012   561 tablet_replica.cc:333] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.077402   561 raft_consensus.cc:2243] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 LEADER]: Raft consensus shutting down.
I20260709 12:27:02.079591   561 raft_consensus.cc:2272] T 908089442ca44109875257bfbc9d06c2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.081826 12594 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:02.082415 12594 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:02.084924 12594 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:27:02.086138  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:02.086898  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c5f3d785e05841fbb0efa210c3d9675c" candidate_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:27:02.088318   561 tablet_replica.cc:333] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.088937  1771 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6084fbc6918e4e0b95a9bd0206a620d8; no voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61
I20260709 12:27:02.089269   561 raft_consensus.cc:2243] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.090427 12594 raft_consensus.cc:2706] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Received pre-election callback for term 36 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 12:27:02.095037   561 raft_consensus.cc:2272] T c5f3d785e05841fbb0efa210c3d9675c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.098299   561 tablet_replica.cc:333] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.099035   561 raft_consensus.cc:2243] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.104821   561 raft_consensus.cc:2272] T bbbcc16cebc244769e7f1a1b6e29594c P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.110903   561 tablet_replica.cc:333] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.111658   561 raft_consensus.cc:2243] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 LEADER]: Raft consensus shutting down.
I20260709 12:27:02.113075   561 raft_consensus.cc:2272] T 36bc2f66a73947468d61113c226ed318 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.122606   561 tablet_replica.cc:333] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.123322   561 raft_consensus.cc:2243] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.123967   561 raft_consensus.cc:2272] T 76f6bc3c4cce45d7855e6b3429341da2 P 6084fbc6918e4e0b95a9bd0206a620d8 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.127959   561 tablet_replica.cc:333] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8: stopping tablet replica
I20260709 12:27:02.132951   561 raft_consensus.cc:2243] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.134091   561 raft_consensus.cc:2272] T b3034677490f4f0fac2a4ed3e2a8142b P 6084fbc6918e4e0b95a9bd0206a620d8 [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:02.335172  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:02.338826  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:02.393455  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:02.401196  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:02.513746 12599 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.514613 12586 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.514871 12599 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:02.515748 12586 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:02.518772 12586 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:02.524915 12595 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.525944 12595 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:02.530562 12582 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.535074 12582 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:27:02.532588  2382 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:02.539309 12582 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:02.540880 12595 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:02.548256  2383 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:02.548782  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:02.550225 12586 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:27:02.554514 12599 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:27:02.556195  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
W20260709 12:27:02.557521  1844 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:02.559378  1844 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:02.585147  1845 leader_election.cc:336] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:02.585807  1845 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:02.586705 12595 raft_consensus.cc:2749] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:02.588238  1845 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:02.588734  1845 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:02.589519 12595 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:27:02.591622  1845 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:02.592182  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:02.596716 12595 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:27:02.599947  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:02.602099 12567 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.603116 12567 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:02.611260 12567 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:02.613569  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:27:02.615098  2382 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:02.617795  2383 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:02.620195  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:02.621126 12567 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:02.628260  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:02.690431 12567 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.691396 12567 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:02.701436 12567 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:02.712297  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:27:02.713128  1877 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 31.
W20260709 12:27:02.722303  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:02.725152  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:27:02.729399 12603 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:27:02.736290 12603 raft_consensus.cc:493] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.736900 12603 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:27:02.728686  2383 leader_election.cc:336] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:27:02.766465  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:02.777348  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:02.784240  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:02.795244 12603 raft_consensus.cc:515] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:02.806394 12599 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.807293 12599 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:02.809808 12599 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:02.812817  1845 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:02.816839  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c072b820694347cabf7868a1b3805f76" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
W20260709 12:27:02.817353  2383 leader_election.cc:336] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:02.817412  1877 raft_consensus.cc:3060] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:27:02.817994 12603 leader_election.cc:290] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:02.826254  1877 raft_consensus.cc:2468] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 32.
W20260709 12:27:02.828439  1844 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:02.835089  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:02.835325  2383 leader_election.cc:304] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:02.836018 12599 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:27:02.837538 12603 raft_consensus.cc:2804] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:27:02.838210 12603 raft_consensus.cc:697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 32 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
I20260709 12:27:02.839610 12603 consensus_queue.cc:237] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:02.850756 12595 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.851689 12595 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:02.856452 12595 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:27:02.858182  1845 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:02.858702  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:02.859412  2417 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
W20260709 12:27:02.861260  1844 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:02.861793  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:02.863742 12595 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:27:02.868464 12599 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.869099 12599 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:27:02.869913  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:02.878443  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:02.890951 12599 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:02.894805  1845 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:27:02.900074  1625 catalog_manager.cc:5697] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 31 to 32, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 32 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:27:02.921365  1844 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:02.922068  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:02.933523 12599 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:02.974638  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:02.998795 12595 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:02.999629 12595 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:03.006742 12595 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:03.010998  1845 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:03.018210  1844 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:03.018883  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.020362 12595 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:27:03.092213 12582 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.092900 12582 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:27:03.094050  1844 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:03.095561 12582 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:03.100198  1844 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:03.102469  1845 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.103055  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.103996 12582 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:03.138233  1844 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:03.140393 12603 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.142746 12603 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.145593 12603 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:03.146591  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:03.149026  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:27:03.150331  2383 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.151371  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:03.152552 12603 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:27:03.172492 12603 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.173276 12603 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.175719 12603 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:03.183530 12604 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.184309 12604 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.186863 12604 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:03.187553  2382 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:03.187892 12599 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.188742 12599 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:27:03.201244  1845 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.207306 12599 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:03.215711  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:27:03.215924  2383 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.217260  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.218142 12604 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:27:03.221599 12604 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.222326 12604 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.225164 12604 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:03.236832  2382 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:03.238693  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:27:03.239473  1844 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:03.240170  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:03.241037  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.244894  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:03.248250  2383 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:03.248957 12604 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:27:03.250406 12595 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:27:03.250548 12604 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:27:03.252034  2383 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.252625 12604 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.256703 12604 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:03.257280  1844 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:03.259390 12604 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:03.262326 12595 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.262447  1877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:27:03.263356 12595 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.266198 12595 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:27:03.269724 12595 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.270308 12595 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.273370 12582 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.274050 12582 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.276578 12582 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:27:03.279575 12599 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.280179 12599 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:03.282600 12599 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:03.283761 12599 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.284426 12599 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:03.286991 12582 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.287669 12582 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:27:03.288744  1844 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:03.289876 12582 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:03.291657  1844 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:03.295146  1845 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:03.299108  1844 leader_election.cc:336] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:03.300055  1845 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:03.304631  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:03.300629  1845 leader_election.cc:336] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.305150  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:03.306171 12604 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:27:03.308079  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:03.305490  1845 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.309253  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.286770 12599 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:03.312108 12595 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:27:03.315450  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:03.316524  2417 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 31.
I20260709 12:27:03.317848 12599 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:27:03.321213 12612 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.321758  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:03.321970 12612 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.322474  2417 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 29.
I20260709 12:27:03.323944  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:03.324937 12612 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:27:03.329440  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.333117 12599 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:27:03.333577 12599 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.334023 12599 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:27:03.331337  1845 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.335114  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:03.336836  1845 leader_election.cc:336] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.337854  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.340507  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.350395 12613 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:27:03.350510  2383 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:03.337426 12595 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:27:03.352701 12612 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:27:03.351259 12613 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.358650 12613 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:27:03.359834  1845 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.366637  1846 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 123095127f334445889c21a42ba1d41e, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.371142  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:03.371824  2417 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
W20260709 12:27:03.376869  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:03.382565 12612 raft_consensus.cc:2749] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:27:03.389396 12599 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:03.395391  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:03.396200  1844 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:03.396663  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:03.399924  1845 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.395397 12613 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.405447  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8c92163822f64868a647b688599f49bd" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:27:03.406111  2417 raft_consensus.cc:3060] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:27:03.407657 12613 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:27:03.411051  1845 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.412216 12614 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:27:03.417012  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae23688e76624153b5a7089586669bb0" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 32 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:27:03.417774  2413 raft_consensus.cc:3060] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:27:03.423729 12599 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:27:03.428148 12614 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.428968 12614 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.432102 12614 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:27:03.435384  2411 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:03.436347  2411 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:27:03.437568  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:27:03.438071  2413 raft_consensus.cc:2468] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 32.
I20260709 12:27:03.438467 12599 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:27:03.439981  1846 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.440486  2417 raft_consensus.cc:2468] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 30.
I20260709 12:27:03.441946 12614 raft_consensus.cc:2804] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:27:03.442585 12614 raft_consensus.cc:697] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 32 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:27:03.443537  1877 raft_consensus.cc:1275] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:27:03.445374 12614 consensus_queue.cc:237] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 31.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.447335 12603 consensus_queue.cc:1048] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:27:03.462618  1846 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:03.464277  1845 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.464777 12582 raft_consensus.cc:2804] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:27:03.466192 12582 raft_consensus.cc:697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 30 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:27:03.467418 12582 consensus_queue.cc:237] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:03.477512  1845 consensus_peers.cc:597] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:03.481647 12613 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.439111 12599 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.483054 12599 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:27:03.483872 12613 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:03.486433 12613 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:03.495697 12603 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:27:03.499567  1844 leader_election.cc:336] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:03.511102 12603 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.532316  1625 catalog_manager.cc:5697] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 31 to 32, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 32 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:03.533811  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:03.536514 12603 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:03.539707  1845 leader_election.cc:336] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.540567  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.558542 12614 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:03.561412 12604 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.562108 12604 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.564321 12604 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:03.565508 12599 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:03.567643  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:03.567730 12599 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
I20260709 12:27:03.570258  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "62f288182b21420f8ef28e25fc10cd70" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e"
I20260709 12:27:03.571128  2417 raft_consensus.cc:3060] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:27:03.571589  1845 consensus_peers.cc:597] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.573246  1845 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.574260  2382 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:03.578236 12618 log.cc:927] Time spent T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:27:03.578568  2417 raft_consensus.cc:2468] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 28.
W20260709 12:27:03.580596  1845 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:03.581993  1845 consensus_peers.cc:597] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:03.583950  1846 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.601344  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:27:03.602319  1881 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 30.
I20260709 12:27:03.587508 12614 raft_consensus.cc:2804] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:27:03.612187 12614 raft_consensus.cc:697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Becoming Leader. State: Replica: a20e99e125b84513a44e88860667ed61, State: Running, Role: LEADER
I20260709 12:27:03.613400 12614 consensus_queue.cc:237] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:03.621394  1845 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:03.609684  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 
I20260709 12:27:03.632323 12603 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:27:03.634922 12603 raft_consensus.cc:493] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting leader election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:03.635334 12603 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:27:03.639557  2383 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.640156  2383 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.647087  2383 leader_election.cc:336] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:03.650020  2383 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.650432  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:03.652688  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:03.653832 12621 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:03.653187  2383 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:03.675235 12603 raft_consensus.cc:515] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:03.687932  2383 leader_election.cc:336] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:03.693107 12603 leader_election.cc:290] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: Requested vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:03.693527  1881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9875eb35aaed48d7b2670b5ea57e3948" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61"
I20260709 12:27:03.694386  1881 raft_consensus.cc:3060] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:27:03.707659  1881 raft_consensus.cc:2468] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 31.
W20260709 12:27:03.709009  1844 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:03.707265  1625 catalog_manager.cc:5697] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 29 to 30, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 30 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:03.712256  1625 catalog_manager.cc:5697] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 reported cstate change: term changed from 27 to 28, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to a20e99e125b84513a44e88860667ed61 (127.0.140.67). New cstate: current_term: 28 leader_uuid: "a20e99e125b84513a44e88860667ed61" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:03.725845  2383 leader_election.cc:304] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:03.726989 12603 raft_consensus.cc:2804] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:27:03.727620 12603 raft_consensus.cc:697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 31 LEADER]: Becoming Leader. State: Replica: 123095127f334445889c21a42ba1d41e, State: Running, Role: LEADER
W20260709 12:27:03.728523  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:03.728658 12603 consensus_queue.cc:237] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:03.759934  1625 catalog_manager.cc:5697] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e reported cstate change: term changed from 30 to 31, leader changed from 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66) to 123095127f334445889c21a42ba1d41e (127.0.140.68). New cstate: current_term: 31 leader_uuid: "123095127f334445889c21a42ba1d41e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:27:03.777956  2383 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.789552  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:03.805442  1845 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.829260  2383 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.849225  2383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111) [suppressed 593 similar messages]
W20260709 12:27:03.865660  2383 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.874631  1845 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.896960  1845 consensus_peers.cc:597] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.904816  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.906023  1845 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.930872  1845 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.935031  1845 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.940834   561 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1828)
W20260709 12:27:03.967809  1845 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.991870  2383 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:03.992023  1845 consensus_peers.cc:597] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:04.029660  1845 consensus_peers.cc:597] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:04.043601  2417 raft_consensus.cc:1275] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:27:04.056897 12582 consensus_queue.cc:1048] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:27:04.060364  1845 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:04.095269  1845 consensus_peers.cc:597] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:04.100006  2413 raft_consensus.cc:1275] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:27:04.101877 12614 consensus_queue.cc:1048] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
W20260709 12:27:04.146145  2383 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:04.177928  2411 raft_consensus.cc:1275] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Refusing update from remote peer a20e99e125b84513a44e88860667ed61: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:27:04.187840 12614 consensus_queue.cc:1048] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:27:04.224205  1845 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:04.264562  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:04.269325  1845 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:04.269671  1881 raft_consensus.cc:1275] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Refusing update from remote peer 123095127f334445889c21a42ba1d41e: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
W20260709 12:27:04.270028  1845 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:04.282279 12603 consensus_queue.cc:1048] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:27:04.288415  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:04.290154  2383 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:04.317498  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:04.422111 12603 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:04.422920 12603 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:04.431238 12615 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:04.431962 12615 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:04.434481 12615 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:04.437228  2382 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:04.439831 12615 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:04.440654 12615 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:04.443010 12603 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:04.449350  2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:04.450179  2417 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
I20260709 12:27:04.444959 12615 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:27:04.456699  2383 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:04.457316  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:04.471042 12603 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:04.478614  1845 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:04.485090  1845 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:04.485628  1845 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:04.553668 12582 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:04.554389 12582 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:04.560633 12582 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:04.562999  1845 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:04.701355 12615 raft_consensus.cc:493] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:04.701982 12615 raft_consensus.cc:515] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:04.707690 12615 leader_election.cc:290] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:04.710109  1845 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:04.706292  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:04.802428  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:04.850484 12603 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:04.851109 12603 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:04.853823 12603 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:04.872419  2383 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:04.874140  2382 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:04.876299  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:04.887616 12603 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:27:04.937285 12582 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:27:04.939298  1844 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:27:04.937850 12582 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:04.936959 12614 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:04.941038 12615 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:04.941627 12615 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:04.942272 12582 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:04.940068  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:27:04.943958 12615 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:04.941628   561 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1828)
I20260709 12:27:04.941056 12614 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:04.949208 12614 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:04.955951  1844 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:04.956569  1844 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:04.961613  1845 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:04.964519  1845 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:04.965436 12615 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:27:04.967687  1844 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:04.968173  1844 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:04.969532 12615 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:04.970758  1844 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:04.971194  1844 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:04.972146  1844 leader_election.cc:336] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:04.972572  1844 leader_election.cc:304] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:04.975409  1845 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:04.981561 12615 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:27:04.982517 12615 raft_consensus.cc:2749] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:27:05.017752  1844 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:05.018293  1844 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.019181 12635 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:27:05.019337  1844 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:05.019974  1844 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.020929 12635 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:05.021373  1844 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:05.021988  1844 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.022712 12635 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:05.047762  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:05.063280  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:05.085825  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:05.182670 12635 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:05.183298 12635 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:27:05.198891  1844 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:05.199723  1845 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:05.200109  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.201758 12615 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:27:05.202842 12635 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:05.257504  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:05.297210 12615 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
W20260709 12:27:05.297493  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:05.298026 12615 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:05.303130 12615 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:27:05.306111  1845 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:05.309562 12615 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:05.310287 12615 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:05.312942  2413 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:05.313812  2413 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 27.
I20260709 12:27:05.314082 12615 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:05.319424  1845 leader_election.cc:336] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:05.325337  1844 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:05.326026  1844 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.327684 12615 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:27:05.333078  1844 leader_election.cc:336] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:05.334399  1844 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.335402 12635 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:27:05.344130  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:05.344754 12635 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:05.345394 12635 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:05.347699 12635 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:05.350131  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:05.353705  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:05.355989  1845 leader_election.cc:336] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:05.378103  1844 leader_election.cc:336] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:05.378620  1844 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.392087 12615 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:27:05.404047  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:05.436496  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:27:05.452291 12615 raft_consensus.cc:493] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:05.453233 12615 raft_consensus.cc:515] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:05.455888 12615 leader_election.cc:290] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:05.476965  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:05.486296  1845 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:05.495402  1844 leader_election.cc:336] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:05.495960  1844 leader_election.cc:304] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.503436 12635 raft_consensus.cc:2749] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:05.515965  2383 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:05.520648 12615 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:05.521328 12615 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:05.524006 12615 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:05.543610  1844 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:05.577160  1845 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:05.577713  1845 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.578495 12615 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:27:05.716034  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:05.729800 12621 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:05.730469 12621 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:05.732896 12621 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:05.739562  2383 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:05.755260  2382 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:05.755867  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:05.759071 12621 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:05.787163  1844 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:27:05.803479  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:05.899602  2383 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:05.901790  1845 consensus_peers.cc:597] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:05.984759  2383 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.080773  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.098567  1844 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:06.119256  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:06.122200  1845 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.131508  2383 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.136263  1845 consensus_peers.cc:597] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.147171  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:06.196462  1845 consensus_peers.cc:597] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.212920  2383 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.213521  1845 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.247219  1844 consensus_peers.cc:597] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:06.274928  1844 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:06.286267  2383 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:06.327188 12603 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:06.327721 12603 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:06.329612 12603 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:06.340997  2383 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:06.341878  2382 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:06.342345  2382 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:06.344484 12621 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:06.345731  1845 consensus_peers.cc:597] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.380389  1844 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:06.424636  1845 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.425477  1845 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.429275  1845 consensus_peers.cc:597] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:06.460831  1845 consensus_peers.cc:597] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.468979  1844 consensus_peers.cc:597] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:27:06.477000  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:06.481473  1845 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.501156  1845 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.525586  1845 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.526670  2383 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.541085  2383 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:06.563341  1845 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.564422  1844 consensus_peers.cc:597] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:06.577050  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.593024  2383 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.616595  1844 consensus_peers.cc:597] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:27:06.620575  1845 consensus_peers.cc:597] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.645785  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:06.648077  1845 consensus_peers.cc:597] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:06.648675  1845 consensus_peers.cc:597] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:06.655975  1845 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.782449  1844 consensus_peers.cc:597] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:06.809492 12615 raft_consensus.cc:493] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:06.810187 12615 raft_consensus.cc:515] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:06.812534 12615 leader_election.cc:290] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:06.821480  1844 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:06.824492  1845 leader_election.cc:336] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:06.825026  1845 leader_election.cc:304] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:06.825870 12615 raft_consensus.cc:2749] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:06.831440  1845 consensus_peers.cc:597] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:06.869318  1844 consensus_peers.cc:597] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:06.894690  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:06.905194  2383 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:07.286037 12615 raft_consensus.cc:493] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:07.286736 12615 raft_consensus.cc:515] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:07.288884 12615 leader_election.cc:290] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:07.289438 12615 raft_consensus.cc:493] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:07.289929 12615 raft_consensus.cc:515] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:07.292013 12615 leader_election.cc:290] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:07.330984 12603 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:07.340255 12614 raft_consensus.cc:493] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:07.349810 12603 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:07.356174 12603 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:07.358633 12614 raft_consensus.cc:515] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:07.375397 12614 leader_election.cc:290] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:07.416178  1844 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:07.421650 12621 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:07.422427 12621 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:07.426734 12621 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:07.428828  1845 consensus_peers.cc:597] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:07.416935  1844 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:07.433728  1844 consensus_peers.cc:597] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:07.435122  1844 consensus_peers.cc:597] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:07.443492 12637 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:07.444931 12637 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:07.447992 12637 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), 123095127f334445889c21a42ba1d41e (127.0.140.68:37665)
W20260709 12:27:07.449891  1844 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:07.458225  1845 leader_election.cc:336] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:07.458866  1845 leader_election.cc:304] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:07.460005  1845 leader_election.cc:336] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:07.460421  1845 leader_election.cc:304] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:07.461206 12639 raft_consensus.cc:2749] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:27:07.461784 12638 raft_consensus.cc:2749] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:27:07.462762  2415 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "a20e99e125b84513a44e88860667ed61" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "123095127f334445889c21a42ba1d41e" is_pre_election: true
I20260709 12:27:07.463726  2415 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a20e99e125b84513a44e88860667ed61 in term 25.
W20260709 12:27:07.484243  2383 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:07.484282  1844 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:07.486626 12639 raft_consensus.cc:493] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:07.487298 12639 raft_consensus.cc:515] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:07.488312  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bb813e4fab14a61b810a24b56899586" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
I20260709 12:27:07.489043  1880 raft_consensus.cc:2468] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 25.
I20260709 12:27:07.489640 12639 leader_election.cc:290] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:07.498948  1844 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:07.507086  1880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "908089442ca44109875257bfbc9d06c2" candidate_uuid: "123095127f334445889c21a42ba1d41e" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "a20e99e125b84513a44e88860667ed61" is_pre_election: true
W20260709 12:27:07.507458  2382 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:07.508140  1880 raft_consensus.cc:2468] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 123095127f334445889c21a42ba1d41e in term 27.
W20260709 12:27:07.508430  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:07.509583  2382 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:07.510026  2382 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:07.510963 12645 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:27:07.512697  1845 leader_election.cc:336] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:07.515039  1845 leader_election.cc:304] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:07.515889 12638 raft_consensus.cc:2749] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:07.515918  2383 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:07.516599  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:07.517345 12645 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:07.530148  1845 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:07.530872  1845 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e, a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:07.531742 12638 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:07.539368  1845 leader_election.cc:336] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:07.540351 12638 raft_consensus.cc:493] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:07.541033 12638 raft_consensus.cc:515] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:07.543638 12638 leader_election.cc:290] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:07.550359  1844 leader_election.cc:336] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:07.554935  1845 leader_election.cc:304] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:07.571952 12638 raft_consensus.cc:2749] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:27:07.583313   561 tablet_server.cc:196] TabletServer@127.0.140.66:0 shutdown complete.
W20260709 12:27:07.586370  1845 leader_election.cc:336] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:07.587231  1845 leader_election.cc:304] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:07.588388 12638 raft_consensus.cc:2749] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:27:07.724489  1844 consensus_peers.cc:597] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:27:07.758642   561 tablet_server.cc:179] TabletServer@127.0.140.67:0 shutting down...
W20260709 12:27:07.811116  2383 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:07.855934  1844 consensus_peers.cc:597] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:27:07.872138  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:07.890403  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:07.918165  1661 debug-util.cc:398] Leaking SignalData structure 0x7b0800173a00 after lost signal to thread 564
W20260709 12:27:07.919476  1661 debug-util.cc:398] Leaking SignalData structure 0x7b0800397bc0 after lost signal to thread 1664
W20260709 12:27:07.920773  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:27:07.920933  1661 debug-util.cc:398] Leaking SignalData structure 0x7b0800050500 after lost signal to thread 1905
W20260709 12:27:07.922586  1661 debug-util.cc:398] Leaking SignalData structure 0x7b08003e7b20 after lost signal to thread 2443
W20260709 12:27:07.926647  1844 consensus_peers.cc:597] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:07.927464  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:07.954355  2383 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:07.957218  2383 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:07.973577  2383 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.001230  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.002040  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.003927  2383 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.030171  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:08.037273  2383 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.038026  2383 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.038676  2383 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.040681  2383 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.059381  2383 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:08.068689  1844 consensus_peers.cc:597] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:27:08.070292 12650 raft_consensus.cc:493] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:08.071046 12650 raft_consensus.cc:515] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:08.074147 12650 leader_election.cc:290] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:08.081065  1844 leader_election.cc:336] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:08.088040  1845 leader_election.cc:336] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:08.088660  1845 leader_election.cc:304] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:08.090472   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:08.091044 12650 raft_consensus.cc:2749] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:27:08.101171   561 tablet_replica.cc:333] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.102560   561 raft_consensus.cc:2243] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.109849   561 raft_consensus.cc:2272] T 34263595c9ed4e0a9eba4bfe41c0702e P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.114074   561 tablet_replica.cc:333] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.123515   561 raft_consensus.cc:2243] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.124462   561 raft_consensus.cc:2272] T 84a31d3e62ae470d9c01e48ea29fbaee P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:08.137176  1614 catalog_manager.cc:4755] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9585ba97eaf14e069dcd2dcaf726a09b: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:08.139532   561 tablet_replica.cc:333] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.140719   561 raft_consensus.cc:2243] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.141539   561 raft_consensus.cc:2272] T 5e573c99734f4d91bb30d1fe72b56cf7 P a20e99e125b84513a44e88860667ed61 [term 22 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.145030   561 tablet_replica.cc:333] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.146026   561 raft_consensus.cc:2243] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.146759   561 raft_consensus.cc:2272] T 4f27ba9b299f46b7a89ef2d660780f8d P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.150066   561 tablet_replica.cc:333] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.151285   561 raft_consensus.cc:2243] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.152118   561 raft_consensus.cc:2272] T 82c4d9259f1a44119e802a40207d64b5 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.159520   561 tablet_replica.cc:333] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.160753   561 raft_consensus.cc:2243] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.162232   561 raft_consensus.cc:2272] T ae12891834034d968900107cc58fa821 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.165601   561 tablet_replica.cc:333] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.166822   561 raft_consensus.cc:2243] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.167799   561 raft_consensus.cc:2272] T 23caea311a774ba3b29b7749bd3f1cd6 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.188469   561 tablet_replica.cc:333] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.189720   561 raft_consensus.cc:2243] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.190747   561 raft_consensus.cc:2272] T 2b797b0728dc4e01976ab61ef164fa90 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.193962   561 tablet_replica.cc:333] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.195341   561 raft_consensus.cc:2243] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.196154   561 raft_consensus.cc:2272] T 7295d6494bad4378b63658f8274dfde8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.208004   561 tablet_replica.cc:333] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.209033   561 raft_consensus.cc:2243] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.210240   561 raft_consensus.cc:2272] T 8bb813e4fab14a61b810a24b56899586 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.211338 12652 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:08.211984 12652 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:08.248510  2383 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:08.251814 12652 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:08.255970  2382 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:08.257330  2382 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:08.261296 12652 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:27:08.292647 12635 raft_consensus.cc:493] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:08.293375 12635 raft_consensus.cc:515] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
W20260709 12:27:08.295912  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:08.296991 12635 leader_election.cc:290] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:08.310559   561 tablet_replica.cc:333] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.318431   561 raft_consensus.cc:2243] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.322909   561 raft_consensus.cc:2272] T c072b820694347cabf7868a1b3805f76 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.326905   561 tablet_replica.cc:333] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.328509   561 raft_consensus.cc:2243] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Raft consensus shutting down.
W20260709 12:27:08.330441  1844 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:08.335877  1845 consensus_peers.cc:597] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:08.337508  1845 leader_election.cc:336] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:08.338155  1845 leader_election.cc:304] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:08.336843   561 raft_consensus.cc:2272] T aca584e2f1114809bcadeaf6faad34f8 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.349133 12615 raft_consensus.cc:2749] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:08.387606  2383 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:08.399700   561 tablet_replica.cc:333] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.408879   561 raft_consensus.cc:2243] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.410212   561 raft_consensus.cc:2272] T 80ed6fcb806a4023b94ecd7b266b421e P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:08.416858  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:08.422328  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 12:27:08.435572   561 tablet_replica.cc:333] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.436511   561 raft_consensus.cc:2243] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.437193   561 raft_consensus.cc:2272] T a534e4794d9548638a2781e4dc451bc6 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.440335   561 tablet_replica.cc:333] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.441206   561 raft_consensus.cc:2243] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.442914   561 raft_consensus.cc:2272] T 6388b1a5243d463589750794e8ac5511 P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.445888   561 tablet_replica.cc:333] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.449110   561 raft_consensus.cc:2243] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.450917   561 raft_consensus.cc:2272] T b1cb2a27e8444930b93f1e21c2deb19c P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.454180 12615 raft_consensus.cc:493] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:08.454690 12615 raft_consensus.cc:515] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } }
I20260709 12:27:08.456768 12615 leader_election.cc:290] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:08.457448  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:27:08.466897   561 tablet_replica.cc:333] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.467928   561 raft_consensus.cc:2243] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 32 LEADER]: Raft consensus shutting down.
W20260709 12:27:08.471740  1845 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:08.477155   561 raft_consensus.cc:2272] T 9585ba97eaf14e069dcd2dcaf726a09b P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.480593   561 tablet_replica.cc:333] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.481549   561 raft_consensus.cc:2243] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.482240   561 raft_consensus.cc:2272] T bb48ef23a7434834929994987a36f360 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.485021   561 tablet_replica.cc:333] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.485770   561 raft_consensus.cc:2243] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.487015   561 raft_consensus.cc:2272] T b9c190a9868442e4a7e6d84d752329f3 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.492519   561 tablet_replica.cc:333] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.493453   561 raft_consensus.cc:2243] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.494490   561 raft_consensus.cc:2272] T f15e08b3d9804a00b9836e5223042266 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.497344   561 tablet_replica.cc:333] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.498163   561 raft_consensus.cc:2243] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.498904   561 raft_consensus.cc:2272] T 2e89c188ae524548ae35a90336c0c92c P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:08.502434  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20260709 12:27:08.502954   561 tablet_replica.cc:333] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.503755   561 raft_consensus.cc:2243] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.504887   561 raft_consensus.cc:2272] T 8c92163822f64868a647b688599f49bd P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.507493   561 tablet_replica.cc:333] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.508306   561 raft_consensus.cc:2243] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.509155   561 raft_consensus.cc:2272] T bc4fe4ebd84a44b9bee2535ccd62ba22 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:08.511495  1844 leader_election.cc:336] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:08.511994  1844 leader_election.cc:304] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a20e99e125b84513a44e88860667ed61; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:08.513140 12638 raft_consensus.cc:2749] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:27:08.516616   561 tablet_replica.cc:333] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.517561   561 raft_consensus.cc:2243] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.518959   561 raft_consensus.cc:2272] T 5280e62b42a141359d53bd4f241659ef P a20e99e125b84513a44e88860667ed61 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.522334   561 tablet_replica.cc:333] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.523275   561 raft_consensus.cc:2243] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 25 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.524850   561 raft_consensus.cc:2272] T 65799458c3fb44e7a162099338c0bb62 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.528131   561 tablet_replica.cc:333] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.528987   561 raft_consensus.cc:2243] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.530432   561 raft_consensus.cc:2272] T 1bd4f4834ed2497290417394b678edc0 P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.542028   561 tablet_replica.cc:333] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.542868   561 raft_consensus.cc:2243] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.543977   561 raft_consensus.cc:2272] T 513580259d0a4e6d96fb16084d5a90dc P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.546846   561 tablet_replica.cc:333] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.547717   561 raft_consensus.cc:2243] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.548825   561 raft_consensus.cc:2272] T 034c167cd4e24138be60353036ce8092 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.552121   561 tablet_replica.cc:333] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.552977   561 raft_consensus.cc:2243] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.554279   561 raft_consensus.cc:2272] T 62f288182b21420f8ef28e25fc10cd70 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.557476   561 tablet_replica.cc:333] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.558318   561 raft_consensus.cc:2243] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.558959   561 raft_consensus.cc:2272] T 36f1f37b7d544bb29cd9cace84f6e946 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.561959   561 tablet_replica.cc:333] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.562753   561 raft_consensus.cc:2243] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.565299   561 raft_consensus.cc:2272] T 908089442ca44109875257bfbc9d06c2 P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.568432   561 tablet_replica.cc:333] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.569351   561 raft_consensus.cc:2243] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.570726   561 raft_consensus.cc:2272] T ae23688e76624153b5a7089586669bb0 P a20e99e125b84513a44e88860667ed61 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.573854   561 tablet_replica.cc:333] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.574720   561 raft_consensus.cc:2243] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.575439   561 raft_consensus.cc:2272] T 9b8cae5d30dd4a68a4fba5846e83873e P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.578329   561 tablet_replica.cc:333] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.579188   561 raft_consensus.cc:2243] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.580173   561 raft_consensus.cc:2272] T 9875eb35aaed48d7b2670b5ea57e3948 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.582896   561 tablet_replica.cc:333] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.583691   561 raft_consensus.cc:2243] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.584968   561 raft_consensus.cc:2272] T 3e3f88cbac3d4fa4bdb3a23367a0e160 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.587970   561 tablet_replica.cc:333] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.588896   561 raft_consensus.cc:2243] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.590277   561 raft_consensus.cc:2272] T 1b33fb206f534e1e890a4546d3d43e84 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.593030   561 tablet_replica.cc:333] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.594736   561 raft_consensus.cc:2243] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.595450   561 raft_consensus.cc:2272] T 6f86194bda894372ab64d2c6a9937a5b P a20e99e125b84513a44e88860667ed61 [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.598194   561 tablet_replica.cc:333] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.598922   561 raft_consensus.cc:2243] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.600128   561 raft_consensus.cc:2272] T bc103cf2bf8545cf8f5f5f34e38ad0e1 P a20e99e125b84513a44e88860667ed61 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.602967   561 tablet_replica.cc:333] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.603688   561 raft_consensus.cc:2243] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.605386   561 raft_consensus.cc:2272] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.608501   561 tablet_replica.cc:333] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.609328   561 raft_consensus.cc:2243] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.610067   561 raft_consensus.cc:2272] T efbb8c187a4e410aa09e81ceb9ebecc3 P a20e99e125b84513a44e88860667ed61 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.612779   561 tablet_replica.cc:333] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.613517   561 raft_consensus.cc:2243] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.614822   561 raft_consensus.cc:2272] T c5f3d785e05841fbb0efa210c3d9675c P a20e99e125b84513a44e88860667ed61 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:08.616511  2383 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:08.619165   561 tablet_replica.cc:333] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.620157   561 raft_consensus.cc:2243] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 23 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.621335   561 raft_consensus.cc:2272] T bbbcc16cebc244769e7f1a1b6e29594c P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.624500   561 tablet_replica.cc:333] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.625255   561 raft_consensus.cc:2243] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.625949   561 raft_consensus.cc:2272] T 36bc2f66a73947468d61113c226ed318 P a20e99e125b84513a44e88860667ed61 [term 23 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.628798   561 tablet_replica.cc:333] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.629474   561 raft_consensus.cc:2243] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.630347   561 raft_consensus.cc:2272] T ea5d8f652fc448d6af846ff6850004b8 P a20e99e125b84513a44e88860667ed61 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.633270   561 tablet_replica.cc:333] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.634004   561 raft_consensus.cc:2243] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.635753   561 raft_consensus.cc:2272] T 76f6bc3c4cce45d7855e6b3429341da2 P a20e99e125b84513a44e88860667ed61 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.638738   561 tablet_replica.cc:333] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.639470   561 raft_consensus.cc:2243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.640136   561 raft_consensus.cc:2272] T a4c45f93428e4cc68f85f17ea3ec3ad0 P a20e99e125b84513a44e88860667ed61 [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.643249   561 tablet_replica.cc:333] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.643965   561 raft_consensus.cc:2243] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:08.645272   561 raft_consensus.cc:2272] T bd91edf20ef54a30a2f5af0c0b682350 P a20e99e125b84513a44e88860667ed61 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.648061   561 tablet_replica.cc:333] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61: stopping tablet replica
I20260709 12:27:08.648929   561 raft_consensus.cc:2243] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:27:08.650260   561 raft_consensus.cc:2272] T b3034677490f4f0fac2a4ed3e2a8142b P a20e99e125b84513a44e88860667ed61 [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:08.673518 12645 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:08.674126 12645 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:08.676667 12645 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:08.688109  2382 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:08.690714  2383 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:08.691407  2383 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:08.692387 12645 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:08.811517  2383 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:08.857021  2383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111) [suppressed 647 similar messages]
W20260709 12:27:08.931689  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:08.932533  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:08.985030  2383 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:09.001597  2383 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:09.024745 12652 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:09.025455 12652 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:09.028738 12652 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:09.038733  2382 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:09.049705  2383 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:09.055091  2383 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:09.055622  2383 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:09.065372 12652 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:27:09.067687  2383 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:09.163415  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:09.268290  2383 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:09.269600  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:09.339002  2383 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:09.375921  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:09.395403  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:09.414610  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260709 12:27:09.438804 12652 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.439414 12652 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:09.441615 12652 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:09.469070  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:09.472165  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:09.472713  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:09.473500 12652 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:27:09.474735  2383 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:09.475546  2383 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:09.519440 12652 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.520059 12652 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:09.522356 12652 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:09.535353  2383 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:09.538407  2383 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:09.538897  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:09.539762 12652 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:27:09.749761 12656 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.750453 12656 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:09.759202  2382 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:09.760578 12656 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:09.768927  2383 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:09.769448  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:09.770941 12656 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:09.793663   561 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 1902)
I20260709 12:27:09.806464 12656 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.807168 12656 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:09.824033  2382 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:09.825063  2383 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:09.825605  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:09.825104 12656 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:09.826423 12652 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:27:09.832036 12652 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.832693 12652 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:09.835318 12652 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:09.855381  2383 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:09.859509 12652 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.860229 12652 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:09.864940  2383 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:09.865520  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:09.866456 12656 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:27:09.867141 12645 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.867460 12656 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.867740 12645 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:09.867983 12656 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:09.870081 12645 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:09.870049 12656 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:09.875976 12652 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:09.908685 12657 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:09.909374 12657 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:09.918092 12657 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:09.918671  2382 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:09.929114  2383 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:09.929842  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:09.930505  2383 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:09.933662  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:09.934180  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:09.935151 12657 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:27:09.935359  2383 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:09.936106  2383 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:09.937050 12657 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:27:09.938364  2383 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:09.938966  2383 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:09.939874 12657 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:27:09.948625  2383 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:09.949378  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:09.950381 12657 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:09.955257  2383 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:09.966737 12657 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.967401 12657 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:09.969929 12657 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:09.976641  2382 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:09.977890 12652 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:09.978699 12652 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:09.981132 12652 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:10.007748  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:10.009493  2383 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:10.010248  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:10.011579 12652 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:27:10.013381  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:10.014410  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:10.022372  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:10.023368 12652 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:27:10.036995 12652 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:10.037695 12652 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:10.045557 12652 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:10.054394  2382 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:10.090431  2383 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:10.091147  2383 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:10.092113 12652 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:10.096594  2383 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:10.177819  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:10.245759  2383 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.344194  2383 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.371716  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:10.400729 12657 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:10.401397 12657 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:10.404035 12657 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:10.410243  2382 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:10.426678  2383 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:10.427701  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:10.428637 12657 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:10.434302  2383 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.435029  2383 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:10.444967 12657 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:10.445642 12657 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:10.458040 12657 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:10.460436  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:10.464787  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:10.468714  2382 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:10.493826  2383 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.504658  2383 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:10.506310  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
W20260709 12:27:10.505676  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260709 12:27:10.507630 12657 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:27:10.518859  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.519752  2383 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.532984  2383 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.570693  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:10.611109  2383 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.636828  2383 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.794344   561 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 1902)
W20260709 12:27:10.810200  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:10.827396  2383 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:10.886246  2383 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:10.941789  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:10.982734  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:27:11.042474  2383 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:11.062127  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:11.191592  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 12:27:11.309388 12658 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:11.310019 12658 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:11.313793 12658 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:11.320312  2382 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:11.330579  2383 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:11.336899  2383 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:11.337555  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:11.338421 12658 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:11.389298  2383 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:11.437343  2383 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:11.439386  2383 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:27:11.469554 12657 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:11.470171 12657 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:11.477211 12657 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:11.478765  2382 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:11.487355  2383 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:11.487960  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:11.488969 12657 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:27:11.572924  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:11.578966  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:27:11.650151 12657 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:11.650772 12657 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:11.653373 12657 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:11.668220 12657 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:11.669164 12657 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:11.685410 12657 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:11.686046 12658 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:11.687558 12658 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:11.698061 12658 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:11.686830  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:11.713284  2382 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:11.714373 12657 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:11.715565 12657 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:11.718606 12657 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:11.725135 12658 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:11.725833 12658 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:11.728592 12658 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:11.730235  2383 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:11.742901  2383 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:11.747097  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:11.748833 12658 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:11.763562  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:11.765787  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:11.766487  2383 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:11.766970  2383 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:11.767879 12657 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:11.774971  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:11.775614  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:11.776424 12657 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:27:11.777634  2383 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:11.789603  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:11.790355  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:11.795871   561 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 1902)
W20260709 12:27:11.815569  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:11.816319  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:11.817248 12657 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:27:11.818384  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:11.820782  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:11.821617 12657 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:27:11.835390  2383 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:11.840355  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:27:11.845520 12657 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:11.846253 12657 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:11.848980 12657 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:11.866314  2383 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:11.870052  2383 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:11.870657  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:11.871803 12657 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:27:11.875552  2383 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:12.437079 12657 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:12.444291 12657 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:12.454005 12657 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:12.478695 12657 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:12.488531 12657 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:12.495209 12661 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:12.496361 12661 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:12.510411 12661 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
I20260709 12:27:12.526242 12657 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:12.540000 12669 raft_consensus.cc:493] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:12.541077 12669 raft_consensus.cc:515] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } attrs { replace: true } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:12.544623 12669 leader_election.cc:290] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:12.549948  2383 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:12.555778  2382 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:12.556366  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20260709 12:27:12.557081  2382 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:12.562223  2383 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:12.562669  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260709 12:27:12.562822  2383 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:12.564061 12671 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:27:12.564742  2383 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:12.572729  2383 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:12.577075  2383 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:12.577610  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:12.578477 12671 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:27:12.580116  2383 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:12.580670  2383 leader_election.cc:304] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:12.581453 12671 raft_consensus.cc:2749] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:12.581635  2383 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:12.582245  2383 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:12.583057 12671 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:27:12.583392  2383 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:12.586622  2383 leader_election.cc:336] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:12.796617   561 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 1902)
W20260709 12:27:12.920450  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:12.927407  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260709 12:27:12.928541  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260709 12:27:12.932507  2383 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:12.934989 12671 raft_consensus.cc:493] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:12.935928 12671 raft_consensus.cc:515] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:12.939746 12671 leader_election.cc:290] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:12.944221  2382 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:12.947813  2383 leader_election.cc:336] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:12.948437  2383 leader_election.cc:304] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:12.949347 12657 raft_consensus.cc:2749] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:12.951851  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260709 12:27:12.963474  2383 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:13.027614  2383 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:13.039942 12657 raft_consensus.cc:493] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:13.041147 12657 raft_consensus.cc:515] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:13.043972 12657 leader_election.cc:290] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:13.046568  2383 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:13.051721  2383 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:13.053006  2382 leader_election.cc:336] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:13.053506  2382 leader_election.cc:304] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:13.054370 12657 raft_consensus.cc:2749] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:13.144508  2383 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:13.325443  2440 debug-util.cc:398] Leaking SignalData structure 0x7b08002e7160 after lost signal to thread 564
W20260709 12:27:13.327705  2440 debug-util.cc:398] Leaking SignalData structure 0x7b08003b21a0 after lost signal to thread 1664
W20260709 12:27:13.329794  2440 debug-util.cc:398] Leaking SignalData structure 0x7b08002df960 after lost signal to thread 2443
W20260709 12:27:13.388652  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:13.408878  2383 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:13.443878 12657 raft_consensus.cc:493] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:13.444790 12657 raft_consensus.cc:515] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:13.447476 12657 leader_election.cc:290] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:13.454401  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:13.460170  2382 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:13.464999  2383 leader_election.cc:336] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:13.465611  2383 leader_election.cc:304] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:13.466513 12657 raft_consensus.cc:2749] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:27:13.467878  2383 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:13.502892  2382 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260709 12:27:13.511134  2383 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:13.517971  2383 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:13.544790  2382 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260709 12:27:13.602398  2382 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260709 12:27:13.623302  2383 consensus_peers.cc:597] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:13.720116 12671 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:13.721238 12671 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
W20260709 12:27:13.733151  2382 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:13.735466 12671 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:13.749302  2383 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:13.749867  2383 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:13.750862 12671 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:27:13.752231  2383 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:13.797240   561 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 1902)
W20260709 12:27:13.808213  2382 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260709 12:27:13.816937 12671 raft_consensus.cc:493] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:13.817832 12671 raft_consensus.cc:515] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:13.820492 12671 leader_election.cc:290] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:13.855093  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:13.857138  2383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111) [suppressed 441 similar messages]
W20260709 12:27:13.858860  2383 leader_election.cc:336] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:13.859283  2383 leader_election.cc:304] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:13.860023 12671 raft_consensus.cc:2749] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:27:13.910454  2383 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:27:13.967746 12671 raft_consensus.cc:493] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:13.968832 12671 raft_consensus.cc:515] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } attrs { replace: true } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:13.978487 12671 leader_election.cc:290] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:13.984213  2382 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:14.013295  2382 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20260709 12:27:14.025516  2382 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260709 12:27:14.031616  2383 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:14.032315  2383 leader_election.cc:304] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:14.033154 12671 raft_consensus.cc:2749] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:27:14.035810  2383 leader_election.cc:336] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:14.038582 12671 raft_consensus.cc:493] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:14.039464 12671 raft_consensus.cc:515] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:14.048065 12671 leader_election.cc:290] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:14.048837  2382 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:14.072711  2383 leader_election.cc:336] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:14.086921  2382 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20260709 12:27:14.087845  2383 leader_election.cc:304] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:14.088699 12673 raft_consensus.cc:2749] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:27:14.090672  2383 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:14.135340  2383 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:14.135951  2383 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:14.286427  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:14.338604  2382 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260709 12:27:14.346338  2382 consensus_peers.cc:597] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20260709 12:27:14.416790  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:14.443879  2382 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260709 12:27:14.517757 12671 raft_consensus.cc:493] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:14.518692 12671 raft_consensus.cc:515] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:14.520967 12671 leader_election.cc:290] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:14.537020  2383 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:14.537770  2383 leader_election.cc:336] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:14.538184  2383 leader_election.cc:304] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:14.539006 12671 raft_consensus.cc:2749] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:27:14.550287 12671 raft_consensus.cc:493] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:14.550966 12671 raft_consensus.cc:515] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:14.553092 12671 leader_election.cc:290] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:14.560568 12673 raft_consensus.cc:493] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:14.561326 12673 raft_consensus.cc:515] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:14.563606 12673 leader_election.cc:290] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
I20260709 12:27:14.564343 12657 raft_consensus.cc:493] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:14.565249 12657 raft_consensus.cc:515] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:14.567675 12657 leader_election.cc:290] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:14.585400  2383 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:14.587177  2383 leader_election.cc:336] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:14.587837  2383 leader_election.cc:304] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:14.588759 12657 raft_consensus.cc:2749] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:27:14.597886  2383 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:14.598542  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
W20260709 12:27:14.601258  2383 leader_election.cc:336] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:14.608405  2383 leader_election.cc:304] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:14.609166 12657 raft_consensus.cc:2749] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:27:14.610275  2383 consensus_peers.cc:597] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:14.610770  2383 leader_election.cc:336] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:14.611223  2383 leader_election.cc:304] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:14.612033 12657 raft_consensus.cc:2749] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:27:14.626490  2383 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:27:14.627115  2383 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:14.672078  2383 consensus_peers.cc:597] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:14.716089  2382 consensus_peers.cc:597] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260709 12:27:15.029115  2382 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20260709 12:27:15.044483 12657 raft_consensus.cc:493] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:15.045274 12657 raft_consensus.cc:515] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:15.047436 12657 leader_election.cc:290] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:15.079972  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:15.080788  2383 leader_election.cc:336] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:15.081182  2383 leader_election.cc:304] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:15.081910 12657 raft_consensus.cc:2749] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:27:15.088593  2383 consensus_peers.cc:597] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:15.117264  2383 consensus_peers.cc:597] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e -> Peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Couldn't send request to peer 6084fbc6918e4e0b95a9bd0206a620d8. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:27:15.167166  2383 consensus_peers.cc:597] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:15.173811  2383 consensus_peers.cc:597] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:15.257409  2382 consensus_peers.cc:597] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260709 12:27:15.265962  2383 consensus_peers.cc:597] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:15.358938 12676 raft_consensus.cc:493] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:15.359689 12676 raft_consensus.cc:515] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:15.362320 12676 leader_election.cc:290] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:15.379158  2383 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
W20260709 12:27:15.381599  2383 leader_election.cc:336] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:15.382136  2383 leader_election.cc:304] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 6084fbc6918e4e0b95a9bd0206a620d8, a20e99e125b84513a44e88860667ed61
I20260709 12:27:15.383091 12676 raft_consensus.cc:2749] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:27:15.509848  2382 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260709 12:27:15.566592  2383 consensus_peers.cc:597] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:15.568780  2382 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20260709 12:27:15.607051 12657 raft_consensus.cc:493] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:15.607683 12657 raft_consensus.cc:515] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } attrs { replace: true } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:15.610193 12657 leader_election.cc:290] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:15.619402  2382 consensus_peers.cc:597] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e -> Peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Couldn't send request to peer 075704438992440697732181e49ccd09. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20260709 12:27:15.619977  2383 consensus_peers.cc:597] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:15.629050  2382 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:15.637135  2383 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:15.638406  2383 leader_election.cc:304] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:15.640772 12657 raft_consensus.cc:2749] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:27:15.641974  2383 leader_election.cc:336] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:15.681092 12657 raft_consensus.cc:493] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 6084fbc6918e4e0b95a9bd0206a620d8)
I20260709 12:27:15.681738 12657 raft_consensus.cc:515] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "6084fbc6918e4e0b95a9bd0206a620d8" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 40361 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:15.683880 12657 leader_election.cc:290] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361)
W20260709 12:27:15.690861  2382 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
I20260709 12:27:15.698580   561 tablet_server.cc:196] TabletServer@127.0.140.67:0 shutdown complete.
W20260709 12:27:15.702525  2383 leader_election.cc:336] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:15.716548  2383 leader_election.cc:304] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, 6084fbc6918e4e0b95a9bd0206a620d8
I20260709 12:27:15.717629 12657 raft_consensus.cc:2749] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:27:15.729856  2383 consensus_peers.cc:597] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:15.856294  2383 consensus_peers.cc:597] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:15.872728  2383 consensus_peers.cc:597] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:15.896392   561 tablet_server.cc:179] TabletServer@127.0.140.68:0 shutting down...
W20260709 12:27:15.949373  2383 consensus_peers.cc:597] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:15.951509 12657 raft_consensus.cc:493] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election (detected failure of leader a20e99e125b84513a44e88860667ed61)
I20260709 12:27:15.952071 12657 raft_consensus.cc:515] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "075704438992440697732181e49ccd09" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 35067 } } peers { permanent_uuid: "a20e99e125b84513a44e88860667ed61" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35761 } } peers { permanent_uuid: "123095127f334445889c21a42ba1d41e" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 37665 } attrs { promote: false } }
I20260709 12:27:15.954241 12657 leader_election.cc:290] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 075704438992440697732181e49ccd09 (127.0.140.65:35067), a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761)
W20260709 12:27:15.958987  2382 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 075704438992440697732181e49ccd09 (127.0.140.65:35067): Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:15.966281  2383 consensus_peers.cc:597] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:15.966907  2383 leader_election.cc:336] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111)
I20260709 12:27:15.967322  2383 leader_election.cc:304] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 123095127f334445889c21a42ba1d41e; no voters: 075704438992440697732181e49ccd09, a20e99e125b84513a44e88860667ed61
I20260709 12:27:15.968122 12657 raft_consensus.cc:2749] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:27:15.970743   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:15.973100   561 tablet_replica.cc:333] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:15.974251   561 raft_consensus.cc:2243] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.975528   561 raft_consensus.cc:2272] T 12cfc1bb936b41fbbc9cdfafc0cd9470 P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:15.979403  2383 consensus_peers.cc:597] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e -> Peer a20e99e125b84513a44e88860667ed61 (127.0.140.67:35761): Couldn't send request to peer a20e99e125b84513a44e88860667ed61. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:35761: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:27:15.980494   561 tablet_replica.cc:333] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:15.981312   561 raft_consensus.cc:2243] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.982681   561 raft_consensus.cc:2272] T 908089442ca44109875257bfbc9d06c2 P 123095127f334445889c21a42ba1d41e [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.986261   561 tablet_replica.cc:333] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:15.987215   561 raft_consensus.cc:2243] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:15.988404   561 raft_consensus.cc:2272] T c5f3d785e05841fbb0efa210c3d9675c P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.991048   561 tablet_replica.cc:333] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:15.991662   561 raft_consensus.cc:2243] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.992911   561 raft_consensus.cc:2272] T ea5d8f652fc448d6af846ff6850004b8 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:15.995549   561 tablet_replica.cc:333] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:15.996273   561 raft_consensus.cc:2243] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Raft consensus shutting down.
I20260709 12:27:15.997644   561 raft_consensus.cc:2272] T def0036446324543ba07e132e885133a P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.000327   561 tablet_replica.cc:333] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.001006   561 raft_consensus.cc:2243] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.002743   561 raft_consensus.cc:2272] T d891ed30ca204d09b7b083088037b63d P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.006233   561 tablet_replica.cc:333] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.007110   561 raft_consensus.cc:2243] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.007890   561 raft_consensus.cc:2272] T 8bb813e4fab14a61b810a24b56899586 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.010329   561 tablet_replica.cc:333] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.011022   561 raft_consensus.cc:2243] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.012317   561 raft_consensus.cc:2272] T a4c45f93428e4cc68f85f17ea3ec3ad0 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.014662   561 tablet_replica.cc:333] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.015353   561 raft_consensus.cc:2243] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.016396   561 raft_consensus.cc:2272] T 6388b1a5243d463589750794e8ac5511 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.018497   561 tablet_replica.cc:333] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.019166   561 raft_consensus.cc:2243] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 24 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.020354   561 raft_consensus.cc:2272] T 66d3405528d44edb8c0be3b296ef72be P 123095127f334445889c21a42ba1d41e [term 24 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.023006   561 tablet_replica.cc:333] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.023663   561 raft_consensus.cc:2243] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.024571   561 raft_consensus.cc:2272] T 80ed6fcb806a4023b94ecd7b266b421e P 123095127f334445889c21a42ba1d41e [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.027369   561 tablet_replica.cc:333] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.028012   561 raft_consensus.cc:2243] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.028672   561 raft_consensus.cc:2272] T b9c190a9868442e4a7e6d84d752329f3 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.031258   561 tablet_replica.cc:333] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.031853   561 raft_consensus.cc:2243] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.033028   561 raft_consensus.cc:2272] T 8f285ce2457a47ebbd05920b42b35fba P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.035267   561 tablet_replica.cc:333] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.035863   561 raft_consensus.cc:2243] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.037060   561 raft_consensus.cc:2272] T 7a2d4ac0ec9e4adcb3adb743647e0e78 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.039793   561 tablet_replica.cc:333] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.040624   561 raft_consensus.cc:2243] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.042107   561 raft_consensus.cc:2272] T 2b797b0728dc4e01976ab61ef164fa90 P 123095127f334445889c21a42ba1d41e [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.045055   561 tablet_replica.cc:333] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.045871   561 raft_consensus.cc:2243] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.047348   561 raft_consensus.cc:2272] T c072b820694347cabf7868a1b3805f76 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.049773   561 tablet_replica.cc:333] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.050367   561 raft_consensus.cc:2243] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.051445   561 raft_consensus.cc:2272] T ae23688e76624153b5a7089586669bb0 P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.053624   561 tablet_replica.cc:333] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.054291   561 raft_consensus.cc:2243] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 26 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.055572   561 raft_consensus.cc:2272] T 9b8cae5d30dd4a68a4fba5846e83873e P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.057611   561 tablet_replica.cc:333] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.058182   561 raft_consensus.cc:2243] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.059386   561 raft_consensus.cc:2272] T bd91edf20ef54a30a2f5af0c0b682350 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.061595   561 tablet_replica.cc:333] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.062186   561 raft_consensus.cc:2243] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 25 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.063302   561 raft_consensus.cc:2272] T 4a57b66dec794db09a7f9048a3d0de10 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.065315   561 tablet_replica.cc:333] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.065866   561 raft_consensus.cc:2243] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.066429   561 raft_consensus.cc:2272] T 1bd4f4834ed2497290417394b678edc0 P 123095127f334445889c21a42ba1d41e [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.068567   561 tablet_replica.cc:333] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.069142   561 raft_consensus.cc:2243] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.069761   561 raft_consensus.cc:2272] T 65799458c3fb44e7a162099338c0bb62 P 123095127f334445889c21a42ba1d41e [term 25 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.071813   561 tablet_replica.cc:333] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.072417   561 raft_consensus.cc:2243] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.073586   561 raft_consensus.cc:2272] T 7b550688b5e84ddda5fcf049010ac1e8 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.075837   561 tablet_replica.cc:333] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.076435   561 raft_consensus.cc:2243] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.077533   561 raft_consensus.cc:2272] T 23caea311a774ba3b29b7749bd3f1cd6 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.080129   561 tablet_replica.cc:333] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.080916   561 raft_consensus.cc:2243] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.081749   561 raft_consensus.cc:2272] T 513580259d0a4e6d96fb16084d5a90dc P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.084051   561 tablet_replica.cc:333] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.084692   561 raft_consensus.cc:2243] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.085731   561 raft_consensus.cc:2272] T 9585ba97eaf14e069dcd2dcaf726a09b P 123095127f334445889c21a42ba1d41e [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.088202   561 tablet_replica.cc:333] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.088733   561 raft_consensus.cc:2243] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.089923   561 raft_consensus.cc:2272] T 36f1f37b7d544bb29cd9cace84f6e946 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.092061   561 tablet_replica.cc:333] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.092659   561 raft_consensus.cc:2243] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.093572   561 raft_consensus.cc:2272] T 62f288182b21420f8ef28e25fc10cd70 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.095712   561 tablet_replica.cc:333] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.096268   561 raft_consensus.cc:2243] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.097424   561 raft_consensus.cc:2272] T 4f27ba9b299f46b7a89ef2d660780f8d P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.099458   561 tablet_replica.cc:333] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.100042   561 raft_consensus.cc:2243] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.101446   561 raft_consensus.cc:2272] T 9875eb35aaed48d7b2670b5ea57e3948 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.103672   561 tablet_replica.cc:333] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.104228   561 raft_consensus.cc:2243] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.105348   561 raft_consensus.cc:2272] T 79f5c05e64b945cbacc02914f31c5762 P 123095127f334445889c21a42ba1d41e [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.107725   561 tablet_replica.cc:333] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.108264   561 raft_consensus.cc:2243] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.108973   561 raft_consensus.cc:2272] T 8c92163822f64868a647b688599f49bd P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.111104   561 tablet_replica.cc:333] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.111620   561 raft_consensus.cc:2243] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.112577   561 raft_consensus.cc:2272] T 82eacf0c2a8c4d8db4052327304a8793 P 123095127f334445889c21a42ba1d41e [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.114467   561 tablet_replica.cc:333] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.115021   561 raft_consensus.cc:2243] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.115760   561 raft_consensus.cc:2272] T c22ef2bc934c4b7ea57aad5853ab6122 P 123095127f334445889c21a42ba1d41e [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.117722   561 tablet_replica.cc:333] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.118227   561 raft_consensus.cc:2243] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.118855   561 raft_consensus.cc:2272] T 84ce9896327e4c9d9888e80e8dbeeb0d P 123095127f334445889c21a42ba1d41e [term 26 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.120831   561 tablet_replica.cc:333] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e: stopping tablet replica
I20260709 12:27:16.121296   561 raft_consensus.cc:2243] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.122234   561 raft_consensus.cc:2272] T 034c167cd4e24138be60353036ce8092 P 123095127f334445889c21a42ba1d41e [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.645697   561 tablet_server.cc:196] TabletServer@127.0.140.68:0 shutdown complete.
I20260709 12:27:16.717341   561 master.cc:562] Master@127.0.140.126:43371 shutting down...
W20260709 12:27:16.763574  1678 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 12:27:16.764067  1678 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:27:16.764330  1678 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:27:24.532480  1614 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111) [suppressed 208 similar messages]
W20260709 12:27:24.539458  1614 catalog_manager.cc:4755] TS 075704438992440697732181e49ccd09 (127.0.140.65:35067): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9585ba97eaf14e069dcd2dcaf726a09b: Network error: Client connection negotiation failed: client connection to 127.0.140.65:35067: connect: Connection refused (error 111)
W20260709 12:27:31.336215  1615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260709 12:27:31.340451  1615 catalog_manager.cc:4755] TS 6084fbc6918e4e0b95a9bd0206a620d8 (127.0.140.66:40361): ChangeConfig:REMOVE_PEER RPC failed for tablet 8bb813e4fab14a61b810a24b56899586: Network error: Client connection negotiation failed: client connection to 127.0.140.66:40361: connect: Connection refused (error 111)
I20260709 12:27:33.778330   561 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:33.779642   561 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:33.780031   561 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4002daa3304c4fb4be053abd4780c195: stopping tablet replica
I20260709 12:27:34.041374   561 master.cc:584] Master@127.0.140.126:43371 shutdown complete.
[       OK ] LeaderRebalancerTest.AddTserver (366487 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 12:27:34.179405   561 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.140.126:45447
I20260709 12:27:34.180967   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:34.188460 12684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:34.188800 12683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:34.189123 12686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:34.191296   561 server_base.cc:1061] running on GCE node
I20260709 12:27:34.197029   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:34.197304   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:34.197418   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600054197406 us; error 0 us; skew 500 ppm
I20260709 12:27:34.197990   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:34.201318   561 webserver.cc:533] Webserver started at http://127.0.140.126:34211/ using document root <none> and password file <none>
I20260709 12:27:34.201887   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:34.202070   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:34.202311   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:34.203945   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/master-0-root/instance:
uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.209312   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:27:34.213526 12691 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.214490   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:27:34.215668   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/master-0-root
uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.216020   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:34.241027   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:34.242620   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:34.291738   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.126:45447
I20260709 12:27:34.291878 12742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.126:45447 every 8 connection(s)
I20260709 12:27:34.296607 12743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:34.308576 12743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663: Bootstrap starting.
I20260709 12:27:34.313534 12743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:34.318382 12743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663: No bootstrap required, opened a new log
I20260709 12:27:34.320658 12743 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663" member_type: VOTER }
I20260709 12:27:34.321154 12743 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:34.321345 12743 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8de3ba5eb28a4b36a9640e3eb6cb4663, State: Initialized, Role: FOLLOWER
I20260709 12:27:34.321868 12743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663" member_type: VOTER }
I20260709 12:27:34.322316 12743 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:27:34.322530 12743 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:27:34.322752 12743 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:34.327538 12743 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663" member_type: VOTER }
I20260709 12:27:34.328049 12743 leader_election.cc:304] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8de3ba5eb28a4b36a9640e3eb6cb4663; no voters: 
I20260709 12:27:34.329267 12743 leader_election.cc:290] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:27:34.329962 12746 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:34.331851 12746 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 1 LEADER]: Becoming Leader. State: Replica: 8de3ba5eb28a4b36a9640e3eb6cb4663, State: Running, Role: LEADER
I20260709 12:27:34.332618 12743 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:27:34.332695 12746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663" member_type: VOTER }
I20260709 12:27:34.344048 12748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8de3ba5eb28a4b36a9640e3eb6cb4663. Latest consensus state: current_term: 1 leader_uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663" member_type: VOTER } }
I20260709 12:27:34.344187 12747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8de3ba5eb28a4b36a9640e3eb6cb4663" member_type: VOTER } }
I20260709 12:27:34.345278 12748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:34.345395 12747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:34.358006 12754 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:27:34.359283   561 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:27:34.363855 12754 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:27:34.374740 12754 catalog_manager.cc:1383] Generated new cluster ID: 51525c79fa544f449fca1a5c2167760c
I20260709 12:27:34.375211 12754 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:27:34.393725 12754 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:27:34.395479 12754 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:27:34.420640 12754 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663: Generated new TSK 0
I20260709 12:27:34.421449 12754 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:27:34.491365   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:34.499080 12764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:34.500331 12765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:34.504027 12767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:34.505095   561 server_base.cc:1061] running on GCE node
I20260709 12:27:34.507938   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:34.508152   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:34.508401   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600054508361 us; error 0 us; skew 500 ppm
I20260709 12:27:34.509016   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:34.511968   561 webserver.cc:533] Webserver started at http://127.0.140.65:35837/ using document root <none> and password file <none>
I20260709 12:27:34.512548   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:34.512750   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:34.513046   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:34.514285   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-0-root/instance:
uuid: "b543f973837c43b19b7d17feaaca5325"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.519475   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20260709 12:27:34.523751 12772 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.524703   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:27:34.525101   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "b543f973837c43b19b7d17feaaca5325"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.525520   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:34.540640   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:34.541884   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:34.543548   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:34.546130   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:34.546366   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.546622   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:34.546831   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:27:34.595322   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:40461
I20260709 12:27:34.595474 12834 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:40461 every 8 connection(s)
I20260709 12:27:34.600701   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:34.610277 12839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:34.613093 12840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:34.613951 12842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:34.615370   561 server_base.cc:1061] running on GCE node
I20260709 12:27:34.619987   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:34.620393   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:34.620626   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600054620601 us; error 0 us; skew 500 ppm
I20260709 12:27:34.621400   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:34.621879 12835 heartbeater.cc:344] Connected to a master server at 127.0.140.126:45447
I20260709 12:27:34.622366 12835 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:34.623626 12835 heartbeater.cc:507] Master 127.0.140.126:45447 requested a full tablet report, sending...
I20260709 12:27:34.624738   561 webserver.cc:533] Webserver started at http://127.0.140.66:43407/ using document root <none> and password file <none>
I20260709 12:27:34.625438   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:34.625664   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:34.625957   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:34.626606 12708 ts_manager.cc:194] Registered new tserver with Master: b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:34.627542   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-1-root/instance:
uuid: "8a995dc08b2a4497b71db084a2641e6f"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.629000 12708 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55086
I20260709 12:27:34.633227   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:27:34.638226 12847 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.639556   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:34.639952   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-1-root
uuid: "8a995dc08b2a4497b71db084a2641e6f"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.640281   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:34.660192   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:34.661520   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:34.663231   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:34.665750   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:34.666028   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.666285   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:34.666556   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.715672   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.66:38975
I20260709 12:27:34.715865 12909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.66:38975 every 8 connection(s)
I20260709 12:27:34.721940   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:34.731807 12913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:34.734431 12914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:34.738575   561 server_base.cc:1061] running on GCE node
W20260709 12:27:34.740103 12916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:34.740310 12910 heartbeater.cc:344] Connected to a master server at 127.0.140.126:45447
I20260709 12:27:34.741024 12910 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:34.741613   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:34.741945   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:34.742163   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600054742144 us; error 0 us; skew 500 ppm
I20260709 12:27:34.742204 12910 heartbeater.cc:507] Master 127.0.140.126:45447 requested a full tablet report, sending...
I20260709 12:27:34.743453   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:34.745370 12708 ts_manager.cc:194] Registered new tserver with Master: 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:34.746500   561 webserver.cc:533] Webserver started at http://127.0.140.67:33475/ using document root <none> and password file <none>
I20260709 12:27:34.747232   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:34.747480   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:34.747392 12708 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55092
I20260709 12:27:34.747957   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:34.749365   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-2-root/instance:
uuid: "8012bb9a8b274890bcaad0d192cd30d5"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.754980   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:27:34.758729 12921 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.759724   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:34.760113   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-2-root
uuid: "8012bb9a8b274890bcaad0d192cd30d5"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.760515   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:34.776494   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:34.777825   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:34.779551   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:34.781975   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:34.782222   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.782477   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:34.782651   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.832787   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.67:32861
I20260709 12:27:34.832959 12983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.67:32861 every 8 connection(s)
I20260709 12:27:34.838155   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:34.846767 12987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:34.849030 12988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:34.855069 12990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:34.855986 12984 heartbeater.cc:344] Connected to a master server at 127.0.140.126:45447
I20260709 12:27:34.856492 12984 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:34.856635   561 server_base.cc:1061] running on GCE node
I20260709 12:27:34.857577 12984 heartbeater.cc:507] Master 127.0.140.126:45447 requested a full tablet report, sending...
I20260709 12:27:34.857811   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:34.858204   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:34.858412   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600054858392 us; error 0 us; skew 500 ppm
I20260709 12:27:34.859184   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:34.860422 12708 ts_manager.cc:194] Registered new tserver with Master: 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:34.862823   561 webserver.cc:533] Webserver started at http://127.0.140.68:35193/ using document root <none> and password file <none>
I20260709 12:27:34.862881 12708 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55100
I20260709 12:27:34.863831   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:34.864159   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:34.864506   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:34.865963   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-3-root/instance:
uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.871222   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:27:34.875349 12995 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.876401   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 12:27:34.876802   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-3-root
uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:34.877188   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:34.893098   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:34.894709   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:34.896652   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:34.899518   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:34.899789   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.900075   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:34.900271   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:34.954082   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.68:32909
I20260709 12:27:34.954174 13057 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.68:32909 every 8 connection(s)
I20260709 12:27:34.971618 13058 heartbeater.cc:344] Connected to a master server at 127.0.140.126:45447
I20260709 12:27:34.972084 13058 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:34.973001 13058 heartbeater.cc:507] Master 127.0.140.126:45447 requested a full tablet report, sending...
I20260709 12:27:34.975550 12708 ts_manager.cc:194] Registered new tserver with Master: 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:34.975751   561 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.017517118s
I20260709 12:27:34.977800 12708 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55116
I20260709 12:27:35.011615 12708 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55122:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 12:27:35.289840 13023 tablet_service.cc:1511] Processing CreateTablet for tablet 81ab261b0dc04756a70faccbf14a33d6 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:27:35.291599 13023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81ab261b0dc04756a70faccbf14a33d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.292711 13022 tablet_service.cc:1511] Processing CreateTablet for tablet afd1573aba344096affc47d60971991c (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:27:35.294693 13022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd1573aba344096affc47d60971991c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.296953 12875 tablet_service.cc:1511] Processing CreateTablet for tablet 81ab261b0dc04756a70faccbf14a33d6 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:27:35.297427 12874 tablet_service.cc:1511] Processing CreateTablet for tablet ef0476290f24476eae1cbb574a059780 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:27:35.298662 12875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81ab261b0dc04756a70faccbf14a33d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.300050 12874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0476290f24476eae1cbb574a059780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.304502 12799 tablet_service.cc:1511] Processing CreateTablet for tablet ef0476290f24476eae1cbb574a059780 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:27:35.306209 12799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0476290f24476eae1cbb574a059780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.307279 12800 tablet_service.cc:1511] Processing CreateTablet for tablet 81ab261b0dc04756a70faccbf14a33d6 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:27:35.308802 12800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81ab261b0dc04756a70faccbf14a33d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.315172 12798 tablet_service.cc:1511] Processing CreateTablet for tablet afd1573aba344096affc47d60971991c (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:27:35.316810 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd1573aba344096affc47d60971991c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.317194 12873 tablet_service.cc:1511] Processing CreateTablet for tablet e643b9d35a9f4515abe259109274d191 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:27:35.318970 12873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e643b9d35a9f4515abe259109274d191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.321894 13021 tablet_service.cc:1511] Processing CreateTablet for tablet e643b9d35a9f4515abe259109274d191 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:27:35.323566 13021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e643b9d35a9f4515abe259109274d191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.323985 12797 tablet_service.cc:1511] Processing CreateTablet for tablet 224f359a7c884a7890b05f3098408d31 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:27:35.325455 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224f359a7c884a7890b05f3098408d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.326017 12872 tablet_service.cc:1511] Processing CreateTablet for tablet 224f359a7c884a7890b05f3098408d31 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:27:35.327718 12872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224f359a7c884a7890b05f3098408d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.331127 13020 tablet_service.cc:1511] Processing CreateTablet for tablet 735482fa1588451b93a1735236b1859b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:27:35.332916 13020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 735482fa1588451b93a1735236b1859b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.335882 12796 tablet_service.cc:1511] Processing CreateTablet for tablet 735482fa1588451b93a1735236b1859b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:27:35.337553 12796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 735482fa1588451b93a1735236b1859b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.341635 12871 tablet_service.cc:1511] Processing CreateTablet for tablet 735482fa1588451b93a1735236b1859b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:27:35.343621 12871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 735482fa1588451b93a1735236b1859b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.345237 13019 tablet_service.cc:1511] Processing CreateTablet for tablet 56b78cc7f12d4c3a81aed3af33111913 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:27:35.347121 13019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b78cc7f12d4c3a81aed3af33111913. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:35.352885 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:35.396729 12795 tablet_service.cc:1511] Processing CreateTablet for tablet 56b78cc7f12d4c3a81aed3af33111913 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:27:35.398278 12795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b78cc7f12d4c3a81aed3af33111913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.409936 13018 tablet_service.cc:1511] Processing CreateTablet for tablet eae9dd87d90f4d18b0d36431646603e3 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:27:35.412639 12948 tablet_service.cc:1511] Processing CreateTablet for tablet afd1573aba344096affc47d60971991c (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:27:35.413043 12869 tablet_service.cc:1511] Processing CreateTablet for tablet 82ca562394a94f40ae84d9607fb885f4 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:27:35.425618 13017 tablet_service.cc:1511] Processing CreateTablet for tablet cc6c3b7e72b044a5b0f0806e9a6301cf (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:27:35.418284 12947 tablet_service.cc:1511] Processing CreateTablet for tablet e643b9d35a9f4515abe259109274d191 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:27:35.433069 12944 tablet_service.cc:1511] Processing CreateTablet for tablet eae9dd87d90f4d18b0d36431646603e3 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:27:35.433079 12943 tablet_service.cc:1511] Processing CreateTablet for tablet 82ca562394a94f40ae84d9607fb885f4 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:27:35.414583 12948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd1573aba344096affc47d60971991c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.445119 12943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ca562394a94f40ae84d9607fb885f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.424203 12945 tablet_service.cc:1511] Processing CreateTablet for tablet 56b78cc7f12d4c3a81aed3af33111913 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:27:35.446218 12944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae9dd87d90f4d18b0d36431646603e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.446982 12945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56b78cc7f12d4c3a81aed3af33111913. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.435484 12868 tablet_service.cc:1511] Processing CreateTablet for tablet cc6c3b7e72b044a5b0f0806e9a6301cf (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:27:35.426050 12793 tablet_service.cc:1511] Processing CreateTablet for tablet cc6c3b7e72b044a5b0f0806e9a6301cf (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:27:35.419662 12946 tablet_service.cc:1511] Processing CreateTablet for tablet 224f359a7c884a7890b05f3098408d31 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:27:35.451965 12946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 224f359a7c884a7890b05f3098408d31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.452014 13078 tablet_bootstrap.cc:492] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:35.414503 12870 tablet_service.cc:1511] Processing CreateTablet for tablet eae9dd87d90f4d18b0d36431646603e3 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:27:35.414832 12869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ca562394a94f40ae84d9607fb885f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.460107 12868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc6c3b7e72b044a5b0f0806e9a6301cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.460933 12793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc6c3b7e72b044a5b0f0806e9a6301cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.416404 13018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae9dd87d90f4d18b0d36431646603e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.460937 12870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eae9dd87d90f4d18b0d36431646603e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.467764 13017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc6c3b7e72b044a5b0f0806e9a6301cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.412959 12949 tablet_service.cc:1511] Processing CreateTablet for tablet ef0476290f24476eae1cbb574a059780 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:27:35.416594 12794 tablet_service.cc:1511] Processing CreateTablet for tablet 82ca562394a94f40ae84d9607fb885f4 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:27:35.473156 12794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ca562394a94f40ae84d9607fb885f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.482040 12947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e643b9d35a9f4515abe259109274d191. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.487258 12949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef0476290f24476eae1cbb574a059780. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.496327 13078 tablet_bootstrap.cc:654] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:35.530730 12942 tablet_service.cc:1511] Processing CreateTablet for tablet cfad63c176a643f3bf8bba8f61523f87 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:27:35.534117 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfad63c176a643f3bf8bba8f61523f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.540721 13020 tablet_service.cc:1511] Processing CreateTablet for tablet cfad63c176a643f3bf8bba8f61523f87 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:27:35.543838 13020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfad63c176a643f3bf8bba8f61523f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.546265 12941 tablet_service.cc:1511] Processing CreateTablet for tablet d12b72a433e1403ab5c6d70ccad35e0f (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:27:35.547848 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12b72a433e1403ab5c6d70ccad35e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.549671 12872 tablet_service.cc:1511] Processing CreateTablet for tablet d12b72a433e1403ab5c6d70ccad35e0f (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:27:35.552001 12799 tablet_service.cc:1511] Processing CreateTablet for tablet cfad63c176a643f3bf8bba8f61523f87 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:27:35.553853 12799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfad63c176a643f3bf8bba8f61523f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.561782 13022 tablet_service.cc:1511] Processing CreateTablet for tablet d12b72a433e1403ab5c6d70ccad35e0f (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:27:35.566871 13022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12b72a433e1403ab5c6d70ccad35e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.567529 13079 tablet_bootstrap.cc:492] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:35.562350 12798 tablet_service.cc:1511] Processing CreateTablet for tablet 6c22979583614eb8b61240b2fa2424e9 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:27:35.572808 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c22979583614eb8b61240b2fa2424e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.577229 13079 tablet_bootstrap.cc:654] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:35.591398 13080 tablet_bootstrap.cc:492] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:35.602051 12872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12b72a433e1403ab5c6d70ccad35e0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.625547 13078 tablet_bootstrap.cc:492] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:35.628815 13078 ts_tablet_manager.cc:1403] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.177s	user 0.027s	sys 0.017s
I20260709 12:27:35.630631 12873 tablet_service.cc:1511] Processing CreateTablet for tablet 6c22979583614eb8b61240b2fa2424e9 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:27:35.632303 12873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c22979583614eb8b61240b2fa2424e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.633922 13078 raft_consensus.cc:359] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:35.634846 13078 raft_consensus.cc:385] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:35.635216 13078 raft_consensus.cc:740] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:35.635982 12940 tablet_service.cc:1511] Processing CreateTablet for tablet 6c22979583614eb8b61240b2fa2424e9 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:27:35.636264 13078 consensus_queue.cc:260] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:35.638707 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c22979583614eb8b61240b2fa2424e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.639101 12835 heartbeater.cc:499] Master 127.0.140.126:45447 was elected leader, sending a full tablet report...
I20260709 12:27:35.668624 13080 tablet_bootstrap.cc:654] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:35.672520 13083 tablet_bootstrap.cc:492] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:35.680145 13079 tablet_bootstrap.cc:492] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:35.707896 13079 ts_tablet_manager.cc:1403] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.141s	user 0.035s	sys 0.019s
I20260709 12:27:35.685364 12797 tablet_service.cc:1511] Processing CreateTablet for tablet 35361d4194194047849e55975d5a91b9 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:27:35.711822 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35361d4194194047849e55975d5a91b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.717599 13021 tablet_service.cc:1511] Processing CreateTablet for tablet c4eccc8971ad4aacb553dc8e1bb00cd8 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:27:35.719645 13021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4eccc8971ad4aacb553dc8e1bb00cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.724416 13079 raft_consensus.cc:359] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:35.725232 13079 raft_consensus.cc:385] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:35.725548 13079 raft_consensus.cc:740] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:35.729002 13058 heartbeater.cc:499] Master 127.0.140.126:45447 was elected leader, sending a full tablet report...
I20260709 12:27:35.686517 12873 tablet_service.cc:1511] Processing CreateTablet for tablet 35361d4194194047849e55975d5a91b9 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:27:35.687955 12942 tablet_service.cc:1511] Processing CreateTablet for tablet c4eccc8971ad4aacb553dc8e1bb00cd8 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:27:35.731808 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4eccc8971ad4aacb553dc8e1bb00cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.746608 13078 ts_tablet_manager.cc:1434] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.116s	user 0.007s	sys 0.006s
I20260709 12:27:35.747591 12873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35361d4194194047849e55975d5a91b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.747972 13078 tablet_bootstrap.cc:492] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:35.751472 12910 heartbeater.cc:499] Master 127.0.140.126:45447 was elected leader, sending a full tablet report...
I20260709 12:27:35.697366 13019 tablet_service.cc:1511] Processing CreateTablet for tablet 35361d4194194047849e55975d5a91b9 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:27:35.759148 13019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35361d4194194047849e55975d5a91b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.759864 12868 tablet_service.cc:1511] Processing CreateTablet for tablet 070b536983ee48e1853040e1652619e2 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:27:35.760175 13023 tablet_service.cc:1511] Processing CreateTablet for tablet 070b536983ee48e1853040e1652619e2 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:27:35.775844 12868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 070b536983ee48e1853040e1652619e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.775960 13023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 070b536983ee48e1853040e1652619e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.757251 12794 tablet_service.cc:1511] Processing CreateTablet for tablet c4eccc8971ad4aacb553dc8e1bb00cd8 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:27:35.764079 13080 tablet_bootstrap.cc:492] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:35.779713 12794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4eccc8971ad4aacb553dc8e1bb00cd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.779996 13080 ts_tablet_manager.cc:1403] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.189s	user 0.006s	sys 0.009s
I20260709 12:27:35.751152 13079 consensus_queue.cc:260] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:35.790004 13080 raft_consensus.cc:359] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:35.762001 12943 tablet_service.cc:1511] Processing CreateTablet for tablet 070b536983ee48e1853040e1652619e2 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:27:35.792363 12869 tablet_service.cc:1511] Processing CreateTablet for tablet 8e3f59c4b36b4b49925eaa8d5a4bbf8c (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:27:35.804064 12943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 070b536983ee48e1853040e1652619e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.804185 12869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3f59c4b36b4b49925eaa8d5a4bbf8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.768173 13083 tablet_bootstrap.cc:654] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:35.795943 12793 tablet_service.cc:1511] Processing CreateTablet for tablet 8e3f59c4b36b4b49925eaa8d5a4bbf8c (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:27:35.812104 13022 tablet_service.cc:1511] Processing CreateTablet for tablet 25fae6bafbf34e888a111d96f9fe73d4 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:27:35.812417 12796 tablet_service.cc:1511] Processing CreateTablet for tablet 25fae6bafbf34e888a111d96f9fe73d4 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:27:35.813829 13022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fae6bafbf34e888a111d96f9fe73d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.818354 12796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fae6bafbf34e888a111d96f9fe73d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.821162 12870 tablet_service.cc:1511] Processing CreateTablet for tablet 25fae6bafbf34e888a111d96f9fe73d4 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:27:35.822039 12948 tablet_service.cc:1511] Processing CreateTablet for tablet 8e3f59c4b36b4b49925eaa8d5a4bbf8c (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:27:35.822762 12870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25fae6bafbf34e888a111d96f9fe73d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.821202 12793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3f59c4b36b4b49925eaa8d5a4bbf8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.835811 12948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e3f59c4b36b4b49925eaa8d5a4bbf8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.848477 13086 raft_consensus.cc:493] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:35.849200 13086 raft_consensus.cc:515] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:35.855383 13079 ts_tablet_manager.cc:1434] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.147s	user 0.014s	sys 0.005s
I20260709 12:27:35.858702 13020 tablet_service.cc:1511] Processing CreateTablet for tablet acdd78885d1044a3a523c14603afcb9a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:27:35.861538 13020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acdd78885d1044a3a523c14603afcb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.866168 12984 heartbeater.cc:499] Master 127.0.140.126:45447 was elected leader, sending a full tablet report...
I20260709 12:27:35.871667 13079 tablet_bootstrap.cc:492] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:35.871451 13018 tablet_service.cc:1511] Processing CreateTablet for tablet f9b4619b6db04b159a1d568ba006bdb4 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:27:35.872952 13018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b4619b6db04b159a1d568ba006bdb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.873132 12799 tablet_service.cc:1511] Processing CreateTablet for tablet a0e539dd685147a899e29016abb08e1b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:27:35.874580 12799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e539dd685147a899e29016abb08e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.803007 13080 raft_consensus.cc:385] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:35.875033 13080 raft_consensus.cc:740] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:35.875496 12942 tablet_service.cc:1511] Processing CreateTablet for tablet f9b4619b6db04b159a1d568ba006bdb4 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:27:35.876145 13080 consensus_queue.cc:260] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:35.877111 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b4619b6db04b159a1d568ba006bdb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.865831 12940 tablet_service.cc:1511] Processing CreateTablet for tablet acdd78885d1044a3a523c14603afcb9a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:27:35.883678 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acdd78885d1044a3a523c14603afcb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.883556 12868 tablet_service.cc:1511] Processing CreateTablet for tablet f9b4619b6db04b159a1d568ba006bdb4 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:27:35.885165 12868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9b4619b6db04b159a1d568ba006bdb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.889626 13083 tablet_bootstrap.cc:492] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:35.889845 13078 tablet_bootstrap.cc:654] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:35.890348 13083 ts_tablet_manager.cc:1403] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.218s	user 0.019s	sys 0.004s
I20260709 12:27:35.893585 13083 raft_consensus.cc:359] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:35.908182 13086 leader_election.cc:290] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:35.909932 13083 raft_consensus.cc:385] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:35.942996 12798 tablet_service.cc:1511] Processing CreateTablet for tablet acdd78885d1044a3a523c14603afcb9a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:27:35.944900 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet acdd78885d1044a3a523c14603afcb9a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:35.966607 13059 tablet.cc:2406] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:27:35.973650 12872 tablet_service.cc:1511] Processing CreateTablet for tablet a0e539dd685147a899e29016abb08e1b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:27:35.975453 12872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e539dd685147a899e29016abb08e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:35.981995 13079 tablet_bootstrap.cc:654] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:35.986678 12799 tablet_service.cc:1511] Processing CreateTablet for tablet a2b3043eef1a4f98b328054262cfe08c (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:27:35.988310 12799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b3043eef1a4f98b328054262cfe08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.001572 12941 tablet_service.cc:1511] Processing CreateTablet for tablet a0e539dd685147a899e29016abb08e1b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:27:35.957180 13083 raft_consensus.cc:740] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.003484 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0e539dd685147a899e29016abb08e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.003757 13083 consensus_queue.cc:260] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.010809 13023 tablet_service.cc:1511] Processing CreateTablet for tablet a2b3043eef1a4f98b328054262cfe08c (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:27:36.012382 13023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b3043eef1a4f98b328054262cfe08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.024135 12875 tablet_service.cc:1511] Processing CreateTablet for tablet a2b3043eef1a4f98b328054262cfe08c (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:27:36.025799 12875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2b3043eef1a4f98b328054262cfe08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.067319 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224f359a7c884a7890b05f3098408d31" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:36.070369 12940 tablet_service.cc:1511] Processing CreateTablet for tablet 09c1da90a8da49f6be72a8361eead7df (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:27:36.071164 13078 tablet_bootstrap.cc:492] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:36.071846 13078 ts_tablet_manager.cc:1403] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.324s	user 0.008s	sys 0.012s
I20260709 12:27:36.072052 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c1da90a8da49f6be72a8361eead7df. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:36.076004 12773 leader_election.cc:343] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.082320 13080 ts_tablet_manager.cc:1434] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.302s	user 0.006s	sys 0.011s
I20260709 12:27:36.084030 13078 raft_consensus.cc:359] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:36.084934 13078 raft_consensus.cc:385] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.085273 13078 raft_consensus.cc:740] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.092430 13083 ts_tablet_manager.cc:1434] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.202s	user 0.022s	sys 0.013s
I20260709 12:27:36.096902 13079 tablet_bootstrap.cc:492] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:36.096665 13020 tablet_service.cc:1511] Processing CreateTablet for tablet 09c1da90a8da49f6be72a8361eead7df (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:27:36.097512 13079 ts_tablet_manager.cc:1403] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.226s	user 0.018s	sys 0.006s
I20260709 12:27:36.098505 13020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c1da90a8da49f6be72a8361eead7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.101262 13079 raft_consensus.cc:359] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.102133 13079 raft_consensus.cc:385] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.102456 13079 raft_consensus.cc:740] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.103510 13079 consensus_queue.cc:260] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.106832 13083 tablet_bootstrap.cc:492] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:36.086269 13078 consensus_queue.cc:260] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:36.110774 12793 tablet_service.cc:1511] Processing CreateTablet for tablet 09c1da90a8da49f6be72a8361eead7df (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:27:36.112315 12793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09c1da90a8da49f6be72a8361eead7df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.112635 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224f359a7c884a7890b05f3098408d31" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:36.114760 12776 leader_election.cc:343] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.115326 12776 leader_election.cc:304] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:36.116206 13086 raft_consensus.cc:2749] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:36.118636 13079 ts_tablet_manager.cc:1434] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.005s
I20260709 12:27:36.119556 13084 raft_consensus.cc:493] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.120152 13084 raft_consensus.cc:515] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.123337 13018 tablet_service.cc:1511] Processing CreateTablet for tablet 4f810e611ff745b2a7fa598c418b93db (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:27:36.124927 13018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f810e611ff745b2a7fa598c418b93db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.127677 13079 tablet_bootstrap.cc:492] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:36.127393 12872 tablet_service.cc:1511] Processing CreateTablet for tablet 4f810e611ff745b2a7fa598c418b93db (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:27:36.134244 13084 leader_election.cc:290] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:36.136763 12872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f810e611ff745b2a7fa598c418b93db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.143891 12942 tablet_service.cc:1511] Processing CreateTablet for tablet 4f810e611ff745b2a7fa598c418b93db (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:27:36.145489 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f810e611ff745b2a7fa598c418b93db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.136931 13079 tablet_bootstrap.cc:654] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.156945 12796 tablet_service.cc:1511] Processing CreateTablet for tablet f7237e470a3048a8878e2d0d49208ee1 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:27:36.157994 13078 ts_tablet_manager.cc:1434] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.086s	user 0.007s	sys 0.001s
I20260709 12:27:36.159228 12796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7237e470a3048a8878e2d0d49208ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.160163 13088 raft_consensus.cc:493] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.160735 13088 raft_consensus.cc:515] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.163900 13079 tablet_bootstrap.cc:492] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:36.164574 13079 ts_tablet_manager.cc:1403] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.037s	user 0.011s	sys 0.002s
I20260709 12:27:36.166554 13088 leader_election.cc:290] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:36.168357 13078 tablet_bootstrap.cc:492] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:36.176888 12870 tablet_service.cc:1511] Processing CreateTablet for tablet f7237e470a3048a8878e2d0d49208ee1 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:27:36.179626 12870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7237e470a3048a8878e2d0d49208ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.176059 13079 raft_consensus.cc:359] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.183033 13079 raft_consensus.cc:385] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.183548 13079 raft_consensus.cc:740] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.186164 12940 tablet_service.cc:1511] Processing CreateTablet for tablet f7237e470a3048a8878e2d0d49208ee1 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:27:36.188138 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7237e470a3048a8878e2d0d49208ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.189730 13080 tablet_bootstrap.cc:492] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:36.196596 13020 tablet_service.cc:1511] Processing CreateTablet for tablet ca89f605f7bb4bd9acc811382ac82591 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:27:36.205363 13020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca89f605f7bb4bd9acc811382ac82591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.205941 13079 consensus_queue.cc:260] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.212757 13079 ts_tablet_manager.cc:1434] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.048s	user 0.009s	sys 0.002s
I20260709 12:27:36.210319 12799 tablet_service.cc:1511] Processing CreateTablet for tablet ca89f605f7bb4bd9acc811382ac82591 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:27:36.223224 12875 tablet_service.cc:1511] Processing CreateTablet for tablet ca89f605f7bb4bd9acc811382ac82591 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:27:36.223047 13078 tablet_bootstrap.cc:654] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.224769 12875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca89f605f7bb4bd9acc811382ac82591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.226984 12798 tablet_service.cc:1511] Processing CreateTablet for tablet d74f8401a4854eb2b4fc3bde97c43ff8 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:27:36.228727 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d74f8401a4854eb2b4fc3bde97c43ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.230551 13079 tablet_bootstrap.cc:492] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:36.234577 13023 tablet_service.cc:1511] Processing CreateTablet for tablet d74f8401a4854eb2b4fc3bde97c43ff8 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:27:36.236181 13023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d74f8401a4854eb2b4fc3bde97c43ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.243741 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:36.244715 12810 raft_consensus.cc:2468] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:36.245858 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:36.248647 12998 leader_election.cc:304] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:36.251684 13084 raft_consensus.cc:2804] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:36.252125 13084 raft_consensus.cc:493] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.252481 13084 raft_consensus.cc:3060] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.252929 12942 tablet_service.cc:1511] Processing CreateTablet for tablet d74f8401a4854eb2b4fc3bde97c43ff8 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:27:36.254629 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d74f8401a4854eb2b4fc3bde97c43ff8. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:36.263909 12996 leader_election.cc:343] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.265390 12868 tablet_service.cc:1511] Processing CreateTablet for tablet 34ac366ceb574dd18638a5dcbde9b2de (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:27:36.266991 12868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ac366ceb574dd18638a5dcbde9b2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.269483 12799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca89f605f7bb4bd9acc811382ac82591. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.269567 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e643b9d35a9f4515abe259109274d191" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:36.272536 12851 leader_election.cc:343] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.292829 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e643b9d35a9f4515abe259109274d191" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:36.293130 12942 tablet_service.cc:1511] Processing CreateTablet for tablet 34ac366ceb574dd18638a5dcbde9b2de (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:27:36.294641 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ac366ceb574dd18638a5dcbde9b2de. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:36.295444 12849 leader_election.cc:343] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.296084 12849 leader_election.cc:304] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:36.297060 13088 raft_consensus.cc:2749] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:36.297187 13080 tablet_bootstrap.cc:654] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.300056 13019 tablet_service.cc:1511] Processing CreateTablet for tablet 34ac366ceb574dd18638a5dcbde9b2de (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:27:36.301605 13019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34ac366ceb574dd18638a5dcbde9b2de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.305483 13079 tablet_bootstrap.cc:654] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.308260 13083 tablet_bootstrap.cc:700] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent opening tablet: real 0.182s	user 0.005s	sys 0.000s
I20260709 12:27:36.309082 13083 tablet_bootstrap.cc:654] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.309619 12870 tablet_service.cc:1511] Processing CreateTablet for tablet c4c88c0aa80544daa7bae9c5ac62b0fa (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:27:36.311271 12870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c88c0aa80544daa7bae9c5ac62b0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.311970 12798 tablet_service.cc:1511] Processing CreateTablet for tablet c4c88c0aa80544daa7bae9c5ac62b0fa (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:27:36.313405 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c88c0aa80544daa7bae9c5ac62b0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.321376 12942 tablet_service.cc:1511] Processing CreateTablet for tablet c4c88c0aa80544daa7bae9c5ac62b0fa (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:27:36.322988 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4c88c0aa80544daa7bae9c5ac62b0fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.325688 13078 tablet_bootstrap.cc:492] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:36.326404 13078 ts_tablet_manager.cc:1403] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.158s	user 0.021s	sys 0.012s
I20260709 12:27:36.329919 13078 raft_consensus.cc:359] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.337973 12796 tablet_service.cc:1511] Processing CreateTablet for tablet 8d24c0921438489bb6c87bd7b1b0654d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:27:36.339607 12796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d24c0921438489bb6c87bd7b1b0654d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.330691 13078 raft_consensus.cc:385] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.341972 13078 raft_consensus.cc:740] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.343205 13078 consensus_queue.cc:260] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.344128 12940 tablet_service.cc:1511] Processing CreateTablet for tablet 508c4d341f444ad2867dfb09b3b2a125 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:27:36.345760 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508c4d341f444ad2867dfb09b3b2a125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.345445 13022 tablet_service.cc:1511] Processing CreateTablet for tablet 508c4d341f444ad2867dfb09b3b2a125 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:27:36.347095 13022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508c4d341f444ad2867dfb09b3b2a125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.349261 12793 tablet_service.cc:1511] Processing CreateTablet for tablet 508c4d341f444ad2867dfb09b3b2a125 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:27:36.350842 12793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 508c4d341f444ad2867dfb09b3b2a125. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.354967 12872 tablet_service.cc:1511] Processing CreateTablet for tablet 81eacfb947b741a2a11b5b645c586003 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20260709 12:27:36.355993 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:36.356427 12872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81eacfb947b741a2a11b5b645c586003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.358232 12870 tablet_service.cc:1511] Processing CreateTablet for tablet 8d24c0921438489bb6c87bd7b1b0654d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:27:36.358496 12941 tablet_service.cc:1511] Processing CreateTablet for tablet 81eacfb947b741a2a11b5b645c586003 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:27:36.345316 13023 tablet_service.cc:1511] Processing CreateTablet for tablet 8d24c0921438489bb6c87bd7b1b0654d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:27:36.322041 13084 raft_consensus.cc:515] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.361430 13023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d24c0921438489bb6c87bd7b1b0654d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.363584 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:36.364323 12810 raft_consensus.cc:3060] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.365286 13083 tablet_bootstrap.cc:492] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:36.365917 13083 ts_tablet_manager.cc:1403] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.259s	user 0.013s	sys 0.000s
I20260709 12:27:36.366559 12870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d24c0921438489bb6c87bd7b1b0654d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.382975 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81eacfb947b741a2a11b5b645c586003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.384239 13019 tablet_service.cc:1511] Processing CreateTablet for tablet 81eacfb947b741a2a11b5b645c586003 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:27:36.385703 13019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81eacfb947b741a2a11b5b645c586003. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.386716 13086 raft_consensus.cc:493] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.387326 13086 raft_consensus.cc:515] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.387790 12794 tablet_service.cc:1511] Processing CreateTablet for tablet 2c6179fd5410440a89ea4ca9c5ef3944 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:27:36.388803 12948 tablet_service.cc:1511] Processing CreateTablet for tablet 2c6179fd5410440a89ea4ca9c5ef3944 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:27:36.389302 12794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6179fd5410440a89ea4ca9c5ef3944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.390295 12948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6179fd5410440a89ea4ca9c5ef3944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.391155 13095 raft_consensus.cc:493] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.386703 12873 tablet_service.cc:1511] Processing CreateTablet for tablet 2c6179fd5410440a89ea4ca9c5ef3944 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:27:36.391484 12810 raft_consensus.cc:2468] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:36.391775 13095 raft_consensus.cc:515] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.392582 12873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c6179fd5410440a89ea4ca9c5ef3944. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.392979 12998 leader_election.cc:304] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:36.397148 13095 leader_election.cc:290] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:36.397548 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:36.418064 13105 raft_consensus.cc:493] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.418762 13105 raft_consensus.cc:515] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:36.421905 12797 tablet_service.cc:1511] Processing CreateTablet for tablet 0d4e24d741fa47ce960e848db5967200 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
W20260709 12:27:36.427145 12996 leader_election.cc:343] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.428524 12943 tablet_service.cc:1511] Processing CreateTablet for tablet af4bf74fe4b6486a83af6b41bb30f373 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:27:36.421905 12869 tablet_service.cc:1511] Processing CreateTablet for tablet 0d4e24d741fa47ce960e848db5967200 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:27:36.401814 13084 leader_election.cc:290] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:36.442019 12869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4e24d741fa47ce960e848db5967200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.445012 13083 raft_consensus.cc:359] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.445858 13083 raft_consensus.cc:385] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.446138 13083 raft_consensus.cc:740] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.447217 13083 consensus_queue.cc:260] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.452723 13083 ts_tablet_manager.cc:1434] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.086s	user 0.010s	sys 0.000s
I20260709 12:27:36.453902 13083 tablet_bootstrap.cc:492] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:36.457638 12943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4bf74fe4b6486a83af6b41bb30f373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.459604 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4e24d741fa47ce960e848db5967200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.460045 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:36.460956 13080 tablet_bootstrap.cc:492] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:36.461594 13080 ts_tablet_manager.cc:1403] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.272s	user 0.014s	sys 0.002s
W20260709 12:27:36.461845 12773 leader_election.cc:343] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:36.475066 12996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 59074.8us
I20260709 12:27:36.477639 13020 tablet_service.cc:1511] Processing CreateTablet for tablet 0d4e24d741fa47ce960e848db5967200 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:27:36.478466 13107 raft_consensus.cc:2804] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:36.478152 13018 tablet_service.cc:1511] Processing CreateTablet for tablet af4bf74fe4b6486a83af6b41bb30f373 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:27:36.479094 13107 raft_consensus.cc:697] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:36.479516 13020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4e24d741fa47ce960e848db5967200. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.480304 13018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4bf74fe4b6486a83af6b41bb30f373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.480124 13107 consensus_queue.cc:237] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.481843 13109 raft_consensus.cc:493] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.482434 13109 raft_consensus.cc:515] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.485878 13109 leader_election.cc:290] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:36.486467 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e643b9d35a9f4515abe259109274d191" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:36.487253 12885 raft_consensus.cc:2468] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:36.488757 12996 leader_election.cc:304] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:36.495716 12795 tablet_service.cc:1511] Processing CreateTablet for tablet af4bf74fe4b6486a83af6b41bb30f373 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:27:36.496584 13078 ts_tablet_manager.cc:1434] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.170s	user 0.007s	sys 0.016s
I20260709 12:27:36.497623 12795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4bf74fe4b6486a83af6b41bb30f373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.497877 13078 tablet_bootstrap.cc:492] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
W20260709 12:27:36.436939 12998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 44060.3us
I20260709 12:27:36.458366 13086 leader_election.cc:290] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:36.508494 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:36.509397 13105 leader_election.cc:290] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:36.509408 12959 raft_consensus.cc:2468] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:36.511340 12776 leader_election.cc:304] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:36.517469 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afd1573aba344096affc47d60971991c" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:36.519290 13086 raft_consensus.cc:2804] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:36.519824 13086 raft_consensus.cc:493] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.520262 13086 raft_consensus.cc:3060] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.520591 13079 tablet_bootstrap.cc:492] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:36.521338 13079 ts_tablet_manager.cc:1403] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.291s	user 0.030s	sys 0.028s
I20260709 12:27:36.524523 13080 raft_consensus.cc:359] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.530495 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e643b9d35a9f4515abe259109274d191" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:36.533707 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afd1573aba344096affc47d60971991c" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:36.505800 13109 raft_consensus.cc:2804] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:36.553541 13109 raft_consensus.cc:493] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.554018 13109 raft_consensus.cc:3060] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.539811 12959 raft_consensus.cc:2468] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:36.524997 13079 raft_consensus.cc:359] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:36.555959 13079 raft_consensus.cc:385] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.556375 13079 raft_consensus.cc:740] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
W20260709 12:27:36.552798 12998 leader_election.cc:343] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.557431 13079 consensus_queue.cc:260] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:36.565147 13078 tablet_bootstrap.cc:654] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.558852 13080 raft_consensus.cc:385] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.578959 13080 raft_consensus.cc:740] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.579994 13080 consensus_queue.cc:260] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.582530 13080 ts_tablet_manager.cc:1434] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.121s	user 0.009s	sys 0.000s
I20260709 12:27:36.577953 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:36.598333 13021 tablet_service.cc:1511] Processing CreateTablet for tablet b0eea117b500470093718b8583fa5e96 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:27:36.568643 12870 tablet_service.cc:1511] Processing CreateTablet for tablet f40e0b80665048e09e181f1a8ea7509a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
W20260709 12:27:36.584841 12999 leader_election.cc:343] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.590309 12799 tablet_service.cc:1511] Processing CreateTablet for tablet b32a14229832492682674ac4c282404d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:27:36.600803 13021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0eea117b500470093718b8583fa5e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.601120 12799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32a14229832492682674ac4c282404d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.601491 12870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f40e0b80665048e09e181f1a8ea7509a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.602355 13086 raft_consensus.cc:515] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.595198 12941 tablet_service.cc:1511] Processing CreateTablet for tablet b32a14229832492682674ac4c282404d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:27:36.598392 13020 tablet_service.cc:1511] Processing CreateTablet for tablet f40e0b80665048e09e181f1a8ea7509a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:27:36.590727 12869 tablet_service.cc:1511] Processing CreateTablet for tablet b32a14229832492682674ac4c282404d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:27:36.615411 12872 tablet_service.cc:1511] Processing CreateTablet for tablet b0eea117b500470093718b8583fa5e96 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:27:36.617164 13083 tablet_bootstrap.cc:654] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.620941 13020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f40e0b80665048e09e181f1a8ea7509a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.621671 13017 tablet_service.cc:1511] Processing CreateTablet for tablet 9e1ba3f1907244abbd44d141e09ff65d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:27:36.623366 13017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e1ba3f1907244abbd44d141e09ff65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.622953 12794 tablet_service.cc:1511] Processing CreateTablet for tablet 9e1ba3f1907244abbd44d141e09ff65d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:27:36.624447 12794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e1ba3f1907244abbd44d141e09ff65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.615459 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32a14229832492682674ac4c282404d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.633514 13080 tablet_bootstrap.cc:492] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:36.634591 12869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b32a14229832492682674ac4c282404d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.642417 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:36.602977 12999 leader_election.cc:304] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:36.642298 12872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0eea117b500470093718b8583fa5e96. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:36.644114 12999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 61500us
W20260709 12:27:36.644397 12922 leader_election.cc:343] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.645555 13109 raft_consensus.cc:515] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.654986 12797 tablet_service.cc:1511] Processing CreateTablet for tablet b0eea117b500470093718b8583fa5e96 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:27:36.657842 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0eea117b500470093718b8583fa5e96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.661769 12810 raft_consensus.cc:2393] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8012bb9a8b274890bcaad0d192cd30d5 in current term 1: Already voted for candidate b543f973837c43b19b7d17feaaca5325 in this term.
I20260709 12:27:36.664847 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:36.665580 12924 leader_election.cc:304] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
W20260709 12:27:36.666563 12773 leader_election.cc:343] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.664316 12873 tablet_service.cc:1511] Processing CreateTablet for tablet 9cf46416458c4a9398e8de1c6b8ccb5d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:27:36.667320 13095 raft_consensus.cc:3060] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.667786 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:36.668764 12940 tablet_service.cc:1511] Processing CreateTablet for tablet 9e1ba3f1907244abbd44d141e09ff65d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:27:36.670286 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e1ba3f1907244abbd44d141e09ff65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.668107 12873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf46416458c4a9398e8de1c6b8ccb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.674942 13086 leader_election.cc:290] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:36.678014 13095 raft_consensus.cc:2749] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:36.686017 12708 catalog_manager.cc:5697] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:36.659065 13107 raft_consensus.cc:2749] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:36.706451 13079 ts_tablet_manager.cc:1434] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.185s	user 0.009s	sys 0.011s
I20260709 12:27:36.711674 13079 tablet_bootstrap.cc:492] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:36.700448 12948 tablet_service.cc:1511] Processing CreateTablet for tablet 9cf46416458c4a9398e8de1c6b8ccb5d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:27:36.717424 12948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf46416458c4a9398e8de1c6b8ccb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.710081 12958 raft_consensus.cc:2468] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:36.725358 12776 leader_election.cc:304] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:36.727835 13086 raft_consensus.cc:2804] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:36.735283 13086 raft_consensus.cc:697] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:36.738570 12942 tablet_service.cc:1511] Processing CreateTablet for tablet f40e0b80665048e09e181f1a8ea7509a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:27:36.740288 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f40e0b80665048e09e181f1a8ea7509a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.751056 13018 tablet_service.cc:1511] Processing CreateTablet for tablet 9cf46416458c4a9398e8de1c6b8ccb5d (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:27:36.752877 13018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9cf46416458c4a9398e8de1c6b8ccb5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.757752 12799 tablet_service.cc:1511] Processing CreateTablet for tablet fc164dcd3f514a00a186655a9a7b2215 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:27:36.759428 12799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc164dcd3f514a00a186655a9a7b2215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.761854 13113 raft_consensus.cc:493] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.762462 13113 raft_consensus.cc:515] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:36.766863 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afd1573aba344096affc47d60971991c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:36.774330 12776 leader_election.cc:343] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:36.776736 12868 tablet_service.cc:1511] Processing CreateTablet for tablet fc164dcd3f514a00a186655a9a7b2215 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:27:36.778019 12940 tablet_service.cc:1511] Processing CreateTablet for tablet fc164dcd3f514a00a186655a9a7b2215 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:27:36.779510 12868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc164dcd3f514a00a186655a9a7b2215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.736743 13086 consensus_queue.cc:237] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.779778 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc164dcd3f514a00a186655a9a7b2215. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.766826 13080 tablet_bootstrap.cc:654] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.780781 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e643b9d35a9f4515abe259109274d191" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:36.789568 12885 raft_consensus.cc:3060] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.780416 13109 leader_election.cc:290] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:36.769975 13113 leader_election.cc:290] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:36.781334 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e643b9d35a9f4515abe259109274d191" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:36.811076 12958 raft_consensus.cc:3060] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.812999 13021 tablet_service.cc:1511] Processing CreateTablet for tablet 30cb0580c4514b41b650a6b2d5089f3a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:27:36.814617 13021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30cb0580c4514b41b650a6b2d5089f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.821660 13083 tablet_bootstrap.cc:492] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:36.822316 13083 ts_tablet_manager.cc:1403] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.369s	user 0.037s	sys 0.029s
I20260709 12:27:36.823534 13078 tablet_bootstrap.cc:492] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:36.824216 13078 ts_tablet_manager.cc:1403] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.327s	user 0.010s	sys 0.016s
I20260709 12:27:36.826184 13083 raft_consensus.cc:359] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.827217 13083 raft_consensus.cc:385] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.827582 13083 raft_consensus.cc:740] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.817659 13079 tablet_bootstrap.cc:654] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:36.836527 12794 tablet_service.cc:1511] Processing CreateTablet for tablet 30cb0580c4514b41b650a6b2d5089f3a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:27:36.840631 12794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30cb0580c4514b41b650a6b2d5089f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.827626 13078 raft_consensus.cc:359] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.838557 12885 raft_consensus.cc:2468] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:36.864038 13080 tablet_bootstrap.cc:492] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:36.846695 13078 raft_consensus.cc:385] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.853868 12958 raft_consensus.cc:2468] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:36.866189 13080 ts_tablet_manager.cc:1403] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.233s	user 0.020s	sys 0.015s
I20260709 12:27:36.831385 13083 consensus_queue.cc:260] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.867615 13088 raft_consensus.cc:493] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.867069 13078 raft_consensus.cc:740] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.873504 12996 leader_election.cc:304] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:36.884615 12797 tablet_service.cc:1511] Processing CreateTablet for tablet 38871c668f414df593e0f6d35b1feab7 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:27:36.889371 13109 raft_consensus.cc:2804] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:36.887140 12872 tablet_service.cc:1511] Processing CreateTablet for tablet 30cb0580c4514b41b650a6b2d5089f3a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:27:36.880396 13088 raft_consensus.cc:515] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.890743 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38871c668f414df593e0f6d35b1feab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.892735 12707 catalog_manager.cc:5697] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:36.893769 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735482fa1588451b93a1735236b1859b" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:36.890749 12872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30cb0580c4514b41b650a6b2d5089f3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.899586 13109 raft_consensus.cc:697] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:36.900538 13109 consensus_queue.cc:237] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.895068 13088 leader_election.cc:290] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:36.903858 12941 tablet_service.cc:1511] Processing CreateTablet for tablet 38871c668f414df593e0f6d35b1feab7 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:27:36.904779 13080 raft_consensus.cc:359] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.910655 13020 tablet_service.cc:1511] Processing CreateTablet for tablet 38871c668f414df593e0f6d35b1feab7 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:27:36.911101 13078 consensus_queue.cc:260] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.923540 13033 raft_consensus.cc:2468] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:36.925722 12849 leader_election.cc:304] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:36.926735 13088 raft_consensus.cc:2804] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:36.914048 13083 ts_tablet_manager.cc:1434] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.091s	user 0.006s	sys 0.004s
I20260709 12:27:36.916960 13017 tablet_service.cc:1511] Processing CreateTablet for tablet 3ca75f6a5016474a844f5532614ebc75 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:27:36.919509 13079 tablet_bootstrap.cc:492] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:36.922945 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38871c668f414df593e0f6d35b1feab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.930040 13083 tablet_bootstrap.cc:492] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:36.930517 13079 ts_tablet_manager.cc:1403] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.219s	user 0.016s	sys 0.005s
I20260709 12:27:36.935664 12799 tablet_service.cc:1511] Processing CreateTablet for tablet bdcb0dae3adf465cbbded92399c9cb51 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:27:36.940806 13017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca75f6a5016474a844f5532614ebc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.942414 13018 tablet_service.cc:1511] Processing CreateTablet for tablet 0034a6bef1bf4e119ee686581612f8c0 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:27:36.927251 13088 raft_consensus.cc:493] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.927318 13080 raft_consensus.cc:385] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.948369 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afd1573aba344096affc47d60971991c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:36.949087 13033 raft_consensus.cc:2468] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:36.950592 12774 leader_election.cc:304] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:36.951632 13086 raft_consensus.cc:2804] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:36.952105 13086 raft_consensus.cc:493] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:36.951737 12868 tablet_service.cc:1511] Processing CreateTablet for tablet 3ca75f6a5016474a844f5532614ebc75 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:27:36.952541 13086 raft_consensus.cc:3060] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.947880 13080 raft_consensus.cc:740] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.953219 13018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0034a6bef1bf4e119ee686581612f8c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.954035 13020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38871c668f414df593e0f6d35b1feab7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.953920 13080 consensus_queue.cc:260] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:36.947688 13088 raft_consensus.cc:3060] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:36.958307 12799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdcb0dae3adf465cbbded92399c9cb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.963142 12940 tablet_service.cc:1511] Processing CreateTablet for tablet bdcb0dae3adf465cbbded92399c9cb51 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:27:36.961421 12795 tablet_service.cc:1511] Processing CreateTablet for tablet 0034a6bef1bf4e119ee686581612f8c0 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:27:36.963243 13086 raft_consensus.cc:515] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:36.953336 12868 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca75f6a5016474a844f5532614ebc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.955273 12948 tablet_service.cc:1511] Processing CreateTablet for tablet 3ca75f6a5016474a844f5532614ebc75 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:27:36.968112 12795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0034a6bef1bf4e119ee686581612f8c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.968657 13078 ts_tablet_manager.cc:1434] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.144s	user 0.003s	sys 0.007s
I20260709 12:27:36.977648 12870 tablet_service.cc:1511] Processing CreateTablet for tablet 0034a6bef1bf4e119ee686581612f8c0 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:27:36.975116 13079 raft_consensus.cc:359] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:36.979166 12942 tablet_service.cc:1511] Processing CreateTablet for tablet 8bc4a9064e4143f7a4f1ce3a3a4bb860 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:27:36.979625 13095 raft_consensus.cc:493] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:36.991015 13079 raft_consensus.cc:385] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:36.997256 13079 raft_consensus.cc:740] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:36.992951 13095 raft_consensus.cc:515] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:36.999820 12870 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0034a6bef1bf4e119ee686581612f8c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:36.995438 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdcb0dae3adf465cbbded92399c9cb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.006557 13020 tablet_service.cc:1511] Processing CreateTablet for tablet 8bc4a9064e4143f7a4f1ce3a3a4bb860 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:27:37.011072 13020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc4a9064e4143f7a4f1ce3a3a4bb860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.003309 13078 tablet_bootstrap.cc:492] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:37.013803 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae9dd87d90f4d18b0d36431646603e3" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:37.015633 12922 leader_election.cc:343] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.016180 12943 tablet_service.cc:1511] Processing CreateTablet for tablet 7d28ffbbc352420bb35bfab9e1847e67 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:27:36.998329 13079 consensus_queue.cc:260] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.019523 12793 tablet_service.cc:1511] Processing CreateTablet for tablet 068448e5fa294d199668e58a170cbee3 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:27:37.019917 13021 tablet_service.cc:1511] Processing CreateTablet for tablet 7d28ffbbc352420bb35bfab9e1847e67 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:27:37.019981 13109 consensus_queue.cc:1048] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.020849 13079 ts_tablet_manager.cc:1434] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.076s	user 0.006s	sys 0.002s
I20260709 12:27:37.021323 12793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068448e5fa294d199668e58a170cbee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.022091 13079 tablet_bootstrap.cc:492] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:37.028810 13021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d28ffbbc352420bb35bfab9e1847e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.030063 12794 tablet_service.cc:1511] Processing CreateTablet for tablet 8bc4a9064e4143f7a4f1ce3a3a4bb860 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:27:37.031620 12794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc4a9064e4143f7a4f1ce3a3a4bb860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.034111 12943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d28ffbbc352420bb35bfab9e1847e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.036849 13095 leader_election.cc:290] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:37.039002 12869 tablet_service.cc:1511] Processing CreateTablet for tablet 7d28ffbbc352420bb35bfab9e1847e67 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:27:37.040930 12869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d28ffbbc352420bb35bfab9e1847e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.047811 13086 leader_election.cc:290] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:37.049084 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afd1573aba344096affc47d60971991c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:37.049753 13033 raft_consensus.cc:3060] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.052477 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae9dd87d90f4d18b0d36431646603e3" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:37.053011 13088 raft_consensus.cc:515] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
W20260709 12:27:37.054749 12923 leader_election.cc:343] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.055341 12923 leader_election.cc:304] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:37.055797 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afd1573aba344096affc47d60971991c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:37.056689 13095 raft_consensus.cc:2749] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.057773 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735482fa1588451b93a1735236b1859b" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
W20260709 12:27:37.058122 12776 leader_election.cc:343] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.058473 13032 raft_consensus.cc:3060] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.059785 13088 leader_election.cc:290] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:37.061874 13033 raft_consensus.cc:2468] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:37.063462 12774 leader_election.cc:304] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:37.064735 12873 tablet_service.cc:1511] Processing CreateTablet for tablet bdcb0dae3adf465cbbded92399c9cb51 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:27:37.066732 12873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdcb0dae3adf465cbbded92399c9cb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.065622 13086 raft_consensus.cc:2804] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.067385 13086 raft_consensus.cc:697] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:37.068287 13086 consensus_queue.cc:237] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.072525 12875 tablet_service.cc:1511] Processing CreateTablet for tablet 068448e5fa294d199668e58a170cbee3 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:27:37.076176 12941 tablet_service.cc:1511] Processing CreateTablet for tablet 068448e5fa294d199668e58a170cbee3 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:27:37.075971 13019 tablet_service.cc:1511] Processing CreateTablet for tablet 423d51d99d504d708ad4af4d083eba48 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:27:37.077696 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068448e5fa294d199668e58a170cbee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.077430 12798 tablet_service.cc:1511] Processing CreateTablet for tablet 423d51d99d504d708ad4af4d083eba48 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:27:37.079131 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423d51d99d504d708ad4af4d083eba48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.077703 13019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423d51d99d504d708ad4af4d083eba48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.087147 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735482fa1588451b93a1735236b1859b" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
W20260709 12:27:37.088927 12850 leader_election.cc:343] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.086612 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735482fa1588451b93a1735236b1859b" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
W20260709 12:27:37.091723 12850 leader_election.cc:343] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.094008 13079 tablet_bootstrap.cc:654] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.097419 13032 raft_consensus.cc:2468] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:37.099195 12948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ca75f6a5016474a844f5532614ebc75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.116744 13083 tablet_bootstrap.cc:654] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.132356 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8bc4a9064e4143f7a4f1ce3a3a4bb860. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.121574 12875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 068448e5fa294d199668e58a170cbee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.138365 12849 leader_election.cc:304] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:37.144097 12874 tablet_service.cc:1511] Processing CreateTablet for tablet 423d51d99d504d708ad4af4d083eba48 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:27:37.158730 12871 tablet_service.cc:1511] Processing CreateTablet for tablet ed10b80d23ae40019ba46ed7087ca05a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:27:37.160202 12946 tablet_service.cc:1511] Processing CreateTablet for tablet fd444f6f267d4255873e372d6e3e3a43 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:27:37.162086 12946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd444f6f267d4255873e372d6e3e3a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.164410 12797 tablet_service.cc:1511] Processing CreateTablet for tablet fd444f6f267d4255873e372d6e3e3a43 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:27:37.165870 13080 ts_tablet_manager.cc:1434] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.292s	user 0.008s	sys 0.010s
I20260709 12:27:37.167204 13080 tablet_bootstrap.cc:492] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:37.167732 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd444f6f267d4255873e372d6e3e3a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.171240 13078 tablet_bootstrap.cc:654] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.171612 12942 tablet_service.cc:1511] Processing CreateTablet for tablet ed10b80d23ae40019ba46ed7087ca05a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:27:37.173091 12942 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed10b80d23ae40019ba46ed7087ca05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.175048 13125 raft_consensus.cc:2804] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.175858 13125 raft_consensus.cc:697] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:37.176963 13125 consensus_queue.cc:237] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.180966 12874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 423d51d99d504d708ad4af4d083eba48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.164410 13022 tablet_service.cc:1511] Processing CreateTablet for tablet fd444f6f267d4255873e372d6e3e3a43 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:27:37.181916 12871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed10b80d23ae40019ba46ed7087ca05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.182821 13022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd444f6f267d4255873e372d6e3e3a43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.192065 13017 tablet_service.cc:1511] Processing CreateTablet for tablet ed10b80d23ae40019ba46ed7087ca05a (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:27:37.197898 13017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed10b80d23ae40019ba46ed7087ca05a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:37.255072 12996 consensus_peers.cc:597] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:37.260035 12793 tablet_service.cc:1511] Processing CreateTablet for tablet d9b2c51994d34d189bd05462fa15c84b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:27:37.261754 12793 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b2c51994d34d189bd05462fa15c84b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.263236 13088 raft_consensus.cc:493] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.263875 13088 raft_consensus.cc:515] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.264889 12885 raft_consensus.cc:3060] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.266315 13088 leader_election.cc:290] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:37.291400 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:37.292539 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:37.293721 12958 raft_consensus.cc:2393] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate b543f973837c43b19b7d17feaaca5325 in this term.
I20260709 12:27:37.294596 13079 tablet_bootstrap.cc:492] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:37.295235 13079 ts_tablet_manager.cc:1403] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.273s	user 0.018s	sys 0.008s
I20260709 12:27:37.300776 12850 leader_election.cc:304] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:37.300640 13086 consensus_queue.cc:1048] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.317239 13080 tablet_bootstrap.cc:654] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.320200 12708 catalog_manager.cc:5697] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:37.320118 13125 raft_consensus.cc:2749] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.300172 13079 raft_consensus.cc:359] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.322214 13079 raft_consensus.cc:385] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.322558 13079 raft_consensus.cc:740] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.323693 13079 consensus_queue.cc:260] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.326226 13079 ts_tablet_manager.cc:1434] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.000s
I20260709 12:27:37.327911 13079 tablet_bootstrap.cc:492] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:37.337241 13083 tablet_bootstrap.cc:492] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:37.338001 13083 ts_tablet_manager.cc:1403] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.408s	user 0.015s	sys 0.050s
I20260709 12:27:37.338766 13086 consensus_queue.cc:1048] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.342073 13083 raft_consensus.cc:359] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.343043 13083 raft_consensus.cc:385] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.343361 13083 raft_consensus.cc:740] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.344383 12946 tablet_service.cc:1511] Processing CreateTablet for tablet d9b2c51994d34d189bd05462fa15c84b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:27:37.344413 13083 consensus_queue.cc:260] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.346088 12794 tablet_service.cc:1511] Processing CreateTablet for tablet c1144634236d42f5b2d3656f6eb55389 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:27:37.347440 12794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1144634236d42f5b2d3656f6eb55389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.347899 13083 ts_tablet_manager.cc:1434] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
I20260709 12:27:37.354748 12706 catalog_manager.cc:5697] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 1 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:37.356276 12869 tablet_service.cc:1511] Processing CreateTablet for tablet c1144634236d42f5b2d3656f6eb55389 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:27:37.346112 12946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b2c51994d34d189bd05462fa15c84b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.341454 13017 tablet_service.cc:1511] Processing CreateTablet for tablet c1144634236d42f5b2d3656f6eb55389 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:27:37.366321 12869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1144634236d42f5b2d3656f6eb55389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.368078 12941 tablet_service.cc:1511] Processing CreateTablet for tablet 303cfb4dc35a4ac094447486c4419179 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
W20260709 12:27:37.369282 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:37.351390 12874 tablet_service.cc:1511] Processing CreateTablet for tablet d9b2c51994d34d189bd05462fa15c84b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:27:37.369792 13113 raft_consensus.cc:493] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.370535 13113 raft_consensus.cc:515] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.371039 12874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b2c51994d34d189bd05462fa15c84b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.373147 13113 leader_election.cc:290] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:37.375121 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735482fa1588451b93a1735236b1859b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:37.375639 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735482fa1588451b93a1735236b1859b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:37.376173 13032 raft_consensus.cc:2393] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b543f973837c43b19b7d17feaaca5325 in current term 1: Already voted for candidate 8a995dc08b2a4497b71db084a2641e6f in this term.
I20260709 12:27:37.378098 12774 leader_election.cc:304] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:37.374107 13083 tablet_bootstrap.cc:492] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:37.379825 13017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1144634236d42f5b2d3656f6eb55389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.380362 13086 raft_consensus.cc:3060] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.387933 13109 consensus_queue.cc:1048] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.392957 12941 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 303cfb4dc35a4ac094447486c4419179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.381597 13078 tablet_bootstrap.cc:492] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:37.409262 13019 tablet_service.cc:1511] Processing CreateTablet for tablet 303cfb4dc35a4ac094447486c4419179 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:27:37.411278 13019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 303cfb4dc35a4ac094447486c4419179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.412004 13078 ts_tablet_manager.cc:1403] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.409s	user 0.013s	sys 0.026s
I20260709 12:27:37.414583 12797 tablet_service.cc:1511] Processing CreateTablet for tablet 303cfb4dc35a4ac094447486c4419179 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:27:37.416136 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 303cfb4dc35a4ac094447486c4419179. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.415848 12873 tablet_service.cc:1511] Processing CreateTablet for tablet bf54d438996844a1b6028dabd88f727e (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:27:37.401569 12708 catalog_manager.cc:5697] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:37.404615 13079 tablet_bootstrap.cc:654] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.421692 12873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf54d438996844a1b6028dabd88f727e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.442272 12948 tablet_service.cc:1511] Processing CreateTablet for tablet bf54d438996844a1b6028dabd88f727e (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:27:37.444267 12948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf54d438996844a1b6028dabd88f727e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.446020 13113 raft_consensus.cc:493] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.449092 13113 raft_consensus.cc:515] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.451339 13113 leader_election.cc:290] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:37.453540 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b78cc7f12d4c3a81aed3af33111913" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:37.453621 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b78cc7f12d4c3a81aed3af33111913" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:37.456782 12774 leader_election.cc:343] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:37.458191 12776 leader_election.cc:343] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.458547 13107 consensus_queue.cc:1048] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.458691 12776 leader_election.cc:304] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:37.460280 13113 raft_consensus.cc:2749] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.461627 13078 raft_consensus.cc:359] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.462457 13078 raft_consensus.cc:385] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.462767 13078 raft_consensus.cc:740] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.439298 13086 raft_consensus.cc:2749] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.479045 13022 tablet_service.cc:1511] Processing CreateTablet for tablet bf54d438996844a1b6028dabd88f727e (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:27:37.501737 12871 tablet_service.cc:1511] Processing CreateTablet for tablet e2cbb9487e7d4b9a90f2a4aa06956329 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:27:37.503284 12798 tablet_service.cc:1511] Processing CreateTablet for tablet e2cbb9487e7d4b9a90f2a4aa06956329 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:27:37.504837 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2cbb9487e7d4b9a90f2a4aa06956329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.506994 13022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf54d438996844a1b6028dabd88f727e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.503646 12871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2cbb9487e7d4b9a90f2a4aa06956329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.483558 13078 consensus_queue.cc:260] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.531149 13079 tablet_bootstrap.cc:492] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:37.531777 13079 ts_tablet_manager.cc:1403] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.204s	user 0.017s	sys 0.007s
I20260709 12:27:37.536113 13079 raft_consensus.cc:359] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.536852 13079 raft_consensus.cc:385] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.537140 13079 raft_consensus.cc:740] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.537942 13079 consensus_queue.cc:260] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.555297 13079 ts_tablet_manager.cc:1434] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:27:37.572345 13078 ts_tablet_manager.cc:1434] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.160s	user 0.007s	sys 0.004s
I20260709 12:27:37.573427 12940 tablet_service.cc:1511] Processing CreateTablet for tablet e2cbb9487e7d4b9a90f2a4aa06956329 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:27:37.559652 13080 tablet_bootstrap.cc:492] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:37.574702 13080 ts_tablet_manager.cc:1403] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.408s	user 0.021s	sys 0.003s
I20260709 12:27:37.574862 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2cbb9487e7d4b9a90f2a4aa06956329. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.578403 13078 tablet_bootstrap.cc:492] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:37.578682 13080 raft_consensus.cc:359] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
W20260709 12:27:37.578279 12776 consensus_peers.cc:597] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:37.588647 13083 tablet_bootstrap.cc:654] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.564452 13079 tablet_bootstrap.cc:492] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:37.609368 13078 tablet_bootstrap.cc:654] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.608192 13080 raft_consensus.cc:385] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.621404 13080 raft_consensus.cc:740] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.633970 13019 tablet_service.cc:1511] Processing CreateTablet for tablet dc65da4fd7234bbf96ecc59d58baa5c1 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:27:37.642237 13019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc65da4fd7234bbf96ecc59d58baa5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.646518 13022 tablet_service.cc:1511] Processing CreateTablet for tablet ff6968296a8740f4b8c329c728a64b1b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:27:37.647995 13022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff6968296a8740f4b8c329c728a64b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.627156 12874 tablet_service.cc:1511] Processing CreateTablet for tablet dc65da4fd7234bbf96ecc59d58baa5c1 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:27:37.649423 12874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc65da4fd7234bbf96ecc59d58baa5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.649991 12940 tablet_service.cc:1511] Processing CreateTablet for tablet ff6968296a8740f4b8c329c728a64b1b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:27:37.650954 13079 tablet_bootstrap.cc:654] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.651379 13144 raft_consensus.cc:493] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.651356 12940 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff6968296a8740f4b8c329c728a64b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.652031 12797 tablet_service.cc:1511] Processing CreateTablet for tablet dc65da4fd7234bbf96ecc59d58baa5c1 (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:27:37.653807 12797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc65da4fd7234bbf96ecc59d58baa5c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.652040 13144 raft_consensus.cc:515] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.660097 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6c3b7e72b044a5b0f0806e9a6301cf" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:37.660218 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6c3b7e72b044a5b0f0806e9a6301cf" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:37.662611 13144 leader_election.cc:290] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:37.667475 12773 leader_election.cc:343] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.668773 13088 consensus_queue.cc:1048] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.672142 12798 tablet_service.cc:1511] Processing CreateTablet for tablet ff6968296a8740f4b8c329c728a64b1b (DEFAULT_TABLE table=test-workload [id=3e89391ac09f4845b3d7c802b31e0682]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:27:37.673640 12798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff6968296a8740f4b8c329c728a64b1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:37.698040 13095 raft_consensus.cc:493] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.698849 13095 raft_consensus.cc:515] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.702907 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:37.703554 13080 consensus_queue.cc:260] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.708541 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:37.708904 12774 leader_election.cc:343] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.709394 12774 leader_election.cc:304] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:37.715217 13144 consensus_queue.cc:1048] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260709 12:27:37.720826 12924 leader_election.cc:343] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.710932 13113 raft_consensus.cc:2749] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.705416 13083 tablet_bootstrap.cc:492] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:37.723865 13083 ts_tablet_manager.cc:1403] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.364s	user 0.023s	sys 0.006s
I20260709 12:27:37.727561 13080 ts_tablet_manager.cc:1434] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.152s	user 0.006s	sys 0.003s
W20260709 12:27:37.727900 12923 leader_election.cc:343] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.728927 13080 tablet_bootstrap.cc:492] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:37.728976 12923 leader_election.cc:304] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:37.705648 13095 leader_election.cc:290] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:37.736116 13079 tablet_bootstrap.cc:492] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:37.736783 13079 ts_tablet_manager.cc:1403] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.180s	user 0.030s	sys 0.020s
I20260709 12:27:37.732718 13095 raft_consensus.cc:2749] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.747238 13079 raft_consensus.cc:359] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.748162 13079 raft_consensus.cc:385] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.748337 13083 raft_consensus.cc:359] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.749631 13083 raft_consensus.cc:385] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.750028 13083 raft_consensus.cc:740] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.748493 13079 raft_consensus.cc:740] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.751543 13079 consensus_queue.cc:260] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.746106 13078 tablet_bootstrap.cc:492] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:37.753324 13078 ts_tablet_manager.cc:1403] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.175s	user 0.012s	sys 0.024s
I20260709 12:27:37.754232 13079 ts_tablet_manager.cc:1434] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:27:37.756807 13078 raft_consensus.cc:359] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:37.762662 13079 tablet_bootstrap.cc:492] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:37.770000 13079 tablet_bootstrap.cc:654] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.761686 13078 raft_consensus.cc:385] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.771435 13078 raft_consensus.cc:740] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.771139 13083 consensus_queue.cc:260] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.775385 13080 tablet_bootstrap.cc:654] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.787431 13141 raft_consensus.cc:493] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.788297 13141 raft_consensus.cc:515] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.792330 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b78cc7f12d4c3a81aed3af33111913" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:37.793232 12957 raft_consensus.cc:2468] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:37.794581 13141 leader_election.cc:290] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:37.795475 13125 consensus_queue.cc:1048] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:37.795948 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b78cc7f12d4c3a81aed3af33111913" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:37.796645 12810 raft_consensus.cc:2468] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:37.795537 12999 leader_election.cc:304] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:37.798870 13141 raft_consensus.cc:2804] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:37.799518 13141 raft_consensus.cc:493] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:37.799976 13141 raft_consensus.cc:3060] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.778699 13078 consensus_queue.cc:260] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:37.776883 13083 ts_tablet_manager.cc:1434] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.001s
I20260709 12:27:37.809358 13083 tablet_bootstrap.cc:492] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:37.815889 13078 ts_tablet_manager.cc:1434] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.062s	user 0.008s	sys 0.000s
I20260709 12:27:37.817196 13078 tablet_bootstrap.cc:492] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:37.819380 13083 tablet_bootstrap.cc:654] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.831298 13141 raft_consensus.cc:515] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.832959 13083 tablet_bootstrap.cc:492] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:37.833534 13083 ts_tablet_manager.cc:1403] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.005s
I20260709 12:27:37.836823 13083 raft_consensus.cc:359] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.837080 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b78cc7f12d4c3a81aed3af33111913" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:37.837450 13078 tablet_bootstrap.cc:654] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.837639 13083 raft_consensus.cc:385] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.837891 12810 raft_consensus.cc:3060] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.838037 13083 raft_consensus.cc:740] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.841137 13141 leader_election.cc:290] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:37.845293 13079 tablet_bootstrap.cc:492] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:37.843609 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56b78cc7f12d4c3a81aed3af33111913" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:37.846240 13079 ts_tablet_manager.cc:1403] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.084s	user 0.031s	sys 0.018s
I20260709 12:27:37.846361 12957 raft_consensus.cc:3060] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:37.849159 13079 raft_consensus.cc:359] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.849931 13079 raft_consensus.cc:385] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.850190 13079 raft_consensus.cc:740] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.843499 13083 consensus_queue.cc:260] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.852416 13079 consensus_queue.cc:260] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.855165 12810 raft_consensus.cc:2468] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:37.855495 13083 ts_tablet_manager.cc:1434] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.004s
I20260709 12:27:37.855309 12957 raft_consensus.cc:2468] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:37.857012 12998 leader_election.cc:304] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:37.858016 13141 raft_consensus.cc:2804] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:37.858757 13079 ts_tablet_manager.cc:1434] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:27:37.859448 13141 raft_consensus.cc:697] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:37.859895 13079 tablet_bootstrap.cc:492] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:37.860507 13141 consensus_queue.cc:237] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.857013 13083 tablet_bootstrap.cc:492] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:37.862313 13080 tablet_bootstrap.cc:492] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:37.863827 13078 tablet_bootstrap.cc:492] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:37.867102 13078 ts_tablet_manager.cc:1403] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.050s	user 0.009s	sys 0.007s
I20260709 12:27:37.870527 13078 raft_consensus.cc:359] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:37.880203 13079 tablet_bootstrap.cc:654] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.880319 13078 raft_consensus.cc:385] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.877720 12706 catalog_manager.cc:5697] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:37.880718 13078 raft_consensus.cc:740] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.880959 13083 tablet_bootstrap.cc:654] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.881819 13078 consensus_queue.cc:260] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:37.889606 13078 ts_tablet_manager.cc:1434] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.002s
I20260709 12:27:37.867568 13080 ts_tablet_manager.cc:1403] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.139s	user 0.023s	sys 0.012s
I20260709 12:27:37.897118 13083 tablet_bootstrap.cc:492] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:37.897204 13079 tablet_bootstrap.cc:492] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:37.897931 13083 ts_tablet_manager.cc:1403] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.041s	user 0.011s	sys 0.009s
I20260709 12:27:37.898000 13079 ts_tablet_manager.cc:1403] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.038s	user 0.015s	sys 0.008s
I20260709 12:27:37.899227 13080 raft_consensus.cc:359] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.900095 13080 raft_consensus.cc:385] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.900413 13080 raft_consensus.cc:740] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.901295 13078 tablet_bootstrap.cc:492] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:37.901584 13083 raft_consensus.cc:359] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.902312 13083 raft_consensus.cc:385] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.902619 13083 raft_consensus.cc:740] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.903844 13079 raft_consensus.cc:359] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.904564 13079 raft_consensus.cc:385] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.903616 13083 consensus_queue.cc:260] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.904852 13079 raft_consensus.cc:740] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.905716 13079 consensus_queue.cc:260] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.907604 13079 ts_tablet_manager.cc:1434] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:27:37.908582 13079 tablet_bootstrap.cc:492] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:37.908955 13078 tablet_bootstrap.cc:654] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.913947 13083 ts_tablet_manager.cc:1434] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.001s
I20260709 12:27:37.915329 13079 tablet_bootstrap.cc:654] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.915599 13083 tablet_bootstrap.cc:492] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:37.917969 13080 consensus_queue.cc:260] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.926091 13080 ts_tablet_manager.cc:1434] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.030s	user 0.007s	sys 0.000s
I20260709 12:27:37.927731 13080 tablet_bootstrap.cc:492] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:37.936880 13080 tablet_bootstrap.cc:654] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.940044 13083 tablet_bootstrap.cc:654] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.941007 13086 raft_consensus.cc:493] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:37.941603 13086 raft_consensus.cc:515] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:37.942971 13078 tablet_bootstrap.cc:492] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:37.943553 13078 ts_tablet_manager.cc:1403] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.051s	user 0.005s	sys 0.030s
I20260709 12:27:37.944087 13086 leader_election.cc:290] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:37.945295 13079 tablet_bootstrap.cc:492] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:37.945941 13079 ts_tablet_manager.cc:1403] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.038s	user 0.010s	sys 0.013s
I20260709 12:27:37.946095 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c22979583614eb8b61240b2fa2424e9" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:37.946431 13078 raft_consensus.cc:359] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.947945 13078 raft_consensus.cc:385] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.948244 13078 raft_consensus.cc:740] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.948925 13078 consensus_queue.cc:260] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.948964 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c22979583614eb8b61240b2fa2424e9" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:37.950362 12773 leader_election.cc:343] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.950914 13078 ts_tablet_manager.cc:1434] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.002s
I20260709 12:27:37.951653 13079 raft_consensus.cc:359] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
W20260709 12:27:37.955111 12776 leader_election.cc:343] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:37.955989 13078 tablet_bootstrap.cc:492] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:37.956519 12776 leader_election.cc:304] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:37.958415 13086 raft_consensus.cc:2749] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:37.955277 13079 raft_consensus.cc:385] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.960052 13079 raft_consensus.cc:740] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.961653 13079 consensus_queue.cc:260] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.970611 13078 tablet_bootstrap.cc:654] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:37.973341 13083 tablet_bootstrap.cc:492] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:37.973963 13083 ts_tablet_manager.cc:1403] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.059s	user 0.007s	sys 0.015s
I20260709 12:27:37.977170 13083 raft_consensus.cc:359] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.977990 13083 raft_consensus.cc:385] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.978302 13083 raft_consensus.cc:740] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.981714 13083 consensus_queue.cc:260] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:37.985543 13079 ts_tablet_manager.cc:1434] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.039s	user 0.002s	sys 0.007s
I20260709 12:27:37.985592 13083 ts_tablet_manager.cc:1434] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.011s	user 0.000s	sys 0.010s
I20260709 12:27:37.987964 13083 tablet_bootstrap.cc:492] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:37.989071 13079 tablet_bootstrap.cc:492] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:37.991681 13078 tablet_bootstrap.cc:492] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:37.992327 13078 ts_tablet_manager.cc:1403] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.037s	user 0.015s	sys 0.002s
I20260709 12:27:37.994045 13080 tablet_bootstrap.cc:492] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:37.994853 13080 ts_tablet_manager.cc:1403] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.067s	user 0.021s	sys 0.029s
I20260709 12:27:37.995716 13078 raft_consensus.cc:359] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.996695 13078 raft_consensus.cc:385] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:37.997056 13078 raft_consensus.cc:740] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:37.997905 13078 consensus_queue.cc:260] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:37.999930 13083 tablet_bootstrap.cc:654] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.000375 13078 ts_tablet_manager.cc:1434] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:38.002017 13080 raft_consensus.cc:359] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.002863 13080 raft_consensus.cc:385] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.003280 13080 raft_consensus.cc:740] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.004587 13080 consensus_queue.cc:260] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.007526 13080 ts_tablet_manager.cc:1434] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:27:38.008615 13086 raft_consensus.cc:493] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.009418 13079 tablet_bootstrap.cc:654] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.009218 13086 raft_consensus.cc:515] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.010452 13083 tablet_bootstrap.cc:492] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:38.011006 13083 ts_tablet_manager.cc:1403] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.023s	user 0.013s	sys 0.003s
I20260709 12:27:38.011693 13086 leader_election.cc:290] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:38.014161 13078 tablet_bootstrap.cc:492] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:38.014429 13083 raft_consensus.cc:359] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.015552 13083 raft_consensus.cc:385] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.015887 13083 raft_consensus.cc:740] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.017014 13083 consensus_queue.cc:260] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.018201 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82ca562394a94f40ae84d9607fb885f4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:38.018889 12957 raft_consensus.cc:2468] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:38.019466 13083 ts_tablet_manager.cc:1434] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:38.019742 13141 raft_consensus.cc:493] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.020228 13141 raft_consensus.cc:515] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.022416 13141 leader_election.cc:290] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:38.022821 13078 tablet_bootstrap.cc:654] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.024225 13083 tablet_bootstrap.cc:492] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:38.024705 12776 leader_election.cc:304] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:38.025593 13086 raft_consensus.cc:2804] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.025995 13086 raft_consensus.cc:493] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.026315 13086 raft_consensus.cc:3060] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.031252 13083 tablet_bootstrap.cc:654] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.034991 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:38.035996 12810 raft_consensus.cc:2468] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:38.037557 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:38.038038 13125 raft_consensus.cc:493] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.038264 12957 raft_consensus.cc:2468] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:38.038527 13125 raft_consensus.cc:515] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.039851 12999 leader_election.cc:304] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:38.040696 13125 leader_election.cc:290] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:38.041574 13142 raft_consensus.cc:493] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.042088 13142 raft_consensus.cc:515] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.044462 13142 leader_election.cc:290] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:38.044502 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82ca562394a94f40ae84d9607fb885f4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:38.046326 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:38.047904 12959 raft_consensus.cc:3060] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.048007 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:38.051518 12849 leader_election.cc:304] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:38.051971 13141 raft_consensus.cc:2804] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.052659 13141 raft_consensus.cc:493] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.053123 13141 raft_consensus.cc:3060] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.053367 13125 raft_consensus.cc:2749] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:38.055860 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224f359a7c884a7890b05f3098408d31" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:38.056912 12885 raft_consensus.cc:2468] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:38.059439 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224f359a7c884a7890b05f3098408d31" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:38.060969 12957 raft_consensus.cc:2468] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
W20260709 12:27:38.055101 12773 leader_election.cc:343] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.063042 12776 leader_election.cc:304] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:38.064344 13142 raft_consensus.cc:2804] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.064810 13142 raft_consensus.cc:493] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.065173 13142 raft_consensus.cc:3060] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.074213 13142 raft_consensus.cc:515] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.076617 13142 leader_election.cc:290] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:38.077898 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224f359a7c884a7890b05f3098408d31" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:38.079126 12885 raft_consensus.cc:3060] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.085835 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224f359a7c884a7890b05f3098408d31" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:38.087556 12957 raft_consensus.cc:3060] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.091964 13080 tablet_bootstrap.cc:492] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:38.101187 13107 raft_consensus.cc:493] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.102382 13107 raft_consensus.cc:515] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.106863 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:38.109027 12885 raft_consensus.cc:2468] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
W20260709 12:27:38.109400 12996 leader_election.cc:343] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.113718 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:38.097002 13086 raft_consensus.cc:515] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:38.116653 12999 leader_election.cc:343] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.117718 12999 leader_election.cc:304] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:38.116569 12773 leader_election.cc:304] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:38.118772 13109 raft_consensus.cc:2749] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:38.112243 13107 leader_election.cc:290] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:38.119515 13142 raft_consensus.cc:2804] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:38.120463 13080 tablet_bootstrap.cc:654] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.120637 13142 raft_consensus.cc:697] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:38.121622 13078 tablet_bootstrap.cc:492] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:38.122253 13078 ts_tablet_manager.cc:1403] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.108s	user 0.037s	sys 0.045s
I20260709 12:27:38.122152 13142 consensus_queue.cc:237] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.126904 13078 raft_consensus.cc:359] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.127737 13078 raft_consensus.cc:385] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.127992 13078 raft_consensus.cc:740] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.128762 13078 consensus_queue.cc:260] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.135752 13109 raft_consensus.cc:493] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.140640 13109 raft_consensus.cc:515] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.146019 12885 raft_consensus.cc:3060] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.149891 13079 tablet_bootstrap.cc:492] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:38.150758 13079 ts_tablet_manager.cc:1403] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.162s	user 0.016s	sys 0.006s
I20260709 12:27:38.151387 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6c3b7e72b044a5b0f0806e9a6301cf" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:38.152470 12810 raft_consensus.cc:2468] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:38.154578 12998 leader_election.cc:304] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:38.155107 13109 leader_election.cc:290] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:38.133912 13141 raft_consensus.cc:515] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.156035 13107 raft_consensus.cc:2804] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.159904 13107 raft_consensus.cc:493] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.160295 13107 raft_consensus.cc:3060] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.160449 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6c3b7e72b044a5b0f0806e9a6301cf" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:38.162271 12996 leader_election.cc:343] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.166404 13144 raft_consensus.cc:493] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.166957 13144 raft_consensus.cc:515] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.169036 13086 leader_election.cc:290] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:38.172482 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35361d4194194047849e55975d5a91b9" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:38.173367 13109 raft_consensus.cc:493] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.173907 13109 raft_consensus.cc:515] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.175163 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82ca562394a94f40ae84d9607fb885f4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:38.175900 12958 raft_consensus.cc:3060] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.178345 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:38.173941 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35361d4194194047849e55975d5a91b9" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:38.180465 12998 leader_election.cc:343] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.181078 13109 leader_election.cc:290] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:38.181565 13125 raft_consensus.cc:493] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.180886 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:38.182106 13125 raft_consensus.cc:515] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.185312 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:38.186170 13125 leader_election.cc:290] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:38.186548 12955 raft_consensus.cc:3060] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.187426 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae9dd87d90f4d18b0d36431646603e3" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:38.188207 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae9dd87d90f4d18b0d36431646603e3" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:38.189033 13032 raft_consensus.cc:2468] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:38.189143 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82ca562394a94f40ae84d9607fb885f4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:38.169050 13144 leader_election.cc:290] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:38.188283 12954 raft_consensus.cc:2468] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:38.191520 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:38.192163 12810 raft_consensus.cc:3060] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.191001 12849 leader_election.cc:304] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
W20260709 12:27:38.183604 12773 leader_election.cc:343] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.182480 12956 raft_consensus.cc:2468] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
W20260709 12:27:38.194955 12774 leader_election.cc:343] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.196293 12774 leader_election.cc:304] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
W20260709 12:27:38.194502 12773 leader_election.cc:343] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.196962 12999 leader_election.cc:304] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:38.206374 13125 raft_consensus.cc:2804] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.208413 13125 raft_consensus.cc:493] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.205359 13144 raft_consensus.cc:2749] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:38.209625 12955 raft_consensus.cc:2468] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:38.209020 13125 raft_consensus.cc:3060] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.211984 12999 leader_election.cc:304] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:38.205691 13109 raft_consensus.cc:2804] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.213805 13109 raft_consensus.cc:493] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.214216 13109 raft_consensus.cc:3060] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.224309 13141 leader_election.cc:290] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:38.224583 13171 raft_consensus.cc:2804] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:38.225240 13171 raft_consensus.cc:697] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:38.227694 13078 ts_tablet_manager.cc:1434] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.105s	user 0.018s	sys 0.045s
I20260709 12:27:38.229064 13078 tablet_bootstrap.cc:492] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:38.229036 13079 raft_consensus.cc:359] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.165935 12957 raft_consensus.cc:2468] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:38.216398 13107 raft_consensus.cc:515] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.229539 13083 tablet_bootstrap.cc:492] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:38.226156 13171 consensus_queue.cc:237] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.235754 12958 raft_consensus.cc:2468] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:38.241087 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6c3b7e72b044a5b0f0806e9a6301cf" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:38.243029 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6c3b7e72b044a5b0f0806e9a6301cf" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:38.252530 12809 raft_consensus.cc:3060] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:38.253353 12996 leader_election.cc:343] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.237244 13083 ts_tablet_manager.cc:1403] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.213s	user 0.045s	sys 0.110s
I20260709 12:27:38.238133 13078 tablet_bootstrap.cc:654] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.257319 13083 raft_consensus.cc:359] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.258167 13083 raft_consensus.cc:385] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.258474 13083 raft_consensus.cc:740] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.259334 13083 consensus_queue.cc:260] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.261885 13083 ts_tablet_manager.cc:1434] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:27:38.261938 12809 raft_consensus.cc:2468] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:38.263118 12776 leader_election.cc:304] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:38.263770 12810 raft_consensus.cc:2468] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:38.264452 13142 raft_consensus.cc:2804] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:38.265086 13142 raft_consensus.cc:697] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:38.265519 13083 tablet_bootstrap.cc:492] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:38.265718 12998 leader_election.cc:304] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:38.266161 13142 consensus_queue.cc:237] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.251801 13107 leader_election.cc:290] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:38.230221 13079 raft_consensus.cc:385] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.267707 13141 raft_consensus.cc:2804] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:38.268311 13080 tablet_bootstrap.cc:492] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:38.246039 13125 raft_consensus.cc:515] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.271193 13080 ts_tablet_manager.cc:1403] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.180s	user 0.019s	sys 0.007s
I20260709 12:27:38.259801 12706 catalog_manager.cc:5697] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:38.272948 13083 tablet_bootstrap.cc:654] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.273742 13141 raft_consensus.cc:697] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:38.274886 13141 consensus_queue.cc:237] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.277555 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae9dd87d90f4d18b0d36431646603e3" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:38.278282 12958 raft_consensus.cc:3060] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.279392 13079 raft_consensus.cc:740] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.279831 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eae9dd87d90f4d18b0d36431646603e3" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:38.280248 13079 consensus_queue.cc:260] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.280843 13032 raft_consensus.cc:3060] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.283723 13125 leader_election.cc:290] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:38.316102 13080 raft_consensus.cc:359] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.324791 13079 ts_tablet_manager.cc:1434] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.174s	user 0.004s	sys 0.006s
I20260709 12:27:38.326560 13079 tablet_bootstrap.cc:492] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:38.331214 13083 tablet_bootstrap.cc:492] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:38.334039 13083 ts_tablet_manager.cc:1403] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.069s	user 0.032s	sys 0.024s
I20260709 12:27:38.319579 13080 raft_consensus.cc:385] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.345762 13078 tablet_bootstrap.cc:492] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:38.346611 13078 ts_tablet_manager.cc:1403] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.118s	user 0.025s	sys 0.006s
I20260709 12:27:38.341357 13080 raft_consensus.cc:740] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.358896 13078 raft_consensus.cc:359] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.359668 13078 raft_consensus.cc:385] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.360026 13078 raft_consensus.cc:740] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.362289 13078 consensus_queue.cc:260] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.353819 13141 consensus_queue.cc:1048] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:27:38.363722 13142 raft_consensus.cc:493] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.364969 13142 raft_consensus.cc:515] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.365906 13078 ts_tablet_manager.cc:1434] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:27:38.374640 13078 tablet_bootstrap.cc:492] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:38.376719 13079 tablet_bootstrap.cc:654] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.377360 13109 raft_consensus.cc:515] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.370438 13032 raft_consensus.cc:2468] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:38.379860 13080 consensus_queue.cc:260] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:38.394002 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:38.376219 12958 raft_consensus.cc:2468] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:38.395545 13083 raft_consensus.cc:359] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.397549 13083 raft_consensus.cc:385] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.397980 13083 raft_consensus.cc:740] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.399080 12849 leader_election.cc:304] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:38.399864 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:38.400189 13178 raft_consensus.cc:2804] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:38.400637 12959 raft_consensus.cc:3060] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.400964 13178 raft_consensus.cc:697] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:38.401978 13178 consensus_queue.cc:237] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.398684 13083 consensus_queue.cc:260] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.407933 13142 leader_election.cc:290] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:38.406986 12708 catalog_manager.cc:5697] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:38.408890 13080 ts_tablet_manager.cc:1434] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.137s	user 0.011s	sys 0.000s
I20260709 12:27:38.416468 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:38.417037 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
W20260709 12:27:38.418545 12998 leader_election.cc:343] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:38.418552 12773 leader_election.cc:343] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.422844 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:38.424459 12776 leader_election.cc:343] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.425041 12776 leader_election.cc:304] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:38.414629 13109 leader_election.cc:290] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:38.427538 13080 tablet_bootstrap.cc:492] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:38.430071 12706 catalog_manager.cc:5697] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:38.432873 13144 raft_consensus.cc:2749] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:38.434026 12959 raft_consensus.cc:2468] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:38.436475 12999 leader_election.cc:304] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:38.437599 13141 raft_consensus.cc:2804] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:38.434963 13078 tablet_bootstrap.cc:654] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.438421 13141 raft_consensus.cc:697] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:38.439661 13141 consensus_queue.cc:237] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.435191 13083 ts_tablet_manager.cc:1434] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.100s	user 0.009s	sys 0.000s
I20260709 12:27:38.445307 13083 tablet_bootstrap.cc:492] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:38.450999 13079 tablet_bootstrap.cc:492] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:38.451242 13080 tablet_bootstrap.cc:654] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.451584 13079 ts_tablet_manager.cc:1403] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.125s	user 0.015s	sys 0.000s
I20260709 12:27:38.458973 13079 raft_consensus.cc:359] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.459667 13079 raft_consensus.cc:385] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.459923 13079 raft_consensus.cc:740] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.463125 13109 consensus_queue.cc:1048] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:38.464270 12708 catalog_manager.cc:5697] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:38.472982 12707 catalog_manager.cc:5697] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 1 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:38.476591 13083 tablet_bootstrap.cc:654] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.477444 13079 consensus_queue.cc:260] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.484997 13079 ts_tablet_manager.cc:1434] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.000s
I20260709 12:27:38.486451 13079 tablet_bootstrap.cc:492] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:38.506914 13079 tablet_bootstrap.cc:654] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.524550 12708 catalog_manager.cc:5697] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:38.530632 13079 tablet_bootstrap.cc:492] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:38.531250 13079 ts_tablet_manager.cc:1403] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.045s	user 0.022s	sys 0.003s
I20260709 12:27:38.534474 13080 tablet_bootstrap.cc:492] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:38.534255 13079 raft_consensus.cc:359] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.535059 13080 ts_tablet_manager.cc:1403] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.108s	user 0.029s	sys 0.034s
I20260709 12:27:38.535177 13079 raft_consensus.cc:385] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.535530 13079 raft_consensus.cc:740] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.534369 13078 tablet_bootstrap.cc:492] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:38.537238 13078 ts_tablet_manager.cc:1403] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.163s	user 0.009s	sys 0.007s
I20260709 12:27:38.536258 13079 consensus_queue.cc:260] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.538112 13080 raft_consensus.cc:359] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.539465 13080 raft_consensus.cc:385] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.539746 13080 raft_consensus.cc:740] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.540419 13080 consensus_queue.cc:260] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.540761 13083 tablet_bootstrap.cc:492] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:38.540544 13078 raft_consensus.cc:359] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.541345 13083 ts_tablet_manager.cc:1403] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.096s	user 0.015s	sys 0.000s
I20260709 12:27:38.541514 13078 raft_consensus.cc:385] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.541949 13078 raft_consensus.cc:740] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.542814 13078 consensus_queue.cc:260] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.544384 13083 raft_consensus.cc:359] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.545188 13083 raft_consensus.cc:385] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.545449 13083 raft_consensus.cc:740] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.546401 13078 ts_tablet_manager.cc:1434] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:27:38.546900 13083 consensus_queue.cc:260] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.547727 13078 tablet_bootstrap.cc:492] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:38.549407 13152 raft_consensus.cc:493] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.550096 13152 raft_consensus.cc:515] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.552455 13080 ts_tablet_manager.cc:1434] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.017s	user 0.004s	sys 0.002s
I20260709 12:27:38.549640 13083 ts_tablet_manager.cc:1434] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:38.554001 13080 tablet_bootstrap.cc:492] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:38.554574 13079 ts_tablet_manager.cc:1434] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 12:27:38.555740 13079 tablet_bootstrap.cc:492] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:38.559681 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:38.561492 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:38.562420 13032 raft_consensus.cc:2468] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:38.567857 13080 tablet_bootstrap.cc:654] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:38.568405 12922 leader_election.cc:343] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.569841 12923 leader_election.cc:304] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:38.571280 13152 leader_election.cc:290] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:38.572296 13152 raft_consensus.cc:2804] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.573927 13152 raft_consensus.cc:493] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.574898 13152 raft_consensus.cc:3060] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.577301 13079 tablet_bootstrap.cc:654] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.579941 13083 tablet_bootstrap.cc:492] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:38.587905 13083 tablet_bootstrap.cc:654] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.595709 13083 tablet_bootstrap.cc:492] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:38.596251 13083 ts_tablet_manager.cc:1403] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.000s
I20260709 12:27:38.596782 13079 tablet_bootstrap.cc:492] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:38.597458 13079 ts_tablet_manager.cc:1403] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.042s	user 0.012s	sys 0.004s
I20260709 12:27:38.605647 13144 consensus_queue.cc:1048] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:38.606444 13080 tablet_bootstrap.cc:492] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:38.606990 13080 ts_tablet_manager.cc:1403] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.053s	user 0.010s	sys 0.010s
I20260709 12:27:38.602468 13079 raft_consensus.cc:359] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.609123 13079 raft_consensus.cc:385] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.609431 13079 raft_consensus.cc:740] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.609927 13080 raft_consensus.cc:359] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.610247 13079 consensus_queue.cc:260] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.610635 13080 raft_consensus.cc:385] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.613466 13079 ts_tablet_manager.cc:1434] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:27:38.613796 13078 tablet_bootstrap.cc:654] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.614599 13079 tablet_bootstrap.cc:492] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:38.611588 13080 raft_consensus.cc:740] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.618630 13080 consensus_queue.cc:260] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.622071 13176 raft_consensus.cc:493] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.622709 13176 raft_consensus.cc:515] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.623697 13080 ts_tablet_manager.cc:1434] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.001s
I20260709 12:27:38.625480 13080 tablet_bootstrap.cc:492] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:38.626205 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82ca562394a94f40ae84d9607fb885f4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:38.628448 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82ca562394a94f40ae84d9607fb885f4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:38.629395 12959 raft_consensus.cc:2393] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate b543f973837c43b19b7d17feaaca5325 in this term.
I20260709 12:27:38.630810 12851 leader_election.cc:304] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:38.627085 13176 leader_election.cc:290] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:38.633141 13079 tablet_bootstrap.cc:654] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.634004 13176 raft_consensus.cc:3060] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.640784 13083 raft_consensus.cc:359] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.643589 13083 raft_consensus.cc:385] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.644001 13083 raft_consensus.cc:740] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.645210 13152 raft_consensus.cc:515] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.660192 13078 tablet_bootstrap.cc:492] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:38.660856 13078 ts_tablet_manager.cc:1403] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.113s	user 0.010s	sys 0.010s
I20260709 12:27:38.661350 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:38.648411 13083 consensus_queue.cc:260] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.667546 13080 tablet_bootstrap.cc:654] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:38.676339 12922 leader_election.cc:343] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.677728 13079 tablet_bootstrap.cc:492] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:38.678339 13079 ts_tablet_manager.cc:1403] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.064s	user 0.010s	sys 0.005s
I20260709 12:27:38.678697 13152 leader_election.cc:290] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:38.680222 13083 ts_tablet_manager.cc:1434] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.084s	user 0.009s	sys 0.000s
I20260709 12:27:38.676246 13176 raft_consensus.cc:2749] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:38.681320 13083 tablet_bootstrap.cc:492] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:38.682719 13079 raft_consensus.cc:359] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.683547 13079 raft_consensus.cc:385] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.683878 13079 raft_consensus.cc:740] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.684641 13079 consensus_queue.cc:260] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.685014 13078 raft_consensus.cc:359] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.685680 13078 raft_consensus.cc:385] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.685966 13078 raft_consensus.cc:740] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.682878 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:38.686635 13078 consensus_queue.cc:260] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.686734 13032 raft_consensus.cc:3060] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.693377 13083 tablet_bootstrap.cc:654] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.687058 13079 ts_tablet_manager.cc:1434] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.008s	user 0.000s	sys 0.007s
I20260709 12:27:38.696292 13079 tablet_bootstrap.cc:492] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:38.688872 13078 ts_tablet_manager.cc:1434] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.000s
I20260709 12:27:38.696796 13152 raft_consensus.cc:493] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.697772 13078 tablet_bootstrap.cc:492] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:38.697507 13152 raft_consensus.cc:515] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.691641 13142 raft_consensus.cc:493] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.699738 13142 raft_consensus.cc:515] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.705365 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:38.706171 13142 leader_election.cc:290] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:38.706439 12959 raft_consensus.cc:2393] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b543f973837c43b19b7d17feaaca5325 in current term 1: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
I20260709 12:27:38.708755 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:38.710433 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c22979583614eb8b61240b2fa2424e9" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:38.711303 12809 raft_consensus.cc:2468] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:38.713989 12776 leader_election.cc:304] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:38.714922 13079 tablet_bootstrap.cc:654] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.715778 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c22979583614eb8b61240b2fa2424e9" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:38.716624 12885 raft_consensus.cc:2468] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:38.719051 12924 leader_election.cc:304] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:38.720170 13196 raft_consensus.cc:2804] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.720495 13086 raft_consensus.cc:3060] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.720696 13196 raft_consensus.cc:493] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.721204 13196 raft_consensus.cc:3060] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.708140 13152 leader_election.cc:290] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:38.726222 13171 raft_consensus.cc:493] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.726660 13078 tablet_bootstrap.cc:654] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.726984 13171 raft_consensus.cc:515] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.733606 13171 leader_election.cc:290] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:38.728797 13032 raft_consensus.cc:2468] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:38.742141 13142 consensus_queue.cc:1048] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:38.744663 12923 leader_election.cc:304] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:38.745958 13152 raft_consensus.cc:2804] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:38.747004 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "070b536983ee48e1853040e1652619e2" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:38.748705 12996 leader_election.cc:343] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.749071 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "070b536983ee48e1853040e1652619e2" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:38.749372 13144 consensus_queue.cc:1048] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:38.749974 12959 raft_consensus.cc:2468] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:38.759555 12999 leader_election.cc:304] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:38.763561 13109 raft_consensus.cc:2804] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.763994 13109 raft_consensus.cc:493] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.765772 13083 tablet_bootstrap.cc:492] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:38.766471 13083 ts_tablet_manager.cc:1403] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.085s	user 0.006s	sys 0.026s
I20260709 12:27:38.775226 13142 consensus_queue.cc:1048] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:38.760951 13152 raft_consensus.cc:697] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:38.776517 13080 tablet_bootstrap.cc:492] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:38.764346 13109 raft_consensus.cc:3060] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.782243 13176 raft_consensus.cc:493] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.781635 13080 ts_tablet_manager.cc:1403] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.156s	user 0.014s	sys 0.006s
I20260709 12:27:38.778560 13086 raft_consensus.cc:2749] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:38.783902 13176 raft_consensus.cc:515] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.800745 13079 tablet_bootstrap.cc:492] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:38.801755 13079 ts_tablet_manager.cc:1403] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.106s	user 0.034s	sys 0.007s
I20260709 12:27:38.772159 13196 raft_consensus.cc:515] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.813825 12884 raft_consensus.cc:3060] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.808919 13141 consensus_queue.cc:1048] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:27:38.823634 13141 consensus_queue.cc:1048] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:38.831139 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6c3b7e72b044a5b0f0806e9a6301cf" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:38.805788 13152 consensus_queue.cc:237] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.835487 13171 consensus_queue.cc:1048] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:38.835807 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc6c3b7e72b044a5b0f0806e9a6301cf" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:38.836647 13083 raft_consensus.cc:359] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.837502 13083 raft_consensus.cc:385] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.837772 13083 raft_consensus.cc:740] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.838510 13083 consensus_queue.cc:260] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.846524 13079 raft_consensus.cc:359] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.848538 13079 raft_consensus.cc:385] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.808802 13206 raft_consensus.cc:493] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.848887 13079 raft_consensus.cc:740] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.829839 13176 leader_election.cc:290] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:38.852721 13079 consensus_queue.cc:260] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.854693 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c22979583614eb8b61240b2fa2424e9" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:38.859836 13083 ts_tablet_manager.cc:1434] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.093s	user 0.008s	sys 0.000s
I20260709 12:27:38.861770 13083 tablet_bootstrap.cc:492] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:38.851617 13141 raft_consensus.cc:493] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:38.856863 13171 consensus_queue.cc:1048] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:27:38.860980 12810 raft_consensus.cc:3060] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.872074 13079 ts_tablet_manager.cc:1434] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.070s	user 0.008s	sys 0.002s
I20260709 12:27:38.872478 13078 tablet_bootstrap.cc:492] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:38.873003 13078 ts_tablet_manager.cc:1403] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.175s	user 0.009s	sys 0.014s
I20260709 12:27:38.874531 13079 tablet_bootstrap.cc:492] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:38.875927 13080 raft_consensus.cc:359] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.877362 13080 raft_consensus.cc:385] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.877640 13080 raft_consensus.cc:740] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.878401 13080 consensus_queue.cc:260] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.880031 13083 tablet_bootstrap.cc:654] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.880558 13080 ts_tablet_manager.cc:1434] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.097s	user 0.005s	sys 0.002s
I20260709 12:27:38.881682 13080 tablet_bootstrap.cc:492] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:38.869026 13141 raft_consensus.cc:515] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:38.864670 12849 leader_election.cc:304] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:38.892254 13141 leader_election.cc:290] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:38.894351 13125 raft_consensus.cc:2749] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:38.883538 13196 leader_election.cc:290] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:38.855633 13206 raft_consensus.cc:515] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:38.906905 13078 raft_consensus.cc:359] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.908152 13078 raft_consensus.cc:385] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:38.908447 13078 raft_consensus.cc:740] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:38.910213 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c22979583614eb8b61240b2fa2424e9" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:38.910993 12884 raft_consensus.cc:3060] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.917091 13206 leader_election.cc:290] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:38.918025 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acdd78885d1044a3a523c14603afcb9a" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:38.918874 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acdd78885d1044a3a523c14603afcb9a" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:38.919711 12809 raft_consensus.cc:2468] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
W20260709 12:27:38.920125 12923 leader_election.cc:343] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.921219 12924 leader_election.cc:304] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:38.923367 13080 tablet_bootstrap.cc:654] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.924378 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35361d4194194047849e55975d5a91b9" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:38.926365 12996 leader_election.cc:343] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:38.926472 13206 raft_consensus.cc:2804] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.927201 13206 raft_consensus.cc:493] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.927448 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35361d4194194047849e55975d5a91b9" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:38.928334 12809 raft_consensus.cc:2468] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:38.928416 12884 raft_consensus.cc:2468] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:38.929855 12922 leader_election.cc:304] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:38.930980 13209 raft_consensus.cc:2804] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:38.930886 12998 leader_election.cc:304] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:38.931757 13209 raft_consensus.cc:697] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:38.927675 13206 raft_consensus.cc:3060] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:38.932457 13125 consensus_queue.cc:1048] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:38.932796 13209 consensus_queue.cc:237] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:38.937165 13079 tablet_bootstrap.cc:654] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:38.938705 13078 consensus_queue.cc:260] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:38.952103 13171 consensus_queue.cc:1048] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:27:38.953528 12810 raft_consensus.cc:2468] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:38.975033 13171 raft_consensus.cc:2804] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:38.975572 13171 raft_consensus.cc:493] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:38.975927 13171 raft_consensus.cc:3060] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.017848 13109 raft_consensus.cc:515] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.023994 13083 tablet_bootstrap.cc:492] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:39.025323 13083 ts_tablet_manager.cc:1403] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.165s	user 0.015s	sys 0.004s
I20260709 12:27:39.033725 13079 tablet_bootstrap.cc:492] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:39.034346 13079 ts_tablet_manager.cc:1403] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.161s	user 0.021s	sys 0.000s
I20260709 12:27:39.037326 13079 raft_consensus.cc:359] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.037994 13079 raft_consensus.cc:385] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.038269 13079 raft_consensus.cc:740] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.040607 13079 consensus_queue.cc:260] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.042408 13079 ts_tablet_manager.cc:1434] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:39.043422 13079 tablet_bootstrap.cc:492] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:39.044780 13125 consensus_queue.cc:1048] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:39.059262 13078 ts_tablet_manager.cc:1434] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.186s	user 0.008s	sys 0.009s
I20260709 12:27:39.062242 13078 tablet_bootstrap.cc:492] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
W20260709 12:27:39.022027 12999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 85577.5us
I20260709 12:27:39.068279 13227 raft_consensus.cc:493] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.070127 13079 tablet_bootstrap.cc:654] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.074502 13221 raft_consensus.cc:493] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.066131 13171 raft_consensus.cc:515] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.069180 13227 raft_consensus.cc:515] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.079138 13224 raft_consensus.cc:493] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.102995 13080 tablet_bootstrap.cc:492] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:39.103685 13080 ts_tablet_manager.cc:1403] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.222s	user 0.015s	sys 0.010s
I20260709 12:27:39.104050 13109 leader_election.cc:290] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:39.091946 13221 raft_consensus.cc:515] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.093977 13206 raft_consensus.cc:515] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:39.083786 12776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 115429us
I20260709 12:27:39.112825 13083 raft_consensus.cc:359] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.114205 13083 raft_consensus.cc:385] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.114562 13083 raft_consensus.cc:740] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.116688 13083 consensus_queue.cc:260] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.106451 13224 raft_consensus.cc:515] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.121589 13229 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.119041 13080 raft_consensus.cc:359] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.122583 13206 leader_election.cc:290] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:39.132843 13080 raft_consensus.cc:385] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.136195 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:39.136500 13080 raft_consensus.cc:740] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.124184 13224 leader_election.cc:290] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:39.126809 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "070b536983ee48e1853040e1652619e2" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:39.137846 12955 raft_consensus.cc:3060] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.130211 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c1da90a8da49f6be72a8361eead7df" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:39.129536 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acdd78885d1044a3a523c14603afcb9a" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:39.139757 12808 raft_consensus.cc:3060] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:39.140655 12774 leader_election.cc:343] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.128719 13221 leader_election.cc:290] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:39.126776 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "070b536983ee48e1853040e1652619e2" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:39.123013 13229 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.127352 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:39.146646 12999 leader_election.cc:343] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.150306 13080 consensus_queue.cc:260] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.151805 12883 raft_consensus.cc:3060] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:39.153635 13217 log.cc:927] Time spent T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5: Append to log took a long time: real 0.121s	user 0.002s	sys 0.000s
I20260709 12:27:39.161585 13083 ts_tablet_manager.cc:1434] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.136s	user 0.009s	sys 0.002s
I20260709 12:27:39.160511 13221 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.163662 13171 leader_election.cc:290] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:39.165314 13229 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:39.166128 13079 tablet_bootstrap.cc:492] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:39.166843 13079 ts_tablet_manager.cc:1403] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.124s	user 0.020s	sys 0.003s
I20260709 12:27:39.169438 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acdd78885d1044a3a523c14603afcb9a" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:39.169962 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35361d4194194047849e55975d5a91b9" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
W20260709 12:27:39.166567 12996 leader_election.cc:343] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.171697 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35361d4194194047849e55975d5a91b9" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:39.172639 12806 raft_consensus.cc:3060] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.172839 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c1da90a8da49f6be72a8361eead7df" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:39.173736 12959 raft_consensus.cc:2468] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:39.178740 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:39.179953 13227 leader_election.cc:290] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:39.180801 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:39.171244 12996 leader_election.cc:304] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.166177 13083 tablet_bootstrap.cc:492] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:39.163697 13221 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.178094 13079 raft_consensus.cc:359] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.182448 12955 raft_consensus.cc:2468] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:39.194506 12999 leader_election.cc:304] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:39.196295 13223 raft_consensus.cc:2804] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:39.196382 13078 tablet_bootstrap.cc:654] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.196920 13223 raft_consensus.cc:697] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:39.197913 13223 consensus_queue.cc:237] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.199096 13142 raft_consensus.cc:493] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.199736 13142 raft_consensus.cc:515] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.201205 13109 raft_consensus.cc:2749] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.201359 13206 raft_consensus.cc:493] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:39.201947 12923 leader_election.cc:343] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.202059 13206 raft_consensus.cc:515] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.203176 12806 raft_consensus.cc:2468] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:39.203504 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:39.204838 13206 leader_election.cc:290] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:39.205886 12996 leader_election.cc:343] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:39.207057 12996 leader_election.cc:343] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.209100 13142 leader_election.cc:290] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:39.209851 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2b3043eef1a4f98b328054262cfe08c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:39.210427 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:39.211989 12774 leader_election.cc:343] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:39.212373 12922 leader_election.cc:343] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.206164 12808 raft_consensus.cc:2468] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:39.190919 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:39.219585 12959 raft_consensus.cc:2468] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:39.219993 12924 leader_election.cc:304] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:39.191026 13171 consensus_queue.cc:1048] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.030s
I20260709 12:27:39.221459 13206 raft_consensus.cc:2804] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:27:39.189002 12773 leader_election.cc:343] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.222457 13206 raft_consensus.cc:697] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:39.190366 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:39.223606 13032 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:39.190544 12810 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:39.223814 13206 consensus_queue.cc:237] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.229941 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2b3043eef1a4f98b328054262cfe08c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:39.231086 12883 raft_consensus.cc:2468] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
W20260709 12:27:39.232102 12773 leader_election.cc:343] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.232362 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:39.232666 12773 leader_election.cc:304] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.233807 13219 raft_consensus.cc:2749] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.234432 12774 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:39.235014 13080 ts_tablet_manager.cc:1434] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.131s	user 0.015s	sys 0.004s
I20260709 12:27:39.236214 13080 tablet_bootstrap.cc:492] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:39.238256 13219 raft_consensus.cc:2804] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:27:39.238660 12773 leader_election.cc:343] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.192253 13221 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:39.238901 13219 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.188225 12776 leader_election.cc:304] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:39.243275 13219 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:39.244182 12776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 56102us
I20260709 12:27:39.192466 13079 raft_consensus.cc:385] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.247084 13079 raft_consensus.cc:740] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.247869 13079 consensus_queue.cc:260] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.241901 13078 tablet_bootstrap.cc:492] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:39.249171 13078 ts_tablet_manager.cc:1403] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.188s	user 0.008s	sys 0.008s
I20260709 12:27:39.250538 13125 raft_consensus.cc:493] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.250743 13079 ts_tablet_manager.cc:1434] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.084s	user 0.007s	sys 0.003s
I20260709 12:27:39.251168 13125 raft_consensus.cc:515] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.251968 13079 tablet_bootstrap.cc:492] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:39.253847 13125 leader_election.cc:290] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:39.253064 13078 raft_consensus.cc:359] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.258587 13078 raft_consensus.cc:385] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.258486 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:39.259263 13078 raft_consensus.cc:740] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.260236 13080 tablet_bootstrap.cc:654] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.260902 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:39.261003 13078 consensus_queue.cc:260] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.262817 13083 tablet_bootstrap.cc:654] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.264015 13078 ts_tablet_manager.cc:1434] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
I20260709 12:27:39.268584 13078 tablet_bootstrap.cc:492] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:39.267912 12703 catalog_manager.cc:5697] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:39.265983 13233 raft_consensus.cc:2804] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:39.271399 13233 raft_consensus.cc:493] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.271584 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:39.272549 12806 raft_consensus.cc:2468] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:39.266860 13219 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.259645 13032 raft_consensus.cc:2393] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:39.270100 12998 leader_election.cc:304] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.270656 12776 leader_election.cc:304] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.259464 13079 tablet_bootstrap.cc:654] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.273523 13233 raft_consensus.cc:3060] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.277163 12924 leader_election.cc:304] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.281726 13221 raft_consensus.cc:2804] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:39.282131 13207 raft_consensus.cc:2804] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:39.282411 13221 raft_consensus.cc:493] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.284678 12849 leader_election.cc:304] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:39.285260 13221 raft_consensus.cc:3060] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.286027 13125 raft_consensus.cc:3060] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.289001 13224 raft_consensus.cc:2804] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:39.289641 13224 raft_consensus.cc:697] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:39.289551 12998 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.290719 13224 consensus_queue.cc:237] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.297250 13232 raft_consensus.cc:2804] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:39.297482 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:39.297894 13232 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.298332 13232 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.298662 13125 raft_consensus.cc:2749] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.307581 13152 consensus_queue.cc:1048] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:39.310226 13219 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:39.312839 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
W20260709 12:27:39.314635 12773 leader_election.cc:343] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.283150 13207 raft_consensus.cc:493] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.308053 13221 raft_consensus.cc:515] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.317065 13207 raft_consensus.cc:3060] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.320772 13141 raft_consensus.cc:493] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.321491 13141 raft_consensus.cc:515] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.323760 13221 leader_election.cc:290] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:39.347496 13232 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.349200 13033 raft_consensus.cc:2393] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b543f973837c43b19b7d17feaaca5325 in current term 1: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
I20260709 12:27:39.352389 13080 tablet_bootstrap.cc:492] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:39.352908 13080 ts_tablet_manager.cc:1403] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.117s	user 0.010s	sys 0.009s
I20260709 12:27:39.351603 12774 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.351986 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:39.359257 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:39.359983 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acdd78885d1044a3a523c14603afcb9a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:39.360010 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
W20260709 12:27:39.362058 12996 leader_election.cc:343] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.360420 13221 raft_consensus.cc:2749] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.353739 13233 raft_consensus.cc:515] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.367436 13078 tablet_bootstrap.cc:654] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:39.368497 12773 leader_election.cc:343] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.354386 13141 leader_election.cc:290] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:39.363587 13232 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:39.372180 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acdd78885d1044a3a523c14603afcb9a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:39.375555 12807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:39.376382 13083 tablet_bootstrap.cc:492] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:39.376982 13083 ts_tablet_manager.cc:1403] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.214s	user 0.016s	sys 0.011s
I20260709 12:27:39.380182 12807 raft_consensus.cc:2393] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate b543f973837c43b19b7d17feaaca5325 in this term.
I20260709 12:27:39.381403 12806 raft_consensus.cc:2393] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:39.382354 13206 consensus_queue.cc:1048] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:39.384523 13083 raft_consensus.cc:359] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.385334 13083 raft_consensus.cc:385] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.385646 13083 raft_consensus.cc:740] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.386895 13083 consensus_queue.cc:260] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.392383 12998 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:39.392017 13080 raft_consensus.cc:359] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.393514 13232 raft_consensus.cc:2749] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.393671 13080 raft_consensus.cc:385] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.394455 13080 raft_consensus.cc:740] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.397154 12998 leader_election.cc:304] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:39.399036 13083 ts_tablet_manager.cc:1434] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.002s
I20260709 12:27:39.400565 13083 tablet_bootstrap.cc:492] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:39.399533 13080 consensus_queue.cc:260] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.402285 13232 raft_consensus.cc:3060] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.408005 13233 leader_election.cc:290] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:39.411095 13232 raft_consensus.cc:2749] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.415336 13083 tablet_bootstrap.cc:654] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.416071 13080 ts_tablet_manager.cc:1434] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.063s	user 0.006s	sys 0.005s
I20260709 12:27:39.414546 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c1da90a8da49f6be72a8361eead7df" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:39.418048 12957 raft_consensus.cc:3060] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.418664 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c1da90a8da49f6be72a8361eead7df" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
W20260709 12:27:39.420866 12774 leader_election.cc:343] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.440016 13079 tablet_bootstrap.cc:492] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:39.440747 13079 ts_tablet_manager.cc:1403] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.189s	user 0.019s	sys 0.020s
I20260709 12:27:39.441259 13080 tablet_bootstrap.cc:492] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:39.434131 12957 raft_consensus.cc:2468] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:39.434991 13207 raft_consensus.cc:515] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.446856 13079 raft_consensus.cc:359] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.447695 13079 raft_consensus.cc:385] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.448050 13079 raft_consensus.cc:740] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.448627 13207 leader_election.cc:290] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:39.449559 12959 raft_consensus.cc:2393] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b543f973837c43b19b7d17feaaca5325 in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:39.451016 13080 tablet_bootstrap.cc:654] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.448906 13079 consensus_queue.cc:260] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.451797 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:39.458117 13083 tablet_bootstrap.cc:492] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:39.459702 13083 ts_tablet_manager.cc:1403] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.059s	user 0.013s	sys 0.001s
I20260709 12:27:39.461078 12881 raft_consensus.cc:3060] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.462608 13207 consensus_queue.cc:1048] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:39.461194 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:39.467711 12806 raft_consensus.cc:2393] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8012bb9a8b274890bcaad0d192cd30d5 in current term 1: Already voted for candidate b543f973837c43b19b7d17feaaca5325 in this term.
I20260709 12:27:39.468538 13079 ts_tablet_manager.cc:1434] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.002s
I20260709 12:27:39.470063 13079 tablet_bootstrap.cc:492] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:39.477990 13083 raft_consensus.cc:359] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.480017 13083 raft_consensus.cc:385] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.481885 13078 tablet_bootstrap.cc:492] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:39.482848 13219 raft_consensus.cc:493] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.483403 13219 raft_consensus.cc:515] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.483680 13083 raft_consensus.cc:740] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.486994 13207 consensus_queue.cc:1048] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:39.487860 13083 consensus_queue.cc:260] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.488062 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:39.474151 12776 leader_election.cc:304] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.491084 13083 ts_tablet_manager.cc:1434] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.031s	user 0.009s	sys 0.000s
I20260709 12:27:39.492195 13083 tablet_bootstrap.cc:492] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:39.492921 13221 raft_consensus.cc:2749] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.497632 13078 ts_tablet_manager.cc:1403] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.229s	user 0.017s	sys 0.006s
W20260709 12:27:39.495673 12773 leader_election.cc:343] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.512720 13219 leader_election.cc:290] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:39.513808 13233 raft_consensus.cc:493] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.514358 13078 raft_consensus.cc:359] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.515314 13078 raft_consensus.cc:385] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.515584 13078 raft_consensus.cc:740] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.516232 13078 consensus_queue.cc:260] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.507761 12776 leader_election.cc:304] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:39.518687 13078 ts_tablet_manager.cc:1434] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260709 12:27:39.519021 12881 raft_consensus.cc:2468] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:39.520033 13078 tablet_bootstrap.cc:492] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
W20260709 12:27:39.522720 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:39.523007 13079 tablet_bootstrap.cc:654] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.532087 13083 tablet_bootstrap.cc:654] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.518646 13219 raft_consensus.cc:2804] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:39.516573 13233 raft_consensus.cc:515] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.535985 13078 tablet_bootstrap.cc:654] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.530648 12922 leader_election.cc:304] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:39.544627 13152 raft_consensus.cc:2804] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:39.545149 13152 raft_consensus.cc:697] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:39.545928 13152 consensus_queue.cc:237] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.546881 13079 tablet_bootstrap.cc:492] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:39.548521 13079 ts_tablet_manager.cc:1403] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.079s	user 0.010s	sys 0.014s
I20260709 12:27:39.553617 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:39.558804 13080 tablet_bootstrap.cc:492] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:39.558959 13079 raft_consensus.cc:359] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.560010 13079 raft_consensus.cc:385] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.560444 13079 raft_consensus.cc:740] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.561235 13079 consensus_queue.cc:260] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.563979 13079 ts_tablet_manager.cc:1434] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:27:39.564425 13233 leader_election.cc:290] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:39.565058 13079 tablet_bootstrap.cc:492] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:39.566321 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:39.568034 12774 leader_election.cc:343] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.572045 13079 tablet_bootstrap.cc:654] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.543424 13219 raft_consensus.cc:697] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:39.559496 13080 ts_tablet_manager.cc:1403] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.122s	user 0.006s	sys 0.020s
I20260709 12:27:39.573923 13219 consensus_queue.cc:237] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:39.576212 12776 leader_election.cc:343] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.576802 12776 leader_election.cc:304] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.577767 13221 raft_consensus.cc:2749] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.582398 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:39.586031 12708 catalog_manager.cc:5697] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } }
W20260709 12:27:39.597429 12776 leader_election.cc:343] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.597025 13080 raft_consensus.cc:359] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.602226 13080 raft_consensus.cc:385] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.602538 13232 consensus_queue.cc:1048] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:39.598222 12776 leader_election.cc:304] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:39.605656 13142 raft_consensus.cc:2749] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.607796 13080 raft_consensus.cc:740] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.607542 12705 catalog_manager.cc:5697] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:39.608680 13080 consensus_queue.cc:260] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.610070 12705 catalog_manager.cc:5697] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:39.611109 13080 ts_tablet_manager.cc:1434] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.033s	user 0.005s	sys 0.002s
I20260709 12:27:39.612330 13080 tablet_bootstrap.cc:492] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:39.626453 13078 tablet_bootstrap.cc:492] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:39.627127 13078 ts_tablet_manager.cc:1403] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.107s	user 0.018s	sys 0.008s
I20260709 12:27:39.629132 13083 tablet_bootstrap.cc:492] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:39.629830 13083 ts_tablet_manager.cc:1403] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.138s	user 0.024s	sys 0.024s
I20260709 12:27:39.630564 13080 tablet_bootstrap.cc:654] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.632440 13078 raft_consensus.cc:359] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.632932 13083 raft_consensus.cc:359] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.633261 13078 raft_consensus.cc:385] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.633762 13083 raft_consensus.cc:385] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.633970 13078 raft_consensus.cc:740] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.634205 13083 raft_consensus.cc:740] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.635185 13083 consensus_queue.cc:260] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.635391 13078 consensus_queue.cc:260] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.637578 13083 ts_tablet_manager.cc:1434] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:39.638670 13083 tablet_bootstrap.cc:492] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:39.643332 13078 ts_tablet_manager.cc:1434] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.001s
I20260709 12:27:39.644964 13078 tablet_bootstrap.cc:492] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:39.683917 13254 raft_consensus.cc:493] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.678732 13079 tablet_bootstrap.cc:492] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:39.684612 13254 raft_consensus.cc:515] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.689141 13083 tablet_bootstrap.cc:654] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.694252 13078 tablet_bootstrap.cc:654] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.698472 13079 ts_tablet_manager.cc:1403] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.134s	user 0.012s	sys 0.010s
I20260709 12:27:39.711632 13079 raft_consensus.cc:359] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.712569 13079 raft_consensus.cc:385] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.712883 13079 raft_consensus.cc:740] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.713074 13234 consensus_queue.cc:1048] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:39.713714 13079 consensus_queue.cc:260] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.717093 12881 raft_consensus.cc:3060] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.723359 12703 catalog_manager.cc:5697] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:39.729928 13079 ts_tablet_manager.cc:1434] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.003s
I20260709 12:27:39.731112 13079 tablet_bootstrap.cc:492] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:39.738306 13079 tablet_bootstrap.cc:654] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.739979 13078 tablet_bootstrap.cc:492] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:39.740432 13254 leader_election.cc:290] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:39.740712 13078 ts_tablet_manager.cc:1403] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.096s	user 0.009s	sys 0.005s
I20260709 12:27:39.746826 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:39.747745 13033 raft_consensus.cc:2468] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:39.748971 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:39.750659 12923 leader_election.cc:304] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
W20260709 12:27:39.751447 12922 leader_election.cc:343] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.751822 13254 raft_consensus.cc:2804] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:39.752282 13254 raft_consensus.cc:493] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.752624 13254 raft_consensus.cc:3060] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.755581 13234 consensus_queue.cc:1048] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:39.760983 13078 raft_consensus.cc:359] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.762144 13078 raft_consensus.cc:385] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.768107 13152 consensus_queue.cc:1048] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:39.774338 13152 consensus_queue.cc:1048] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:39.762550 13078 raft_consensus.cc:740] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.777262 13254 raft_consensus.cc:515] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.780043 13232 raft_consensus.cc:493] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.780947 13232 raft_consensus.cc:515] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.785754 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c1da90a8da49f6be72a8361eead7df" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:39.786717 12957 raft_consensus.cc:2393] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate b543f973837c43b19b7d17feaaca5325 in this term.
I20260709 12:27:39.787231 13152 raft_consensus.cc:493] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.787770 13080 tablet_bootstrap.cc:492] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:39.788447 13080 ts_tablet_manager.cc:1403] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.176s	user 0.034s	sys 0.025s
I20260709 12:27:39.787967 13152 raft_consensus.cc:515] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.789366 13224 raft_consensus.cc:493] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.790036 13224 raft_consensus.cc:515] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.789521 13232 leader_election.cc:290] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:39.776571 13078 consensus_queue.cc:260] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.791335 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c1da90a8da49f6be72a8361eead7df" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:39.791455 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:39.793376 12922 leader_election.cc:343] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.793527 12998 leader_election.cc:304] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:39.794188 13224 leader_election.cc:290] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:39.794833 13232 raft_consensus.cc:3060] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.795585 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2b3043eef1a4f98b328054262cfe08c" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:39.791534 13080 raft_consensus.cc:359] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.801182 13080 raft_consensus.cc:385] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.801519 13080 raft_consensus.cc:740] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.802497 13080 consensus_queue.cc:260] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.817039 13078 ts_tablet_manager.cc:1434] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.076s	user 0.009s	sys 0.000s
I20260709 12:27:39.818567 13078 tablet_bootstrap.cc:492] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:39.823518 13080 ts_tablet_manager.cc:1434] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.035s	user 0.008s	sys 0.000s
I20260709 12:27:39.815783 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2b3043eef1a4f98b328054262cfe08c" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:39.834084 13080 tablet_bootstrap.cc:492] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:39.797858 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:39.816964 13152 leader_election.cc:290] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:39.838586 12810 raft_consensus.cc:2468] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:39.849187 13232 raft_consensus.cc:2749] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:27:39.831861 12996 leader_election.cc:343] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:39.851082 12923 leader_election.cc:343] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.851652 12923 leader_election.cc:304] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.851696 12998 leader_election.cc:304] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.854991 13152 raft_consensus.cc:2749] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.855911 13232 raft_consensus.cc:2804] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:39.856393 13232 raft_consensus.cc:493] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.856774 13232 raft_consensus.cc:3060] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.860961 13219 raft_consensus.cc:493] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.861884 13219 raft_consensus.cc:515] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.864248 13232 raft_consensus.cc:515] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.866634 13232 leader_election.cc:290] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:39.871937 13232 consensus_queue.cc:1048] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:39.872742 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2b3043eef1a4f98b328054262cfe08c" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:39.874352 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a2b3043eef1a4f98b328054262cfe08c" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:39.875089 12810 raft_consensus.cc:3060] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.881099 13219 leader_election.cc:290] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:39.882925 12996 leader_election.cc:343] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.882915 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d24c0921438489bb6c87bd7b1b0654d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:39.884366 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d24c0921438489bb6c87bd7b1b0654d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:39.885696 12774 leader_election.cc:343] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:39.886977 12773 leader_election.cc:343] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.888406 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:39.888633 12773 leader_election.cc:304] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.889086 13032 raft_consensus.cc:3060] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.889830 13142 raft_consensus.cc:2749] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:39.886703 13254 leader_election.cc:290] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:39.891067 13078 tablet_bootstrap.cc:654] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.887589 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:39.888034 12707 catalog_manager.cc:5697] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:39.893359 12922 leader_election.cc:343] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:39.901679 13079 tablet_bootstrap.cc:492] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:39.902249 13079 ts_tablet_manager.cc:1403] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.171s	user 0.038s	sys 0.037s
I20260709 12:27:39.906927 13079 raft_consensus.cc:359] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.907640 13079 raft_consensus.cc:385] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.907938 13079 raft_consensus.cc:740] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.908644 13079 consensus_queue.cc:260] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.910631 13080 tablet_bootstrap.cc:654] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:39.912560 12810 raft_consensus.cc:2468] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:39.913909 13083 tablet_bootstrap.cc:492] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:39.914477 13083 ts_tablet_manager.cc:1403] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.276s	user 0.015s	sys 0.000s
I20260709 12:27:39.914414 12998 leader_election.cc:304] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:39.916476 13224 raft_consensus.cc:2804] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:39.917502 13224 raft_consensus.cc:697] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:39.918764 13224 consensus_queue.cc:237] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:39.921619 13083 raft_consensus.cc:359] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.922860 13083 raft_consensus.cc:385] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.923127 13083 raft_consensus.cc:740] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.911183 13078 tablet_bootstrap.cc:492] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:39.911088 13079 ts_tablet_manager.cc:1434] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:27:39.929324 13078 ts_tablet_manager.cc:1403] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.111s	user 0.013s	sys 0.003s
I20260709 12:27:39.929785 13079 tablet_bootstrap.cc:492] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:39.924089 13083 consensus_queue.cc:260] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:39.937705 13083 ts_tablet_manager.cc:1434] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 12:27:39.939023 13083 tablet_bootstrap.cc:492] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:39.938853 13142 raft_consensus.cc:493] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:39.943364 13078 raft_consensus.cc:359] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.944114 13078 raft_consensus.cc:385] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:39.944442 13078 raft_consensus.cc:740] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:39.943404 13142 raft_consensus.cc:515] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:39.945195 13078 consensus_queue.cc:260] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:39.948206 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:39.948915 12957 raft_consensus.cc:2468] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:39.953459 12776 leader_election.cc:304] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:39.955513 13221 raft_consensus.cc:2804] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:39.955978 13221 raft_consensus.cc:493] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:39.956326 13221 raft_consensus.cc:3060] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:39.954031 13078 ts_tablet_manager.cc:1434] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 12:27:39.967422 13078 tablet_bootstrap.cc:492] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:39.972038 13219 consensus_queue.cc:1048] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:39.946589 13142 leader_election.cc:290] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:39.980737 13032 raft_consensus.cc:2468] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:39.990232 13254 raft_consensus.cc:493] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.007839 13254 raft_consensus.cc:515] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.011423 12807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:40.021327 13206 consensus_queue.cc:1048] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:40.022857 13083 tablet_bootstrap.cc:654] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.004014 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:40.026119 12881 raft_consensus.cc:2468] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
W20260709 12:27:40.014003 13270 log.cc:927] Time spent T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:27:40.035707 13221 raft_consensus.cc:515] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.037606 13221 leader_election.cc:290] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:40.038478 12923 leader_election.cc:304] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.038770 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:40.039764 12883 raft_consensus.cc:3060] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.038643 12807 raft_consensus.cc:2393] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8012bb9a8b274890bcaad0d192cd30d5 in current term 1: Already voted for candidate b543f973837c43b19b7d17feaaca5325 in this term.
I20260709 12:27:40.043351 13206 raft_consensus.cc:2804] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:40.046430 13206 raft_consensus.cc:697] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:40.049953 13254 leader_election.cc:290] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:40.051059 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:40.051756 12957 raft_consensus.cc:3060] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.051946 13080 tablet_bootstrap.cc:492] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:40.052567 13080 ts_tablet_manager.cc:1403] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.219s	user 0.053s	sys 0.024s
I20260709 12:27:40.055884 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:40.056051 13080 raft_consensus.cc:359] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.058387 13080 raft_consensus.cc:385] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.058648 13078 tablet_bootstrap.cc:654] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.058737 13080 raft_consensus.cc:740] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.058005 12922 leader_election.cc:304] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:40.060314 13080 consensus_queue.cc:260] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.054225 13206 consensus_queue.cc:237] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.079303 13079 tablet_bootstrap.cc:654] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.079674 13083 tablet_bootstrap.cc:492] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:40.080611 13083 ts_tablet_manager.cc:1403] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.142s	user 0.013s	sys 0.005s
I20260709 12:27:40.071007 13254 consensus_queue.cc:1048] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:27:40.086961 13229 raft_consensus.cc:493] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.089244 13229 raft_consensus.cc:515] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.091754 13229 leader_election.cc:290] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:40.096364 13083 raft_consensus.cc:359] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.071007 12957 raft_consensus.cc:2468] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:40.097551 13083 raft_consensus.cc:385] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.098090 13083 raft_consensus.cc:740] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.098122 13152 raft_consensus.cc:2749] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:40.098436 13221 raft_consensus.cc:493] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.099092 13221 raft_consensus.cc:515] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.101189 13080 ts_tablet_manager.cc:1434] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.048s	user 0.006s	sys 0.002s
I20260709 12:27:40.103070 13080 tablet_bootstrap.cc:492] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:40.103446 13083 consensus_queue.cc:260] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.108557 13221 leader_election.cc:290] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:40.112911 12776 leader_election.cc:304] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:40.117046 12708 catalog_manager.cc:5697] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:40.120378 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:40.121294 12959 raft_consensus.cc:2468] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:40.114168 13221 raft_consensus.cc:2804] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:40.126690 13221 raft_consensus.cc:697] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:40.127789 13221 consensus_queue.cc:237] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.136934 13083 ts_tablet_manager.cc:1434] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.056s	user 0.012s	sys 0.000s
I20260709 12:27:40.138036 13083 tablet_bootstrap.cc:492] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:40.134142 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4c88c0aa80544daa7bae9c5ac62b0fa" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:40.151943 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:40.152176 13279 raft_consensus.cc:493] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.154455 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4c88c0aa80544daa7bae9c5ac62b0fa" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:40.152848 13279 raft_consensus.cc:515] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.152774 12808 raft_consensus.cc:2468] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:40.160910 13279 leader_election.cc:290] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:40.163674 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:40.166917 12923 leader_election.cc:343] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.167447 12883 raft_consensus.cc:2468] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:40.167877 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:40.165951 12999 leader_election.cc:304] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:40.170159 13141 raft_consensus.cc:2804] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:40.170629 13141 raft_consensus.cc:493] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.171074 13141 raft_consensus.cc:3060] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:40.173552 12773 leader_election.cc:343] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:40.175597 12922 leader_election.cc:343] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.178365 13078 tablet_bootstrap.cc:492] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:40.178674 12922 leader_election.cc:304] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.180917 13280 raft_consensus.cc:2749] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:27:40.178874 12776 leader_election.cc:343] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.184257 12776 leader_election.cc:304] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.187227 13079 tablet_bootstrap.cc:492] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:40.187561 13219 raft_consensus.cc:2749] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:40.187908 13079 ts_tablet_manager.cc:1403] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.258s	user 0.012s	sys 0.017s
I20260709 12:27:40.189085 13083 tablet_bootstrap.cc:654] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.179080 13078 ts_tablet_manager.cc:1403] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.212s	user 0.023s	sys 0.017s
I20260709 12:27:40.208331 13080 tablet_bootstrap.cc:654] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.216406 13219 consensus_queue.cc:1048] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:40.232007 13079 raft_consensus.cc:359] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.233331 13079 raft_consensus.cc:385] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.233706 13079 raft_consensus.cc:740] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.235039 13079 consensus_queue.cc:260] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.242753 13141 raft_consensus.cc:515] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.244686 13079 ts_tablet_manager.cc:1434] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.056s	user 0.008s	sys 0.003s
I20260709 12:27:40.246028 13079 tablet_bootstrap.cc:492] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:40.248781 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:40.251585 13078 raft_consensus.cc:359] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.252650 13078 raft_consensus.cc:385] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.252992 13078 raft_consensus.cc:740] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.250696 12957 raft_consensus.cc:3060] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.257345 13078 consensus_queue.cc:260] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.259991 13141 leader_election.cc:290] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:40.272250 13083 tablet_bootstrap.cc:492] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:40.272857 13083 ts_tablet_manager.cc:1403] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.135s	user 0.018s	sys 0.033s
I20260709 12:27:40.269821 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:40.276731 12810 raft_consensus.cc:3060] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.282148 13078 ts_tablet_manager.cc:1434] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.087s	user 0.005s	sys 0.003s
I20260709 12:27:40.283308 13078 tablet_bootstrap.cc:492] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:40.288326 12705 catalog_manager.cc:5697] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:40.288502 12703 catalog_manager.cc:5697] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:40.295796 12810 raft_consensus.cc:2468] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:40.297807 12998 leader_election.cc:304] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:40.299196 13141 raft_consensus.cc:2804] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:40.300097 13141 raft_consensus.cc:697] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:40.301294 13141 consensus_queue.cc:237] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.301949 13080 tablet_bootstrap.cc:492] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:40.302680 13080 ts_tablet_manager.cc:1403] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.200s	user 0.015s	sys 0.006s
I20260709 12:27:40.307986 12957 raft_consensus.cc:2468] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:40.317196 13078 tablet_bootstrap.cc:654] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.318840 13079 tablet_bootstrap.cc:654] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.305821 13080 raft_consensus.cc:359] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.324887 13083 raft_consensus.cc:359] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.325763 13080 raft_consensus.cc:385] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.325888 13083 raft_consensus.cc:385] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.326220 13080 raft_consensus.cc:740] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.327527 13080 consensus_queue.cc:260] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.326480 13083 raft_consensus.cc:740] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.330716 13080 ts_tablet_manager.cc:1434] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.001s
I20260709 12:27:40.332253 13080 tablet_bootstrap.cc:492] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:40.331092 13083 consensus_queue.cc:260] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.336413 13083 ts_tablet_manager.cc:1434] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.063s	user 0.009s	sys 0.000s
I20260709 12:27:40.337420 13083 tablet_bootstrap.cc:492] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:40.348160 13083 tablet_bootstrap.cc:654] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.351159 12707 catalog_manager.cc:5697] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:40.357308 13080 tablet_bootstrap.cc:654] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.365257 13083 tablet_bootstrap.cc:492] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:40.366159 13083 ts_tablet_manager.cc:1403] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.005s
I20260709 12:27:40.370766 13083 raft_consensus.cc:359] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.371932 13083 raft_consensus.cc:385] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.373353 13083 raft_consensus.cc:740] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.375000 13083 consensus_queue.cc:260] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.379005 13083 ts_tablet_manager.cc:1434] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.002s
I20260709 12:27:40.380548 13083 tablet_bootstrap.cc:492] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:40.390529 13083 tablet_bootstrap.cc:654] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.398241 13206 raft_consensus.cc:493] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.399410 13206 raft_consensus.cc:515] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.403837 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:40.405378 13141 raft_consensus.cc:493] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.406309 13141 raft_consensus.cc:515] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.407732 13221 raft_consensus.cc:493] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.408339 13221 raft_consensus.cc:515] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.409353 13141 leader_election.cc:290] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:40.412257 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:40.412755 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:40.413511 13221 leader_election.cc:290] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:40.411625 12708 catalog_manager.cc:5697] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:40.416673 12774 leader_election.cc:343] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:40.418224 12922 leader_election.cc:343] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.413043 12957 raft_consensus.cc:2468] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:40.418448 13079 tablet_bootstrap.cc:492] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:40.418355 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:40.419682 13079 ts_tablet_manager.cc:1403] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.174s	user 0.020s	sys 0.040s
I20260709 12:27:40.420368 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:40.421177 12999 leader_election.cc:304] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
W20260709 12:27:40.422112 12996 leader_election.cc:343] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.422161 13141 raft_consensus.cc:2804] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:40.423372 13141 raft_consensus.cc:493] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.423077 13079 raft_consensus.cc:359] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.423998 13141 raft_consensus.cc:3060] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.424131 13079 raft_consensus.cc:385] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.425401 13079 raft_consensus.cc:740] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.426260 13079 consensus_queue.cc:260] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:40.428156 12776 leader_election.cc:343] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.428701 13079 ts_tablet_manager.cc:1434] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:40.430022 13079 tablet_bootstrap.cc:492] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:40.430917 13206 leader_election.cc:290] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:40.428671 12776 leader_election.cc:304] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:40.433187 13286 raft_consensus.cc:2749] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:40.433575 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:40.435338 12923 leader_election.cc:343] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.435832 12923 leader_election.cc:304] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.437001 13078 tablet_bootstrap.cc:492] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:40.437556 13206 raft_consensus.cc:2749] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:40.437556 13078 ts_tablet_manager.cc:1403] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.154s	user 0.021s	sys 0.034s
I20260709 12:27:40.439114 13079 tablet_bootstrap.cc:654] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.439568 13083 tablet_bootstrap.cc:492] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:40.440167 13083 ts_tablet_manager.cc:1403] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.060s	user 0.014s	sys 0.007s
I20260709 12:27:40.440320 13141 raft_consensus.cc:515] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.441341 13078 raft_consensus.cc:359] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.442047 13078 raft_consensus.cc:385] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.442315 13078 raft_consensus.cc:740] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.444854 13083 raft_consensus.cc:359] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.445493 13078 consensus_queue.cc:260] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.445937 13083 raft_consensus.cc:385] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.446506 13083 raft_consensus.cc:740] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.450345 13141 leader_election.cc:290] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:40.452977 13234 raft_consensus.cc:493] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.453550 13234 raft_consensus.cc:515] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.453951 13080 tablet_bootstrap.cc:492] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:40.454623 13080 ts_tablet_manager.cc:1403] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.123s	user 0.027s	sys 0.030s
I20260709 12:27:40.453719 13083 consensus_queue.cc:260] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.457438 13083 ts_tablet_manager.cc:1434] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.000s
I20260709 12:27:40.457960 13080 raft_consensus.cc:359] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.458431 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:40.458850 13080 raft_consensus.cc:385] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.459055 13083 tablet_bootstrap.cc:492] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:40.459167 13080 raft_consensus.cc:740] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.448791 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:40.458038 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:40.448535 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:40.460772 13234 leader_election.cc:290] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:40.459148 12957 raft_consensus.cc:2468] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:40.460855 13080 consensus_queue.cc:260] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.447832 13078 ts_tablet_manager.cc:1434] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.002s
W20260709 12:27:40.463131 12996 leader_election.cc:343] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.469810 13078 tablet_bootstrap.cc:492] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:40.464655 12999 leader_election.cc:304] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
W20260709 12:27:40.471411 12996 leader_election.cc:343] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.473785 13080 ts_tablet_manager.cc:1434] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260709 12:27:40.473973 13234 raft_consensus.cc:2804] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:40.474668 13234 raft_consensus.cc:493] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.474931 13080 tablet_bootstrap.cc:492] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:40.475140 13234 raft_consensus.cc:3060] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.481143 13221 raft_consensus.cc:493] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.481724 13080 tablet_bootstrap.cc:654] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.481930 13221 raft_consensus.cc:515] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.485278 13221 leader_election.cc:290] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:40.486225 13078 tablet_bootstrap.cc:654] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.490248 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca89f605f7bb4bd9acc811382ac82591" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:40.491303 13033 raft_consensus.cc:2468] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:40.493260 12774 leader_election.cc:304] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:40.494436 13221 raft_consensus.cc:2804] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:40.467376 13079 tablet_bootstrap.cc:492] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:40.494910 13221 raft_consensus.cc:493] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.495321 13221 raft_consensus.cc:3060] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.495373 13079 ts_tablet_manager.cc:1403] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.066s	user 0.018s	sys 0.016s
I20260709 12:27:40.498971 13079 raft_consensus.cc:359] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.499660 13079 raft_consensus.cc:385] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.499946 13079 raft_consensus.cc:740] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.500730 13079 consensus_queue.cc:260] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.502485 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca89f605f7bb4bd9acc811382ac82591" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:40.468530 13083 tablet_bootstrap.cc:654] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.462550 12955 raft_consensus.cc:3060] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:40.504176 12773 leader_election.cc:343] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.506247 13234 raft_consensus.cc:515] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.508350 13234 leader_election.cc:290] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:40.509985 13221 raft_consensus.cc:515] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.510241 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:40.510170 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:40.516363 12957 raft_consensus.cc:3060] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:40.517899 12996 leader_election.cc:343] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.524762 12957 raft_consensus.cc:2468] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:40.526945 12999 leader_election.cc:304] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.513844 13078 tablet_bootstrap.cc:492] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:40.528290 13078 ts_tablet_manager.cc:1403] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.059s	user 0.013s	sys 0.005s
I20260709 12:27:40.528934 13229 raft_consensus.cc:2804] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:40.531906 13079 ts_tablet_manager.cc:1434] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.001s
I20260709 12:27:40.532922 13079 tablet_bootstrap.cc:492] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:40.546816 13234 consensus_queue.cc:1048] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:40.527606 13221 leader_election.cc:290] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:40.534410 13229 raft_consensus.cc:697] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:40.559744 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca89f605f7bb4bd9acc811382ac82591" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:40.551282 12955 raft_consensus.cc:2468] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:40.556915 13125 raft_consensus.cc:493] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.544597 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca89f605f7bb4bd9acc811382ac82591" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:40.563091 13125 raft_consensus.cc:515] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
W20260709 12:27:40.566105 12773 leader_election.cc:343] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.570915 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:40.572384 13031 raft_consensus.cc:2393] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:40.574983 13286 consensus_queue.cc:1048] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:40.567427 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:40.577829 12851 leader_election.cc:304] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:40.579432 12881 raft_consensus.cc:3060] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.579113 13229 consensus_queue.cc:237] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.563666 13033 raft_consensus.cc:3060] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.590965 13125 leader_election.cc:290] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:40.592725 13078 raft_consensus.cc:359] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.593715 13078 raft_consensus.cc:385] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.594038 13078 raft_consensus.cc:740] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.598332 12999 leader_election.cc:304] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.598364 12882 raft_consensus.cc:3060] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.603281 13141 raft_consensus.cc:2804] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:40.603919 13141 raft_consensus.cc:697] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:40.605185 13033 raft_consensus.cc:2468] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:40.604480 13078 consensus_queue.cc:260] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.613267 13141 consensus_queue.cc:237] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.620779 13079 tablet_bootstrap.cc:654] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.621865 13125 raft_consensus.cc:2749] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:40.625175 13083 tablet_bootstrap.cc:492] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:40.626149 13083 ts_tablet_manager.cc:1403] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.167s	user 0.026s	sys 0.043s
I20260709 12:27:40.634490 12774 leader_election.cc:304] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.636857 13206 consensus_queue.cc:1048] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:40.636891 13142 raft_consensus.cc:2804] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:40.639211 13080 tablet_bootstrap.cc:492] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:40.639567 13142 raft_consensus.cc:697] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:40.639763 13080 ts_tablet_manager.cc:1403] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.165s	user 0.039s	sys 0.034s
I20260709 12:27:40.640602 13142 consensus_queue.cc:237] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.643217 13080 raft_consensus.cc:359] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.644202 13080 raft_consensus.cc:385] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.644606 13080 raft_consensus.cc:740] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.646420 13078 ts_tablet_manager.cc:1434] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.118s	user 0.007s	sys 0.003s
I20260709 12:27:40.646117 13080 consensus_queue.cc:260] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:40.653257 13224 consensus_queue.cc:1048] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:40.656163 13224 raft_consensus.cc:493] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.656690 13224 raft_consensus.cc:515] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.657819 13206 consensus_queue.cc:1048] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:40.666533 13078 tablet_bootstrap.cc:492] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:40.667892 13080 ts_tablet_manager.cc:1434] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.028s	user 0.009s	sys 0.000s
I20260709 12:27:40.670138 13080 tablet_bootstrap.cc:492] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:40.673223 13083 raft_consensus.cc:359] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.676775 13083 raft_consensus.cc:385] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.677126 13083 raft_consensus.cc:740] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.678045 13083 consensus_queue.cc:260] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.680476 13083 ts_tablet_manager.cc:1434] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.054s	user 0.006s	sys 0.002s
I20260709 12:27:40.682287 13083 tablet_bootstrap.cc:492] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:40.676586 13224 leader_election.cc:290] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:40.694334 13079 tablet_bootstrap.cc:492] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
W20260709 12:27:40.716383 12998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 83268.9us
I20260709 12:27:40.697628 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:40.700929 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:40.718456 13079 ts_tablet_manager.cc:1403] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.186s	user 0.020s	sys 0.010s
I20260709 12:27:40.732267 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:40.733134 12810 raft_consensus.cc:2468] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:40.736785 13078 tablet_bootstrap.cc:654] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.739630 13206 raft_consensus.cc:493] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:40.730978 12996 leader_election.cc:343] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.740835 12703 catalog_manager.cc:5697] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:40.743429 12703 catalog_manager.cc:5697] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:40.744050 13286 consensus_queue.cc:1048] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:40.740226 13206 raft_consensus.cc:515] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.755252 12807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:40.755978 12807 raft_consensus.cc:2468] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:40.761632 12924 leader_election.cc:304] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:40.766906 13280 raft_consensus.cc:2804] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:40.767385 13280 raft_consensus.cc:493] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.767719 13280 raft_consensus.cc:3060] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.767860 13206 leader_election.cc:290] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:40.772943 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:40.778519 12922 leader_election.cc:343] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.778959 12998 leader_election.cc:304] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.782326 13080 tablet_bootstrap.cc:654] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.783644 13306 raft_consensus.cc:2804] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:40.784060 13306 raft_consensus.cc:493] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:40.784420 13306 raft_consensus.cc:3060] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.778038 13280 raft_consensus.cc:515] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.778154 13083 tablet_bootstrap.cc:654] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.795993 13079 raft_consensus.cc:359] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.796653 13079 raft_consensus.cc:385] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.796891 13079 raft_consensus.cc:740] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.800936 13079 consensus_queue.cc:260] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.808239 13079 ts_tablet_manager.cc:1434] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.086s	user 0.006s	sys 0.000s
I20260709 12:27:40.809324 13079 tablet_bootstrap.cc:492] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:40.823145 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:40.824002 12810 raft_consensus.cc:3060] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.820339 12705 catalog_manager.cc:5697] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:40.824805 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:40.826012 13301 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.826594 13301 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.828357 13078 tablet_bootstrap.cc:492] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:40.828943 13078 ts_tablet_manager.cc:1403] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.164s	user 0.010s	sys 0.006s
I20260709 12:27:40.829074 13301 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:27:40.831357 12922 leader_election.cc:343] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.831552 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:40.834802 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:40.835179 13033 raft_consensus.cc:2393] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
I20260709 12:27:40.835824 12808 raft_consensus.cc:2393] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate b543f973837c43b19b7d17feaaca5325 in this term.
I20260709 12:27:40.831604 13306 raft_consensus.cc:515] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:40.837332 12850 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:40.839738 13306 leader_election.cc:290] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:40.841176 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:40.849833 13078 raft_consensus.cc:359] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.850958 13078 raft_consensus.cc:385] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:40.851339 13078 raft_consensus.cc:740] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:40.852326 13078 consensus_queue.cc:260] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:40.854444 13078 ts_tablet_manager.cc:1434] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.000s
I20260709 12:27:40.823396 13280 leader_election.cc:290] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:40.875413 13280 raft_consensus.cc:493] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.860170 13078 tablet_bootstrap.cc:492] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:40.868211 12810 raft_consensus.cc:2468] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:40.842805 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:40.878494 12808 raft_consensus.cc:3060] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:40.886186 13078 tablet_bootstrap.cc:654] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.887003 13286 raft_consensus.cc:493] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:40.887789 13286 raft_consensus.cc:515] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.889214 13280 raft_consensus.cc:515] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:40.892186 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6179fd5410440a89ea4ca9c5ef3944" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:40.892534 13280 leader_election.cc:290] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:40.895658 12773 leader_election.cc:343] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.896981 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4c88c0aa80544daa7bae9c5ac62b0fa" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:40.899291 12922 leader_election.cc:343] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.903173 13286 leader_election.cc:290] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:40.904080 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6179fd5410440a89ea4ca9c5ef3944" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:40.906666 12776 leader_election.cc:343] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.907364 12776 leader_election.cc:304] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.909227 13286 raft_consensus.cc:2749] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:40.844681 13301 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:40.918332 13284 log.cc:927] Time spent T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:27:40.920060 13275 log.cc:927] Time spent T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:27:40.922676 12924 leader_election.cc:304] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:40.923956 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4c88c0aa80544daa7bae9c5ac62b0fa" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:40.924804 12806 raft_consensus.cc:2468] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:40.925192 13306 consensus_queue.cc:1048] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:40.910518 13206 raft_consensus.cc:493] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:40.859565 12996 leader_election.cc:343] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:40.925239 13280 raft_consensus.cc:2804] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:40.942003 13079 tablet_bootstrap.cc:654] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:40.960072 13078 tablet_bootstrap.cc:492] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:40.960966 13078 ts_tablet_manager.cc:1403] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.103s	user 0.012s	sys 0.019s
I20260709 12:27:40.929582 13206 raft_consensus.cc:515] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:40.936892 12996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 79416us
I20260709 12:27:40.947773 13280 raft_consensus.cc:697] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:41.009855 13142 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.004709 12924 leader_election.cc:304] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:41.040354 13080 tablet_bootstrap.cc:492] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:41.041234 13080 ts_tablet_manager.cc:1403] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.371s	user 0.017s	sys 0.013s
I20260709 12:27:41.045936 12808 raft_consensus.cc:2468] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:41.052556 13078 raft_consensus.cc:359] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.053453 13078 raft_consensus.cc:385] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.053754 13078 raft_consensus.cc:740] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.056787 13327 raft_consensus.cc:2804] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:41.060866 12882 raft_consensus.cc:3060] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.061357 13301 raft_consensus.cc:2749] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:41.057358 13327 raft_consensus.cc:493] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.018841 13142 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.067579 12998 leader_election.cc:304] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:41.074581 13083 tablet_bootstrap.cc:492] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:41.027195 13280 consensus_queue.cc:237] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.075112 13306 raft_consensus.cc:493] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.075393 13083 ts_tablet_manager.cc:1403] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.393s	user 0.032s	sys 0.027s
I20260709 12:27:41.075804 13306 raft_consensus.cc:515] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.078285 13306 leader_election.cc:290] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:41.073945 13234 raft_consensus.cc:2804] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:41.080073 13234 raft_consensus.cc:697] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:41.080960 13080 raft_consensus.cc:359] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.082636 13080 raft_consensus.cc:385] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.082993 13080 raft_consensus.cc:740] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.065999 13327 raft_consensus.cc:3060] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.090538 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:41.091403 13031 raft_consensus.cc:2468] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:41.092348 13079 tablet_bootstrap.cc:492] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:41.093096 12923 leader_election.cc:304] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:41.093840 13142 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:41.094331 13079 ts_tablet_manager.cc:1403] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.285s	user 0.018s	sys 0.016s
I20260709 12:27:41.085989 13080 consensus_queue.cc:260] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.097041 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:41.097332 13080 ts_tablet_manager.cc:1434] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.056s	user 0.009s	sys 0.000s
I20260709 12:27:41.097779 12881 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:41.099087 13080 tablet_bootstrap.cc:492] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:41.099294 12773 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:41.100309 13221 raft_consensus.cc:2804] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:41.100670 13221 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.100971 13221 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:41.101822 13206 leader_election.cc:290] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:41.106451 13080 tablet_bootstrap.cc:654] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.107198 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:41.108144 13033 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
W20260709 12:27:41.109233 12773 consensus_peers.cc:597] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:41.097333 13079 raft_consensus.cc:359] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.113623 13079 raft_consensus.cc:385] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:27:41.069970 12924 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 83790.4us
I20260709 12:27:41.113982 13079 raft_consensus.cc:740] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.116381 13079 consensus_queue.cc:260] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.117959 13078 consensus_queue.cc:260] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.119927 13083 raft_consensus.cc:359] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.120193 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:41.120713 13083 raft_consensus.cc:385] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.121003 12808 raft_consensus.cc:2468] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:41.121650 13279 raft_consensus.cc:2804] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:41.122092 13279 raft_consensus.cc:493] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.122501 13279 raft_consensus.cc:3060] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.122653 12998 leader_election.cc:304] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:41.124064 13306 raft_consensus.cc:2804] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:41.121078 13083 raft_consensus.cc:740] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.137576 13083 consensus_queue.cc:260] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.124483 13306 raft_consensus.cc:493] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.143059 13078 ts_tablet_manager.cc:1434] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.182s	user 0.012s	sys 0.003s
I20260709 12:27:41.145547 13078 tablet_bootstrap.cc:492] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:41.146847 13083 ts_tablet_manager.cc:1434] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.071s	user 0.007s	sys 0.008s
I20260709 12:27:41.147286 13079 ts_tablet_manager.cc:1434] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.000s
I20260709 12:27:41.122120 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:41.082463 13234 consensus_queue.cc:237] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.148581 13083 tablet_bootstrap.cc:492] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:41.154554 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:41.155398 13219 consensus_queue.cc:1048] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:41.141208 13306 raft_consensus.cc:3060] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.148550 13079 tablet_bootstrap.cc:492] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:41.171901 13221 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.175961 13080 tablet_bootstrap.cc:492] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:41.176769 13080 ts_tablet_manager.cc:1403] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.078s	user 0.016s	sys 0.005s
I20260709 12:27:41.177155 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:41.177870 13031 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:41.180666 13080 raft_consensus.cc:359] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.184635 13080 raft_consensus.cc:385] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.185184 13080 raft_consensus.cc:740] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.186367 13080 consensus_queue.cc:260] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.190415 13079 tablet_bootstrap.cc:654] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.193421 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:41.194128 12885 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:41.194926 13301 raft_consensus.cc:493] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.195747 13301 raft_consensus.cc:515] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:41.198945 13221 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:41.165338 12810 raft_consensus.cc:2468] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:41.204972 13141 consensus_queue.cc:1048] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:41.210980 12885 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:27:41.212471 12773 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:41.213449 13221 raft_consensus.cc:2804] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:41.213950 13221 raft_consensus.cc:697] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:41.214901 13221 consensus_queue.cc:237] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.216017 13307 consensus_queue.cc:1048] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:41.217723 13307 consensus_queue.cc:1048] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:41.235760 12708 catalog_manager.cc:5697] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:41.249686 13079 tablet_bootstrap.cc:492] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:41.250211 13080 ts_tablet_manager.cc:1434] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.073s	user 0.010s	sys 0.000s
I20260709 12:27:41.250378 13079 ts_tablet_manager.cc:1403] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.102s	user 0.015s	sys 0.011s
I20260709 12:27:41.251837 13083 tablet_bootstrap.cc:654] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.256033 13080 tablet_bootstrap.cc:492] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:41.261013 13301 leader_election.cc:290] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
W20260709 12:27:41.273444 12996 consensus_peers.cc:597] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:41.274360 13219 raft_consensus.cc:493] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.275022 13219 raft_consensus.cc:515] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.279069 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:41.280977 12774 leader_election.cc:343] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.282480 13219 leader_election.cc:290] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:41.283980 13078 tablet_bootstrap.cc:654] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.290132 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:41.296495 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:41.262938 13279 raft_consensus.cc:515] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.322180 13080 tablet_bootstrap.cc:654] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.299823 13031 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:27:41.331709 13327 raft_consensus.cc:515] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.343356 13306 raft_consensus.cc:515] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.344638 13327 leader_election.cc:290] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:41.345407 12776 leader_election.cc:343] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.345512 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4c88c0aa80544daa7bae9c5ac62b0fa" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:41.329828 13079 raft_consensus.cc:359] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.345858 13219 raft_consensus.cc:493] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.347570 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4c88c0aa80544daa7bae9c5ac62b0fa" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:41.365705 12810 raft_consensus.cc:3060] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.366914 12776 leader_election.cc:304] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:41.348510 13279 leader_election.cc:290] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:41.349720 12958 raft_consensus.cc:2393] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:41.349972 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:41.351330 13234 consensus_queue.cc:1048] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:41.354763 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:41.361188 13079 raft_consensus.cc:385] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.375942 13079 raft_consensus.cc:740] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.371750 13286 raft_consensus.cc:2749] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:41.370994 12883 raft_consensus.cc:3060] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.372227 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:41.383080 13079 consensus_queue.cc:260] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.374796 12808 raft_consensus.cc:3060] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.370288 13219 raft_consensus.cc:515] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.388125 13279 raft_consensus.cc:493] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.411217 13032 raft_consensus.cc:2393] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8012bb9a8b274890bcaad0d192cd30d5 in current term 1: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
I20260709 12:27:41.396957 13224 raft_consensus.cc:493] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.415838 13224 raft_consensus.cc:515] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.416707 13219 leader_election.cc:290] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:41.419338 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b32a14229832492682674ac4c282404d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:41.420544 13279 raft_consensus.cc:515] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.420994 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d24c0921438489bb6c87bd7b1b0654d" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:41.421731 12806 raft_consensus.cc:2468] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
W20260709 12:27:41.424029 12773 leader_election.cc:343] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.424155 12998 leader_election.cc:304] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:41.425544 13083 tablet_bootstrap.cc:492] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:41.425696 13306 leader_election.cc:290] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:41.426193 13083 ts_tablet_manager.cc:1403] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.278s	user 0.008s	sys 0.020s
I20260709 12:27:41.428892 13306 raft_consensus.cc:2804] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:41.429352 13306 raft_consensus.cc:493] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.429842 13306 raft_consensus.cc:3060] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.429361 12807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6179fd5410440a89ea4ca9c5ef3944" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:41.435510 12807 raft_consensus.cc:2468] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:41.439055 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:41.440558 12956 raft_consensus.cc:2393] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:41.442437 12849 leader_election.cc:304] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:41.443058 13078 tablet_bootstrap.cc:492] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:41.443235 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:41.446344 13301 raft_consensus.cc:2749] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:41.447131 13078 ts_tablet_manager.cc:1403] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.303s	user 0.032s	sys 0.037s
I20260709 12:27:41.446753 12705 catalog_manager.cc:5697] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 2, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:41.450151 13078 raft_consensus.cc:359] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.450950 13078 raft_consensus.cc:385] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.451241 13078 raft_consensus.cc:740] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.451493 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d24c0921438489bb6c87bd7b1b0654d" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:41.451772 13279 leader_election.cc:290] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:41.453084 12996 leader_election.cc:343] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.452857 13078 consensus_queue.cc:260] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.453960 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6179fd5410440a89ea4ca9c5ef3944" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:41.444613 13083 raft_consensus.cc:359] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.439332 12924 leader_election.cc:304] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:41.453653 12708 catalog_manager.cc:5697] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:41.455011 12808 raft_consensus.cc:2468] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:41.418562 13224 leader_election.cc:290] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:41.482272 13083 raft_consensus.cc:385] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:27:41.482362 12922 leader_election.cc:343] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.483832 13279 raft_consensus.cc:2804] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:41.445652 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b32a14229832492682674ac4c282404d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:41.484925 13279 raft_consensus.cc:493] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.485438 13279 raft_consensus.cc:3060] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.473949 12810 raft_consensus.cc:2468] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:41.455248 13078 ts_tablet_manager.cc:1434] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:41.488301 12883 raft_consensus.cc:2468] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
W20260709 12:27:41.488821 12776 leader_election.cc:343] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.489321 13083 raft_consensus.cc:740] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.475176 13306 raft_consensus.cc:515] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.490523 13083 consensus_queue.cc:260] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.488973 13078 tablet_bootstrap.cc:492] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:41.497828 13079 ts_tablet_manager.cc:1434] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.244s	user 0.024s	sys 0.033s
I20260709 12:27:41.503242 13079 tablet_bootstrap.cc:492] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:41.488384 12882 raft_consensus.cc:3060] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.468742 13286 raft_consensus.cc:493] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.507813 13083 ts_tablet_manager.cc:1434] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.081s	user 0.017s	sys 0.001s
I20260709 12:27:41.508718 13083 tablet_bootstrap.cc:492] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:41.489435 12776 leader_election.cc:304] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:41.512197 13080 tablet_bootstrap.cc:492] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:41.512791 13080 ts_tablet_manager.cc:1403] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.262s	user 0.018s	sys 0.022s
W20260709 12:27:41.505409 12924 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 66238.8us
I20260709 12:27:41.509697 13286 raft_consensus.cc:515] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.518105 13306 leader_election.cc:290] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:41.523697 13286 leader_election.cc:290] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:41.523819 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0eea117b500470093718b8583fa5e96" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:41.527341 12774 leader_election.cc:343] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.528707 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0eea117b500470093718b8583fa5e96" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:41.504215 12922 leader_election.cc:304] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:41.534940 13083 tablet_bootstrap.cc:654] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.535948 12999 leader_election.cc:304] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:41.537686 13234 raft_consensus.cc:2749] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:41.523818 13080 raft_consensus.cc:359] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.538570 13080 raft_consensus.cc:385] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.539753 13079 tablet_bootstrap.cc:654] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.536186 13327 raft_consensus.cc:2804] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:41.527943 13279 raft_consensus.cc:515] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.542541 13327 raft_consensus.cc:697] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:41.543601 13327 consensus_queue.cc:237] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.543915 12924 leader_election.cc:304] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:41.544214 13279 leader_election.cc:290] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:41.546151 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6179fd5410440a89ea4ca9c5ef3944" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
W20260709 12:27:41.547405 12776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 58637.5us
I20260709 12:27:41.550002 13142 raft_consensus.cc:2749] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:41.552270 13206 raft_consensus.cc:2804] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:41.552873 13206 raft_consensus.cc:697] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:41.553781 13206 consensus_queue.cc:237] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:41.559523 12922 leader_election.cc:343] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:41.561226 12773 leader_election.cc:343] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.538887 13080 raft_consensus.cc:740] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.561941 12773 leader_election.cc:304] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:41.564978 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d24c0921438489bb6c87bd7b1b0654d" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:41.566394 13221 raft_consensus.cc:2749] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:27:41.566938 12996 leader_election.cc:343] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.568521 13080 consensus_queue.cc:260] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.572129 13280 consensus_queue.cc:1048] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:41.572711 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d24c0921438489bb6c87bd7b1b0654d" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:41.573370 12810 raft_consensus.cc:3060] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.580420 13080 ts_tablet_manager.cc:1434] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.067s	user 0.005s	sys 0.004s
I20260709 12:27:41.581913 13080 tablet_bootstrap.cc:492] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:41.590689 13079 tablet_bootstrap.cc:492] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:41.591404 13079 ts_tablet_manager.cc:1403] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.088s	user 0.010s	sys 0.021s
I20260709 12:27:41.591661 13080 tablet_bootstrap.cc:654] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.592723 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6179fd5410440a89ea4ca9c5ef3944" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:41.593504 12806 raft_consensus.cc:3060] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.594280 13079 raft_consensus.cc:359] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.595090 13079 raft_consensus.cc:385] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.595458 13079 raft_consensus.cc:740] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.596293 13079 consensus_queue.cc:260] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.597609 13301 raft_consensus.cc:493] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.598249 13301 raft_consensus.cc:515] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.601598 13301 leader_election.cc:290] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:41.604559 13221 raft_consensus.cc:493] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.605188 13221 raft_consensus.cc:515] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.605363 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:41.607878 13221 leader_election.cc:290] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:41.609194 13079 ts_tablet_manager.cc:1434] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:27:41.610013 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:41.611923 13078 tablet_bootstrap.cc:654] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.610301 13079 tablet_bootstrap.cc:492] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
W20260709 12:27:41.614151 12776 leader_election.cc:343] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.615162 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:41.617823 12774 leader_election.cc:343] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:41.618382 12774 leader_election.cc:304] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:41.621444 13079 tablet_bootstrap.cc:654] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.624156 13221 raft_consensus.cc:2749] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:41.641002 13280 consensus_queue.cc:1048] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:41.651599 13080 tablet_bootstrap.cc:492] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:41.652211 13080 ts_tablet_manager.cc:1403] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.071s	user 0.009s	sys 0.007s
I20260709 12:27:41.656929 12810 raft_consensus.cc:2468] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:41.672103 12806 raft_consensus.cc:2468] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:41.676036 13078 tablet_bootstrap.cc:492] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:41.677018 13078 ts_tablet_manager.cc:1403] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.188s	user 0.021s	sys 0.002s
I20260709 12:27:41.646664 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:41.682350 13083 tablet_bootstrap.cc:492] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:41.683343 13083 ts_tablet_manager.cc:1403] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.175s	user 0.021s	sys 0.025s
W20260709 12:27:41.686810 12996 consensus_peers.cc:597] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:41.687088 13083 raft_consensus.cc:359] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.688360 13083 raft_consensus.cc:385] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.688814 13083 raft_consensus.cc:740] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.688623 12998 leader_election.cc:304] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:41.690737 13306 raft_consensus.cc:2804] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:41.691385 13306 raft_consensus.cc:697] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:41.692461 13306 consensus_queue.cc:237] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.700649 12706 catalog_manager.cc:5697] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:41.703299 12851 leader_election.cc:304] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:41.704933 13083 consensus_queue.cc:260] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.703896 13080 raft_consensus.cc:359] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:41.714823 12924 leader_election.cc:304] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:41.716439 13080 raft_consensus.cc:385] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.708266 12706 catalog_manager.cc:5697] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:41.716797 13206 raft_consensus.cc:2804] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:41.717854 13206 raft_consensus.cc:697] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:41.709836 13078 raft_consensus.cc:359] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:41.716868 13080 raft_consensus.cc:740] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.720870 13078 raft_consensus.cc:385] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.708446 13083 ts_tablet_manager.cc:1434] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.025s	user 0.008s	sys 0.000s
I20260709 12:27:41.723430 13083 tablet_bootstrap.cc:492] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:41.722028 13080 consensus_queue.cc:260] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:41.719581 13206 consensus_queue.cc:237] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.721731 13078 raft_consensus.cc:740] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.729511 13078 consensus_queue.cc:260] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:41.737124 13080 ts_tablet_manager.cc:1434] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.085s	user 0.009s	sys 0.000s
I20260709 12:27:41.738845 13080 tablet_bootstrap.cc:492] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:41.751854 13078 ts_tablet_manager.cc:1434] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.074s	user 0.008s	sys 0.002s
I20260709 12:27:41.755232 13078 tablet_bootstrap.cc:492] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:41.776548 13079 tablet_bootstrap.cc:492] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:41.777248 13079 ts_tablet_manager.cc:1403] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.167s	user 0.007s	sys 0.017s
I20260709 12:27:41.779125 12882 raft_consensus.cc:3060] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.751492 13301 raft_consensus.cc:2749] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:41.788689 12885 raft_consensus.cc:1275] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:41.741221 13307 consensus_queue.cc:1048] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260709 12:27:41.790360 13347 consensus_queue.cc:1048] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:41.792703 13078 tablet_bootstrap.cc:654] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.799564 13079 raft_consensus.cc:359] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.800405 13079 raft_consensus.cc:385] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.800662 13079 raft_consensus.cc:740] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.801355 13079 consensus_queue.cc:260] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:41.804517 13079 ts_tablet_manager.cc:1434] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.003s
I20260709 12:27:41.805636 13079 tablet_bootstrap.cc:492] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
W20260709 12:27:41.765269 12998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 44929.4us
I20260709 12:27:41.880348 13080 tablet_bootstrap.cc:654] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.879679 12708 catalog_manager.cc:5697] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:41.891799 13280 raft_consensus.cc:493] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.896597 13280 raft_consensus.cc:515] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.882476 13361 raft_consensus.cc:493] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.899415 13361 raft_consensus.cc:515] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:41.904197 13079 tablet_bootstrap.cc:654] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.908414 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:41.909276 12957 raft_consensus.cc:2468] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:41.911245 12999 leader_election.cc:304] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:41.912667 13306 raft_consensus.cc:2804] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:41.913146 13306 raft_consensus.cc:493] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.913528 13306 raft_consensus.cc:3060] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.902447 13361 leader_election.cc:290] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:41.923530 13078 tablet_bootstrap.cc:492] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:41.924280 13078 ts_tablet_manager.cc:1403] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.169s	user 0.032s	sys 0.022s
I20260709 12:27:41.929116 13078 raft_consensus.cc:359] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.930043 13078 raft_consensus.cc:385] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:41.930356 13078 raft_consensus.cc:740] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:41.930459 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:41.931353 13280 leader_election.cc:290] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:41.931298 13078 consensus_queue.cc:260] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:41.933789 12807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:41.934607 12807 raft_consensus.cc:2468] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:41.935470 13078 ts_tablet_manager.cc:1434] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:27:41.936898 13078 tablet_bootstrap.cc:492] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:41.951378 13029 raft_consensus.cc:1275] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:27:41.954154 13142 consensus_queue.cc:1048] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:41.954668 13083 tablet_bootstrap.cc:654] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.955641 12706 catalog_manager.cc:5697] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:41.942487 12924 leader_election.cc:304] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:41.961684 13206 raft_consensus.cc:2804] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:41.962677 13206 raft_consensus.cc:493] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:41.965385 13206 raft_consensus.cc:3060] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:41.956442 13280 raft_consensus.cc:493] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:41.967172 13280 raft_consensus.cc:515] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:41.971014 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:41.973716 13078 tablet_bootstrap.cc:654] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:41.975534 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:41.976336 12810 raft_consensus.cc:2468] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:42.008855 13306 raft_consensus.cc:515] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.013819 13280 leader_election.cc:290] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:42.010445 13206 raft_consensus.cc:515] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.024289 13032 raft_consensus.cc:2393] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8012bb9a8b274890bcaad0d192cd30d5 in current term 1: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
I20260709 12:27:42.027629 12807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:42.028926 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:42.029278 12807 raft_consensus.cc:3060] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.029740 13033 raft_consensus.cc:2468] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:42.031682 13083 tablet_bootstrap.cc:492] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:42.031791 12923 leader_election.cc:304] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:42.032481 13083 ts_tablet_manager.cc:1403] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.309s	user 0.018s	sys 0.004s
I20260709 12:27:42.033672 13279 raft_consensus.cc:2804] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:42.034128 13279 raft_consensus.cc:493] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.034459 13279 raft_consensus.cc:3060] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.037237 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:42.040061 13083 raft_consensus.cc:359] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.040912 13083 raft_consensus.cc:385] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.041167 13083 raft_consensus.cc:740] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.041893 13083 consensus_queue.cc:260] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.042668 13327 consensus_queue.cc:1048] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:42.043341 13219 raft_consensus.cc:493] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.043834 13219 raft_consensus.cc:515] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:42.044229 13327 consensus_queue.cc:1048] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:42.044209 13083 ts_tablet_manager.cc:1434] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:27:42.046036 13219 leader_election.cc:290] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:42.047363 13083 tablet_bootstrap.cc:492] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:42.048045 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc164dcd3f514a00a186655a9a7b2215" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:42.050410 12773 leader_election.cc:343] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.050773 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc164dcd3f514a00a186655a9a7b2215" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:42.054052 12776 leader_election.cc:343] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.054544 13330 raft_consensus.cc:493] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.055087 13083 tablet_bootstrap.cc:654] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.054616 12776 leader_election.cc:304] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:42.055632 13330 raft_consensus.cc:515] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.056322 13346 raft_consensus.cc:2749] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:27:42.058296 12922 leader_election.cc:343] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.058594 13330 leader_election.cc:290] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:42.061960 12957 raft_consensus.cc:2393] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:42.061904 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:42.067751 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
W20260709 12:27:42.068496 12923 leader_election.cc:343] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.070343 13206 leader_election.cc:290] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:42.026373 13306 leader_election.cc:290] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:42.068656 13033 raft_consensus.cc:2393] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8012bb9a8b274890bcaad0d192cd30d5 in current term 1: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
I20260709 12:27:42.053632 13279 raft_consensus.cc:515] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.085844 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:42.087030 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "af4bf74fe4b6486a83af6b41bb30f373" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:42.087050 12808 raft_consensus.cc:2468] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:42.101301 13206 consensus_queue.cc:1048] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:42.110888 13079 tablet_bootstrap.cc:492] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:42.111636 13079 ts_tablet_manager.cc:1403] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.306s	user 0.011s	sys 0.007s
I20260709 12:27:42.114843 13079 raft_consensus.cc:359] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.115479 13079 raft_consensus.cc:385] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.115734 13079 raft_consensus.cc:740] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.116521 13079 consensus_queue.cc:260] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.118896 13079 ts_tablet_manager.cc:1434] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:42.120097 13079 tablet_bootstrap.cc:492] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:42.128019 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
W20260709 12:27:42.130061 12922 leader_election.cc:343] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.131052 12924 leader_election.cc:304] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:42.128064 13080 tablet_bootstrap.cc:492] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:42.132041 13206 raft_consensus.cc:2749] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:42.132270 13080 ts_tablet_manager.cc:1403] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.394s	user 0.017s	sys 0.014s
I20260709 12:27:42.132589 13279 leader_election.cc:290] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:42.131310 12881 raft_consensus.cc:3060] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.081430 12807 raft_consensus.cc:2468] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:42.149165 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:42.150002 13029 raft_consensus.cc:3060] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.139386 13306 consensus_queue.cc:1048] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260709 12:27:42.158226 12998 leader_election.cc:304] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:42.158830 12924 leader_election.cc:304] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:42.162276 13080 raft_consensus.cc:359] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.164183 13206 raft_consensus.cc:2804] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:42.164362 13080 raft_consensus.cc:385] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.164629 13206 raft_consensus.cc:493] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.164696 13080 raft_consensus.cc:740] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.165061 13206 raft_consensus.cc:3060] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.165731 13080 consensus_queue.cc:260] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.166426 13224 raft_consensus.cc:2804] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:42.167165 13224 raft_consensus.cc:697] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:42.168432 13080 ts_tablet_manager.cc:1434] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.036s	user 0.008s	sys 0.000s
I20260709 12:27:42.169665 13080 tablet_bootstrap.cc:492] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:42.168021 13224 consensus_queue.cc:237] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.172089 13083 tablet_bootstrap.cc:492] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:42.172649 13083 ts_tablet_manager.cc:1403] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.126s	user 0.026s	sys 0.027s
I20260709 12:27:42.176870 13079 tablet_bootstrap.cc:654] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.177407 13206 raft_consensus.cc:515] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:42.199000 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:42.199144 13301 raft_consensus.cc:493] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.199946 13301 raft_consensus.cc:515] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.200611 13330 consensus_queue.cc:1048] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:42.202328 13301 leader_election.cc:290] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
W20260709 12:27:42.223997 12922 consensus_peers.cc:597] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:42.226519 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:42.217808 13083 raft_consensus.cc:359] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:42.244068 13083 raft_consensus.cc:385] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.244609 13083 raft_consensus.cc:740] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.252821 13206 leader_election.cc:290] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:42.253670 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
W20260709 12:27:42.258951 12923 leader_election.cc:343] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.264299 13280 raft_consensus.cc:493] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.264922 13280 raft_consensus.cc:515] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.267454 13280 leader_election.cc:290] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:42.270870 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f40e0b80665048e09e181f1a8ea7509a" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:42.272814 12923 leader_election.cc:343] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.278264 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f40e0b80665048e09e181f1a8ea7509a" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:42.281699 12922 leader_election.cc:343] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.282471 12922 leader_election.cc:304] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:42.284009 13206 raft_consensus.cc:2749] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:42.286620 13078 tablet_bootstrap.cc:492] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:42.287427 13078 ts_tablet_manager.cc:1403] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.351s	user 0.026s	sys 0.003s
I20260709 12:27:42.287932 13080 tablet_bootstrap.cc:654] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.246949 13083 consensus_queue.cc:260] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:42.304287 13083 ts_tablet_manager.cc:1434] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.131s	user 0.009s	sys 0.001s
I20260709 12:27:42.307322 13083 tablet_bootstrap.cc:492] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
W20260709 12:27:42.317710 12996 consensus_peers.cc:597] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:42.320408 13079 tablet_bootstrap.cc:492] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:42.321051 13079 ts_tablet_manager.cc:1403] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.201s	user 0.040s	sys 0.041s
I20260709 12:27:42.324050 13079 raft_consensus.cc:359] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:42.324790 13079 raft_consensus.cc:385] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.325086 13079 raft_consensus.cc:740] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.325806 13079 consensus_queue.cc:260] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:42.308638 13279 consensus_queue.cc:1048] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 12:27:42.337677 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:42.338423 12809 raft_consensus.cc:3060] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.345394 13083 tablet_bootstrap.cc:654] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.346516 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:42.347347 13221 raft_consensus.cc:493] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.347553 12959 raft_consensus.cc:2393] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:42.347970 13221 raft_consensus.cc:515] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.354102 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30cb0580c4514b41b650a6b2d5089f3a" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:42.349200 13079 ts_tablet_manager.cc:1434] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.002s
I20260709 12:27:42.356843 13079 tablet_bootstrap.cc:492] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
W20260709 12:27:42.359385 12774 leader_election.cc:343] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.366453 13362 raft_consensus.cc:493] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 18cf4829b3bc4fafb3ef7f1b98f853fe)
I20260709 12:27:42.355589 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30cb0580c4514b41b650a6b2d5089f3a" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:42.356124 13221 leader_election.cc:290] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:42.367641 13362 raft_consensus.cc:515] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.377454 13234 raft_consensus.cc:493] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.378422 13234 raft_consensus.cc:515] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.379760 12851 leader_election.cc:304] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:42.381321 13301 raft_consensus.cc:3060] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.382189 13029 raft_consensus.cc:2468] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
W20260709 12:27:42.383714 12773 leader_election.cc:343] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.384265 12773 leader_election.cc:304] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:42.386394 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0eea117b500470093718b8583fa5e96" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:42.387341 12808 raft_consensus.cc:2468] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:42.388484 13234 leader_election.cc:290] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:42.390594 12923 leader_election.cc:304] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:42.393566 13301 raft_consensus.cc:2749] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:42.394660 13219 raft_consensus.cc:2749] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:42.394979 13379 raft_consensus.cc:2804] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:42.399837 13379 raft_consensus.cc:697] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:42.400753 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:42.401082 13379 consensus_queue.cc:237] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.391911 12998 leader_election.cc:304] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:42.403817 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:42.404826 13224 raft_consensus.cc:2804] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:42.405366 13224 raft_consensus.cc:493] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.405787 13224 raft_consensus.cc:3060] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.406054 12849 leader_election.cc:304] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:42.398054 12809 raft_consensus.cc:2468] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:42.399915 13362 leader_election.cc:290] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:42.397563 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0eea117b500470093718b8583fa5e96" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:42.396852 13079 tablet_bootstrap.cc:654] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.409309 13301 raft_consensus.cc:2749] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:42.414251 13078 raft_consensus.cc:359] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.415243 13078 raft_consensus.cc:385] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.415547 13078 raft_consensus.cc:740] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.416314 13078 consensus_queue.cc:260] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.428262 13078 ts_tablet_manager.cc:1434] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.137s	user 0.007s	sys 0.001s
I20260709 12:27:42.431885 13078 tablet_bootstrap.cc:492] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
W20260709 12:27:42.433596 12996 leader_election.cc:343] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.438319 13083 tablet_bootstrap.cc:492] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:42.439254 13083 ts_tablet_manager.cc:1403] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.132s	user 0.015s	sys 0.004s
I20260709 12:27:42.443172 13080 tablet_bootstrap.cc:492] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:42.443753 13080 ts_tablet_manager.cc:1403] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.274s	user 0.015s	sys 0.001s
I20260709 12:27:42.448486 12706 catalog_manager.cc:5697] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:42.427989 12924 leader_election.cc:304] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:42.459951 13327 raft_consensus.cc:2804] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:42.460556 13327 raft_consensus.cc:697] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:42.461444 13327 consensus_queue.cc:237] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.466461 13080 raft_consensus.cc:359] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.473071 13224 raft_consensus.cc:515] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.474609 13080 raft_consensus.cc:385] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.474982 13080 raft_consensus.cc:740] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.477075 13080 consensus_queue.cc:260] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.484537 13078 tablet_bootstrap.cc:654] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.485203 13079 tablet_bootstrap.cc:492] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:42.485894 13079 ts_tablet_manager.cc:1403] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.129s	user 0.017s	sys 0.015s
I20260709 12:27:42.486608 13327 raft_consensus.cc:493] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.487293 13327 raft_consensus.cc:515] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:42.489795 12924 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 65309.1us
I20260709 12:27:42.487377 13083 raft_consensus.cc:359] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.493175 13224 leader_election.cc:290] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:42.493010 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0eea117b500470093718b8583fa5e96" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:42.493309 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0eea117b500470093718b8583fa5e96" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:42.493077 13083 raft_consensus.cc:385] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.493872 13079 raft_consensus.cc:359] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.495687 13079 raft_consensus.cc:385] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.495982 13079 raft_consensus.cc:740] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.495251 13083 raft_consensus.cc:740] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.498579 13083 consensus_queue.cc:260] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.499816 13080 ts_tablet_manager.cc:1434] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.056s	user 0.008s	sys 0.001s
I20260709 12:27:42.501217 13080 tablet_bootstrap.cc:492] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:42.502019 12808 raft_consensus.cc:3060] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:42.504266 12996 leader_election.cc:343] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.521395 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b32a14229832492682674ac4c282404d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:42.523459 12922 leader_election.cc:343] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.524255 13327 leader_election.cc:290] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:42.533627 13083 ts_tablet_manager.cc:1434] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.094s	user 0.009s	sys 0.000s
I20260709 12:27:42.535523 13083 tablet_bootstrap.cc:492] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:42.539945 13346 raft_consensus.cc:493] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.541831 13079 consensus_queue.cc:260] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.543242 13346 raft_consensus.cc:515] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:42.547569 13346 leader_election.cc:290] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:42.543960 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b32a14229832492682674ac4c282404d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:42.551225 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:42.551496 12806 raft_consensus.cc:2468] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:42.554383 13079 ts_tablet_manager.cc:1434] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.068s	user 0.010s	sys 0.001s
I20260709 12:27:42.555585 13079 tablet_bootstrap.cc:492] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:42.560989 13079 tablet_bootstrap.cc:654] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.564236 13078 tablet_bootstrap.cc:492] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:42.564863 13078 ts_tablet_manager.cc:1403] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.135s	user 0.019s	sys 0.029s
I20260709 12:27:42.567011 13080 tablet_bootstrap.cc:654] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.570192 13078 raft_consensus.cc:359] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.571578 13078 raft_consensus.cc:385] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.571900 13078 raft_consensus.cc:740] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.572973 13078 consensus_queue.cc:260] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.575107 13078 ts_tablet_manager.cc:1434] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:27:42.576646 13083 tablet_bootstrap.cc:654] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.576361 13078 tablet_bootstrap.cc:492] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:42.586320 13079 tablet_bootstrap.cc:492] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:42.586961 13079 ts_tablet_manager.cc:1403] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.032s	user 0.014s	sys 0.000s
I20260709 12:27:42.587045 13080 tablet_bootstrap.cc:492] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:42.587666 13080 ts_tablet_manager.cc:1403] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.087s	user 0.019s	sys 0.007s
I20260709 12:27:42.589417 13078 tablet_bootstrap.cc:654] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.590031 13083 tablet_bootstrap.cc:492] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:42.590108 13079 raft_consensus.cc:359] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.590549 13080 raft_consensus.cc:359] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.590971 13083 ts_tablet_manager.cc:1403] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.056s	user 0.014s	sys 0.000s
I20260709 12:27:42.591216 13080 raft_consensus.cc:385] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.592000 13080 raft_consensus.cc:740] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.591121 13079 raft_consensus.cc:385] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.592710 13079 raft_consensus.cc:740] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.592864 13080 consensus_queue.cc:260] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.593408 13079 consensus_queue.cc:260] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.560817 12808 raft_consensus.cc:2468] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:42.670652 13078 tablet_bootstrap.cc:492] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:42.673511 13078 ts_tablet_manager.cc:1403] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.097s	user 0.003s	sys 0.016s
I20260709 12:27:42.674206 13079 ts_tablet_manager.cc:1434] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.087s	user 0.006s	sys 0.002s
I20260709 12:27:42.677356 13079 tablet_bootstrap.cc:492] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:42.683797 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:42.687250 13346 raft_consensus.cc:493] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.688329 12998 leader_election.cc:304] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:42.688709 13346 raft_consensus.cc:515] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.690876 13307 raft_consensus.cc:2804] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:42.660622 12924 leader_election.cc:304] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
W20260709 12:27:42.705602 12924 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 45753.9us
I20260709 12:27:42.705672 12704 catalog_manager.cc:5697] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:42.708432 12773 leader_election.cc:343] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.698021 13078 raft_consensus.cc:359] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
W20260709 12:27:42.716254 12776 leader_election.cc:343] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.726948 13346 leader_election.cc:290] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:42.714736 13224 raft_consensus.cc:493] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.728173 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0034a6bef1bf4e119ee686581612f8c0" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:42.728672 13224 raft_consensus.cc:515] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.715346 13078 raft_consensus.cc:385] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:27:42.733032 12774 leader_election.cc:343] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.710983 12704 catalog_manager.cc:5697] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:42.738613 13224 leader_election.cc:290] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:42.747294 13080 ts_tablet_manager.cc:1434] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.159s	user 0.011s	sys 0.000s
I20260709 12:27:42.748095 13379 raft_consensus.cc:2804] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:42.740279 13083 raft_consensus.cc:359] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:42.748872 13083 raft_consensus.cc:385] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.748777 13379 raft_consensus.cc:493] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.749347 13083 raft_consensus.cc:740] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.752399 13083 consensus_queue.cc:260] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:42.751382 13080 tablet_bootstrap.cc:492] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:42.749960 13379 raft_consensus.cc:3060] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.768631 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:42.753031 12998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 65401.4us
I20260709 12:27:42.716857 13307 raft_consensus.cc:697] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:42.762712 13280 raft_consensus.cc:493] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.781999 13280 raft_consensus.cc:515] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.730964 13078 raft_consensus.cc:740] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.790012 13307 consensus_queue.cc:237] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.794030 13280 leader_election.cc:290] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:42.784948 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0034a6bef1bf4e119ee686581612f8c0" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:42.796669 12883 raft_consensus.cc:3060] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.798533 13078 consensus_queue.cc:260] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.801458 13083 ts_tablet_manager.cc:1434] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.209s	user 0.013s	sys 0.002s
I20260709 12:27:42.785183 12776 leader_election.cc:304] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:42.805852 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf46416458c4a9398e8de1c6b8ccb5d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:42.807679 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf46416458c4a9398e8de1c6b8ccb5d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:42.814239 13142 raft_consensus.cc:2749] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:42.822928 13080 tablet_bootstrap.cc:654] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.815218 13032 raft_consensus.cc:2468] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:42.826493 13362 raft_consensus.cc:493] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:42.826895 12776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 119392us
I20260709 12:27:42.828343 13079 tablet_bootstrap.cc:654] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.831112 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:42.832665 12809 raft_consensus.cc:2393] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:42.833519 13219 raft_consensus.cc:493] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.834456 13219 raft_consensus.cc:515] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:42.827268 13362 raft_consensus.cc:515] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:42.840171 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:42.844887 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:42.847620 13219 leader_election.cc:290] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
W20260709 12:27:42.851380 12774 leader_election.cc:343] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.851359 12923 leader_election.cc:304] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
W20260709 12:27:42.853075 12922 leader_election.cc:343] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.853518 13327 raft_consensus.cc:2804] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:42.854071 13327 raft_consensus.cc:493] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.854486 13327 raft_consensus.cc:3060] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:42.859244 12773 leader_election.cc:343] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.860239 12773 leader_election.cc:304] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:42.861791 13142 raft_consensus.cc:2749] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:42.838343 13362 leader_election.cc:290] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:42.865556 13224 consensus_queue.cc:1048] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:42.868767 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:42.870831 12850 leader_election.cc:304] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:42.872799 12885 raft_consensus.cc:3060] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.861805 13078 ts_tablet_manager.cc:1434] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.188s	user 0.009s	sys 0.009s
I20260709 12:27:42.876638 13078 tablet_bootstrap.cc:492] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:42.878386 13083 tablet_bootstrap.cc:492] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:42.882475 13307 raft_consensus.cc:493] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:42.883514 13307 raft_consensus.cc:515] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.895318 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:42.899024 12923 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.68:32909, user_credentials={real_user=slave}} blocked reactor thread for 47981.9us
I20260709 12:27:42.909665 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f40e0b80665048e09e181f1a8ea7509a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:42.912552 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f40e0b80665048e09e181f1a8ea7509a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:42.913806 12959 raft_consensus.cc:2468] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:42.919976 12999 leader_election.cc:304] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:42.921175 13234 raft_consensus.cc:2804] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:42.921715 13234 raft_consensus.cc:493] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:42.922078 13234 raft_consensus.cc:3060] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:42.928076 12996 leader_election.cc:343] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.940891 13083 tablet_bootstrap.cc:654] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.911734 13307 leader_election.cc:290] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
W20260709 12:27:42.943285 12776 leader_election.cc:343] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:42.949658 13327 raft_consensus.cc:515] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:42.945454 12776 leader_election.cc:304] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:42.959991 13032 raft_consensus.cc:3060] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.970320 13142 raft_consensus.cc:2749] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:42.911180 12998 leader_election.cc:304] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:42.992664 13206 consensus_queue.cc:1048] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:43.006738 13079 tablet_bootstrap.cc:492] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:43.007503 13079 ts_tablet_manager.cc:1403] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.331s	user 0.010s	sys 0.014s
I20260709 12:27:43.008263 13080 tablet_bootstrap.cc:492] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:43.008934 13080 ts_tablet_manager.cc:1403] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.258s	user 0.024s	sys 0.032s
W20260709 12:27:42.996124 12998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 85425us
I20260709 12:27:43.016942 13363 consensus_queue.cc:1048] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:43.026392 13206 consensus_queue.cc:1048] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:43.016680 13079 raft_consensus.cc:359] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.030568 13079 raft_consensus.cc:385] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.031106 13079 raft_consensus.cc:740] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.032263 13078 tablet_bootstrap.cc:654] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.032331 13079 consensus_queue.cc:260] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.035672 13079 ts_tablet_manager.cc:1434] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.028s	user 0.005s	sys 0.005s
I20260709 12:27:43.040112 13234 raft_consensus.cc:515] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:43.041601 13079 tablet_bootstrap.cc:492] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:43.040760 12704 catalog_manager.cc:5697] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:43.062917 13234 leader_election.cc:290] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:43.068846 13224 raft_consensus.cc:2749] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:43.069730 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f40e0b80665048e09e181f1a8ea7509a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:43.090994 12959 raft_consensus.cc:3060] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.113368 13206 consensus_queue.cc:1048] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260709 12:27:43.085052 13379 raft_consensus.cc:515] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.122879 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f40e0b80665048e09e181f1a8ea7509a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:43.151012 13327 leader_election.cc:290] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:43.162022 13083 tablet_bootstrap.cc:492] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:43.162566 13083 ts_tablet_manager.cc:1403] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.343s	user 0.021s	sys 0.021s
I20260709 12:27:43.167846 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf46416458c4a9398e8de1c6b8ccb5d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:43.168725 13078 tablet_bootstrap.cc:492] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:43.143173 13280 consensus_queue.cc:1048] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260709 12:27:43.158653 13379 leader_election.cc:290] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:43.161561 13080 raft_consensus.cc:359] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.171435 13080 raft_consensus.cc:385] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.171841 13080 raft_consensus.cc:740] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.169749 13078 ts_tablet_manager.cc:1403] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.294s	user 0.009s	sys 0.030s
W20260709 12:27:43.176093 12922 leader_election.cc:343] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.172753 13362 raft_consensus.cc:2749] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:43.178056 13083 raft_consensus.cc:359] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.178843 13083 raft_consensus.cc:385] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.179144 13083 raft_consensus.cc:740] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.180292 13080 consensus_queue.cc:260] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.181385 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b32a14229832492682674ac4c282404d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:43.183530 13080 ts_tablet_manager.cc:1434] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.174s	user 0.008s	sys 0.003s
I20260709 12:27:43.184139 13079 tablet_bootstrap.cc:654] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.190260 13083 consensus_queue.cc:260] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.183574 12881 raft_consensus.cc:3060] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.192445 13083 ts_tablet_manager.cc:1434] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.000s
I20260709 12:27:43.193754 13083 tablet_bootstrap.cc:492] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:43.194278 13078 raft_consensus.cc:359] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:43.196240 13078 raft_consensus.cc:385] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.196659 13078 raft_consensus.cc:740] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
W20260709 12:27:43.198679 12922 leader_election.cc:343] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.213282 13080 tablet_bootstrap.cc:492] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:43.216375 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b32a14229832492682674ac4c282404d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
W20260709 12:27:43.217006 12996 leader_election.cc:343] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.217880 12806 raft_consensus.cc:3060] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.218969 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf46416458c4a9398e8de1c6b8ccb5d" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:43.219745 13031 raft_consensus.cc:3060] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.230041 13031 raft_consensus.cc:2468] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:43.236496 13078 consensus_queue.cc:260] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
W20260709 12:27:43.202251 12998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 82760.2us
I20260709 12:27:43.254314 13219 raft_consensus.cc:493] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.259275 13219 raft_consensus.cc:515] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.264902 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "068448e5fa294d199668e58a170cbee3" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:43.268378 12776 leader_election.cc:343] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.261579 13219 leader_election.cc:290] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:43.291999 12923 leader_election.cc:304] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:43.293025 13080 tablet_bootstrap.cc:654] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.293725 13279 raft_consensus.cc:2804] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.296515 13279 raft_consensus.cc:697] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:43.297729 13279 consensus_queue.cc:237] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:43.305256 13379 raft_consensus.cc:493] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.342185 13078 ts_tablet_manager.cc:1434] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.168s	user 0.007s	sys 0.002s
I20260709 12:27:43.343845 13078 tablet_bootstrap.cc:492] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:43.347225 13379 raft_consensus.cc:515] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.373805 12806 raft_consensus.cc:2468] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:43.374466 13083 tablet_bootstrap.cc:654] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.377774 12959 raft_consensus.cc:2468] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:43.393751 13079 tablet_bootstrap.cc:492] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:43.394490 13079 ts_tablet_manager.cc:1403] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.353s	user 0.016s	sys 0.023s
I20260709 12:27:43.400658 13379 leader_election.cc:290] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:43.403214 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc164dcd3f514a00a186655a9a7b2215" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:43.405139 13306 raft_consensus.cc:493] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.405737 13306 raft_consensus.cc:515] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:43.408092 13306 leader_election.cc:290] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:43.409025 12922 leader_election.cc:343] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.411906 13234 consensus_queue.cc:1048] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:43.415934 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30cb0580c4514b41b650a6b2d5089f3a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:43.416622 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30cb0580c4514b41b650a6b2d5089f3a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:43.417095 12808 raft_consensus.cc:2468] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
W20260709 12:27:43.418988 12996 leader_election.cc:343] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.436219 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "068448e5fa294d199668e58a170cbee3" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:43.438410 13079 raft_consensus.cc:359] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:43.438979 12773 leader_election.cc:343] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.439429 13079 raft_consensus.cc:385] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.439599 12773 leader_election.cc:304] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:43.441221 13142 raft_consensus.cc:2749] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:43.439811 13079 raft_consensus.cc:740] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.444183 13079 consensus_queue.cc:260] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.446108 13079 ts_tablet_manager.cc:1434] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.051s	user 0.005s	sys 0.000s
I20260709 12:27:43.447446 13079 tablet_bootstrap.cc:492] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:43.449729 13078 tablet_bootstrap.cc:654] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.481704 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc164dcd3f514a00a186655a9a7b2215" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:43.482136 12924 leader_election.cc:304] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:43.482623 12810 raft_consensus.cc:2468] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:43.484879 13279 raft_consensus.cc:2804] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.485543 13279 raft_consensus.cc:697] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:43.514541 13079 tablet_bootstrap.cc:654] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.521190 12924 leader_election.cc:304] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:43.487304 12998 leader_election.cc:304] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:43.489228 12999 leader_election.cc:304] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:43.541710 13080 tablet_bootstrap.cc:492] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:43.542481 13080 ts_tablet_manager.cc:1403] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.332s	user 0.009s	sys 0.007s
W20260709 12:27:43.546913 12996 consensus_peers.cc:597] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:43.558516 12999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 90738.3us
I20260709 12:27:43.557991 12705 catalog_manager.cc:5697] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:43.559619 13306 raft_consensus.cc:2804] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.543184 13224 raft_consensus.cc:2804] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.572465 13224 raft_consensus.cc:493] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.572978 13224 raft_consensus.cc:3060] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.519572 13279 consensus_queue.cc:237] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.577841 13306 raft_consensus.cc:697] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:43.579074 13306 consensus_queue.cc:237] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:43.585615 13424 raft_consensus.cc:2804] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.586206 13424 raft_consensus.cc:493] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.586563 13424 raft_consensus.cc:3060] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:43.562134 12998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 98930.2us
I20260709 12:27:43.600998 13362 raft_consensus.cc:493] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8012bb9a8b274890bcaad0d192cd30d5)
I20260709 12:27:43.601823 13362 raft_consensus.cc:515] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.621457 13206 raft_consensus.cc:493] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.622062 13206 raft_consensus.cc:515] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.631273 13362 leader_election.cc:290] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:43.632481 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:43.634229 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6179fd5410440a89ea4ca9c5ef3944" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:43.634620 12807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:43.635380 12807 raft_consensus.cc:2468] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
W20260709 12:27:43.641734 12923 leader_election.cc:343] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.631549 13206 leader_election.cc:290] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:43.655908 13346 raft_consensus.cc:493] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.656782 13346 raft_consensus.cc:515] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.657159 13224 raft_consensus.cc:515] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:43.660228 13224 leader_election.cc:290] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:43.660982 13346 leader_election.cc:290] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:43.661713 12805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30cb0580c4514b41b650a6b2d5089f3a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:43.662647 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4a9064e4143f7a4f1ce3a3a4bb860" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:43.663185 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30cb0580c4514b41b650a6b2d5089f3a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:43.663352 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4a9064e4143f7a4f1ce3a3a4bb860" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:43.664958 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c6179fd5410440a89ea4ca9c5ef3944" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:43.671933 13080 raft_consensus.cc:359] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.713406 13080 raft_consensus.cc:385] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.714358 13080 raft_consensus.cc:740] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.710461 12805 raft_consensus.cc:3060] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:43.720202 12996 leader_election.cc:343] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.723660 13079 tablet_bootstrap.cc:492] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:43.723660 13078 tablet_bootstrap.cc:492] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:43.724575 13078 ts_tablet_manager.cc:1403] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.381s	user 0.027s	sys 0.032s
W20260709 12:27:43.725577 12776 leader_election.cc:343] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.728262 13078 raft_consensus.cc:359] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.732568 13078 raft_consensus.cc:385] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.733103 13078 raft_consensus.cc:740] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.734917 13080 consensus_queue.cc:260] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.724575 13079 ts_tablet_manager.cc:1403] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.277s	user 0.049s	sys 0.055s
I20260709 12:27:43.741654 13079 raft_consensus.cc:359] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:43.742543 13079 raft_consensus.cc:385] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.742844 13079 raft_consensus.cc:740] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.744750 12924 leader_election.cc:304] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:43.741658 13078 consensus_queue.cc:260] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.751259 12705 catalog_manager.cc:5697] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.749140 13206 raft_consensus.cc:2804] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.753773 12706 catalog_manager.cc:5697] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.754555 13206 raft_consensus.cc:493] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.766175 13206 raft_consensus.cc:3060] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.744263 12850 leader_election.cc:304] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:43.776139 13362 raft_consensus.cc:2749] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:43.779583 13080 ts_tablet_manager.cc:1434] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.237s	user 0.009s	sys 0.000s
I20260709 12:27:43.780004 13206 raft_consensus.cc:515] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:43.780959 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:27:43.761842 12774 leader_election.cc:343] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.759207 12805 raft_consensus.cc:2468] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:43.765172 13424 raft_consensus.cc:515] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.781509 12774 leader_election.cc:304] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:43.783085 13142 raft_consensus.cc:2749] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:43.785296 13080 tablet_bootstrap.cc:492] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:43.804145 12805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:43.806373 12805 raft_consensus.cc:3060] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.808816 13080 tablet_bootstrap.cc:654] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.761373 13083 tablet_bootstrap.cc:492] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:43.786075 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc164dcd3f514a00a186655a9a7b2215" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:43.786607 13424 leader_election.cc:290] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:43.812711 12922 leader_election.cc:343] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.788061 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc164dcd3f514a00a186655a9a7b2215" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:43.814025 12808 raft_consensus.cc:3060] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.814726 13083 ts_tablet_manager.cc:1403] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.621s	user 0.048s	sys 0.026s
I20260709 12:27:43.780153 13079 consensus_queue.cc:260] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:43.763345 13078 ts_tablet_manager.cc:1434] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.038s	user 0.008s	sys 0.000s
I20260709 12:27:43.791158 12998 leader_election.cc:304] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:43.820043 13078 tablet_bootstrap.cc:492] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:43.795176 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:43.820415 13224 raft_consensus.cc:2804] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.820739 13029 raft_consensus.cc:3060] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.821040 13224 raft_consensus.cc:697] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:43.792261 13206 leader_election.cc:290] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:43.821934 13224 consensus_queue.cc:237] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:43.824131 13079 ts_tablet_manager.cc:1434] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.087s	user 0.008s	sys 0.000s
I20260709 12:27:43.825496 13079 tablet_bootstrap.cc:492] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:43.818502 12805 raft_consensus.cc:2468] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:43.827383 12924 leader_election.cc:304] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:43.828284 13206 raft_consensus.cc:2804] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.828830 13206 raft_consensus.cc:697] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:43.829361 13029 raft_consensus.cc:2468] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:43.829701 13206 consensus_queue.cc:237] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.852986 13078 tablet_bootstrap.cc:654] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.856094 13362 raft_consensus.cc:493] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.856926 13362 raft_consensus.cc:515] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.861172 12805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b32a14229832492682674ac4c282404d" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:43.862847 12805 raft_consensus.cc:2393] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:43.865140 13362 leader_election.cc:290] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:43.863122 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b32a14229832492682674ac4c282404d" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:43.867297 12851 leader_election.cc:304] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:43.868937 13423 raft_consensus.cc:3060] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.870275 12808 raft_consensus.cc:2468] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:43.877878 13079 tablet_bootstrap.cc:654] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:43.878537 12922 consensus_peers.cc:597] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:43.819057 13083 raft_consensus.cc:359] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.887562 12924 leader_election.cc:304] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:43.888044 13083 raft_consensus.cc:385] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.888926 13379 raft_consensus.cc:2804] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.889936 13379 raft_consensus.cc:697] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:43.891215 13379 consensus_queue.cc:237] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:43.892274 12703 catalog_manager.cc:5697] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:43.888921 13083 raft_consensus.cc:740] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.897344 13083 consensus_queue.cc:260] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:43.900813 13083 ts_tablet_manager.cc:1434] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.085s	user 0.008s	sys 0.000s
I20260709 12:27:43.902311 13083 tablet_bootstrap.cc:492] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:43.906029 13080 tablet_bootstrap.cc:492] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:43.906610 13080 ts_tablet_manager.cc:1403] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.122s	user 0.039s	sys 0.047s
I20260709 12:27:43.907698 13423 raft_consensus.cc:2749] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:43.908383 13083 tablet_bootstrap.cc:654] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.909340 13080 raft_consensus.cc:359] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:43.910010 13080 raft_consensus.cc:385] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.910269 13080 raft_consensus.cc:740] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.910950 13080 consensus_queue.cc:260] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:43.914633 13080 ts_tablet_manager.cc:1434] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:43.915982 13080 tablet_bootstrap.cc:492] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:43.926083 13080 tablet_bootstrap.cc:654] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.926606 13379 consensus_queue.cc:1048] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:43.931432 13438 log.cc:927] Time spent T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:27:43.939725 12706 catalog_manager.cc:5697] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:43.968578 13079 tablet_bootstrap.cc:492] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:43.982231 13079 ts_tablet_manager.cc:1403] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.157s	user 0.020s	sys 0.005s
I20260709 12:27:43.969193 13078 tablet_bootstrap.cc:492] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:43.986770 13078 ts_tablet_manager.cc:1403] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.167s	user 0.009s	sys 0.017s
I20260709 12:27:43.994061 12708 catalog_manager.cc:5697] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.997144 13079 raft_consensus.cc:359] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.001614 13079 raft_consensus.cc:385] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.002161 13079 raft_consensus.cc:740] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.003440 13079 consensus_queue.cc:260] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.006603 13079 ts_tablet_manager.cc:1434] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 12:27:44.023450 13142 raft_consensus.cc:493] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.031911 13142 raft_consensus.cc:515] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.032423 12884 raft_consensus.cc:3060] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.035362 13142 leader_election.cc:290] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:44.035439 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d51d99d504d708ad4af4d083eba48" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:44.035497 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d51d99d504d708ad4af4d083eba48" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:44.037129 13079 tablet_bootstrap.cc:492] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
W20260709 12:27:44.037384 12773 leader_election.cc:343] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.038650 13078 raft_consensus.cc:359] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.042074 13078 raft_consensus.cc:385] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.042429 13078 raft_consensus.cc:740] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.058068 13078 consensus_queue.cc:260] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
W20260709 12:27:44.055459 12774 leader_election.cc:343] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.060729 12774 leader_election.cc:304] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:44.070850 13080 tablet_bootstrap.cc:492] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:44.071102 13142 raft_consensus.cc:2749] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.071765 13080 ts_tablet_manager.cc:1403] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.156s	user 0.051s	sys 0.055s
I20260709 12:27:44.075697 13083 tablet_bootstrap.cc:492] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:44.076448 13083 ts_tablet_manager.cc:1403] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.175s	user 0.011s	sys 0.003s
I20260709 12:27:44.075729 13080 raft_consensus.cc:359] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.077116 13080 raft_consensus.cc:385] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.077416 13080 raft_consensus.cc:740] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.077915 13078 ts_tablet_manager.cc:1434] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.091s	user 0.009s	sys 0.003s
I20260709 12:27:44.078540 13080 consensus_queue.cc:260] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.080863 13080 ts_tablet_manager.cc:1434] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:27:44.082180 13080 tablet_bootstrap.cc:492] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:44.090454 13079 tablet_bootstrap.cc:654] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.092063 13080 tablet_bootstrap.cc:654] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.092507 13307 consensus_queue.cc:1048] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.079357 13083 raft_consensus.cc:359] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.094609 13083 raft_consensus.cc:385] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.095024 13083 raft_consensus.cc:740] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.099740 13083 consensus_queue.cc:260] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.094080 13078 tablet_bootstrap.cc:492] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:44.107327 13206 raft_consensus.cc:493] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.107831 13206 raft_consensus.cc:515] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.119628 13419 consensus_queue.cc:1048] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.121905 13280 consensus_queue.cc:1048] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.119628 13080 tablet_bootstrap.cc:492] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:44.128697 13080 ts_tablet_manager.cc:1403] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.047s	user 0.012s	sys 0.006s
I20260709 12:27:44.132488 13206 leader_election.cc:290] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:44.133661 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed10b80d23ae40019ba46ed7087ca05a" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:44.135486 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed10b80d23ae40019ba46ed7087ca05a" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:44.136193 13083 ts_tablet_manager.cc:1434] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.000s
I20260709 12:27:44.145345 13083 tablet_bootstrap.cc:492] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:44.132354 13080 raft_consensus.cc:359] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.140652 13079 tablet_bootstrap.cc:492] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:44.151273 13079 ts_tablet_manager.cc:1403] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.114s	user 0.014s	sys 0.000s
I20260709 12:27:44.154037 13079 raft_consensus.cc:359] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.154701 13079 raft_consensus.cc:385] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.155014 13079 raft_consensus.cc:740] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.155659 13079 consensus_queue.cc:260] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.157876 13079 ts_tablet_manager.cc:1434] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:44.158252 13080 raft_consensus.cc:385] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.158537 13080 raft_consensus.cc:740] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.159296 13080 consensus_queue.cc:260] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.161237 13080 ts_tablet_manager.cc:1434] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.032s	user 0.002s	sys 0.004s
I20260709 12:27:44.142571 13078 tablet_bootstrap.cc:654] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.163864 13080 tablet_bootstrap.cc:492] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:44.165381 13206 consensus_queue.cc:1048] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.164799 13079 tablet_bootstrap.cc:492] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
W20260709 12:27:44.143467 12922 leader_election.cc:343] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:44.169175 12923 leader_election.cc:343] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.172247 12923 leader_election.cc:304] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:44.173449 13379 raft_consensus.cc:2749] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.176780 13083 tablet_bootstrap.cc:654] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.192528 13080 tablet_bootstrap.cc:654] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.194429 13083 tablet_bootstrap.cc:492] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:44.195060 13083 ts_tablet_manager.cc:1403] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.050s	user 0.012s	sys 0.002s
I20260709 12:27:44.207887 13083 raft_consensus.cc:359] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.208669 13083 raft_consensus.cc:385] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.208441 13079 tablet_bootstrap.cc:654] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.209010 13083 raft_consensus.cc:740] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.219383 13078 tablet_bootstrap.cc:492] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:44.222366 13078 ts_tablet_manager.cc:1403] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.129s	user 0.020s	sys 0.024s
I20260709 12:27:44.225265 13078 raft_consensus.cc:359] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.225980 13078 raft_consensus.cc:385] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.226261 13078 raft_consensus.cc:740] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.226325 13362 raft_consensus.cc:493] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.226928 13362 raft_consensus.cc:515] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.227167 13078 consensus_queue.cc:260] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.229208 13362 leader_election.cc:290] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:44.230248 13080 tablet_bootstrap.cc:492] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:44.230839 13080 ts_tablet_manager.cc:1403] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.069s	user 0.010s	sys 0.011s
I20260709 12:27:44.231623 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0eea117b500470093718b8583fa5e96" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:44.235749 13078 ts_tablet_manager.cc:1434] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:27:44.236522 13080 raft_consensus.cc:359] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.237233 13080 raft_consensus.cc:385] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.237483 13080 raft_consensus.cc:740] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.238148 13080 consensus_queue.cc:260] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.240280 13080 ts_tablet_manager.cc:1434] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:27:44.241314 13080 tablet_bootstrap.cc:492] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:44.239054 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0eea117b500470093718b8583fa5e96" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:44.245064 13083 consensus_queue.cc:260] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.246593 12850 leader_election.cc:304] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:44.247644 13078 tablet_bootstrap.cc:492] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:44.248093 13083 ts_tablet_manager.cc:1434] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.053s	user 0.010s	sys 0.000s
I20260709 12:27:44.249362 13083 tablet_bootstrap.cc:492] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:44.254284 13078 tablet_bootstrap.cc:654] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.257623 13080 tablet_bootstrap.cc:654] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.274525 13362 raft_consensus.cc:2749] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.276434 13379 raft_consensus.cc:493] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.277141 13379 raft_consensus.cc:515] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.281227 13080 tablet_bootstrap.cc:492] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:44.281808 13080 ts_tablet_manager.cc:1403] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.041s	user 0.007s	sys 0.011s
I20260709 12:27:44.281694 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4a9064e4143f7a4f1ce3a3a4bb860" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:44.282637 13379 leader_election.cc:290] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:27:44.284433 12923 leader_election.cc:343] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.284642 12805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4a9064e4143f7a4f1ce3a3a4bb860" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:44.284934 13080 raft_consensus.cc:359] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.285457 12805 raft_consensus.cc:2468] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:44.285729 13080 raft_consensus.cc:385] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.286105 13080 raft_consensus.cc:740] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.286890 12924 leader_election.cc:304] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:44.287413 13079 tablet_bootstrap.cc:492] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:44.287786 13379 raft_consensus.cc:2804] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:44.287946 13079 ts_tablet_manager.cc:1403] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.124s	user 0.012s	sys 0.019s
I20260709 12:27:44.288269 13379 raft_consensus.cc:493] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:44.288446 13078 tablet_bootstrap.cc:492] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:44.288801 13379 raft_consensus.cc:3060] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.289088 13078 ts_tablet_manager.cc:1403] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.042s	user 0.016s	sys 0.006s
I20260709 12:27:44.291169 13079 raft_consensus.cc:359] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.291878 13079 raft_consensus.cc:385] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.292160 13079 raft_consensus.cc:740] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.294087 13419 raft_consensus.cc:493] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.292965 13079 consensus_queue.cc:260] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.294886 13419 raft_consensus.cc:515] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.299173 13379 raft_consensus.cc:515] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.300586 13078 raft_consensus.cc:359] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.302672 13078 raft_consensus.cc:385] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.303030 13078 raft_consensus.cc:740] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.303768 13078 consensus_queue.cc:260] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.304833 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca75f6a5016474a844f5532614ebc75" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:44.305755 13080 consensus_queue.cc:260] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.307964 13080 ts_tablet_manager.cc:1434] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.001s
I20260709 12:27:44.309155 13080 tablet_bootstrap.cc:492] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:44.301839 13083 tablet_bootstrap.cc:654] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:44.310740 12996 leader_election.cc:343] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.305007 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4a9064e4143f7a4f1ce3a3a4bb860" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:44.305326 13078 ts_tablet_manager.cc:1434] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.016s	user 0.004s	sys 0.000s
I20260709 12:27:44.314502 13419 leader_election.cc:290] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
W20260709 12:27:44.315938 12923 leader_election.cc:343] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.319101 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca75f6a5016474a844f5532614ebc75" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:44.320619 13078 tablet_bootstrap.cc:492] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
W20260709 12:27:44.327294 12999 leader_election.cc:343] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.328004 12999 leader_election.cc:304] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:44.329799 13224 raft_consensus.cc:2749] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.331177 13079 ts_tablet_manager.cc:1434] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.043s	user 0.008s	sys 0.000s
I20260709 12:27:44.332043 13379 leader_election.cc:290] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:44.333287 13078 tablet_bootstrap.cc:654] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.334043 13079 tablet_bootstrap.cc:492] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:44.336177 13083 tablet_bootstrap.cc:492] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:44.337518 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4a9064e4143f7a4f1ce3a3a4bb860" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:44.339021 13080 tablet_bootstrap.cc:654] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.342983 13083 ts_tablet_manager.cc:1403] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.094s	user 0.013s	sys 0.001s
I20260709 12:27:44.340623 13224 consensus_queue.cc:1048] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:44.349495 13083 raft_consensus.cc:359] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.350571 13083 raft_consensus.cc:385] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.350915 13083 raft_consensus.cc:740] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.352041 13083 consensus_queue.cc:260] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.354863 13083 ts_tablet_manager.cc:1434] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.003s
I20260709 12:27:44.358178 12808 raft_consensus.cc:3060] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.362838 13083 tablet_bootstrap.cc:492] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:44.373967 13078 tablet_bootstrap.cc:492] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:44.374521 13078 ts_tablet_manager.cc:1403] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.054s	user 0.024s	sys 0.009s
W20260709 12:27:44.376333 12996 consensus_peers.cc:597] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:44.378518 13078 raft_consensus.cc:359] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.379283 13078 raft_consensus.cc:385] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.379603 13078 raft_consensus.cc:740] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.379385 13419 raft_consensus.cc:493] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.380896 13419 raft_consensus.cc:515] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.383378 13419 leader_election.cc:290] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:44.384014 12882 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d28ffbbc352420bb35bfab9e1847e67" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:44.385017 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d28ffbbc352420bb35bfab9e1847e67" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:44.385797 12955 raft_consensus.cc:2468] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
W20260709 12:27:44.385891 12996 leader_election.cc:343] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.387339 12999 leader_election.cc:304] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:44.389026 13224 raft_consensus.cc:2804] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:44.389449 13224 raft_consensus.cc:493] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:44.389798 13224 raft_consensus.cc:3060] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.384048 13078 consensus_queue.cc:260] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.392599 13419 raft_consensus.cc:493] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.393339 13419 raft_consensus.cc:515] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.394747 13079 tablet_bootstrap.cc:654] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.400200 13083 tablet_bootstrap.cc:654] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.393929 13078 ts_tablet_manager.cc:1434] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:27:44.416038 13206 raft_consensus.cc:493] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.416806 13206 raft_consensus.cc:515] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.427966 12882 raft_consensus.cc:3060] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.429299 12884 raft_consensus.cc:3060] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.439795 13454 raft_consensus.cc:493] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.439980 13078 tablet_bootstrap.cc:492] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:44.419502 13142 raft_consensus.cc:493] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.440452 13454 raft_consensus.cc:515] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.441277 13142 raft_consensus.cc:515] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.443439 13142 leader_election.cc:290] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:44.445009 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:44.445274 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:44.445955 13206 leader_election.cc:290] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:44.447311 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:44.447512 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:44.447940 12774 leader_election.cc:343] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.448778 13142 raft_consensus.cc:493] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:27:44.449019 12773 leader_election.cc:343] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.449333 13142 raft_consensus.cc:515] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.446288 12956 raft_consensus.cc:2468] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:44.452575 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1144634236d42f5b2d3656f6eb55389" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:44.453938 12776 leader_election.cc:343] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:44.454181 12774 leader_election.cc:343] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.454492 12776 leader_election.cc:304] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:44.455978 12776 leader_election.cc:304] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:44.456720 13346 raft_consensus.cc:2749] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.459328 13142 leader_election.cc:290] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:44.446182 13454 leader_election.cc:290] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:44.460435 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1144634236d42f5b2d3656f6eb55389" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:44.460856 12879 raft_consensus.cc:3060] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.464488 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:44.465325 12773 leader_election.cc:343] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.465911 12773 leader_election.cc:304] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:44.466358 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:44.467176 13142 raft_consensus.cc:2804] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:44.467629 13142 raft_consensus.cc:493] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:44.467949 13142 raft_consensus.cc:3060] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:44.474576 12922 leader_election.cc:343] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:44.470641 13464 log.cc:927] Time spent T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:27:44.476481 13079 tablet_bootstrap.cc:492] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:44.477015 13454 raft_consensus.cc:2749] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.477950 13454 raft_consensus.cc:493] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.477051 13079 ts_tablet_manager.cc:1403] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.143s	user 0.021s	sys 0.008s
I20260709 12:27:44.478463 13454 raft_consensus.cc:515] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.482765 13279 consensus_queue.cc:1048] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.488593 13079 raft_consensus.cc:359] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.490635 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "303cfb4dc35a4ac094447486c4419179" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:44.491006 13224 raft_consensus.cc:515] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.489328 13454 leader_election.cc:290] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
W20260709 12:27:44.494699 12774 leader_election.cc:343] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.493534 13079 raft_consensus.cc:385] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.488611 13080 tablet_bootstrap.cc:492] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:44.495436 13079 raft_consensus.cc:740] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.495779 13080 ts_tablet_manager.cc:1403] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.187s	user 0.021s	sys 0.035s
I20260709 12:27:44.496328 13079 consensus_queue.cc:260] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.501147 12808 raft_consensus.cc:2468] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:44.501878 13419 leader_election.cc:290] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:44.497686 13224 leader_election.cc:290] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:44.504099 13080 raft_consensus.cc:359] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.515003 13080 raft_consensus.cc:385] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.515842 12805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4a9064e4143f7a4f1ce3a3a4bb860" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:44.516752 12805 raft_consensus.cc:2393] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:44.517515 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d28ffbbc352420bb35bfab9e1847e67" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:44.518095 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8bc4a9064e4143f7a4f1ce3a3a4bb860" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:44.518344 12958 raft_consensus.cc:3060] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.519019 12957 raft_consensus.cc:2393] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:44.515362 13080 raft_consensus.cc:740] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.521057 13080 consensus_queue.cc:260] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.505342 13079 ts_tablet_manager.cc:1434] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.001s
I20260709 12:27:44.510630 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "303cfb4dc35a4ac094447486c4419179" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:44.505685 13279 consensus_queue.cc:1048] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:27:44.504074 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d28ffbbc352420bb35bfab9e1847e67" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
W20260709 12:27:44.524072 12776 leader_election.cc:343] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.524633 12776 leader_election.cc:304] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:44.525152 12924 leader_election.cc:304] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe
I20260709 12:27:44.526002 13206 raft_consensus.cc:2804] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:44.526504 13206 raft_consensus.cc:697] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:44.512250 13142 raft_consensus.cc:515] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.523193 13079 tablet_bootstrap.cc:492] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:44.527385 13206 consensus_queue.cc:237] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:44.532238 12996 leader_election.cc:343] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.532594 12810 raft_consensus.cc:2393] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8012bb9a8b274890bcaad0d192cd30d5 in current term 1: Already voted for candidate b543f973837c43b19b7d17feaaca5325 in this term.
I20260709 12:27:44.532599 13424 consensus_queue.cc:1048] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:27:44.535516 12999 leader_election.cc:304] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:44.539358 13080 ts_tablet_manager.cc:1434] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.043s	user 0.005s	sys 0.003s
I20260709 12:27:44.541776 12924 leader_election.cc:304] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:44.554543 13083 tablet_bootstrap.cc:492] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:44.556296 13083 ts_tablet_manager.cc:1403] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.194s	user 0.017s	sys 0.006s
I20260709 12:27:44.529982 13454 raft_consensus.cc:2749] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.555017 13330 raft_consensus.cc:3060] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.543615 13307 raft_consensus.cc:3060] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.561862 13142 leader_election.cc:290] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:44.567209 13080 tablet_bootstrap.cc:492] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:44.592206 13279 consensus_queue.cc:1048] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:27:44.608098 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
W20260709 12:27:44.609836 12773 leader_election.cc:343] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.610116 13079 tablet_bootstrap.cc:654] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.610437 12958 raft_consensus.cc:2468] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:44.590160 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:44.616151 13330 raft_consensus.cc:2749] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.616711 13362 raft_consensus.cc:493] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.622071 13078 tablet_bootstrap.cc:654] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.625383 13083 raft_consensus.cc:359] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.626174 13083 raft_consensus.cc:385] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:44.628782 13362 raft_consensus.cc:515] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.626374 13083 raft_consensus.cc:740] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:44.637329 13083 consensus_queue.cc:260] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.648011 12999 leader_election.cc:304] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:44.649308 13306 raft_consensus.cc:2804] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:44.649825 13306 raft_consensus.cc:697] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:44.650158 13083 ts_tablet_manager.cc:1434] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.093s	user 0.006s	sys 0.004s
I20260709 12:27:44.650750 13306 consensus_queue.cc:237] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:44.652009 13083 tablet_bootstrap.cc:492] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:44.649704 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc164dcd3f514a00a186655a9a7b2215" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:44.654512 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc164dcd3f514a00a186655a9a7b2215" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:44.696826 13423 raft_consensus.cc:493] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.697404 13423 raft_consensus.cc:515] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.699671 13423 leader_election.cc:290] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:44.701270 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0034a6bef1bf4e119ee686581612f8c0" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:44.702013 13029 raft_consensus.cc:2468] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:44.703528 12849 leader_election.cc:304] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:44.704432 12850 leader_election.cc:304] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:44.708724 13473 raft_consensus.cc:493] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.709378 13473 raft_consensus.cc:515] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:44.711321 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0034a6bef1bf4e119ee686581612f8c0" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:44.712093 12808 raft_consensus.cc:2468] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:44.714644 13423 raft_consensus.cc:2804] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:44.715421 13423 raft_consensus.cc:493] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:44.715939 13423 raft_consensus.cc:3060] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.637606 13362 leader_election.cc:290] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:44.744700 13473 leader_election.cc:290] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:44.745967 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9b2c51994d34d189bd05462fa15c84b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:44.747794 12776 leader_election.cc:343] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.765228 13080 tablet_bootstrap.cc:654] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:44.772078 12703 catalog_manager.cc:5697] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.840940 12955 raft_consensus.cc:2468] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:44.847431 12776 leader_election.cc:304] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:44.855645 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9b2c51994d34d189bd05462fa15c84b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:44.800187 13423 raft_consensus.cc:515] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:44.830699 13362 raft_consensus.cc:2749] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:44.861325 13424 raft_consensus.cc:493] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:44.864919 13454 raft_consensus.cc:2804] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:27:44.865605 12851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 148036us
W20260709 12:27:44.867110 12850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 138913us
I20260709 12:27:44.899070 13423 leader_election.cc:290] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:44.885588 13307 raft_consensus.cc:2749] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.915200 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0034a6bef1bf4e119ee686581612f8c0" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:44.915968 13031 raft_consensus.cc:3060] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.893755 13454 raft_consensus.cc:697] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:44.917896 13454 consensus_queue.cc:237] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:44.893585 12773 leader_election.cc:343] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:44.927719 12773 leader_election.cc:304] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:44.928854 13346 raft_consensus.cc:2749] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:44.889430 13424 raft_consensus.cc:515] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:44.940141 13424 leader_election.cc:290] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:27:44.945590 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:44.979713 13078 tablet_bootstrap.cc:492] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:44.988626 13078 ts_tablet_manager.cc:1403] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.549s	user 0.022s	sys 0.042s
I20260709 12:27:44.996819 13031 raft_consensus.cc:2468] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:45.015425 13079 tablet_bootstrap.cc:492] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:45.016352 13079 ts_tablet_manager.cc:1403] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.493s	user 0.016s	sys 0.024s
I20260709 12:27:45.029165 12805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0034a6bef1bf4e119ee686581612f8c0" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:45.030292 12805 raft_consensus.cc:3060] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.032876 13083 tablet_bootstrap.cc:654] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.034950 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:45.036020 12808 raft_consensus.cc:2468] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:45.029208 13079 raft_consensus.cc:359] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:45.047372 13079 raft_consensus.cc:385] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.047768 13079 raft_consensus.cc:740] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.067399 13080 tablet_bootstrap.cc:492] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:45.067938 13080 ts_tablet_manager.cc:1403] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.505s	user 0.015s	sys 0.010s
I20260709 12:27:45.067138 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:45.067938 13078 raft_consensus.cc:359] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:45.069271 13078 raft_consensus.cc:385] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.069665 13078 raft_consensus.cc:740] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.070928 13079 consensus_queue.cc:260] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:45.072054 12776 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 173804us
I20260709 12:27:45.075749 13078 consensus_queue.cc:260] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:45.082571 13031 raft_consensus.cc:2468] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:45.093652 12849 leader_election.cc:304] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:45.094870 13478 raft_consensus.cc:2804] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:45.095474 13478 raft_consensus.cc:697] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:45.096329 13478 consensus_queue.cc:237] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.096362 13330 consensus_queue.cc:1048] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:27:45.102279 13078 ts_tablet_manager.cc:1434] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.111s	user 0.006s	sys 0.004s
I20260709 12:27:45.103925 13079 ts_tablet_manager.cc:1434] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.087s	user 0.010s	sys 0.000s
I20260709 12:27:45.112016 13079 tablet_bootstrap.cc:492] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:45.112116 13482 raft_consensus.cc:493] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.112862 13482 raft_consensus.cc:515] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:45.111907 13080 raft_consensus.cc:359] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.113909 13080 raft_consensus.cc:385] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.114205 13080 raft_consensus.cc:740] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.115479 13078 tablet_bootstrap.cc:492] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:45.115913 13482 leader_election.cc:290] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:45.116591 13080 consensus_queue.cc:260] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.128404 12924 leader_election.cc:304] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:45.145780 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "068448e5fa294d199668e58a170cbee3" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:45.167833 13080 ts_tablet_manager.cc:1434] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.099s	user 0.010s	sys 0.000s
I20260709 12:27:45.183776 13080 tablet_bootstrap.cc:492] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:45.194281 13078 tablet_bootstrap.cc:654] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.167236 13474 raft_consensus.cc:2804] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:45.198225 13474 raft_consensus.cc:493] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.198704 13474 raft_consensus.cc:3060] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.206002 13079 tablet_bootstrap.cc:654] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.209106 13206 consensus_queue.cc:1048] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:45.214869 13142 raft_consensus.cc:493] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.215515 13142 raft_consensus.cc:515] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:45.167955 12805 raft_consensus.cc:2468] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:45.181622 12808 raft_consensus.cc:2468] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:45.247393 12850 leader_election.cc:304] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:45.251089 13482 raft_consensus.cc:2804] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:45.251556 13482 raft_consensus.cc:493] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.251920 13482 raft_consensus.cc:3060] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.254490 13083 tablet_bootstrap.cc:492] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:45.255364 13083 ts_tablet_manager.cc:1403] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.604s	user 0.016s	sys 0.017s
I20260709 12:27:45.259415 13142 leader_election.cc:290] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:45.223846 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "068448e5fa294d199668e58a170cbee3" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:45.271885 12880 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:45.276628 12773 leader_election.cc:343] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:45.289754 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
W20260709 12:27:45.304244 12996 consensus_peers.cc:597] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:45.313694 12776 leader_election.cc:343] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:45.329077 13080 tablet_bootstrap.cc:654] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:45.329277 13501 log.cc:927] Time spent T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f: Append to log took a long time: real 0.065s	user 0.002s	sys 0.000s
I20260709 12:27:45.289670 13083 raft_consensus.cc:359] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:45.337926 13083 raft_consensus.cc:385] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.338275 13083 raft_consensus.cc:740] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.332435 13360 consensus_queue.cc:1048] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:27:45.319097 13474 raft_consensus.cc:515] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:45.314460 12776 leader_election.cc:304] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:45.342454 13454 raft_consensus.cc:2749] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:45.340770 13083 consensus_queue.cc:260] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:45.349714 12956 raft_consensus.cc:2468] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:45.359998 13083 ts_tablet_manager.cc:1434] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.104s	user 0.010s	sys 0.000s
I20260709 12:27:45.361706 13083 tablet_bootstrap.cc:492] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:45.365198 13474 leader_election.cc:290] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:45.365600 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:45.367552 12806 raft_consensus.cc:3060] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.372063 13028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:45.373065 13028 raft_consensus.cc:3060] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.377010 13078 tablet_bootstrap.cc:492] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:45.377667 13078 ts_tablet_manager.cc:1403] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.263s	user 0.010s	sys 0.008s
I20260709 12:27:45.354292 13482 raft_consensus.cc:515] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:45.386281 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "068448e5fa294d199668e58a170cbee3" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:45.387272 12809 raft_consensus.cc:3060] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.390488 12806 raft_consensus.cc:2468] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:45.392303 12924 leader_election.cc:304] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:45.397202 13083 tablet_bootstrap.cc:654] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.314167 12703 catalog_manager.cc:5697] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 1 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:45.380926 13078 raft_consensus.cc:359] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.398828 13495 raft_consensus.cc:2804] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:45.410799 13078 raft_consensus.cc:385] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.416813 13078 raft_consensus.cc:740] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.424616 13078 consensus_queue.cc:260] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.414698 13028 raft_consensus.cc:2468] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:45.427052 13495 raft_consensus.cc:697] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:45.438946 13078 ts_tablet_manager.cc:1434] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.061s	user 0.010s	sys 0.000s
I20260709 12:27:45.440335 13078 tablet_bootstrap.cc:492] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:45.443419 13495 consensus_queue.cc:237] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:45.447896 13078 tablet_bootstrap.cc:654] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.443738 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "068448e5fa294d199668e58a170cbee3" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:45.454245 12956 raft_consensus.cc:3060] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:45.466199 12773 consensus_peers.cc:597] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:45.467785 13454 consensus_queue.cc:1048] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:45.469331 13234 raft_consensus.cc:493] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.469980 13234 raft_consensus.cc:515] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.472591 12809 raft_consensus.cc:2468] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:45.473240 13234 leader_election.cc:290] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:45.473973 12850 leader_election.cc:304] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:45.436270 13482 leader_election.cc:290] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:45.475356 13478 raft_consensus.cc:2804] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:45.476378 13478 raft_consensus.cc:697] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:45.483009 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed10b80d23ae40019ba46ed7087ca05a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:45.483551 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed10b80d23ae40019ba46ed7087ca05a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:45.484038 12959 raft_consensus.cc:2468] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:45.488968 12999 leader_election.cc:304] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:45.490088 13360 raft_consensus.cc:2804] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:45.490521 13360 raft_consensus.cc:493] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.490919 13360 raft_consensus.cc:3060] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:45.491120 12996 leader_election.cc:343] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:45.496583 13079 tablet_bootstrap.cc:492] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:45.477772 13478 consensus_queue.cc:237] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:45.509369 13482 raft_consensus.cc:493] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.510047 13482 raft_consensus.cc:515] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.512063 13079 ts_tablet_manager.cc:1403] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.400s	user 0.041s	sys 0.024s
I20260709 12:27:45.513875 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d51d99d504d708ad4af4d083eba48" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:45.520432 13080 tablet_bootstrap.cc:492] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:45.524047 13080 ts_tablet_manager.cc:1403] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.346s	user 0.033s	sys 0.030s
I20260709 12:27:45.517498 13482 leader_election.cc:290] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:45.514856 12809 raft_consensus.cc:2468] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:45.536319 13079 raft_consensus.cc:359] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.537343 13079 raft_consensus.cc:385] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.537766 13079 raft_consensus.cc:740] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.541596 13079 consensus_queue.cc:260] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.549798 13360 raft_consensus.cc:515] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.551957 12850 leader_election.cc:304] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:45.552071 12704 catalog_manager.cc:5697] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.557420 13028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d51d99d504d708ad4af4d083eba48" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:45.561327 12956 raft_consensus.cc:2468] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:45.564474 13478 raft_consensus.cc:2804] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:45.581312 13080 raft_consensus.cc:359] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.570880 13079 ts_tablet_manager.cc:1434] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.058s	user 0.002s	sys 0.007s
I20260709 12:27:45.582324 13080 raft_consensus.cc:385] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.582732 13080 raft_consensus.cc:740] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.585008 13078 tablet_bootstrap.cc:492] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:45.585543 13078 ts_tablet_manager.cc:1403] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.145s	user 0.033s	sys 0.020s
I20260709 12:27:45.587970 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed10b80d23ae40019ba46ed7087ca05a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:45.589957 12958 raft_consensus.cc:3060] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:45.584838 12849 leader_election.cc:343] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:45.582821 13360 leader_election.cc:290] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:45.582924 13478 raft_consensus.cc:493] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.595161 13478 raft_consensus.cc:3060] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.599234 13079 tablet_bootstrap.cc:492] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:45.599099 13080 consensus_queue.cc:260] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.600997 13083 tablet_bootstrap.cc:492] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:45.601600 13083 ts_tablet_manager.cc:1403] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.240s	user 0.024s	sys 0.014s
I20260709 12:27:45.606649 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed10b80d23ae40019ba46ed7087ca05a" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
W20260709 12:27:45.608068 12996 leader_election.cc:343] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:45.609275 13078 raft_consensus.cc:359] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.610014 13080 ts_tablet_manager.cc:1434] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.082s	user 0.006s	sys 0.002s
I20260709 12:27:45.610047 13078 raft_consensus.cc:385] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.611928 13482 raft_consensus.cc:493] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.612027 12705 catalog_manager.cc:5697] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 1 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.613163 13482 raft_consensus.cc:515] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.614629 13083 raft_consensus.cc:359] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.615485 13083 raft_consensus.cc:385] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.615779 13083 raft_consensus.cc:740] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.616524 13083 consensus_queue.cc:260] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.616952 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca75f6a5016474a844f5532614ebc75" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:45.617647 12954 raft_consensus.cc:2468] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:45.618072 13080 tablet_bootstrap.cc:492] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:45.619179 12851 leader_election.cc:304] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:45.622529 13362 raft_consensus.cc:2804] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:45.612881 13078 raft_consensus.cc:740] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.623731 13078 consensus_queue.cc:260] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.625774 13080 tablet_bootstrap.cc:654] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.625123 12703 catalog_manager.cc:5697] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.626768 13079 tablet_bootstrap.cc:654] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.623044 13362 raft_consensus.cc:493] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:45.627370 13362 raft_consensus.cc:3060] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.631765 13482 leader_election.cc:290] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:45.632614 13478 raft_consensus.cc:515] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.639338 13028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca75f6a5016474a844f5532614ebc75" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:45.640273 13028 raft_consensus.cc:2468] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:45.635133 12958 raft_consensus.cc:2468] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:45.642795 13078 ts_tablet_manager.cc:1434] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.057s	user 0.005s	sys 0.004s
I20260709 12:27:45.643796 12999 leader_election.cc:304] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:45.645092 13307 raft_consensus.cc:2804] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:45.645890 13307 raft_consensus.cc:697] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:45.646510 13078 tablet_bootstrap.cc:492] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:45.646873 13307 consensus_queue.cc:237] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.648677 13083 ts_tablet_manager.cc:1434] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.047s	user 0.007s	sys 0.001s
I20260709 12:27:45.651896 13478 leader_election.cc:290] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:45.653761 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d51d99d504d708ad4af4d083eba48" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:45.654454 12809 raft_consensus.cc:3060] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.656657 13079 tablet_bootstrap.cc:492] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: No bootstrap required, opened a new log
I20260709 12:27:45.657315 13079 ts_tablet_manager.cc:1403] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.073s	user 0.010s	sys 0.009s
I20260709 12:27:45.659080 13078 tablet_bootstrap.cc:654] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.643769 12703 catalog_manager.cc:5697] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } }
I20260709 12:27:45.655357 13362 raft_consensus.cc:515] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:45.667229 13079 raft_consensus.cc:359] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:45.668027 13079 raft_consensus.cc:385] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.668288 13079 raft_consensus.cc:740] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.669019 13079 consensus_queue.cc:260] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:45.670765 13079 ts_tablet_manager.cc:1434] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.001s
I20260709 12:27:45.669026 13028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "423d51d99d504d708ad4af4d083eba48" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:45.675925 13028 raft_consensus.cc:3060] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.678123 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca75f6a5016474a844f5532614ebc75" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:45.678819 12958 raft_consensus.cc:3060] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.678553 13362 leader_election.cc:290] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:45.680311 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ca75f6a5016474a844f5532614ebc75" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:45.681053 13029 raft_consensus.cc:3060] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:45.698982 13083 tablet_bootstrap.cc:492] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:45.712005 13478 consensus_queue.cc:1048] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:45.713068 13454 raft_consensus.cc:493] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:45.713672 13454 raft_consensus.cc:515] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:45.716045 13454 leader_election.cc:290] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:45.734913 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff6968296a8740f4b8c329c728a64b1b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:45.734920 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff6968296a8740f4b8c329c728a64b1b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:45.739010 12958 raft_consensus.cc:2468] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:45.740406 13083 tablet_bootstrap.cc:654] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.740592 12851 leader_election.cc:304] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:45.741679 13478 raft_consensus.cc:2804] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:45.742202 13478 raft_consensus.cc:697] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:45.743120 13478 consensus_queue.cc:237] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
W20260709 12:27:45.742754 12774 leader_election.cc:343] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:45.762358 12776 leader_election.cc:343] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:45.765283 12776 leader_election.cc:304] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:45.768143 13078 tablet_bootstrap.cc:492] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:45.768718 13078 ts_tablet_manager.cc:1403] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.123s	user 0.016s	sys 0.000s
I20260709 12:27:45.767025 13454 raft_consensus.cc:2749] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:45.770741 13362 consensus_queue.cc:1048] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:45.742861 13029 raft_consensus.cc:2468] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:45.796121 13078 raft_consensus.cc:359] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:45.796937 13078 raft_consensus.cc:385] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.797225 13078 raft_consensus.cc:740] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.800454 13028 raft_consensus.cc:2468] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:45.802387 12849 leader_election.cc:304] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:45.803431 13362 raft_consensus.cc:2804] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:45.803936 13362 raft_consensus.cc:697] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:45.804816 13362 consensus_queue.cc:237] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.790997 12809 raft_consensus.cc:2468] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:45.807615 13078 consensus_queue.cc:260] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:45.809409 13078 ts_tablet_manager.cc:1434] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.040s	user 0.008s	sys 0.000s
I20260709 12:27:45.823925 13078 tablet_bootstrap.cc:492] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:45.843847 13080 tablet_bootstrap.cc:492] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:45.847882 13080 ts_tablet_manager.cc:1403] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.236s	user 0.016s	sys 0.003s
I20260709 12:27:45.864248 13080 raft_consensus.cc:359] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:45.871165 13080 raft_consensus.cc:385] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.872924 13080 raft_consensus.cc:740] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.879768 13080 consensus_queue.cc:260] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
W20260709 12:27:45.863108 12849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:32909, user_credentials={real_user=slave}} blocked reactor thread for 51055.5us
I20260709 12:27:45.900262 13080 ts_tablet_manager.cc:1434] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.052s	user 0.008s	sys 0.002s
I20260709 12:27:45.904240 13083 tablet_bootstrap.cc:492] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:45.905601 13078 tablet_bootstrap.cc:654] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:45.911327 13083 ts_tablet_manager.cc:1403] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.213s	user 0.016s	sys 0.001s
I20260709 12:27:45.908028 12703 catalog_manager.cc:5697] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 1 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:45.921774 13279 consensus_queue.cc:1048] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:45.922353 13083 raft_consensus.cc:359] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:45.921983 12706 catalog_manager.cc:5697] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:45.923199 13083 raft_consensus.cc:385] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:45.925449 13083 raft_consensus.cc:740] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:45.926142 13083 consensus_queue.cc:260] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:45.932322 13083 ts_tablet_manager.cc:1434] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.017s	user 0.003s	sys 0.003s
I20260709 12:27:45.933521 13083 tablet_bootstrap.cc:492] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:45.976965 13080 tablet_bootstrap.cc:492] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:45.939674 13478 consensus_queue.cc:1048] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260709 12:27:46.017220 13279 consensus_queue.cc:1048] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:46.036450 13080 tablet_bootstrap.cc:654] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.047173 13083 tablet_bootstrap.cc:654] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.043418 12704 catalog_manager.cc:5697] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 1 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:46.048812 13454 raft_consensus.cc:493] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.092022 13495 raft_consensus.cc:493] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.093573 13495 raft_consensus.cc:515] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.086228 13454 raft_consensus.cc:515] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:46.102459 13078 tablet_bootstrap.cc:492] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:46.102758 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:46.103202 13078 ts_tablet_manager.cc:1403] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.282s	user 0.018s	sys 0.030s
I20260709 12:27:46.103492 12810 raft_consensus.cc:2468] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:46.105170 12924 leader_election.cc:304] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:46.115645 13522 consensus_queue.cc:1048] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:46.120553 12996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 67140.1us
I20260709 12:27:46.124495 13478 raft_consensus.cc:493] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.129145 13529 raft_consensus.cc:2804] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:46.147646 13078 raft_consensus.cc:359] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.151351 13078 raft_consensus.cc:385] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.152869 13078 raft_consensus.cc:740] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.140162 13529 raft_consensus.cc:493] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.156280 13078 consensus_queue.cc:260] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.155402 13529 raft_consensus.cc:3060] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.137177 13478 raft_consensus.cc:515] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
W20260709 12:27:46.181268 12996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 44793.6us
I20260709 12:27:46.185969 13083 tablet_bootstrap.cc:492] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:46.188920 12881 raft_consensus.cc:3060] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.197739 13078 ts_tablet_manager.cc:1434] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.094s	user 0.010s	sys 0.000s
I20260709 12:27:46.199259 13083 ts_tablet_manager.cc:1403] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.254s	user 0.017s	sys 0.009s
I20260709 12:27:46.206993 13538 raft_consensus.cc:493] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.207669 13538 raft_consensus.cc:515] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.208467 13454 leader_election.cc:290] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:46.208901 13478 leader_election.cc:290] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:46.209892 13538 leader_election.cc:290] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:46.210765 13080 tablet_bootstrap.cc:492] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:46.211328 13080 ts_tablet_manager.cc:1403] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.239s	user 0.033s	sys 0.005s
I20260709 12:27:46.211774 13028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30cb0580c4514b41b650a6b2d5089f3a" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:46.212191 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "30cb0580c4514b41b650a6b2d5089f3a" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:46.210990 13529 raft_consensus.cc:515] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.214411 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:46.215270 13531 log.cc:927] Time spent T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:27:46.216209 13529 leader_election.cc:290] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:46.198844 13234 consensus_queue.cc:1048] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
W20260709 12:27:46.220168 12773 leader_election.cc:343] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:46.217783 12850 leader_election.cc:304] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:46.225703 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:46.226362 12806 raft_consensus.cc:3060] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.214707 13080 raft_consensus.cc:359] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.237202 13080 raft_consensus.cc:385] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.237826 13080 raft_consensus.cc:740] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.238989 13080 consensus_queue.cc:260] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.240516 13495 leader_election.cc:290] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:46.252277 13078 tablet_bootstrap.cc:492] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:46.264518 13080 ts_tablet_manager.cc:1434] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.053s	user 0.009s	sys 0.000s
I20260709 12:27:46.266013 13080 tablet_bootstrap.cc:492] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:46.265741 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "303cfb4dc35a4ac094447486c4419179" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:46.274823 13080 tablet_bootstrap.cc:654] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.275822 12806 raft_consensus.cc:2468] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:46.291559 13524 raft_consensus.cc:2749] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:46.312237 12924 leader_election.cc:304] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:46.315282 13482 consensus_queue.cc:1048] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:46.321971 13078 tablet_bootstrap.cc:654] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.332777 13279 raft_consensus.cc:2804] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:46.337126 13522 consensus_queue.cc:1048] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:27:46.341281 13083 raft_consensus.cc:359] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.341851 13083 raft_consensus.cc:385] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.342037 13083 raft_consensus.cc:740] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.343912 13083 consensus_queue.cc:260] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.338390 13279 raft_consensus.cc:697] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:46.346385 13279 consensus_queue.cc:237] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.347128 13083 ts_tablet_manager.cc:1434] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.147s	user 0.006s	sys 0.002s
I20260709 12:27:46.360153 13496 raft_consensus.cc:493] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.360781 13496 raft_consensus.cc:515] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.370673 13083 tablet_bootstrap.cc:492] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:46.370524 12807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "303cfb4dc35a4ac094447486c4419179" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:46.371620 13080 tablet_bootstrap.cc:492] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:46.371742 12807 raft_consensus.cc:2468] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:46.372180 13080 ts_tablet_manager.cc:1403] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.106s	user 0.025s	sys 0.012s
I20260709 12:27:46.377168 13080 raft_consensus.cc:359] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.391772 13080 raft_consensus.cc:385] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.392244 13080 raft_consensus.cc:740] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.397230 13080 consensus_queue.cc:260] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.399350 13552 consensus_queue.cc:1048] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:46.402199 13083 tablet_bootstrap.cc:654] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:46.405920 12996 consensus_peers.cc:597] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:46.394223 12851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 56126.7us
I20260709 12:27:46.395475 13496 leader_election.cc:290] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
W20260709 12:27:46.412952 12849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:32909, user_credentials={real_user=slave}} blocked reactor thread for 73038.7us
I20260709 12:27:46.431595 12998 leader_election.cc:304] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:46.436149 13540 raft_consensus.cc:2804] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:46.436810 13540 raft_consensus.cc:493] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.437260 13540 raft_consensus.cc:3060] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.451927 13080 ts_tablet_manager.cc:1434] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.079s	user 0.018s	sys 0.013s
W20260709 12:27:46.453159 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:46.453171 13080 tablet_bootstrap.cc:492] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:46.456707 12955 raft_consensus.cc:2468] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:46.460973 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf54d438996844a1b6028dabd88f727e" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:46.469851 13083 tablet_bootstrap.cc:492] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:46.470498 13083 ts_tablet_manager.cc:1403] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.101s	user 0.013s	sys 0.008s
I20260709 12:27:46.485311 13478 raft_consensus.cc:493] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.485954 13478 raft_consensus.cc:515] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:46.482090 12999 leader_election.cc:343] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:46.494226 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf54d438996844a1b6028dabd88f727e" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:46.494313 13478 leader_election.cc:290] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
W20260709 12:27:46.497023 12996 leader_election.cc:343] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:46.497491 12996 leader_election.cc:304] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:46.503146 13083 raft_consensus.cc:359] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.504487 13083 raft_consensus.cc:385] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.504750 13083 raft_consensus.cc:740] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.507450 13538 raft_consensus.cc:2749] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:46.507848 13083 consensus_queue.cc:260] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.509593 13083 ts_tablet_manager.cc:1434] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.039s	user 0.006s	sys 0.000s
I20260709 12:27:46.510624 13083 tablet_bootstrap.cc:492] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:46.524801 13083 tablet_bootstrap.cc:654] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.540500 13078 tablet_bootstrap.cc:492] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:46.541044 13078 ts_tablet_manager.cc:1403] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.311s	user 0.021s	sys 0.024s
I20260709 12:27:46.540851 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:46.542930 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:46.548779 13078 raft_consensus.cc:359] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:46.549520 13078 raft_consensus.cc:385] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.549788 13078 raft_consensus.cc:740] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.550704 13078 consensus_queue.cc:260] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:46.582544 13078 ts_tablet_manager.cc:1434] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.041s	user 0.003s	sys 0.006s
I20260709 12:27:46.586334 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:46.588001 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:46.604951 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:46.564523 13540 raft_consensus.cc:515] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:46.629805 12996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 61884.7us
I20260709 12:27:46.647217 13560 consensus_queue.cc:1048] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:27:46.648485 13080 tablet_bootstrap.cc:654] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.653851 12851 leader_election.cc:304] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:46.655514 13560 raft_consensus.cc:2749] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:46.622530 13559 raft_consensus.cc:493] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.658525 13559 raft_consensus.cc:515] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.661031 13559 leader_election.cc:290] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:46.665105 12703 catalog_manager.cc:5697] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:46.665972 12922 leader_election.cc:343] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:46.667147 12922 leader_election.cc:343] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:46.672479 12774 leader_election.cc:343] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:46.673237 12774 leader_election.cc:304] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:46.677197 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d28ffbbc352420bb35bfab9e1847e67" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:46.678206 13540 leader_election.cc:290] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:46.679028 13142 raft_consensus.cc:2749] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:46.682083 13083 tablet_bootstrap.cc:492] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:46.684712 13078 tablet_bootstrap.cc:492] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:46.682682 13083 ts_tablet_manager.cc:1403] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.172s	user 0.020s	sys 0.009s
I20260709 12:27:46.703863 13567 raft_consensus.cc:493] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.707077 13567 raft_consensus.cc:515] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:46.697695 13083 raft_consensus.cc:359] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.709589 13083 raft_consensus.cc:385] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.709929 13083 raft_consensus.cc:740] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.711921 13567 leader_election.cc:290] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:46.713215 13078 tablet_bootstrap.cc:654] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.713459 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1144634236d42f5b2d3656f6eb55389" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:46.715967 12773 leader_election.cc:343] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:46.710670 13083 consensus_queue.cc:260] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.722036 13083 ts_tablet_manager.cc:1434] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.000s
I20260709 12:27:46.723202 13083 tablet_bootstrap.cc:492] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5: Bootstrap starting.
I20260709 12:27:46.723508 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d28ffbbc352420bb35bfab9e1847e67" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:46.726464 13379 raft_consensus.cc:493] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.727330 13379 raft_consensus.cc:515] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:46.728120 12774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:32909, user_credentials={real_user=slave}} blocked reactor thread for 53563.2us
W20260709 12:27:46.729264 12999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 161812us
I20260709 12:27:46.731346 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1144634236d42f5b2d3656f6eb55389" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:46.732085 13031 raft_consensus.cc:2468] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 0.
I20260709 12:27:46.741686 13379 leader_election.cc:290] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:46.738205 12774 leader_election.cc:304] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:46.749173 13142 raft_consensus.cc:2804] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:46.745369 12851 leader_election.cc:304] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:46.749610 13142 raft_consensus.cc:493] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.742985 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9b2c51994d34d189bd05462fa15c84b" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:46.750124 13142 raft_consensus.cc:3060] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.750497 12810 raft_consensus.cc:2468] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:46.751652 13550 raft_consensus.cc:2749] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:46.753983 12924 leader_election.cc:304] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:46.757393 13279 raft_consensus.cc:2804] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:46.757804 13279 raft_consensus.cc:493] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.758143 13279 raft_consensus.cc:3060] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.759585 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9b2c51994d34d189bd05462fa15c84b" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
W20260709 12:27:46.764375 12922 leader_election.cc:343] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:46.764534 13142 raft_consensus.cc:515] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:46.767670 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1144634236d42f5b2d3656f6eb55389" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:46.768325 13031 raft_consensus.cc:3060] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.768738 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1144634236d42f5b2d3656f6eb55389" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:46.769583 13142 leader_election.cc:290] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:46.770315 12773 leader_election.cc:343] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:46.775418 13031 raft_consensus.cc:2468] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:46.776731 13279 raft_consensus.cc:515] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.776813 12774 leader_election.cc:304] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:46.779768 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9b2c51994d34d189bd05462fa15c84b" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:46.780915 12881 raft_consensus.cc:3060] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:46.781152 12922 leader_election.cc:343] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:46.779927 13569 raft_consensus.cc:2804] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:46.783227 13569 raft_consensus.cc:697] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:46.784192 13569 consensus_queue.cc:237] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:46.790208 13080 tablet_bootstrap.cc:492] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:46.790874 13080 ts_tablet_manager.cc:1403] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.338s	user 0.015s	sys 0.004s
I20260709 12:27:46.796593 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "303cfb4dc35a4ac094447486c4419179" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:46.797189 13078 tablet_bootstrap.cc:492] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:46.797405 12956 raft_consensus.cc:3060] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.797794 13078 ts_tablet_manager.cc:1403] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.176s	user 0.012s	sys 0.000s
I20260709 12:27:46.801065 13078 raft_consensus.cc:359] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.801803 13078 raft_consensus.cc:385] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.802105 13078 raft_consensus.cc:740] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.806325 12809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9b2c51994d34d189bd05462fa15c84b" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:46.807075 12809 raft_consensus.cc:3060] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.794420 13080 raft_consensus.cc:359] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.808228 13080 raft_consensus.cc:385] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.808573 13080 raft_consensus.cc:740] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.811318 13078 consensus_queue.cc:260] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.813159 13080 consensus_queue.cc:260] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.815073 13083 tablet_bootstrap.cc:654] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.815727 13080 ts_tablet_manager.cc:1434] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 12:27:46.817077 13080 tablet_bootstrap.cc:492] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:46.821907 13078 ts_tablet_manager.cc:1434] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.000s
I20260709 12:27:46.823362 13078 tablet_bootstrap.cc:492] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Bootstrap starting.
I20260709 12:27:46.825248 12956 raft_consensus.cc:2468] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:46.826635 12999 leader_election.cc:304] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:46.827598 13565 raft_consensus.cc:2804] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:46.828126 13565 raft_consensus.cc:697] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:46.828965 13565 consensus_queue.cc:237] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.837199 12804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "303cfb4dc35a4ac094447486c4419179" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:46.838064 12804 raft_consensus.cc:3060] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.848377 12804 raft_consensus.cc:2468] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:46.854151 13083 tablet_bootstrap.cc:492] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5: No bootstrap required, opened a new log
I20260709 12:27:46.866204 13083 ts_tablet_manager.cc:1403] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5: Time spent bootstrapping tablet: real 0.143s	user 0.006s	sys 0.006s
I20260709 12:27:46.867033 12809 raft_consensus.cc:2468] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:46.878659 12924 leader_election.cc:304] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:46.878669 13544 consensus_queue.cc:1048] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:46.881351 13542 raft_consensus.cc:2804] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:46.881930 13542 raft_consensus.cc:697] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:46.882908 13542 consensus_queue.cc:237] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.884207 13078 tablet_bootstrap.cc:654] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.888415 13080 tablet_bootstrap.cc:654] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:46.888820 13492 raft_consensus.cc:493] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.889500 13492 raft_consensus.cc:515] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.892418 13492 leader_election.cc:290] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:46.893185 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff6968296a8740f4b8c329c728a64b1b" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:46.895237 12806 raft_consensus.cc:2468] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:46.871874 13083 raft_consensus.cc:359] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.896328 13083 raft_consensus.cc:385] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.896488 12998 leader_election.cc:304] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:46.897444 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff6968296a8740f4b8c329c728a64b1b" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:46.897693 13534 raft_consensus.cc:2804] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:46.898171 13534 raft_consensus.cc:493] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:46.898540 13534 raft_consensus.cc:3060] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:46.902499 12922 consensus_peers.cc:597] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:46.905282 13550 raft_consensus.cc:493] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:46.905901 13550 raft_consensus.cc:515] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:46.908969 13550 leader_election.cc:290] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:46.909988 13080 tablet_bootstrap.cc:492] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:46.910408 13028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed10b80d23ae40019ba46ed7087ca05a" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:46.896616 13083 raft_consensus.cc:740] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.913057 13080 ts_tablet_manager.cc:1403] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.094s	user 0.007s	sys 0.003s
I20260709 12:27:46.914124 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed10b80d23ae40019ba46ed7087ca05a" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:46.916772 13083 consensus_queue.cc:260] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.917557 13534 raft_consensus.cc:515] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:46.924451 13083 ts_tablet_manager.cc:1434] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5: Time spent starting tablet: real 0.058s	user 0.006s	sys 0.001s
I20260709 12:27:46.927332 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff6968296a8740f4b8c329c728a64b1b" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:46.928427 12705 catalog_manager.cc:5697] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 0 to 1, leader changed from <none> to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:46.921356 12851 leader_election.cc:304] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:46.923141 12958 raft_consensus.cc:2468] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:46.961427 13550 raft_consensus.cc:2749] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:46.970943 13078 tablet_bootstrap.cc:492] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe: No bootstrap required, opened a new log
I20260709 12:27:46.971633 13078 ts_tablet_manager.cc:1403] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent bootstrapping tablet: real 0.149s	user 0.036s	sys 0.019s
I20260709 12:27:46.783617 13279 leader_election.cc:290] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:46.920233 13080 raft_consensus.cc:359] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.974629 13080 raft_consensus.cc:385] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.975060 13080 raft_consensus.cc:740] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.975857 13080 consensus_queue.cc:260] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:46.975929 13078 raft_consensus.cc:359] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:46.977150 13078 raft_consensus.cc:385] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:46.977573 13078 raft_consensus.cc:740] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Initialized, Role: FOLLOWER
I20260709 12:27:46.978206 13080 ts_tablet_manager.cc:1434] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.065s	user 0.007s	sys 0.000s
I20260709 12:27:46.933496 12807 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff6968296a8740f4b8c329c728a64b1b" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:46.934494 13534 leader_election.cc:290] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:46.979766 12807 raft_consensus.cc:3060] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.987176 12884 raft_consensus.cc:3060] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.993739 13080 tablet_bootstrap.cc:492] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:46.997253 12885 raft_consensus.cc:3060] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:46.978654 13078 consensus_queue.cc:260] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:47.009907 13550 raft_consensus.cc:493] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.010561 13550 raft_consensus.cc:515] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:47.016270 12804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9b2c51994d34d189bd05462fa15c84b" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:47.016208 13550 leader_election.cc:290] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:47.017408 12804 raft_consensus.cc:2393] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
W20260709 12:27:47.019611 12922 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 47524us
I20260709 12:27:47.021154 13078 ts_tablet_manager.cc:1434] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe: Time spent starting tablet: real 0.049s	user 0.008s	sys 0.001s
I20260709 12:27:47.022745 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9b2c51994d34d189bd05462fa15c84b" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:47.022972 12807 raft_consensus.cc:2468] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:47.024333 12998 leader_election.cc:304] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:47.022689 12706 catalog_manager.cc:5697] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:47.028175 13534 raft_consensus.cc:2804] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:47.028762 13534 raft_consensus.cc:697] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:47.029655 13534 consensus_queue.cc:237] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:47.033210 12851 leader_election.cc:304] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:47.034382 13559 raft_consensus.cc:3060] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.039477 13080 tablet_bootstrap.cc:654] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.055792 13559 raft_consensus.cc:2749] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:47.065734 12703 catalog_manager.cc:5697] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:47.072665 13080 tablet_bootstrap.cc:492] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:47.073331 13080 ts_tablet_manager.cc:1403] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.080s	user 0.015s	sys 0.001s
I20260709 12:27:47.085783 13080 raft_consensus.cc:359] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:47.086539 13080 raft_consensus.cc:385] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.086858 13080 raft_consensus.cc:740] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.088403 13080 consensus_queue.cc:260] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:47.090545 13544 raft_consensus.cc:493] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.091355 13544 raft_consensus.cc:515] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:47.094049 13544 leader_election.cc:290] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:47.090688 13080 ts_tablet_manager.cc:1434] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.000s
I20260709 12:27:47.094938 13028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf54d438996844a1b6028dabd88f727e" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:47.095827 13028 raft_consensus.cc:2468] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:47.097605 13080 tablet_bootstrap.cc:492] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:47.100878 12923 leader_election.cc:304] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:47.101231 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf54d438996844a1b6028dabd88f727e" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:47.101912 13542 raft_consensus.cc:2804] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.102108 12884 raft_consensus.cc:2468] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 0.
I20260709 12:27:47.102348 13542 raft_consensus.cc:493] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.102844 13542 raft_consensus.cc:3060] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.113175 13542 raft_consensus.cc:515] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:47.115409 13542 leader_election.cc:290] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:47.116665 13080 tablet_bootstrap.cc:654] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.116930 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf54d438996844a1b6028dabd88f727e" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:47.117642 12884 raft_consensus.cc:3060] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.117964 13028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bf54d438996844a1b6028dabd88f727e" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:47.123719 13028 raft_consensus.cc:3060] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.128554 12884 raft_consensus.cc:2468] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:47.129974 12922 leader_election.cc:304] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:47.130985 13542 raft_consensus.cc:2804] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:47.131897 13542 raft_consensus.cc:697] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:47.115895 12704 catalog_manager.cc:5697] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:47.132787 13542 consensus_queue.cc:237] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:47.152029 13080 tablet_bootstrap.cc:492] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:47.152647 13080 ts_tablet_manager.cc:1403] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.055s	user 0.014s	sys 0.016s
I20260709 12:27:47.156091 13080 raft_consensus.cc:359] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:47.156831 13080 raft_consensus.cc:385] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.157109 13080 raft_consensus.cc:740] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.157917 13080 consensus_queue.cc:260] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:47.158852 13028 raft_consensus.cc:2468] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:47.160789 13080 ts_tablet_manager.cc:1434] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 12:27:47.162062 13080 tablet_bootstrap.cc:492] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:47.165449 12706 catalog_manager.cc:5697] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:47.169209 13080 tablet_bootstrap.cc:654] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.182286 13080 tablet_bootstrap.cc:492] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:47.182858 13080 ts_tablet_manager.cc:1403] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.001s
I20260709 12:27:47.185509 13080 raft_consensus.cc:359] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:47.186031 13080 raft_consensus.cc:385] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.186307 13080 raft_consensus.cc:740] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.186987 13080 consensus_queue.cc:260] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:47.189060 13080 ts_tablet_manager.cc:1434] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:27:47.190146 13080 tablet_bootstrap.cc:492] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f: Bootstrap starting.
I20260709 12:27:47.195384 13080 tablet_bootstrap.cc:654] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.200011 13080 tablet_bootstrap.cc:492] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f: No bootstrap required, opened a new log
I20260709 12:27:47.200507 13080 ts_tablet_manager.cc:1403] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f: Time spent bootstrapping tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:27:47.202986 13080 raft_consensus.cc:359] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:47.203441 13080 raft_consensus.cc:385] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.203675 13080 raft_consensus.cc:740] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.204286 13080 consensus_queue.cc:260] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:47.206704 13080 ts_tablet_manager.cc:1434] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:27:47.224985 12884 raft_consensus.cc:3060] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.237354 13142 consensus_queue.cc:1048] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:47.250692 13568 consensus_queue.cc:1048] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:47.285945 13279 consensus_queue.cc:1048] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:47.299636 13495 consensus_queue.cc:1048] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:47.359902 13559 raft_consensus.cc:493] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.360641 13559 raft_consensus.cc:515] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:47.441008 13559 leader_election.cc:290] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:47.452478 13533 raft_consensus.cc:493] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.453378 13533 raft_consensus.cc:515] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:47.474129 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:47.475227 13031 raft_consensus.cc:2468] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 0.
I20260709 12:27:47.511345 13533 leader_election.cc:290] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:47.530774 13522 raft_consensus.cc:493] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.531677 13522 raft_consensus.cc:515] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:47.535476 13522 leader_election.cc:290] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
W20260709 12:27:47.539397 12849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:32909, user_credentials={real_user=slave}} blocked reactor thread for 78135us
I20260709 12:27:47.576767 12885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:47.577699 12885 raft_consensus.cc:2468] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
I20260709 12:27:47.669384 12885 raft_consensus.cc:3060] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.050699 12808 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:48.051620 12808 raft_consensus.cc:2468] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 0.
W20260709 12:27:48.214202 12885 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.537s	user 0.013s	sys 0.000s
W20260709 12:27:48.221233 12885 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.014s	sys 0.000s
I20260709 12:27:48.218406 13601 consensus_queue.cc:1048] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.217056 12998 leader_election.cc:304] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:48.226619 13600 raft_consensus.cc:2804] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.227099 13600 raft_consensus.cc:493] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.227427 13600 raft_consensus.cc:3060] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.240681 13598 consensus_queue.cc:1048] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.243943 13600 raft_consensus.cc:515] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:48.250387 13600 leader_election.cc:290] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:48.252305 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:48.253288 12810 raft_consensus.cc:3060] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.260071 12849 leader_election.cc:304] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:48.269650 12810 raft_consensus.cc:2468] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
I20260709 12:27:48.273138 13522 raft_consensus.cc:2804] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.274047 13522 raft_consensus.cc:493] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.274555 13522 raft_consensus.cc:3060] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.278417 12998 leader_election.cc:304] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:48.283864 13534 raft_consensus.cc:2804] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.285312 13534 raft_consensus.cc:697] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:48.286141 13534 consensus_queue.cc:237] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:48.323511 13605 raft_consensus.cc:493] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8012bb9a8b274890bcaad0d192cd30d5)
I20260709 12:27:48.324177 13605 raft_consensus.cc:515] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:48.326340 13605 leader_election.cc:290] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:48.327561 13522 raft_consensus.cc:515] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:48.332068 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acdd78885d1044a3a523c14603afcb9a" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:48.335858 12850 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.749s	user 0.005s	sys 0.009s
W20260709 12:27:48.336472 12850 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.006s	sys 0.010s
I20260709 12:27:48.348670 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:48.335773 13522 leader_election.cc:290] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:48.357769 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:48.360939 13609 raft_consensus.cc:493] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8012bb9a8b274890bcaad0d192cd30d5)
I20260709 12:27:48.361696 13609 raft_consensus.cc:515] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:48.364197 13609 leader_election.cc:290] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:48.368252 12955 raft_consensus.cc:3060] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.381125 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:48.391710 13544 consensus_queue.cc:1048] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.398828 13607 raft_consensus.cc:493] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8012bb9a8b274890bcaad0d192cd30d5)
I20260709 12:27:48.399662 13607 raft_consensus.cc:515] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:48.400286 13602 raft_consensus.cc:493] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8a995dc08b2a4497b71db084a2641e6f)
I20260709 12:27:48.401001 13602 raft_consensus.cc:515] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:48.404516 13602 leader_election.cc:290] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:48.404748 13607 leader_election.cc:290] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:48.411190 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735482fa1588451b93a1735236b1859b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:48.413404 13602 raft_consensus.cc:493] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8012bb9a8b274890bcaad0d192cd30d5)
I20260709 12:27:48.414072 13602 raft_consensus.cc:515] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:48.416954 13602 leader_election.cc:290] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:48.418054 12806 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:48.420305 12810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:48.421090 13609 raft_consensus.cc:493] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8012bb9a8b274890bcaad0d192cd30d5)
I20260709 12:27:48.421083 12810 raft_consensus.cc:2393] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
I20260709 12:27:48.421803 13609 raft_consensus.cc:515] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:27:48.422938 13234 raft_consensus.cc:493] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8012bb9a8b274890bcaad0d192cd30d5)
I20260709 12:27:48.423470 13234 raft_consensus.cc:515] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:48.425629 13234 leader_election.cc:290] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:27:48.433423 12756 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:48.438047 13033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:48.440481 13028 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:48.441305 13028 raft_consensus.cc:2468] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 1.
I20260709 12:27:48.443116 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "acdd78885d1044a3a523c14603afcb9a" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:48.443193 12877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:48.444175 12877 raft_consensus.cc:2393] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 1: Already voted for candidate 8a995dc08b2a4497b71db084a2641e6f in this term.
I20260709 12:27:48.444657 12801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325"
I20260709 12:27:48.445226 13601 raft_consensus.cc:493] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8012bb9a8b274890bcaad0d192cd30d5)
I20260709 12:27:48.445567 12801 raft_consensus.cc:2393] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 1: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
I20260709 12:27:48.445784 13601 raft_consensus.cc:515] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:48.447314 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "735482fa1588451b93a1735236b1859b" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:48.448143 13601 leader_election.cc:290] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:48.449842 12774 leader_election.cc:304] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:27:48.451162 13602 raft_consensus.cc:2804] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:48.451562 13602 raft_consensus.cc:493] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8012bb9a8b274890bcaad0d192cd30d5)
I20260709 12:27:48.451912 13602 raft_consensus.cc:3060] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.452641 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:48.457058 12773 leader_election.cc:304] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:48.458547 13607 raft_consensus.cc:2749] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:48.461876 13609 leader_election.cc:290] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:48.461658 12850 leader_election.cc:304] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:48.469403 13522 raft_consensus.cc:2749] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.458319 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:48.471210 12801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:48.466274 12877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81eacfb947b741a2a11b5b645c586003" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:48.467190 12851 leader_election.cc:304] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:48.473760 13522 raft_consensus.cc:2749] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.477065 12802 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b543f973837c43b19b7d17feaaca5325" is_pre_election: true
I20260709 12:27:48.479817 13602 raft_consensus.cc:515] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:48.482547 13602 leader_election.cc:290] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:48.483143 13031 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:48.483796 13031 raft_consensus.cc:3060] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.495654 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:48.496142 13279 consensus_queue.cc:1048] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.502151 13234 consensus_queue.cc:1048] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.502269 13534 consensus_queue.cc:1048] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.504920 12999 leader_election.cc:304] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:48.505832 13565 raft_consensus.cc:2749] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:48.525738 12704 catalog_manager.cc:5697] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 1 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.527882 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:48.525863 12776 leader_election.cc:304] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:48.531905 13607 raft_consensus.cc:2749] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:48.535254 12776 leader_election.cc:304] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:48.536074 13602 raft_consensus.cc:2749] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:48.512650 13031 raft_consensus.cc:2468] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:27:48.547271 13028 raft_consensus.cc:1240] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Rejecting Update request from peer 8012bb9a8b274890bcaad0d192cd30d5 for earlier term 1. Current term is 2. Ops: []
I20260709 12:27:48.555114 13615 consensus_queue.cc:1059] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:48.556306 13615 raft_consensus.cc:3055] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:48.556766 13615 raft_consensus.cc:740] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:48.557670 13615 consensus_queue.cc:260] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:48.561779 12774 leader_election.cc:304] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:48.567126 13602 raft_consensus.cc:2804] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:48.567678 13602 raft_consensus.cc:697] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:27:48.568511 13602 consensus_queue.cc:237] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:48.575137 12924 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 75545.8us
I20260709 12:27:48.558772 13615 raft_consensus.cc:3060] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.588688 12955 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "508c4d341f444ad2867dfb09b3b2a125" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:48.590109 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:48.591820 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9b4619b6db04b159a1d568ba006bdb4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:48.595738 12999 leader_election.cc:304] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:48.598950 12999 leader_election.cc:304] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:27:48.601886 13234 raft_consensus.cc:2749] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:48.603229 13601 raft_consensus.cc:2749] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:48.628687 12956 raft_consensus.cc:2468] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 2.
W20260709 12:27:48.662911 12774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:32909, user_credentials={real_user=slave}} blocked reactor thread for 55070.6us
W20260709 12:27:48.681061 12999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 65446.1us
I20260709 12:27:48.685566 12999 leader_election.cc:304] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:48.689579 13234 raft_consensus.cc:2764] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260709 12:27:48.773547 12706 catalog_manager.cc:5697] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:48.780356 13601 consensus_queue.cc:1048] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.878361 13601 consensus_queue.cc:1048] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:48.896876 12850 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 57135.8us
W20260709 12:27:49.028769 12831 debug-util.cc:398] Leaking SignalData structure 0x7b08002e9060 after lost signal to thread 564
W20260709 12:27:49.030246 12831 debug-util.cc:398] Leaking SignalData structure 0x7b080031e280 after lost signal to thread 12742
W20260709 12:27:49.032264 12831 debug-util.cc:398] Leaking SignalData structure 0x7b0800385da0 after lost signal to thread 12834
W20260709 12:27:49.034013 12831 debug-util.cc:398] Leaking SignalData structure 0x7b08003c7f40 after lost signal to thread 12909
W20260709 12:27:49.035670 12831 debug-util.cc:398] Leaking SignalData structure 0x7b08003f2560 after lost signal to thread 12983
W20260709 12:27:49.037261 12831 debug-util.cc:398] Leaking SignalData structure 0x7b0800226ba0 after lost signal to thread 13057
I20260709 12:27:49.059263 12957 raft_consensus.cc:1275] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.061163 13607 consensus_queue.cc:1048] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:49.083760 13033 raft_consensus.cc:1275] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.085444 13607 consensus_queue.cc:1048] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:49.348656 12774 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:32909, user_credentials={real_user=slave}} blocked reactor thread for 45572.5us
I20260709 12:27:50.119015   561 tablet_server.cc:179] TabletServer@127.0.140.65:0 shutting down...
W20260709 12:27:50.166654 12924 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:27:50.237114 12924 consensus_peers.cc:597] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.242692 12924 consensus_peers.cc:597] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.255704 12924 consensus_peers.cc:597] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.263166 12999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 51523.8us
W20260709 12:27:50.310012 12924 consensus_peers.cc:597] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.346439 12924 consensus_peers.cc:597] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.382294 12998 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:40461, user_credentials={real_user=slave}} blocked reactor thread for 47015.5us
W20260709 12:27:50.424623 12924 consensus_peers.cc:597] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.425794 12924 consensus_peers.cc:597] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.438026 12924 consensus_peers.cc:597] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.438649 12924 consensus_peers.cc:597] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.441586 12924 consensus_peers.cc:597] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.452752 12998 consensus_peers.cc:597] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.456164 12924 consensus_peers.cc:597] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.460238 12998 consensus_peers.cc:597] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.462132 12998 consensus_peers.cc:597] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.462109 12849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.68:32909, user_credentials={real_user=slave}} blocked reactor thread for 67481.9us
W20260709 12:27:50.463133 12998 consensus_peers.cc:597] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.461392 12850 consensus_peers.cc:597] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.465271 12998 consensus_peers.cc:597] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.467973 12998 consensus_peers.cc:597] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.472039 12998 consensus_peers.cc:597] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.485643 12850 consensus_peers.cc:597] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.491578 12924 consensus_peers.cc:597] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.492559 12924 consensus_peers.cc:597] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.514418 12850 consensus_peers.cc:597] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.513661 12998 consensus_peers.cc:597] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.515231 12850 consensus_peers.cc:597] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.519323 12998 consensus_peers.cc:597] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.520627 12998 consensus_peers.cc:597] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.539482 12998 consensus_peers.cc:597] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.542994 12924 consensus_peers.cc:597] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.543457 12998 consensus_peers.cc:597] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.546084 12998 consensus_peers.cc:597] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.552690 12998 consensus_peers.cc:597] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.598582 12998 consensus_peers.cc:597] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:50.635902 12998 consensus_peers.cc:597] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:50.652230   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:50.657745   561 tablet_replica.cc:333] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.664604   561 raft_consensus.cc:2243] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.665315   561 raft_consensus.cc:2272] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.668203   561 tablet_replica.cc:333] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.675163   561 raft_consensus.cc:2243] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:50.677134   561 raft_consensus.cc:2272] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.683144   561 tablet_replica.cc:333] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.684454   561 raft_consensus.cc:2243] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.685055   561 raft_consensus.cc:2272] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.687978   561 tablet_replica.cc:333] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.688884   561 raft_consensus.cc:2243] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:50.689957   561 raft_consensus.cc:2272] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.692842   561 tablet_replica.cc:333] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.693980   561 raft_consensus.cc:2243] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.694550   561 raft_consensus.cc:2272] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.697408   561 tablet_replica.cc:333] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.698272   561 raft_consensus.cc:2243] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.698841   561 raft_consensus.cc:2272] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.701526   561 tablet_replica.cc:333] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.702433   561 raft_consensus.cc:2243] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.703118   561 raft_consensus.cc:2272] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.706842   561 tablet_replica.cc:333] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.708163   561 raft_consensus.cc:2243] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.708715   561 raft_consensus.cc:2272] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.711624   561 tablet_replica.cc:333] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.712837   561 raft_consensus.cc:2243] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:50.713896   561 raft_consensus.cc:2272] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.718899   561 tablet_replica.cc:333] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.719707   561 raft_consensus.cc:2243] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.720216   561 raft_consensus.cc:2272] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.728915   561 tablet_replica.cc:333] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.729889   561 raft_consensus.cc:2243] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.730557   561 raft_consensus.cc:2272] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.733167   561 tablet_replica.cc:333] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.733974   561 raft_consensus.cc:2243] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.734508   561 raft_consensus.cc:2272] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.737260   561 tablet_replica.cc:333] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.738255   561 raft_consensus.cc:2243] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.738940   561 raft_consensus.cc:2272] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.741886   561 tablet_replica.cc:333] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.742801   561 raft_consensus.cc:2243] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.743387   561 raft_consensus.cc:2272] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.746465   561 tablet_replica.cc:333] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.747349   561 raft_consensus.cc:2243] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.747917   561 raft_consensus.cc:2272] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.750834   561 tablet_replica.cc:333] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.752637   561 raft_consensus.cc:2243] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.753258   561 raft_consensus.cc:2272] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.756987   561 tablet_replica.cc:333] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.757835   561 raft_consensus.cc:2243] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.758355   561 raft_consensus.cc:2272] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.761736   561 tablet_replica.cc:333] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.762533   561 raft_consensus.cc:2243] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:50.763671   561 raft_consensus.cc:2272] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.766521   561 tablet_replica.cc:333] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.767457   561 raft_consensus.cc:2243] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:50.768743   561 raft_consensus.cc:2272] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:50.769176 12739 debug-util.cc:398] Leaking SignalData structure 0x7b0800048fa0 after lost signal to thread 564
W20260709 12:27:50.770426 12739 debug-util.cc:398] Leaking SignalData structure 0x7b0800083de0 after lost signal to thread 12742
I20260709 12:27:50.771636   561 tablet_replica.cc:333] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
W20260709 12:27:50.771898 12739 debug-util.cc:398] Leaking SignalData structure 0x7b0800323f40 after lost signal to thread 12834
I20260709 12:27:50.772547   561 raft_consensus.cc:2243] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:27:50.773167 12739 debug-util.cc:398] Leaking SignalData structure 0x7b080022e940 after lost signal to thread 12909
I20260709 12:27:50.773644   561 raft_consensus.cc:2272] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:50.774454 12739 debug-util.cc:398] Leaking SignalData structure 0x7b080039f4e0 after lost signal to thread 12983
W20260709 12:27:50.775825 12739 debug-util.cc:398] Leaking SignalData structure 0x7b08002b5ac0 after lost signal to thread 13057
I20260709 12:27:50.776962   561 tablet_replica.cc:333] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.777849   561 raft_consensus.cc:2243] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.778381   561 raft_consensus.cc:2272] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.781003   561 tablet_replica.cc:333] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.781970   561 raft_consensus.cc:2243] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.782599   561 raft_consensus.cc:2272] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.785856   561 tablet_replica.cc:333] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.786881   561 raft_consensus.cc:2243] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.787447   561 raft_consensus.cc:2272] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.807044 13607 raft_consensus.cc:493] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 18cf4829b3bc4fafb3ef7f1b98f853fe)
I20260709 12:27:50.807606 13607 raft_consensus.cc:515] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:50.809687 13607 leader_election.cc:290] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:50.812170   561 tablet_replica.cc:333] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.811643 13648 raft_consensus.cc:493] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 18cf4829b3bc4fafb3ef7f1b98f853fe)
I20260709 12:27:50.812844 13030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:50.813056 13648 raft_consensus.cc:515] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:50.813501   561 raft_consensus.cc:2243] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:50.814993   561 raft_consensus.cc:2272] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.815477 13648 leader_election.cc:290] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:27:50.817830 13030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:50.818501   561 tablet_replica.cc:333] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.819676   561 raft_consensus.cc:2243] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:50.819542 12953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:50.821147   561 raft_consensus.cc:2272] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.821456 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:50.824761   561 tablet_replica.cc:333] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.825546 12776 leader_election.cc:304] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:50.825917   561 raft_consensus.cc:2243] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.826676   561 raft_consensus.cc:2272] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.826941 12776 leader_election.cc:304] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b543f973837c43b19b7d17feaaca5325; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5
I20260709 12:27:50.830325 13607 raft_consensus.cc:2749] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:50.831089 13607 raft_consensus.cc:2749] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:50.838675   561 tablet_replica.cc:333] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.839848   561 raft_consensus.cc:2243] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.840543   561 raft_consensus.cc:2272] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.845288   561 tablet_replica.cc:333] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.846247   561 raft_consensus.cc:2243] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.846886   561 raft_consensus.cc:2272] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.850463   561 tablet_replica.cc:333] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.851423   561 raft_consensus.cc:2243] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:50.852752   561 raft_consensus.cc:2272] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.856256   561 tablet_replica.cc:333] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.857199   561 raft_consensus.cc:2243] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.864301   561 raft_consensus.cc:2272] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.867703   561 tablet_replica.cc:333] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.868660   561 raft_consensus.cc:2243] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.869253   561 raft_consensus.cc:2272] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.872102   561 tablet_replica.cc:333] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.873092   561 raft_consensus.cc:2243] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.879053   561 raft_consensus.cc:2272] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.882433   561 tablet_replica.cc:333] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.884100   561 raft_consensus.cc:2243] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.884722   561 raft_consensus.cc:2272] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.887681   561 tablet_replica.cc:333] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.910035   561 raft_consensus.cc:2243] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.914361   561 raft_consensus.cc:2272] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.939023   561 tablet_replica.cc:333] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.940109   561 raft_consensus.cc:2243] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.940785   561 raft_consensus.cc:2272] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.952908   561 tablet_replica.cc:333] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.955679   561 raft_consensus.cc:2243] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.956480   561 raft_consensus.cc:2272] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.966516   561 tablet_replica.cc:333] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.967518   561 raft_consensus.cc:2243] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.968206   561 raft_consensus.cc:2272] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.972713   561 tablet_replica.cc:333] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.973644   561 raft_consensus.cc:2243] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.974264   561 raft_consensus.cc:2272] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.977684   561 tablet_replica.cc:333] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.978554   561 raft_consensus.cc:2243] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.979205   561 raft_consensus.cc:2272] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.982894   561 tablet_replica.cc:333] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.983803   561 raft_consensus.cc:2243] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.984431   561 raft_consensus.cc:2272] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.987790   561 tablet_replica.cc:333] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.988616   561 raft_consensus.cc:2243] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:50.989974   561 raft_consensus.cc:2272] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.993162   561 tablet_replica.cc:333] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.993974   561 raft_consensus.cc:2243] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:50.994597   561 raft_consensus.cc:2272] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:50.998746   561 tablet_replica.cc:333] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:50.999786   561 raft_consensus.cc:2243] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:51.001014   561 raft_consensus.cc:2272] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.020265   561 tablet_replica.cc:333] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:51.021117   561 raft_consensus.cc:2243] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.023622   561 raft_consensus.cc:2272] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.027022   561 tablet_replica.cc:333] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: stopping tablet replica
I20260709 12:27:51.027777   561 raft_consensus.cc:2243] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:51.028594   561 raft_consensus.cc:2272] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:51.711061 13642 raft_consensus.cc:493] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:51.711705 13642 raft_consensus.cc:515] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:51.714169 13642 leader_election.cc:290] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:51.715584 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:51.734243 12924 leader_election.cc:336] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:51.734848 12924 leader_election.cc:304] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
W20260709 12:27:51.736414 12851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 49186.8us
I20260709 12:27:51.737658 13642 raft_consensus.cc:2749] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:27:51.776748 13662 raft_consensus.cc:493] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:51.777311 13662 raft_consensus.cc:515] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
W20260709 12:27:51.793295 12998 leader_election.cc:336] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:51.795660 13662 leader_election.cc:290] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:51.796840 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1144634236d42f5b2d3656f6eb55389" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:51.798372 12996 leader_election.cc:304] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:51.799417 13609 raft_consensus.cc:2749] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:51.989264 13674 raft_consensus.cc:493] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:51.990018 13674 raft_consensus.cc:515] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:51.990264 13642 raft_consensus.cc:493] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:51.991099 13642 raft_consensus.cc:515] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:51.994041 13642 leader_election.cc:290] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:51.995570 13674 leader_election.cc:290] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:51.996414 13674 raft_consensus.cc:493] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:51.996734 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:51.996986 13674 raft_consensus.cc:515] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:51.997588 12881 raft_consensus.cc:2468] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:51.999297 12922 leader_election.cc:304] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:52.002995 13642 raft_consensus.cc:2804] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:52.007278 13642 raft_consensus.cc:493] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.007727 13642 raft_consensus.cc:3060] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.013952 13660 raft_consensus.cc:493] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.014838 13660 raft_consensus.cc:515] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.017601 13660 leader_election.cc:290] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.024353 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca89f605f7bb4bd9acc811382ac82591" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:52.039820 13678 raft_consensus.cc:493] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.040627 13678 raft_consensus.cc:515] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.043933 13678 leader_election.cc:290] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.050330 13643 raft_consensus.cc:493] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.051355 13643 raft_consensus.cc:515] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.056209 13678 raft_consensus.cc:493] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.056919 13678 raft_consensus.cc:515] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:27:52.071928 12998 leader_election.cc:336] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.072680 12998 leader_election.cc:304] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.075013 13609 raft_consensus.cc:2749] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:52.076932 12958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:52.062371 13643 leader_election.cc:290] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.079907 12959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:52.062280 13678 leader_election.cc:290] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.096788 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afd1573aba344096affc47d60971991c" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:52.101114 12953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c1da90a8da49f6be72a8361eead7df" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:52.102053 12953 raft_consensus.cc:2468] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 1.
W20260709 12:27:52.103497 12924 leader_election.cc:336] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.108626 13652 raft_consensus.cc:493] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.110322 13652 raft_consensus.cc:515] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.079500 13676 raft_consensus.cc:493] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.115100 13662 raft_consensus.cc:493] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.080497 13677 raft_consensus.cc:493] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.122299 13662 raft_consensus.cc:515] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.088503 13682 raft_consensus.cc:493] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.122941 13677 raft_consensus.cc:515] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.123320 13682 raft_consensus.cc:515] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.124734 13662 leader_election.cc:290] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.125355 13677 leader_election.cc:290] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.126111 13683 raft_consensus.cc:493] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.126750 13683 raft_consensus.cc:515] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.117170 12999 leader_election.cc:304] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:52.121286 13676 raft_consensus.cc:515] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.128826 13662 raft_consensus.cc:2804] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:52.129240 13662 raft_consensus.cc:493] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.129601 13662 raft_consensus.cc:3060] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.129683 13683 leader_election.cc:290] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.129968 13678 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.130577 13678 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.131461 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1144634236d42f5b2d3656f6eb55389" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:27:52.131481 12998 leader_election.cc:336] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.132213 13032 raft_consensus.cc:2468] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:52.133795 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:52.133997 12849 leader_election.cc:304] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:52.134438 12884 raft_consensus.cc:2468] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
W20260709 12:27:52.135646 12998 leader_election.cc:336] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.136062 13678 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.138710 13643 raft_consensus.cc:493] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.136126 12998 leader_election.cc:304] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.142521 13684 raft_consensus.cc:2804] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:52.142961 13684 raft_consensus.cc:493] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.143546 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224f359a7c884a7890b05f3098408d31" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:52.144295 13674 leader_election.cc:290] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:52.125357 13682 leader_election.cc:290] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.148387 13676 leader_election.cc:290] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:52.139403 13643 raft_consensus.cc:515] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.149776 13660 raft_consensus.cc:2749] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:52.154106 13662 raft_consensus.cc:515] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.154572 12953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:52.173528 13676 raft_consensus.cc:493] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.174234 12953 raft_consensus.cc:2468] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 2.
I20260709 12:27:52.156335 13652 leader_election.cc:290] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:27:52.175343 12850 leader_election.cc:336] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.141989 13642 raft_consensus.cc:515] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:52.175992 12850 leader_election.cc:336] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.176436 12850 leader_election.cc:304] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.173528 13686 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.159197 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:52.143302 13684 raft_consensus.cc:3060] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.182871 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c1da90a8da49f6be72a8361eead7df" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:52.168671 13682 raft_consensus.cc:493] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.184199 13029 raft_consensus.cc:2393] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8012bb9a8b274890bcaad0d192cd30d5 in current term 2: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
W20260709 12:27:52.169718 12998 leader_election.cc:336] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.170156 12922 leader_election.cc:304] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:52.170921 13677 raft_consensus.cc:493] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.185668 13685 raft_consensus.cc:2804] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:52.186160 13685 raft_consensus.cc:493] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.185798 13677 raft_consensus.cc:515] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.186601 13685 raft_consensus.cc:3060] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:52.187866 12998 leader_election.cc:336] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.171872 13030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca89f605f7bb4bd9acc811382ac82591" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:52.176614 12958 raft_consensus.cc:2393] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 2: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:52.177526 13642 leader_election.cc:290] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:27:52.158329   561 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 12831)
I20260709 12:27:52.178169 13681 raft_consensus.cc:2749] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:52.159183 13643 leader_election.cc:290] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.177604 13662 leader_election.cc:290] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.184298 13682 raft_consensus.cc:515] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.179987 13686 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.195792 13682 leader_election.cc:290] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.196532 12884 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 2.
I20260709 12:27:52.196683 13686 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.174309 13676 raft_consensus.cc:515] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.190569 13030 raft_consensus.cc:2468] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:52.188373 13677 leader_election.cc:290] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.188396 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a0e539dd685147a899e29016abb08e1b" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:52.198770 13684 raft_consensus.cc:515] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.199277 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:52.202534 12849 leader_election.cc:304] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:52.212224 13029 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:27:52.200162 13676 leader_election.cc:290] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:27:52.213430 13684 leader_election.cc:290] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.213728 12849 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:52.214071 13676 raft_consensus.cc:2804] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:52.201162 12883 raft_consensus.cc:3060] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:52.203797 12998 leader_election.cc:336] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.215162 13684 raft_consensus.cc:2804] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:27:52.215826 13684 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.216347 13684 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:52.204654 13685 raft_consensus.cc:515] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:52.200220 12850 leader_election.cc:336] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:27:52.220034 12850 leader_election.cc:336] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:27:52.220533 12850 leader_election.cc:336] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.221377 13685 leader_election.cc:290] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.201669 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82ca562394a94f40ae84d9607fb885f4" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:52.222281 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1144634236d42f5b2d3656f6eb55389" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:52.222738 12884 raft_consensus.cc:2468] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:52.223091 13029 raft_consensus.cc:3060] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:52.225140 12850 leader_election.cc:336] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.227993 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:52.228713 12884 raft_consensus.cc:3060] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:52.232405 12850 leader_election.cc:336] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:27:52.233372 12850 leader_election.cc:336] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:27:52.234171 12850 leader_election.cc:336] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.231773 12922 leader_election.cc:304] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:52.214615 13676 raft_consensus.cc:493] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.235579 13685 raft_consensus.cc:2804] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:52.235972 13676 raft_consensus.cc:3060] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.236260 13685 raft_consensus.cc:493] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.236897 13685 raft_consensus.cc:3060] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.202075 12996 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.240217 13678 raft_consensus.cc:2804] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:27:52.240885 13678 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.241884 13678 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:52.246841 13685 raft_consensus.cc:515] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
W20260709 12:27:52.247448 12996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 45470.5us
I20260709 12:27:52.255528 13684 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.263120 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afd1573aba344096affc47d60971991c" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:52.263816 13032 raft_consensus.cc:2468] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:52.264572 12883 raft_consensus.cc:2468] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 2.
I20260709 12:27:52.269376 13684 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 3 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.269479 12879 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82ca562394a94f40ae84d9607fb885f4" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:52.271046 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
W20260709 12:27:52.275987 12850 leader_election.cc:336] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.280040 13685 leader_election.cc:290] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.307022 12879 raft_consensus.cc:3060] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.312188 12922 leader_election.cc:304] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:52.313810 13685 raft_consensus.cc:2804] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:52.296343 12923 leader_election.cc:304] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: 
I20260709 12:27:52.319346 13685 raft_consensus.cc:697] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:52.321291 13682 raft_consensus.cc:2804] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:52.347553 12999 leader_election.cc:304] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.354938 12884 raft_consensus.cc:2468] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 2.
I20260709 12:27:52.354064 12851 leader_election.cc:304] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.356489 13662 raft_consensus.cc:2804] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:27:52.320559 13685 consensus_queue.cc:237] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.357074 13662 raft_consensus.cc:493] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.357683 13684 raft_consensus.cc:2749] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:52.358088 13662 raft_consensus.cc:3060] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:52.362928 13682 raft_consensus.cc:493] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.363607 13682 raft_consensus.cc:3060] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.365239 13675 raft_consensus.cc:493] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.375274 13675 raft_consensus.cc:515] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.377959 13675 leader_election.cc:290] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.366109 12922 leader_election.cc:304] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:52.380007 13029 raft_consensus.cc:2468] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:27:52.382661 12849 leader_election.cc:304] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.386117 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82ca562394a94f40ae84d9607fb885f4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:52.386559 12953 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:52.387140 12956 raft_consensus.cc:2393] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 2: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:52.387703 12953 raft_consensus.cc:2393] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 2: Already voted for candidate 8012bb9a8b274890bcaad0d192cd30d5 in this term.
I20260709 12:27:52.388019 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224f359a7c884a7890b05f3098408d31" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5" is_pre_election: true
I20260709 12:27:52.388705 13684 raft_consensus.cc:2804] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:52.388784 12957 raft_consensus.cc:2468] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 1.
I20260709 12:27:52.390553 13684 raft_consensus.cc:697] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:52.392000 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "09c1da90a8da49f6be72a8361eead7df" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:52.391861 13684 consensus_queue.cc:237] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.392719 12957 raft_consensus.cc:3060] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.392359 13662 raft_consensus.cc:515] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.396144 13682 raft_consensus.cc:515] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.401501 12957 raft_consensus.cc:2468] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 2.
W20260709 12:27:52.406211 12924 leader_election.cc:336] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.413152 13688 raft_consensus.cc:2804] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:52.413287 12924 leader_election.cc:304] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.413852 13688 raft_consensus.cc:697] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
W20260709 12:27:52.414894 12924 leader_election.cc:336] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.415526 13682 leader_election.cc:290] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:27:52.412065 12922 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 46027.9us
I20260709 12:27:52.398115 13676 raft_consensus.cc:515] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.423975 12851 leader_election.cc:304] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.414719 13675 raft_consensus.cc:2749] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:52.424435 13030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca89f605f7bb4bd9acc811382ac82591" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:52.427644 13030 raft_consensus.cc:3060] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:52.429780 12850 leader_election.cc:336] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.410524 13678 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.431293 13032 raft_consensus.cc:2393] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 8a995dc08b2a4497b71db084a2641e6f in current term 3: Already voted for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in this term.
I20260709 12:27:52.433606 12849 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.435628 13686 raft_consensus.cc:2749] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:27:52.436326 13678 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.437331 13688 consensus_queue.cc:237] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.439726 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
W20260709 12:27:52.439805 12998 leader_election.cc:336] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.447336 13676 leader_election.cc:290] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.447765 12999 leader_election.cc:304] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.448709 12879 raft_consensus.cc:2468] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 2.
I20260709 12:27:52.449786 13679 raft_consensus.cc:2804] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:52.450423 13679 raft_consensus.cc:697] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
W20260709 12:27:52.446254 12924 leader_election.cc:336] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:27:52.457434 12998 consensus_peers.cc:597] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:52.466570 12703 catalog_manager.cc:5697] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 1 to 2, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 2 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:52.469635 12924 leader_election.cc:336] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.456573 12884 raft_consensus.cc:2393] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in current term 3: Already voted for candidate 8a995dc08b2a4497b71db084a2641e6f in this term.
I20260709 12:27:52.412307 13662 leader_election.cc:290] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:52.472776 13676 raft_consensus.cc:2749] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:27:52.473634 12998 leader_election.cc:336] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:27:52.474144 12851 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 50276.2us
I20260709 12:27:52.451440 13679 consensus_queue.cc:237] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.475749 12851 leader_election.cc:304] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
W20260709 12:27:52.470717 12924 leader_election.cc:336] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:27:52.479799 12924 leader_election.cc:336] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.484313 13676 raft_consensus.cc:2749] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:52.485082 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "afd1573aba344096affc47d60971991c" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:52.485823 13029 raft_consensus.cc:3060] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.474306 12998 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
W20260709 12:27:52.489084 12924 leader_election.cc:336] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.491657 12851 leader_election.cc:304] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
W20260709 12:27:52.492681 12998 consensus_peers.cc:597] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:52.493247 13686 raft_consensus.cc:2804] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:52.493743 13686 raft_consensus.cc:493] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.494166 13686 raft_consensus.cc:3060] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.446403 13030 raft_consensus.cc:2468] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:27:52.499127 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd444f6f267d4255873e372d6e3e3a43" candidate_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:52.499867 12957 raft_consensus.cc:3060] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:52.502408 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f" is_pre_election: true
I20260709 12:27:52.504400 12883 raft_consensus.cc:2468] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 1.
I20260709 12:27:52.507372 12957 raft_consensus.cc:2468] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18cf4829b3bc4fafb3ef7f1b98f853fe in term 3.
I20260709 12:27:52.515381 12922 leader_election.cc:304] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.516964 13682 raft_consensus.cc:2804] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:27:52.520391 12924 leader_election.cc:336] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.524101 13682 raft_consensus.cc:697] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:52.526016 12922 leader_election.cc:304] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.497335 13662 raft_consensus.cc:2749] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:27:52.527495 13685 raft_consensus.cc:2804] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:52.527938 13685 raft_consensus.cc:493] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:27:52.528357 13685 raft_consensus.cc:3060] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.524555 12849 leader_election.cc:304] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.531030 13684 raft_consensus.cc:2804] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:52.526185 13682 consensus_queue.cc:237] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.531651 13684 raft_consensus.cc:697] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:52.532884 13684 consensus_queue.cc:237] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.535861 13029 raft_consensus.cc:2468] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 2.
W20260709 12:27:52.536660 12924 leader_election.cc:336] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:27:52.537281 12924 leader_election.cc:336] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.537676 12924 leader_election.cc:304] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.526547 13686 raft_consensus.cc:515] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.555060 13686 leader_election.cc:290] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
W20260709 12:27:52.558193 12924 consensus_peers.cc:597] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:52.538414 12923 leader_election.cc:304] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.559713 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "224f359a7c884a7890b05f3098408d31" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:27:52.560233 12957 raft_consensus.cc:3060] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:52.538731 13687 raft_consensus.cc:2749] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:52.561540 13687 raft_consensus.cc:2804] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:52.562196 13687 raft_consensus.cc:697] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:27:52.564144 13687 consensus_queue.cc:237] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.567878 12957 raft_consensus.cc:2468] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:27:52.569330 12851 leader_election.cc:304] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:52.570363 13686 raft_consensus.cc:2804] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:52.571103 13686 raft_consensus.cc:697] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:52.572165 13686 consensus_queue.cc:237] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:52.578415 12705 catalog_manager.cc:5697] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 1 to 2, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 2 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:52.540701 13685 raft_consensus.cc:515] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:52.543906 12999 leader_election.cc:304] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8012bb9a8b274890bcaad0d192cd30d5; no voters: b543f973837c43b19b7d17feaaca5325
W20260709 12:27:52.591642 12999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 47864us
I20260709 12:27:52.592940 13679 raft_consensus.cc:2804] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:52.593582 13679 raft_consensus.cc:697] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 LEADER]: Becoming Leader. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:27:52.594568 13679 consensus_queue.cc:237] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:52.596827 13685 leader_election.cc:290] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Requested vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:27:52.558589 12706 catalog_manager.cc:5697] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 1 to 2, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:52.600219 12884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:27:52.601011 12884 raft_consensus.cc:3060] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:52.613188 12924 leader_election.cc:336] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.624389 12884 raft_consensus.cc:2468] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8012bb9a8b274890bcaad0d192cd30d5 in term 2.
I20260709 12:27:52.625794 12922 leader_election.cc:304] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:52.627094 13687 raft_consensus.cc:2804] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:52.627715 13687 raft_consensus.cc:697] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Becoming Leader. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
W20260709 12:27:52.628582 12850 leader_election.cc:336] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:52.629067 13687 consensus_queue.cc:237] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
W20260709 12:27:52.629802 12924 consensus_peers.cc:597] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.658001 12998 consensus_peers.cc:597] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.681644 12924 consensus_peers.cc:597] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:52.682492 12703 catalog_manager.cc:5697] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 1 to 2, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 2 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:52.684433 12703 catalog_manager.cc:5697] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 1 to 2, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 2 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:52.704523 12998 consensus_peers.cc:597] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.709343 12924 consensus_peers.cc:597] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:52.715387 12705 catalog_manager.cc:5697] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe reported cstate change: term changed from 2 to 3, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68). New cstate: current_term: 3 leader_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:52.719545 12703 catalog_manager.cc:5697] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 1 to 2, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:52.721645 12703 catalog_manager.cc:5697] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 1 to 2, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:52.723358 12703 catalog_manager.cc:5697] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 1 to 2, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:52.724969 12703 catalog_manager.cc:5697] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 reported cstate change: term changed from 1 to 2, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "8012bb9a8b274890bcaad0d192cd30d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:52.736472 12998 consensus_peers.cc:597] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.739809 12850 consensus_peers.cc:597] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.757045 12924 consensus_peers.cc:597] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.758317 12998 consensus_peers.cc:597] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.758879 12998 consensus_peers.cc:597] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.761253 12924 consensus_peers.cc:597] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.858875 12998 consensus_peers.cc:597] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.859432 12850 consensus_peers.cc:597] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.859988 12998 consensus_peers.cc:597] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.866077 12998 consensus_peers.cc:597] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.866716 12998 consensus_peers.cc:597] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.884833 12998 consensus_peers.cc:597] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:52.896431 13032 raft_consensus.cc:1275] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:52.904312 13686 consensus_queue.cc:1048] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:52.919245 12883 raft_consensus.cc:1275] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer 8012bb9a8b274890bcaad0d192cd30d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:52.924335 13688 consensus_queue.cc:1048] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:52.928216 12884 raft_consensus.cc:1275] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer 8012bb9a8b274890bcaad0d192cd30d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:27:52.928565 12924 consensus_peers.cc:597] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.945127 12850 consensus_peers.cc:597] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:52.946115 12850 consensus_peers.cc:597] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:52.956679 12998 consensus_peers.cc:597] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:52.971187 13687 consensus_queue.cc:1048] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:52.973158 13032 raft_consensus.cc:1275] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:52.989233 12957 raft_consensus.cc:1275] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer 18cf4829b3bc4fafb3ef7f1b98f853fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:27:52.991362 12924 consensus_peers.cc:597] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:52.998522 13684 consensus_queue.cc:1048] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:53.008540 13678 consensus_queue.cc:1048] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:27:53.026417 12924 consensus_peers.cc:597] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:53.027062 12924 consensus_peers.cc:597] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:53.028811 12998 consensus_peers.cc:597] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:53.036119 13032 raft_consensus.cc:1275] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer 8012bb9a8b274890bcaad0d192cd30d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:27:53.045347 12998 consensus_peers.cc:597] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:53.045667 13688 consensus_queue.cc:1048] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:27:53.053016 12850 consensus_peers.cc:597] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:53.109757 12924 consensus_peers.cc:597] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:53.120503 12881 raft_consensus.cc:1275] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer 8012bb9a8b274890bcaad0d192cd30d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:53.123194 13675 consensus_queue.cc:1048] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:53.131754 12884 raft_consensus.cc:1275] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer 8012bb9a8b274890bcaad0d192cd30d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:27:53.136581 12850 consensus_peers.cc:597] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:53.137341 12850 consensus_peers.cc:597] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:53.138603 13688 consensus_queue.cc:1048] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:27:53.192082   561 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 12831)
I20260709 12:27:53.243904 12958 raft_consensus.cc:1275] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Refusing update from remote peer 18cf4829b3bc4fafb3ef7f1b98f853fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:27:53.245187 12998 consensus_peers.cc:597] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:53.245422 12924 consensus_peers.cc:597] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:53.246330 12924 consensus_peers.cc:597] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:53.246927 12924 consensus_peers.cc:597] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:53.247350 12924 consensus_peers.cc:597] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:53.251416 12998 consensus_peers.cc:597] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:53.319247 12998 consensus_peers.cc:597] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:53.338758 12996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 130127us
I20260709 12:27:53.372605 13720 consensus_queue.cc:1048] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:27:53.378594 12924 consensus_peers.cc:597] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:53.389189 12924 consensus_peers.cc:597] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:53.418063 12958 raft_consensus.cc:1275] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:27:53.430096 12924 consensus_peers.cc:597] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:53.431535 12924 consensus_peers.cc:597] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:53.454598 13715 consensus_queue.cc:1048] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:54.105060 13715 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:54.105772 13715 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:54.109323 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:27:54.110239 13029 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 3.
I20260709 12:27:54.116231 12849 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:27:54.129740 13712 raft_consensus.cc:2804] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:27:54.130350 13712 raft_consensus.cc:493] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:54.130802 13712 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:54.157939 13715 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:27:54.178742 12850 leader_election.cc:336] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:27:54.192761   561 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 12831)
I20260709 12:27:54.209362 13712 raft_consensus.cc:515] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:54.213145 13712 leader_election.cc:290] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 4 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:54.214202 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "25fae6bafbf34e888a111d96f9fe73d4" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:27:54.215001 13029 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:27:54.225822 12850 leader_election.cc:336] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:27:54.228397 13029 raft_consensus.cc:2468] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 4.
I20260709 12:27:54.230100 12849 leader_election.cc:304] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: b543f973837c43b19b7d17feaaca5325
I20260709 12:27:54.235090 13712 raft_consensus.cc:2804] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:27:54.237506 13712 raft_consensus.cc:697] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 4 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:27:54.240648 13712 consensus_queue.cc:237] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:54.263355 12706 catalog_manager.cc:5697] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 2 to 4, leader changed from b543f973837c43b19b7d17feaaca5325 (127.0.140.65) to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 4 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:54.701864 12850 consensus_peers.cc:597] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:54.849187 13029 raft_consensus.cc:1275] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 4 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:27:54.851318 13727 consensus_queue.cc:1048] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:27:55.014914 12998 consensus_peers.cc:597] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.035363 12998 consensus_peers.cc:597] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.041798 12924 consensus_peers.cc:597] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.042531 12924 consensus_peers.cc:597] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.050148 12998 consensus_peers.cc:597] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.050880 12998 consensus_peers.cc:597] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.118541 12998 consensus_peers.cc:597] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.134955 12924 consensus_peers.cc:597] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.153472 12924 consensus_peers.cc:597] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.180475 12850 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111) [suppressed 433 similar messages]
W20260709 12:27:55.185267 12850 consensus_peers.cc:597] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.193837   561 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 12831)
W20260709 12:27:55.224735 12850 consensus_peers.cc:597] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.243741 12924 consensus_peers.cc:597] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.272540 12998 consensus_peers.cc:597] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.273275 12998 consensus_peers.cc:597] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.314198 12924 consensus_peers.cc:597] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.315019 12998 consensus_peers.cc:597] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.321679 12998 consensus_peers.cc:597] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.328218 12850 consensus_peers.cc:597] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.329551 12998 consensus_peers.cc:597] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.330410 12998 consensus_peers.cc:597] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.347955 12924 consensus_peers.cc:597] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.350255 12850 consensus_peers.cc:597] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.351990 12850 consensus_peers.cc:597] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.384732 12998 consensus_peers.cc:597] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.434281 12850 consensus_peers.cc:597] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.435024 12924 consensus_peers.cc:597] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.442657 12998 consensus_peers.cc:597] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.448261 12924 consensus_peers.cc:597] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.449083 12924 consensus_peers.cc:597] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.454066 12998 consensus_peers.cc:597] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.473276 12924 consensus_peers.cc:597] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.495874 12850 consensus_peers.cc:597] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.499179 12924 consensus_peers.cc:597] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.574288 12924 consensus_peers.cc:597] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.574987 12924 consensus_peers.cc:597] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.587337 12924 consensus_peers.cc:597] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.601533 12924 consensus_peers.cc:597] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.603024 12998 consensus_peers.cc:597] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.613039 12924 consensus_peers.cc:597] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.675659 12998 consensus_peers.cc:597] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.687959 12924 consensus_peers.cc:597] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.695673 12924 consensus_peers.cc:597] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:55.733403 12998 consensus_peers.cc:597] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:55.748917 12998 consensus_peers.cc:597] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:56.145195 12922 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 71778.5us
W20260709 12:27:56.194471   561 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 12831)
W20260709 12:27:57.166496 12850 consensus_peers.cc:597] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:57.195080   561 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 12831)
W20260709 12:27:57.276192 12924 consensus_peers.cc:597] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.368181 12924 consensus_peers.cc:597] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.394088 12998 consensus_peers.cc:597] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.414310 12998 consensus_peers.cc:597] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.461877 12924 consensus_peers.cc:597] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.531386 12998 consensus_peers.cc:597] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.578070 12924 consensus_peers.cc:597] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.584386 12998 consensus_peers.cc:597] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.603977 12998 consensus_peers.cc:597] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:57.627559 12850 consensus_peers.cc:597] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.651491 12998 consensus_peers.cc:597] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.661221 12850 consensus_peers.cc:597] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:57.668917 12924 consensus_peers.cc:597] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.728155 12998 consensus_peers.cc:597] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.736972 12924 consensus_peers.cc:597] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:57.760450 12998 consensus_peers.cc:597] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.778275 12924 consensus_peers.cc:597] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.779996 12998 consensus_peers.cc:597] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.803701 12998 consensus_peers.cc:597] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.854876 12924 consensus_peers.cc:597] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.874837 12998 consensus_peers.cc:597] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.894594 12998 consensus_peers.cc:597] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.941474 12850 consensus_peers.cc:597] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:57.951984 12924 consensus_peers.cc:597] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:57.998194 12998 consensus_peers.cc:597] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.011109 12924 consensus_peers.cc:597] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.011793 12924 consensus_peers.cc:597] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.014474 12850 consensus_peers.cc:597] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:58.025856 12924 consensus_peers.cc:597] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.040728 12924 consensus_peers.cc:597] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.067463 12850 consensus_peers.cc:597] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.068686 12924 consensus_peers.cc:597] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:58.073951 12998 consensus_peers.cc:597] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.077543 12998 consensus_peers.cc:597] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.078975 12850 consensus_peers.cc:597] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:58.090605 12924 consensus_peers.cc:597] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.125576 12998 consensus_peers.cc:597] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.161392 12998 consensus_peers.cc:597] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.164081 12924 consensus_peers.cc:597] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.174597 12924 consensus_peers.cc:597] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.184173 12998 consensus_peers.cc:597] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:58.195874   561 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 12831)
W20260709 12:27:58.199741 12850 consensus_peers.cc:597] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:27:58.202622 12924 consensus_peers.cc:597] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:58.247239 12924 consensus_peers.cc:597] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:58.298035   561 tablet_server.cc:196] TabletServer@127.0.140.65:0 shutdown complete.
I20260709 12:27:58.432889   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:58.442140 13747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:58.444762 13746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:58.449402 13749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:58.449512   561 server_base.cc:1061] running on GCE node
I20260709 12:27:58.454819   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:58.455188   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:58.455440   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600078455418 us; error 0 us; skew 500 ppm
I20260709 12:27:58.456115   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:58.459568   561 webserver.cc:533] Webserver started at http://127.0.140.65:35837/ using document root <none> and password file <none>
I20260709 12:27:58.460146   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:58.460356   561 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:27:58.465183   561 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.003s	sys 0.002s
I20260709 12:27:58.474191 13754 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:58.475466   561 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20260709 12:27:58.475929   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "b543f973837c43b19b7d17feaaca5325"
format_stamp: "Formatted at 2026-07-09 12:27:34 on dist-test-slave-7mcn"
I20260709 12:27:58.476553   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:58.504283   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:58.506460   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:58.508422   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:58.515853 13761 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
I20260709 12:27:58.808910   561 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 12:27:58.809329   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.295s	user 0.004s	sys 0.008s
I20260709 12:27:58.809755   561 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
I20260709 12:27:58.818452 13761 tablet_bootstrap.cc:492] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:58.843458 13761 tablet_bootstrap.cc:492] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:58.844718 13761 tablet_bootstrap.cc:492] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:58.845530 13761 ts_tablet_manager.cc:1403] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.029s	user 0.009s	sys 0.012s
I20260709 12:27:58.848398 13761 raft_consensus.cc:359] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:58.848963 13761 raft_consensus.cc:740] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.849670 13761 consensus_queue.cc:260] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:58.852340 13761 ts_tablet_manager.cc:1434] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.002s
I20260709 12:27:58.853598 13761 tablet_bootstrap.cc:492] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:58.874511 13761 tablet_bootstrap.cc:492] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:58.875661 13761 tablet_bootstrap.cc:492] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:58.881129 13761 ts_tablet_manager.cc:1403] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.028s	user 0.022s	sys 0.001s
I20260709 12:27:58.884137 13761 raft_consensus.cc:359] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:58.884984 13761 raft_consensus.cc:740] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.885713 13761 consensus_queue.cc:260] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:58.887851 13761 ts_tablet_manager.cc:1434] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:27:58.888810 13761 tablet_bootstrap.cc:492] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:58.907058 13761 tablet_bootstrap.cc:492] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:58.908227 13761 tablet_bootstrap.cc:492] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:58.909003 13761 ts_tablet_manager.cc:1403] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.020s	user 0.017s	sys 0.001s
I20260709 12:27:58.911465 13761 raft_consensus.cc:359] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:58.912190 13761 raft_consensus.cc:740] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.912880 13761 consensus_queue.cc:260] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:58.914831 13761 ts_tablet_manager.cc:1434] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.005s	user 0.002s	sys 0.000s
I20260709 12:27:58.915786 13761 tablet_bootstrap.cc:492] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:58.931066 13761 tablet_bootstrap.cc:492] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:58.932442 13761 tablet_bootstrap.cc:492] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:58.933054 13761 ts_tablet_manager.cc:1403] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.004s
I20260709 12:27:58.935743 13761 raft_consensus.cc:359] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:58.941515 13761 raft_consensus.cc:740] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.943778 13761 consensus_queue.cc:260] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:58.946308 13761 ts_tablet_manager.cc:1434] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:27:58.947619 13761 tablet_bootstrap.cc:492] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:58.968755 13761 tablet_bootstrap.cc:492] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:58.969877 13761 tablet_bootstrap.cc:492] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:58.970609 13761 ts_tablet_manager.cc:1403] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.023s	user 0.007s	sys 0.007s
I20260709 12:27:58.973565 13761 raft_consensus.cc:359] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:58.974382 13761 raft_consensus.cc:740] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:58.975257 13761 consensus_queue.cc:260] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:58.977772 13761 ts_tablet_manager.cc:1434] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:27:58.978750 13761 tablet_bootstrap.cc:492] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:58.995139 13761 tablet_bootstrap.cc:492] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:58.996055 13761 tablet_bootstrap.cc:492] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:58.996803 13761 ts_tablet_manager.cc:1403] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.008s
I20260709 12:27:58.999680 13761 raft_consensus.cc:359] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.000423 13761 raft_consensus.cc:740] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.001206 13761 consensus_queue.cc:260] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.003222 13761 ts_tablet_manager.cc:1434] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:27:59.004109 13761 tablet_bootstrap.cc:492] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.022370 13761 tablet_bootstrap.cc:492] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.023442 13761 tablet_bootstrap.cc:492] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.024161 13761 ts_tablet_manager.cc:1403] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.020s	user 0.015s	sys 0.004s
I20260709 12:27:59.026419 13761 raft_consensus.cc:359] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.027225 13761 raft_consensus.cc:740] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.028345 13761 consensus_queue.cc:260] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.030349 13761 ts_tablet_manager.cc:1434] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:27:59.031261 13761 tablet_bootstrap.cc:492] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.077728   561 ts_tablet_manager.cc:616] Registered 45 tablets
I20260709 12:27:59.078405   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.269s	user 0.241s	sys 0.018s
I20260709 12:27:59.094234 13761 tablet_bootstrap.cc:492] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.095434 13761 tablet_bootstrap.cc:492] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.096263 13761 ts_tablet_manager.cc:1403] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.065s	user 0.034s	sys 0.027s
I20260709 12:27:59.101995 13761 raft_consensus.cc:359] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.104714 13761 raft_consensus.cc:740] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.105660 13761 consensus_queue.cc:260] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.108029 13761 ts_tablet_manager.cc:1434] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.001s
I20260709 12:27:59.109541 13761 tablet_bootstrap.cc:492] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.141037 13761 tablet_bootstrap.cc:492] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.142390 13761 tablet_bootstrap.cc:492] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.143532 13761 ts_tablet_manager.cc:1403] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.034s	user 0.017s	sys 0.012s
I20260709 12:27:59.147042 13761 raft_consensus.cc:359] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.148025 13761 raft_consensus.cc:740] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.148974 13761 consensus_queue.cc:260] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.152232 13761 ts_tablet_manager.cc:1434] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:59.153343 13761 tablet_bootstrap.cc:492] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.254415 13761 tablet_bootstrap.cc:492] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.255620 13761 tablet_bootstrap.cc:492] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.256626 13761 ts_tablet_manager.cc:1403] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.104s	user 0.041s	sys 0.060s
I20260709 12:27:59.260566 13761 raft_consensus.cc:359] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.261667 13761 raft_consensus.cc:740] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.262841 13761 consensus_queue.cc:260] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.265722 13761 ts_tablet_manager.cc:1434] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:27:59.266927 13761 tablet_bootstrap.cc:492] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.294044 13761 tablet_bootstrap.cc:492] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.295490 13761 tablet_bootstrap.cc:492] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.296483 13761 ts_tablet_manager.cc:1403] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.030s	user 0.013s	sys 0.016s
I20260709 12:27:59.300544 13761 raft_consensus.cc:359] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.301553 13761 raft_consensus.cc:740] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.302536 13761 consensus_queue.cc:260] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.305622 13761 ts_tablet_manager.cc:1434] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:59.306841 13761 tablet_bootstrap.cc:492] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.329239 13761 tablet_bootstrap.cc:492] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.330757 13761 tablet_bootstrap.cc:492] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.331357   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:40461
I20260709 12:27:59.331537 13831 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:40461 every 8 connection(s)
I20260709 12:27:59.331763 13761 ts_tablet_manager.cc:1403] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.025s	user 0.020s	sys 0.003s
I20260709 12:27:59.337122 13761 raft_consensus.cc:359] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.338304 13761 raft_consensus.cc:740] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.339479 13761 consensus_queue.cc:260] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.342155 13761 ts_tablet_manager.cc:1434] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:27:59.343210 13761 tablet_bootstrap.cc:492] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.374332 13761 tablet_bootstrap.cc:492] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.376428 13761 tablet_bootstrap.cc:492] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.377573 13761 ts_tablet_manager.cc:1403] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.035s	user 0.020s	sys 0.000s
I20260709 12:27:59.381994 13761 raft_consensus.cc:359] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.382917 13761 raft_consensus.cc:740] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.383744 13761 consensus_queue.cc:260] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.384403 13832 heartbeater.cc:344] Connected to a master server at 127.0.140.126:45447
I20260709 12:27:59.385555 13832 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:59.386475 13761 ts_tablet_manager.cc:1434] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:59.386699 13832 heartbeater.cc:507] Master 127.0.140.126:45447 requested a full tablet report, sending...
I20260709 12:27:59.387619 13761 tablet_bootstrap.cc:492] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.407371 13761 tablet_bootstrap.cc:492] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.408612 13761 tablet_bootstrap.cc:492] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.409487 13761 ts_tablet_manager.cc:1403] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.022s	user 0.018s	sys 0.000s
I20260709 12:27:59.411736 12706 ts_manager.cc:194] Re-registered known tserver with Master: b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:27:59.413512 13761 raft_consensus.cc:359] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.414324 13761 raft_consensus.cc:740] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.415006 13761 consensus_queue.cc:260] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.416549 13761 ts_tablet_manager.cc:1434] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:59.417466 13761 tablet_bootstrap.cc:492] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.434345 13761 tablet_bootstrap.cc:492] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.435530 13761 tablet_bootstrap.cc:492] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.436324 13761 ts_tablet_manager.cc:1403] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.019s	user 0.016s	sys 0.002s
I20260709 12:27:59.438938 13761 raft_consensus.cc:359] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.439615 13761 raft_consensus.cc:740] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.440346 13761 consensus_queue.cc:260] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.442440 13761 ts_tablet_manager.cc:1434] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:27:59.443315 13761 tablet_bootstrap.cc:492] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.444527 13805 raft_consensus.cc:3060] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:59.464303 12706 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44366
I20260709 12:27:59.468568 13761 tablet_bootstrap.cc:492] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.469666 13761 tablet_bootstrap.cc:492] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.470338 13832 heartbeater.cc:499] Master 127.0.140.126:45447 was elected leader, sending a full tablet report...
I20260709 12:27:59.470348 13761 ts_tablet_manager.cc:1403] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.027s	user 0.015s	sys 0.008s
I20260709 12:27:59.473292 13761 raft_consensus.cc:359] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.474010 13761 raft_consensus.cc:740] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.474730 13761 consensus_queue.cc:260] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.476797 13761 ts_tablet_manager.cc:1434] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:59.477769 13761 tablet_bootstrap.cc:492] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.503175 13804 raft_consensus.cc:3060] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:59.504243 13761 tablet_bootstrap.cc:492] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.505328 13761 tablet_bootstrap.cc:492] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.506038 13761 ts_tablet_manager.cc:1403] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.029s	user 0.019s	sys 0.004s
I20260709 12:27:59.508795 13761 raft_consensus.cc:359] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.509528 13761 raft_consensus.cc:740] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.510319 13761 consensus_queue.cc:260] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.512545 13761 ts_tablet_manager.cc:1434] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:27:59.513520 13761 tablet_bootstrap.cc:492] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.546710 13799 raft_consensus.cc:3060] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:59.547122 13761 tablet_bootstrap.cc:492] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.548332 13761 tablet_bootstrap.cc:492] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.549142 13761 ts_tablet_manager.cc:1403] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.011s
I20260709 12:27:59.546890 13805 raft_consensus.cc:3060] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:59.554605 13761 raft_consensus.cc:359] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.555457 13761 raft_consensus.cc:740] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.556241 13761 consensus_queue.cc:260] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.558398 13761 ts_tablet_manager.cc:1434] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:27:59.559307 13761 tablet_bootstrap.cc:492] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.578697 13801 raft_consensus.cc:3060] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:59.595829 13761 tablet_bootstrap.cc:492] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.597185 13761 tablet_bootstrap.cc:492] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.605913 13761 ts_tablet_manager.cc:1403] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.047s	user 0.015s	sys 0.007s
I20260709 12:27:59.610266 13761 raft_consensus.cc:359] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.611297 13761 raft_consensus.cc:740] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.616313 13761 consensus_queue.cc:260] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.618901 13761 ts_tablet_manager.cc:1434] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.000s
I20260709 12:27:59.620151 13761 tablet_bootstrap.cc:492] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.639282 13761 tablet_bootstrap.cc:492] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.640379 13761 tablet_bootstrap.cc:492] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.641266 13761 ts_tablet_manager.cc:1403] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.003s
I20260709 12:27:59.647780 13761 raft_consensus.cc:359] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.648636 13761 raft_consensus.cc:740] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.649461 13761 consensus_queue.cc:260] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.651773 13761 ts_tablet_manager.cc:1434] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 12:27:59.652721 13761 tablet_bootstrap.cc:492] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.669834 13761 tablet_bootstrap.cc:492] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.670893 13761 tablet_bootstrap.cc:492] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.671660 13761 ts_tablet_manager.cc:1403] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.019s	user 0.016s	sys 0.000s
I20260709 12:27:59.679019 13761 raft_consensus.cc:359] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.679853 13761 raft_consensus.cc:740] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.680554 13761 consensus_queue.cc:260] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.682667 13761 ts_tablet_manager.cc:1434] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:27:59.683382 13761 tablet_bootstrap.cc:492] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.700090 13761 tablet_bootstrap.cc:492] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.702603 13761 tablet_bootstrap.cc:492] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.703413 13761 ts_tablet_manager.cc:1403] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.006s
W20260709 12:27:59.705265 12998 consensus_peers.cc:597] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:27:59.706483 13761 raft_consensus.cc:359] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.707191 13761 raft_consensus.cc:740] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.707775 13761 consensus_queue.cc:260] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.709467 13761 ts_tablet_manager.cc:1434] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:27:59.710697 13761 tablet_bootstrap.cc:492] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.725852 13761 tablet_bootstrap.cc:492] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:27:59.726032 12850 consensus_peers.cc:597] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:59.726867 13761 tablet_bootstrap.cc:492] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.727691 13761 ts_tablet_manager.cc:1403] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.017s	user 0.013s	sys 0.003s
I20260709 12:27:59.730294 13761 raft_consensus.cc:359] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.730839 13761 raft_consensus.cc:740] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.731338 13761 consensus_queue.cc:260] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.733057 13761 ts_tablet_manager.cc:1434] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:59.734809 13761 tablet_bootstrap.cc:492] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.751524 13761 tablet_bootstrap.cc:492] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.752621 13761 tablet_bootstrap.cc:492] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.753331 13761 ts_tablet_manager.cc:1403] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.019s	user 0.017s	sys 0.000s
I20260709 12:27:59.755923 13761 raft_consensus.cc:359] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.756469 13761 raft_consensus.cc:740] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.757117 13761 consensus_queue.cc:260] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:27:59.759101 13761 ts_tablet_manager.cc:1434] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:27:59.760820 13761 tablet_bootstrap.cc:492] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.774998 13761 tablet_bootstrap.cc:492] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.775946 13761 tablet_bootstrap.cc:492] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.776762 13761 ts_tablet_manager.cc:1403] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.016s	user 0.012s	sys 0.003s
I20260709 12:27:59.779371 13761 raft_consensus.cc:359] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.780128 13761 raft_consensus.cc:740] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.780990 13761 consensus_queue.cc:260] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.783214 13761 ts_tablet_manager.cc:1434] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:27:59.784123 13761 tablet_bootstrap.cc:492] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.797628 13801 raft_consensus.cc:3060] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:59.800890 13761 tablet_bootstrap.cc:492] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.802248 13761 tablet_bootstrap.cc:492] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.803015 13761 ts_tablet_manager.cc:1403] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.007s
I20260709 12:27:59.805902 13761 raft_consensus.cc:359] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.806591 13761 raft_consensus.cc:740] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.807331 13761 consensus_queue.cc:260] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.809000 13761 ts_tablet_manager.cc:1434] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:59.809782 13761 tablet_bootstrap.cc:492] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.828845 13761 tablet_bootstrap.cc:492] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.830376 13761 tablet_bootstrap.cc:492] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.831166 13761 ts_tablet_manager.cc:1403] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.022s	user 0.017s	sys 0.000s
I20260709 12:27:59.833681 13761 raft_consensus.cc:359] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.834431 13761 raft_consensus.cc:740] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.835182 13761 consensus_queue.cc:260] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.837354 13761 ts_tablet_manager.cc:1434] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:59.838165 13761 tablet_bootstrap.cc:492] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.853392 13761 tablet_bootstrap.cc:492] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.854985 13761 tablet_bootstrap.cc:492] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.855909 13761 ts_tablet_manager.cc:1403] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.018s	user 0.016s	sys 0.000s
I20260709 12:27:59.858845 13761 raft_consensus.cc:359] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.859555 13761 raft_consensus.cc:740] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.860220 13761 consensus_queue.cc:260] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.862362 13761 ts_tablet_manager.cc:1434] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:27:59.864800 13761 tablet_bootstrap.cc:492] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.907382 13761 tablet_bootstrap.cc:492] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.908378 13761 tablet_bootstrap.cc:492] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.908938 13761 ts_tablet_manager.cc:1403] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.044s	user 0.017s	sys 0.012s
I20260709 12:27:59.910728 13761 raft_consensus.cc:359] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.911293 13761 raft_consensus.cc:740] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.911919 13761 consensus_queue.cc:260] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.913830 13761 ts_tablet_manager.cc:1434] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:59.915138 13761 tablet_bootstrap.cc:492] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.929200 13761 tablet_bootstrap.cc:492] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.929970 13761 tablet_bootstrap.cc:492] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.930570 13761 ts_tablet_manager.cc:1403] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.016s	user 0.007s	sys 0.004s
I20260709 12:27:59.932766 13761 raft_consensus.cc:359] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.933315 13761 raft_consensus.cc:740] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.933954 13761 consensus_queue.cc:260] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.935786 13761 ts_tablet_manager.cc:1434] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:59.936896 13761 tablet_bootstrap.cc:492] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.951713 13761 tablet_bootstrap.cc:492] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.952683 13761 tablet_bootstrap.cc:492] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.953284 13761 ts_tablet_manager.cc:1403] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.000s
I20260709 12:27:59.955453 13761 raft_consensus.cc:359] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.955935 13761 raft_consensus.cc:740] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.956554 13761 consensus_queue.cc:260] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:27:59.958036 13761 ts_tablet_manager.cc:1434] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:59.959254 13761 tablet_bootstrap.cc:492] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:27:59.980870 13761 tablet_bootstrap.cc:492] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:59.981871 13761 tablet_bootstrap.cc:492] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:27:59.982550 13761 ts_tablet_manager.cc:1403] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.024s	user 0.012s	sys 0.005s
I20260709 12:27:59.985226 13761 raft_consensus.cc:359] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.986021 13761 raft_consensus.cc:740] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:27:59.986826 13761 consensus_queue.cc:260] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:27:59.989535 13761 ts_tablet_manager.cc:1434] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:27:59.990393 13761 tablet_bootstrap.cc:492] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
W20260709 12:28:00.008394 12998 consensus_peers.cc:597] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:28:00.018627 13761 tablet_bootstrap.cc:492] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.019870 13761 tablet_bootstrap.cc:492] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.020591 13761 ts_tablet_manager.cc:1403] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.030s	user 0.018s	sys 0.008s
I20260709 12:28:00.023339 13761 raft_consensus.cc:359] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.024066 13761 raft_consensus.cc:740] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.024766 13761 consensus_queue.cc:260] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.027102 13761 ts_tablet_manager.cc:1434] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:28:00.028880 13761 tablet_bootstrap.cc:492] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.052374 13761 tablet_bootstrap.cc:492] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.053397 13761 tablet_bootstrap.cc:492] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.054121 13761 ts_tablet_manager.cc:1403] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.025s	user 0.019s	sys 0.003s
I20260709 12:28:00.062140 13761 raft_consensus.cc:359] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.062891 13761 raft_consensus.cc:740] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.063886 13761 consensus_queue.cc:260] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.066076 13761 ts_tablet_manager.cc:1434] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:28:00.067010 13761 tablet_bootstrap.cc:492] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
W20260709 12:28:00.071894 12850 consensus_peers.cc:597] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:28:00.082898 13761 tablet_bootstrap.cc:492] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.084280 13761 tablet_bootstrap.cc:492] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.085246 13761 ts_tablet_manager.cc:1403] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.018s	user 0.008s	sys 0.007s
I20260709 12:28:00.088058 13761 raft_consensus.cc:359] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:00.088824 13761 raft_consensus.cc:740] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.089556 13761 consensus_queue.cc:260] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:00.093600 13761 ts_tablet_manager.cc:1434] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.001s
I20260709 12:28:00.095176 13761 tablet_bootstrap.cc:492] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
W20260709 12:28:00.107349 12998 consensus_peers.cc:597] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:28:00.116376 13761 tablet_bootstrap.cc:492] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.117456 13761 tablet_bootstrap.cc:492] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.118090 13761 ts_tablet_manager.cc:1403] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.023s	user 0.016s	sys 0.000s
I20260709 12:28:00.120659 13761 raft_consensus.cc:359] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:00.121320 13761 raft_consensus.cc:740] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.121953 13761 consensus_queue.cc:260] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:00.123637 13761 ts_tablet_manager.cc:1434] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:28:00.124413 13761 tablet_bootstrap.cc:492] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.163430 13805 raft_consensus.cc:3060] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:00.171113 13804 raft_consensus.cc:3060] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:00.221701 13805 raft_consensus.cc:3060] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Advancing to term 4
W20260709 12:28:00.225386 12996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 72699.1us
W20260709 12:28:00.351046 12996 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:38975, user_credentials={real_user=slave}} blocked reactor thread for 97788.8us
I20260709 12:28:00.384713 13761 tablet_bootstrap.cc:492] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.385921 13761 tablet_bootstrap.cc:492] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.386674 13761 ts_tablet_manager.cc:1403] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.262s	user 0.045s	sys 0.087s
I20260709 12:28:00.392517 13761 raft_consensus.cc:359] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.393244 13761 raft_consensus.cc:740] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.394079 13761 consensus_queue.cc:260] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.397432 13761 ts_tablet_manager.cc:1434] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:28:00.414312 13761 tablet_bootstrap.cc:492] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.445964 13798 raft_consensus.cc:3060] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:00.447918 13761 tablet_bootstrap.cc:492] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.449271 13761 tablet_bootstrap.cc:492] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.452436 13761 ts_tablet_manager.cc:1403] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.038s	user 0.013s	sys 0.015s
W20260709 12:28:00.458705 12998 consensus_peers.cc:597] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 21: this message will repeat every 5th retry.
I20260709 12:28:00.461687 13761 raft_consensus.cc:359] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:00.462700 13761 raft_consensus.cc:740] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.464450 13761 consensus_queue.cc:260] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:00.472015 13761 ts_tablet_manager.cc:1434] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
I20260709 12:28:00.474331 13761 tablet_bootstrap.cc:492] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.496160 13761 tablet_bootstrap.cc:492] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.497381 13761 tablet_bootstrap.cc:492] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.498016 13761 ts_tablet_manager.cc:1403] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.025s	user 0.019s	sys 0.000s
I20260709 12:28:00.500408 13761 raft_consensus.cc:359] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:00.501257 13761 raft_consensus.cc:740] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.502279 13761 consensus_queue.cc:260] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:00.506382 13761 ts_tablet_manager.cc:1434] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:28:00.510391 13761 tablet_bootstrap.cc:492] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.525643 13761 tablet_bootstrap.cc:492] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.526520 13761 tablet_bootstrap.cc:492] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.527289 13761 ts_tablet_manager.cc:1403] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.017s	user 0.016s	sys 0.000s
I20260709 12:28:00.529568 13761 raft_consensus.cc:359] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:00.530210 13761 raft_consensus.cc:740] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.530962 13761 consensus_queue.cc:260] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:00.532980 13761 ts_tablet_manager.cc:1434] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:28:00.534963 13761 tablet_bootstrap.cc:492] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.560102 13761 tablet_bootstrap.cc:492] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.562036 13761 tablet_bootstrap.cc:492] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.562893 13761 ts_tablet_manager.cc:1403] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.028s	user 0.019s	sys 0.003s
I20260709 12:28:00.565510 13761 raft_consensus.cc:359] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:00.566215 13761 raft_consensus.cc:740] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.567018 13761 consensus_queue.cc:260] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:00.572809 13761 ts_tablet_manager.cc:1434] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.010s	user 0.005s	sys 0.000s
I20260709 12:28:00.573760 13761 tablet_bootstrap.cc:492] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.593286 13761 tablet_bootstrap.cc:492] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.594504 13761 tablet_bootstrap.cc:492] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.596191 13761 ts_tablet_manager.cc:1403] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.003s
I20260709 12:28:00.598982 13761 raft_consensus.cc:359] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.599858 13761 raft_consensus.cc:740] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.600651 13761 consensus_queue.cc:260] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.602490 13761 ts_tablet_manager.cc:1434] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:28:00.605281 13761 tablet_bootstrap.cc:492] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.655068 13761 tablet_bootstrap.cc:492] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.656370 13761 tablet_bootstrap.cc:492] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.657414 13761 ts_tablet_manager.cc:1403] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.052s	user 0.025s	sys 0.009s
W20260709 12:28:00.657662 12924 consensus_peers.cc:597] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:28:00.660780 13761 raft_consensus.cc:359] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:00.661640 13761 raft_consensus.cc:740] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.662684 13761 consensus_queue.cc:260] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:00.665299 13761 ts_tablet_manager.cc:1434] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:28:00.666194 13761 tablet_bootstrap.cc:492] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.683676 13761 tablet_bootstrap.cc:492] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.684864 13761 tablet_bootstrap.cc:492] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.685704 13761 ts_tablet_manager.cc:1403] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.003s
I20260709 12:28:00.688750 13761 raft_consensus.cc:359] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.689370 13761 raft_consensus.cc:740] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.689975 13761 consensus_queue.cc:260] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.692030 13761 ts_tablet_manager.cc:1434] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:28:00.693857 13761 tablet_bootstrap.cc:492] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: Bootstrap starting.
I20260709 12:28:00.715698 13761 tablet_bootstrap.cc:492] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:00.718138 13761 tablet_bootstrap.cc:492] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: Bootstrap complete.
I20260709 12:28:00.719050 13761 ts_tablet_manager.cc:1403] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: Time spent bootstrapping tablet: real 0.025s	user 0.015s	sys 0.007s
I20260709 12:28:00.721940 13761 raft_consensus.cc:359] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.722662 13761 raft_consensus.cc:740] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Initialized, Role: FOLLOWER
I20260709 12:28:00.723415 13761 consensus_queue.cc:260] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:00.725705 13761 ts_tablet_manager.cc:1434] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.000s
I20260709 12:28:01.257949 13798 raft_consensus.cc:3060] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.356391   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:09.433290 13030 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "070b536983ee48e1853040e1652619e2"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:09.434233 13030 raft_consensus.cc:606] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:28:09.438046 13030 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:09.438674 13030 raft_consensus.cc:606] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:09.441839 13030 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "35361d4194194047849e55975d5a91b9"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:09.442425 13030 raft_consensus.cc:606] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:09.445516 13030 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:09.446067 13030 raft_consensus.cc:606] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:28:09.460975 12954 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9cf46416458c4a9398e8de1c6b8ccb5d"
dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
mode: GRACEFUL
new_leader_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:52958
I20260709 12:28:09.461608 12954 raft_consensus.cc:606] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Received request to transfer leadership to TS 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:28:09.464707 13030 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:09.465292 13030 raft_consensus.cc:606] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:09.466805   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:28:09.468551   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:09.469908   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:09.513445 13915 raft_consensus.cc:993] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:09.514144 13915 raft_consensus.cc:1081] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:09.519460 13805 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "35361d4194194047849e55975d5a91b9"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:48256
I20260709 12:28:09.520107 13805 raft_consensus.cc:493] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:09.520468 13805 raft_consensus.cc:3060] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.523456 13914 raft_consensus.cc:993] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:09.524328 13912 raft_consensus.cc:1081] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:09.526539 13804 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:48256
I20260709 12:28:09.527271 13804 raft_consensus.cc:493] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:09.527654 13804 raft_consensus.cc:3060] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.529179 13805 raft_consensus.cc:515] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:09.533354 13805 leader_election.cc:290] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:28:09.543859 13804 raft_consensus.cc:515] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:09.546607 13804 leader_election.cc:290] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:28:09.553143 13030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35361d4194194047849e55975d5a91b9" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:09.553514 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:09.554052 13030 raft_consensus.cc:3055] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:09.554086 13029 raft_consensus.cc:3055] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:09.554549 13029 raft_consensus.cc:740] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:09.554504 13030 raft_consensus.cc:740] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:09.555622 13029 consensus_queue.cc:260] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:09.555622 13030 consensus_queue.cc:260] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:09.557401 13029 raft_consensus.cc:3060] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.557458 13030 raft_consensus.cc:3060] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.566586 13029 raft_consensus.cc:2468] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:09.568639 12883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35361d4194194047849e55975d5a91b9" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:28:09.568861 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0d4e24d741fa47ce960e848db5967200" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:28:09.569300 13756 leader_election.cc:304] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:09.569576 12883 raft_consensus.cc:3060] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.569710 12881 raft_consensus.cc:3060] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.572710 13030 raft_consensus.cc:2468] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:09.584810 12883 raft_consensus.cc:2468] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:09.591456 13755 leader_election.cc:304] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:09.601769 13923 raft_consensus.cc:2804] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:09.606249 12881 raft_consensus.cc:2468] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:09.610867 13926 raft_consensus.cc:2804] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:09.613157 13923 raft_consensus.cc:697] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:09.613921 13926 raft_consensus.cc:697] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:09.615784 13923 consensus_queue.cc:237] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:09.616247 13926 consensus_queue.cc:237] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:09.624083 13916 raft_consensus.cc:993] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5: : Instructing follower 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:09.625159 13886 raft_consensus.cc:1081] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Signalling peer 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:09.627766 12881 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9cf46416458c4a9398e8de1c6b8ccb5d"
dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:37324
I20260709 12:28:09.628446 12881 raft_consensus.cc:493] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:09.628821 12881 raft_consensus.cc:3060] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.638298 13912 raft_consensus.cc:993] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:09.637898 12703 catalog_manager.cc:5697] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:09.638988 13912 raft_consensus.cc:1081] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:09.640708 12881 raft_consensus.cc:515] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:09.641237 12703 catalog_manager.cc:5697] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:09.641902 12883 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db"
dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:37310
I20260709 12:28:09.643100 12883 raft_consensus.cc:493] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:09.643594 12883 raft_consensus.cc:3060] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.645998 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf46416458c4a9398e8de1c6b8ccb5d" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:09.646775 12957 raft_consensus.cc:3055] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:09.647135 12957 raft_consensus.cc:740] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:28:09.648175 12957 consensus_queue.cc:260] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:09.649086 13030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9cf46416458c4a9398e8de1c6b8ccb5d" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:09.649650 12957 raft_consensus.cc:3060] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.649816 13030 raft_consensus.cc:3060] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.644200 12881 leader_election.cc:290] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:09.665866 12957 raft_consensus.cc:2468] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:09.666955 13030 raft_consensus.cc:2468] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:09.655073 12883 raft_consensus.cc:515] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:09.668700 12851 leader_election.cc:304] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:28:09.669920 13869 raft_consensus.cc:2804] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:09.670440 13869 raft_consensus.cc:697] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:28:09.670801 12883 leader_election.cc:290] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:09.671746 13869 consensus_queue.cc:237] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:09.675531 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:09.676455 12957 raft_consensus.cc:3060] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.676846 13030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4f810e611ff745b2a7fa598c418b93db" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:09.677949 13030 raft_consensus.cc:3055] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:09.678356 13030 raft_consensus.cc:740] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:09.679504 13030 consensus_queue.cc:260] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:09.684478 13030 raft_consensus.cc:3060] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.696976 12957 raft_consensus.cc:2468] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:09.703076 12851 leader_election.cc:304] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:28:09.704584 12706 catalog_manager.cc:5697] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 1 to 2, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 2 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:09.706634 13030 raft_consensus.cc:2468] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:09.706964 13869 raft_consensus.cc:2804] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:09.707448 13869 raft_consensus.cc:697] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:28:09.708336 13869 consensus_queue.cc:237] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:09.719332 13925 raft_consensus.cc:993] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:09.720232 13914 raft_consensus.cc:1081] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:09.724509 13804 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:48256
I20260709 12:28:09.725418 13804 raft_consensus.cc:493] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:09.725912 13804 raft_consensus.cc:3060] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.763877 13804 raft_consensus.cc:515] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:09.768438 12706 catalog_manager.cc:5697] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 2 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:09.772774 13030 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:09.773459 13030 raft_consensus.cc:3055] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:09.773741 13030 raft_consensus.cc:740] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:09.774554 13030 consensus_queue.cc:260] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:09.775863 13030 raft_consensus.cc:3060] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.784791 13915 raft_consensus.cc:993] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:09.790401 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cfad63c176a643f3bf8bba8f61523f87" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:09.791416 12957 raft_consensus.cc:3060] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.795897 13912 raft_consensus.cc:1081] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:09.810475 12957 raft_consensus.cc:2468] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:09.814066 13758 leader_election.cc:304] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:09.815836 13926 raft_consensus.cc:2804] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:09.816704 13926 raft_consensus.cc:697] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:09.817847 13926 consensus_queue.cc:237] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:09.816005 13804 leader_election.cc:290] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:09.823488 13030 raft_consensus.cc:2468] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:09.830832 12878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "070b536983ee48e1853040e1652619e2"
dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:37310
I20260709 12:28:09.831429 12878 raft_consensus.cc:493] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:09.831777 12878 raft_consensus.cc:3060] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.843926 12878 raft_consensus.cc:515] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:09.841971 12706 catalog_manager.cc:5697] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:09.846324 12878 leader_election.cc:290] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:09.847478 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "070b536983ee48e1853040e1652619e2" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:09.848181 13029 raft_consensus.cc:3055] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:09.848481 13029 raft_consensus.cc:740] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:09.849325 13029 consensus_queue.cc:260] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:09.850742 13029 raft_consensus.cc:3060] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.851230 12957 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "070b536983ee48e1853040e1652619e2" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:09.851838 12957 raft_consensus.cc:3060] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:09.858592 13029 raft_consensus.cc:2468] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:09.859120 12957 raft_consensus.cc:2468] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:09.860179 12849 leader_election.cc:304] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:28:09.860996 13869 raft_consensus.cc:2804] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:09.861425 13869 raft_consensus.cc:697] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:28:09.862423 13869 consensus_queue.cc:237] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:09.874258 12706 catalog_manager.cc:5697] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 2 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:10.020514 13029 raft_consensus.cc:1275] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.021818 12878 raft_consensus.cc:1275] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.022157 13926 consensus_queue.cc:1048] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:10.023548 13923 consensus_queue.cc:1048] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:10.053682 12881 raft_consensus.cc:1275] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.056421 13929 consensus_queue.cc:1048] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:10.061724 13029 raft_consensus.cc:1275] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.064088 13926 consensus_queue.cc:1048] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:10.151851 13027 raft_consensus.cc:1275] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.153673 13869 consensus_queue.cc:1048] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:10.176091 12954 raft_consensus.cc:1275] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.180826 13868 consensus_queue.cc:1048] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:10.224296 12957 raft_consensus.cc:1275] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.226086 13868 consensus_queue.cc:1048] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:10.223513 12954 raft_consensus.cc:1275] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.231386 13929 consensus_queue.cc:1048] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:10.246376 13027 raft_consensus.cc:1275] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.249282 13933 consensus_queue.cc:1048] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:10.255820 13027 raft_consensus.cc:1275] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.257268 13929 consensus_queue.cc:1048] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:10.259622 12957 raft_consensus.cc:1275] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.262015 13933 consensus_queue.cc:1048] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:10.295679 13032 raft_consensus.cc:1275] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:10.297695 13869 consensus_queue.cc:1048] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:10.518477   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:10.555549 13032 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:10.556202 13032 raft_consensus.cc:606] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:28:10.559332 13032 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:10.560112 13032 raft_consensus.cc:606] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:10.564189 13032 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:10.564827 13032 raft_consensus.cc:606] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:10.568797 13032 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:10.569484 13032 raft_consensus.cc:606] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:10.573621 13027 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1"
dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:43156
I20260709 12:28:10.574283 13027 raft_consensus.cc:606] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:10.577453 12954 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1"
dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:52958
I20260709 12:28:10.578112 12954 raft_consensus.cc:606] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:10.579878   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:28:10.581918 13914 raft_consensus.cc:993] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:10.582530 13915 raft_consensus.cc:1081] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:10.583360   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:10.584430   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:10.584970 13798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:48256
I20260709 12:28:10.585731 13798 raft_consensus.cc:493] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:10.586164 13798 raft_consensus.cc:3060] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.594838 13798 raft_consensus.cc:515] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:10.598238 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:10.598862 13798 leader_election.cc:290] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:10.599788 13027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:10.600422 13027 raft_consensus.cc:3055] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:10.600716 13027 raft_consensus.cc:740] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:10.601826 13027 consensus_queue.cc:260] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:10.603227 13027 raft_consensus.cc:3060] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.599014 12954 raft_consensus.cc:3060] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.612915 13027 raft_consensus.cc:2468] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:10.614462 13756 leader_election.cc:304] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:10.615650 13929 raft_consensus.cc:2804] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:10.616154 13929 raft_consensus.cc:697] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:10.617161 13929 consensus_queue.cc:237] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:10.620519 12954 raft_consensus.cc:2468] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:10.629925 12705 catalog_manager.cc:5697] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:10.631613 13914 raft_consensus.cc:993] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:10.632335 13915 raft_consensus.cc:1081] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:10.634852 13798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:48256
I20260709 12:28:10.638103 13798 raft_consensus.cc:493] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:10.639250 13798 raft_consensus.cc:3060] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.648489 13798 raft_consensus.cc:515] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:10.653007 13798 leader_election.cc:290] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:10.655277 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:10.656311 13032 raft_consensus.cc:3055] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:10.656754 13032 raft_consensus.cc:740] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:10.663313 13032 consensus_queue.cc:260] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:10.666530 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c4eccc8971ad4aacb553dc8e1bb00cd8" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:10.667394 12954 raft_consensus.cc:3060] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.667500 13032 raft_consensus.cc:3060] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.676045 12954 raft_consensus.cc:2468] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:10.680260 13758 leader_election.cc:304] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:10.698035 13929 raft_consensus.cc:2804] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:10.698745 13929 raft_consensus.cc:697] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:10.699792 13929 consensus_queue.cc:237] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:10.707098 13032 raft_consensus.cc:2468] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:10.721484 13914 raft_consensus.cc:993] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:10.725169 13971 raft_consensus.cc:1081] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:10.727386 12878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de"
dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:37310
I20260709 12:28:10.728156 12878 raft_consensus.cc:493] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:10.728613 12878 raft_consensus.cc:3060] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.739324 12878 raft_consensus.cc:515] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:10.741593 12703 catalog_manager.cc:5697] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:10.742949 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:10.749056 13032 raft_consensus.cc:3055] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:10.749452 13032 raft_consensus.cc:740] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:10.743513 12878 leader_election.cc:290] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:10.750257 13032 consensus_queue.cc:260] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:10.751587 13032 raft_consensus.cc:3060] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.756036 13971 raft_consensus.cc:993] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:10.756706 13971 raft_consensus.cc:1081] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:10.756976 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "34ac366ceb574dd18638a5dcbde9b2de" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:10.757658 12954 raft_consensus.cc:3060] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.761518 13032 raft_consensus.cc:2468] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:10.761940 13798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:48256
I20260709 12:28:10.763031 12849 leader_election.cc:304] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:28:10.764117 13933 raft_consensus.cc:2804] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:10.764849 13933 raft_consensus.cc:697] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:28:10.765578 12954 raft_consensus.cc:2468] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:10.765815 13933 consensus_queue.cc:237] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:10.762669 13798 raft_consensus.cc:493] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:10.767437 13798 raft_consensus.cc:3060] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.777863 13798 raft_consensus.cc:515] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:10.779791 13798 leader_election.cc:290] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:28:10.781453 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:10.781478 12878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc65da4fd7234bbf96ecc59d58baa5c1" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:28:10.782147 13032 raft_consensus.cc:3055] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:10.782539 13032 raft_consensus.cc:740] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:10.782532 12878 raft_consensus.cc:3060] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.783488 13032 consensus_queue.cc:260] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:10.783784 12703 catalog_manager.cc:5697] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 2 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:10.784803 13032 raft_consensus.cc:3060] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.792346 13032 raft_consensus.cc:2468] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:10.793401 13756 leader_election.cc:304] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:10.793653 12878 raft_consensus.cc:2468] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:10.794409 13929 raft_consensus.cc:2804] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:10.794983 13929 raft_consensus.cc:697] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:10.795915 13929 consensus_queue.cc:237] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:10.814364 12703 catalog_manager.cc:5697] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:10.857445 13973 raft_consensus.cc:993] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:10.858155 13916 raft_consensus.cc:1081] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:10.860173 13798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:47448
I20260709 12:28:10.860762 13798 raft_consensus.cc:493] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:10.861148 13798 raft_consensus.cc:3060] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.866917 13798 raft_consensus.cc:515] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:10.868572 13798 leader_election.cc:290] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:28:10.869747 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:10.870447 12954 raft_consensus.cc:3055] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:10.870370 12878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7237e470a3048a8878e2d0d49208ee1" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:28:10.871142 12954 raft_consensus.cc:740] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:28:10.871411 12878 raft_consensus.cc:3060] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.872067 12954 consensus_queue.cc:260] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:10.873196 12954 raft_consensus.cc:3060] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:10.877995 12878 raft_consensus.cc:2468] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:10.879376 12954 raft_consensus.cc:2468] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:10.879792 13755 leader_election.cc:304] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:10.880869 13929 raft_consensus.cc:2804] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:10.881438 13929 raft_consensus.cc:697] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:10.882395 13929 consensus_queue.cc:237] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:10.896256 12703 catalog_manager.cc:5697] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:11.002534 13032 raft_consensus.cc:1275] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.004176 13929 consensus_queue.cc:1048] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:11.025818 12956 raft_consensus.cc:1275] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.027678 13923 consensus_queue.cc:1048] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:11.048447 13971 raft_consensus.cc:993] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:11.049077 13914 raft_consensus.cc:1081] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:11.050865 13798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:48256
I20260709 12:28:11.051460 13798 raft_consensus.cc:493] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:11.051856 13798 raft_consensus.cc:3060] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:11.058606 13798 raft_consensus.cc:515] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:11.060521 13798 leader_election.cc:290] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:28:11.062089 12878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:28:11.062216 13029 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81ab261b0dc04756a70faccbf14a33d6" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:11.062821 12878 raft_consensus.cc:3060] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:11.063097 13029 raft_consensus.cc:3055] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:11.063508 13029 raft_consensus.cc:740] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 18cf4829b3bc4fafb3ef7f1b98f853fe, State: Running, Role: LEADER
I20260709 12:28:11.064313 13029 consensus_queue.cc:260] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:11.065263 13029 raft_consensus.cc:3060] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:11.070536 12878 raft_consensus.cc:2468] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:11.071884 13755 leader_election.cc:304] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:11.072512 13029 raft_consensus.cc:2468] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:11.072948 13923 raft_consensus.cc:2804] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:11.073547 13923 raft_consensus.cc:697] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:11.074401 13923 consensus_queue.cc:237] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:11.088187 12703 catalog_manager.cc:5697] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:11.107909 13029 raft_consensus.cc:1275] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.109176 13923 consensus_queue.cc:1048] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:11.125283 12956 raft_consensus.cc:1275] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.127436 13928 consensus_queue.cc:1048] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:11.208508 13032 raft_consensus.cc:1275] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.210114 13963 consensus_queue.cc:1048] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:11.256294 12956 raft_consensus.cc:1275] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.261042 13963 consensus_queue.cc:1048] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:11.268787 12956 raft_consensus.cc:1275] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.275066 13923 consensus_queue.cc:1048] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:11.279747 13027 raft_consensus.cc:1275] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.315165 13929 consensus_queue.cc:1048] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:11.339867 12878 raft_consensus.cc:1275] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.351364 13923 consensus_queue.cc:1048] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:11.413429 12881 raft_consensus.cc:1275] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.416888 13923 consensus_queue.cc:1048] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:11.515547 13032 raft_consensus.cc:1275] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.517204 13996 consensus_queue.cc:1048] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:11.537962 12881 raft_consensus.cc:1275] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:11.540040 13997 consensus_queue.cc:1048] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:11.728823   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:11.771234 12954 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7"
dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:52958
I20260709 12:28:11.771911 12954 raft_consensus.cc:606] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:11.774605 12954 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d"
dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:52958
I20260709 12:28:11.775163 12954 raft_consensus.cc:606] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:11.778240 12956 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51"
dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:52958
I20260709 12:28:11.778950 12956 raft_consensus.cc:606] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:11.781934 12956 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f"
dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
mode: GRACEFUL
new_leader_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:52958
I20260709 12:28:11.782550 12956 raft_consensus.cc:606] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Received request to transfer leadership to TS 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:28:11.785157 13989 raft_consensus.cc:993] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:11.785815 13916 raft_consensus.cc:1081] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:11.785887 12956 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ef0476290f24476eae1cbb574a059780"
dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:52958
I20260709 12:28:11.786670 12956 raft_consensus.cc:606] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:11.787971 13800 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:47448
I20260709 12:28:11.788664 13800 raft_consensus.cc:493] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:11.789037 13800 raft_consensus.cc:3060] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:11.789790   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:28:11.791772   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:11.792784   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:11.802913 13800 raft_consensus.cc:515] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:11.809828 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:11.810441 12956 raft_consensus.cc:3055] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:11.810694 12956 raft_consensus.cc:740] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:28:11.811754 12956 consensus_queue.cc:260] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:11.812791 12956 raft_consensus.cc:3060] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:11.824525 12956 raft_consensus.cc:2468] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:11.826577 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9e1ba3f1907244abbd44d141e09ff65d" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:11.826987 13800 leader_election.cc:290] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:11.828480 13758 leader_election.cc:304] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:11.829312 13996 raft_consensus.cc:2804] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:11.829301 13032 raft_consensus.cc:3060] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:11.829918 13996 raft_consensus.cc:697] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:11.830842 13996 consensus_queue.cc:237] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:11.840574 13032 raft_consensus.cc:2468] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:11.862564 12706 catalog_manager.cc:5697] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:11.867704 13916 raft_consensus.cc:993] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:11.868412 13916 raft_consensus.cc:1081] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:11.870270 13798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:47448
I20260709 12:28:11.870832 13798 raft_consensus.cc:493] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:11.871172 13798 raft_consensus.cc:3060] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:11.878078 13798 raft_consensus.cc:515] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:11.879914 13798 leader_election.cc:290] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:11.882074 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:11.882237 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:11.882874 12956 raft_consensus.cc:3055] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:11.882936 13032 raft_consensus.cc:3060] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:11.883405 12956 raft_consensus.cc:740] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:28:11.884393 12956 consensus_queue.cc:260] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:11.891145 12956 raft_consensus.cc:3060] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:11.894518 13032 raft_consensus.cc:2468] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:11.896143 13756 leader_election.cc:304] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:11.897085 13997 raft_consensus.cc:2804] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:11.897745 13997 raft_consensus.cc:697] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:11.898694 13997 consensus_queue.cc:237] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:11.909155 12956 raft_consensus.cc:2468] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:11.910506 12706 catalog_manager.cc:5697] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:12.154054 13973 raft_consensus.cc:993] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5: : Instructing follower 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:12.154950 13916 raft_consensus.cc:1081] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Signalling peer 8a995dc08b2a4497b71db084a2641e6f to start an election
I20260709 12:28:12.157137 12878 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f"
dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
 from {username='slave'} at 127.0.0.1:37324
I20260709 12:28:12.157769 12878 raft_consensus.cc:493] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:12.158180 12878 raft_consensus.cc:3060] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:12.164891 12878 raft_consensus.cc:515] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:12.166927 12878 leader_election.cc:290] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909)
I20260709 12:28:12.167601 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:12.168372 12954 raft_consensus.cc:3055] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:12.168669 12954 raft_consensus.cc:740] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:28:12.168977 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d12b72a433e1403ab5c6d70ccad35e0f" candidate_uuid: "8a995dc08b2a4497b71db084a2641e6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe"
I20260709 12:28:12.169703 13032 raft_consensus.cc:3060] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:12.169565 12954 consensus_queue.cc:260] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:12.170979 12954 raft_consensus.cc:3060] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:12.175199 13032 raft_consensus.cc:2468] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:12.176713 12849 leader_election.cc:304] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, 8a995dc08b2a4497b71db084a2641e6f; no voters: 
I20260709 12:28:12.177313 12954 raft_consensus.cc:2468] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a995dc08b2a4497b71db084a2641e6f in term 2.
I20260709 12:28:12.177614 13987 raft_consensus.cc:2804] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:12.178565 13987 raft_consensus.cc:697] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Becoming Leader. State: Replica: 8a995dc08b2a4497b71db084a2641e6f, State: Running, Role: LEADER
I20260709 12:28:12.179679 13987 consensus_queue.cc:237] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:12.191267 12706 catalog_manager.cc:5697] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f reported cstate change: term changed from 1 to 2, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66). New cstate: current_term: 2 leader_uuid: "8a995dc08b2a4497b71db084a2641e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:12.207131 13927 raft_consensus.cc:993] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:12.207855 13973 raft_consensus.cc:1081] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:12.212692 13798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:47448
I20260709 12:28:12.213446 13798 raft_consensus.cc:493] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:12.213900 13798 raft_consensus.cc:3060] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:12.220170 13798 raft_consensus.cc:515] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:12.222229 13798 leader_election.cc:290] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 3 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:28:12.223599 12878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:28:12.224411 12878 raft_consensus.cc:3060] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:12.223662 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bdcb0dae3adf465cbbded92399c9cb51" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:12.225070 12954 raft_consensus.cc:3055] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:12.225317 12954 raft_consensus.cc:740] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:28:12.226104 12954 consensus_queue.cc:260] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:12.227176 12954 raft_consensus.cc:3060] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:12.231022 12957 raft_consensus.cc:1275] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:12.232837 13996 consensus_queue.cc:1048] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:12.233152 12878 raft_consensus.cc:2468] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 3.
I20260709 12:28:12.234699 13755 leader_election.cc:304] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:12.235821 13996 raft_consensus.cc:2804] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:12.236390 13996 raft_consensus.cc:697] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 3 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:12.237291 13996 consensus_queue.cc:237] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:12.242172 12954 raft_consensus.cc:2468] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 3.
I20260709 12:28:12.259366 13032 raft_consensus.cc:1275] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:12.261065 13929 consensus_queue.cc:1048] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:12.263896 12706 catalog_manager.cc:5697] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 2 to 3, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 3 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:12.287655 13916 raft_consensus.cc:993] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:12.288134 13973 raft_consensus.cc:1081] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:12.290032 13798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ef0476290f24476eae1cbb574a059780"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:47448
I20260709 12:28:12.290607 13798 raft_consensus.cc:493] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:12.290963 13798 raft_consensus.cc:3060] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:12.296314 13798 raft_consensus.cc:515] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:12.299651 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:12.299769 13798 leader_election.cc:290] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 3 election: Requested vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:28:12.300396 12954 raft_consensus.cc:3055] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:12.300746 12954 raft_consensus.cc:740] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:28:12.301744 12878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef0476290f24476eae1cbb574a059780" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:28:12.301597 12954 consensus_queue.cc:260] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:12.302361 12878 raft_consensus.cc:3060] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:12.303053 12954 raft_consensus.cc:3060] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:12.310029 12878 raft_consensus.cc:2468] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 3.
I20260709 12:28:12.310353 12954 raft_consensus.cc:2468] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 3.
I20260709 12:28:12.311342 13755 leader_election.cc:304] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:12.312230 13929 raft_consensus.cc:2804] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:12.312812 13929 raft_consensus.cc:697] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 3 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:12.313691 13929 consensus_queue.cc:237] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:12.325420 12706 catalog_manager.cc:5697] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 2 to 3, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 3 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:12.333397 13032 raft_consensus.cc:1275] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:12.335117 13929 consensus_queue.cc:1048] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:12.356189 12954 raft_consensus.cc:1275] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:12.358361 13996 consensus_queue.cc:1048] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:12.629633 12954 raft_consensus.cc:1275] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:12.631281 13868 consensus_queue.cc:1048] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:12.645246 13032 raft_consensus.cc:1275] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Refusing update from remote peer 8a995dc08b2a4497b71db084a2641e6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:12.647838 14020 consensus_queue.cc:1048] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:12.678421 12881 raft_consensus.cc:1275] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:12.680404 13929 consensus_queue.cc:1048] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:12.701887 12957 raft_consensus.cc:1275] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:12.704140 13929 consensus_queue.cc:1048] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:12.780467 12957 raft_consensus.cc:1275] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:12.781029 12881 raft_consensus.cc:1275] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:12.781983 13996 consensus_queue.cc:1048] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:12.783176 13997 consensus_queue.cc:1048] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:12.852859   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:12.901680 12954 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c"
dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:52958
I20260709 12:28:12.902329 12954 raft_consensus.cc:606] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:12.905555 12954 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329"
dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
mode: GRACEFUL
new_leader_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:52958
I20260709 12:28:12.906242 12954 raft_consensus.cc:606] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Received request to transfer leadership to TS b543f973837c43b19b7d17feaaca5325
I20260709 12:28:12.907596   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:28:12.909197   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:12.910046   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:12.950486 13973 raft_consensus.cc:993] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:12.951220 13927 raft_consensus.cc:1081] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:12.953853 13801 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:47448
I20260709 12:28:12.954553 13801 raft_consensus.cc:493] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:12.955035 13801 raft_consensus.cc:3060] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:12.994491 13801 raft_consensus.cc:515] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:13.011308 12878 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:28:13.013244 13801 leader_election.cc:290] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:28:13.029908 12878 raft_consensus.cc:3060] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:13.046497 12954 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2cbb9487e7d4b9a90f2a4aa06956329" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:13.048200 12954 raft_consensus.cc:3055] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:13.048784 12954 raft_consensus.cc:740] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:28:13.053184 12954 consensus_queue.cc:260] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:13.068759 12954 raft_consensus.cc:3060] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:13.125792 12954 raft_consensus.cc:2468] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:13.133188 13758 leader_election.cc:304] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:13.134191 12878 raft_consensus.cc:2468] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:13.137562 13996 raft_consensus.cc:2804] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:13.143985 13996 raft_consensus.cc:697] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:13.145601 13996 consensus_queue.cc:237] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:13.167914 13973 raft_consensus.cc:993] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5: : Instructing follower b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:13.168712 13973 raft_consensus.cc:1081] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Signalling peer b543f973837c43b19b7d17feaaca5325 to start an election
I20260709 12:28:13.216539 12705 catalog_manager.cc:5697] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:13.220440 13798 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c"
dest_uuid: "b543f973837c43b19b7d17feaaca5325"
 from {username='slave'} at 127.0.0.1:47448
I20260709 12:28:13.238305 13798 raft_consensus.cc:493] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:13.238979 13798 raft_consensus.cc:3060] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:13.291343 12999 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:32861, user_credentials={real_user=slave}} blocked reactor thread for 69143.1us
I20260709 12:28:13.296083 13798 raft_consensus.cc:515] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:13.312150 12956 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8012bb9a8b274890bcaad0d192cd30d5"
I20260709 12:28:13.313098 13798 leader_election.cc:290] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Requested vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:28:13.315480 12881 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8e3f59c4b36b4b49925eaa8d5a4bbf8c" candidate_uuid: "b543f973837c43b19b7d17feaaca5325" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8a995dc08b2a4497b71db084a2641e6f"
I20260709 12:28:13.316517 12881 raft_consensus.cc:3060] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:13.324200 12956 raft_consensus.cc:3055] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:13.325557 12956 raft_consensus.cc:740] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8012bb9a8b274890bcaad0d192cd30d5, State: Running, Role: LEADER
I20260709 12:28:13.326824 12956 consensus_queue.cc:260] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:13.329131 12956 raft_consensus.cc:3060] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:13.339857 12881 raft_consensus.cc:2468] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:13.342108 13755 leader_election.cc:304] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325; no voters: 
I20260709 12:28:13.343278 13996 raft_consensus.cc:2804] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:13.343910 13996 raft_consensus.cc:697] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Becoming Leader. State: Replica: b543f973837c43b19b7d17feaaca5325, State: Running, Role: LEADER
I20260709 12:28:13.344878 13996 consensus_queue.cc:237] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } }
I20260709 12:28:13.347122 12956 raft_consensus.cc:2468] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b543f973837c43b19b7d17feaaca5325 in term 2.
I20260709 12:28:13.358311 12705 catalog_manager.cc:5697] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 reported cstate change: term changed from 1 to 2, leader changed from 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67) to b543f973837c43b19b7d17feaaca5325 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "b543f973837c43b19b7d17feaaca5325" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:13.648378 12878 raft_consensus.cc:1275] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:13.650125 13996 consensus_queue.cc:1048] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:13.662103 12954 raft_consensus.cc:1275] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:13.669126 13996 consensus_queue.cc:1048] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:13.737464 12878 raft_consensus.cc:1275] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:13.739251 13996 consensus_queue.cc:1048] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:13.752617 12954 raft_consensus.cc:1275] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Refusing update from remote peer b543f973837c43b19b7d17feaaca5325: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:13.755756 14042 consensus_queue.cc:1048] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:14.234643   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:14.281169   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:14.282737   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:14.283535   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:14.328212   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:14.372296   561 tablet_server.cc:179] TabletServer@127.0.140.65:40461 shutting down...
W20260709 12:28:14.399976 12924 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 344 similar messages]
W20260709 12:28:14.406467 12924 consensus_peers.cc:597] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.407702 12924 consensus_peers.cc:597] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.421532 12850 consensus_peers.cc:597] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.431499 12850 consensus_peers.cc:597] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.433882 12998 consensus_peers.cc:597] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.439932   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:14.441375   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.442838   561 raft_consensus.cc:2243] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:28:14.444003 12998 consensus_peers.cc:597] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.445606   561 raft_consensus.cc:2272] T c4eccc8971ad4aacb553dc8e1bb00cd8 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.446949 12998 consensus_peers.cc:597] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.451786   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.453349   561 raft_consensus.cc:2243] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.454142   561 raft_consensus.cc:2272] T 82ca562394a94f40ae84d9607fb885f4 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.458351   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.459349   561 raft_consensus.cc:2243] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.460566   561 raft_consensus.cc:2272] T 38871c668f414df593e0f6d35b1feab7 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.464179   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.465026   561 raft_consensus.cc:2243] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.466486   561 raft_consensus.cc:2272] T 9e1ba3f1907244abbd44d141e09ff65d P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.470671   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.471948   561 raft_consensus.cc:2243] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.473560   561 raft_consensus.cc:2272] T 35361d4194194047849e55975d5a91b9 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.477860   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.478950   561 raft_consensus.cc:2243] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.479473   561 raft_consensus.cc:2272] T 068448e5fa294d199668e58a170cbee3 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.483242   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.484325   561 raft_consensus.cc:2243] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.484992   561 raft_consensus.cc:2272] T 6c22979583614eb8b61240b2fa2424e9 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.487291 12998 consensus_peers.cc:597] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.488744   561 tablet_replica.cc:333] stopping tablet replica
W20260709 12:28:14.488890 12998 consensus_peers.cc:597] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.490013   561 raft_consensus.cc:2243] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.491843   561 raft_consensus.cc:2272] T d74f8401a4854eb2b4fc3bde97c43ff8 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.493916 12850 consensus_peers.cc:597] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.495775   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.496886   561 raft_consensus.cc:2243] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.497576   561 raft_consensus.cc:2272] T 423d51d99d504d708ad4af4d083eba48 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.497699 12850 consensus_peers.cc:597] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.500712   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.501824   561 raft_consensus.cc:2243] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.502491   561 raft_consensus.cc:2272] T 735482fa1588451b93a1735236b1859b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.505779   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.506495   561 raft_consensus.cc:2243] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.507468   561 raft_consensus.cc:2272] T e2cbb9487e7d4b9a90f2a4aa06956329 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.511516   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.512655   561 raft_consensus.cc:2243] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.513280   561 raft_consensus.cc:2272] T af4bf74fe4b6486a83af6b41bb30f373 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.516894   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.517868   561 raft_consensus.cc:2243] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.519152   561 raft_consensus.cc:2272] T 0d4e24d741fa47ce960e848db5967200 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.523078   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.524008   561 raft_consensus.cc:2243] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.524626   561 raft_consensus.cc:2272] T 25fae6bafbf34e888a111d96f9fe73d4 P b543f973837c43b19b7d17feaaca5325 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.524667 12998 consensus_peers.cc:597] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.528177   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.528895   561 raft_consensus.cc:2243] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.529883   561 raft_consensus.cc:2272] T ef0476290f24476eae1cbb574a059780 P b543f973837c43b19b7d17feaaca5325 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.533365   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.534346   561 raft_consensus.cc:2243] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.534924   561 raft_consensus.cc:2272] T 2c6179fd5410440a89ea4ca9c5ef3944 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.537536   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.538228   561 raft_consensus.cc:2243] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.539342   561 raft_consensus.cc:2272] T 81ab261b0dc04756a70faccbf14a33d6 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.542629   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.543514   561 raft_consensus.cc:2243] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.544245   561 raft_consensus.cc:2272] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.544642 12924 consensus_peers.cc:597] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.547851   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.549036   561 raft_consensus.cc:2243] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.550444   561 raft_consensus.cc:2272] T cfad63c176a643f3bf8bba8f61523f87 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.554410   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.555445   561 raft_consensus.cc:2243] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.556094   561 raft_consensus.cc:2272] T fc164dcd3f514a00a186655a9a7b2215 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.557978 12998 consensus_peers.cc:597] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.560072   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.560957   561 raft_consensus.cc:2243] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.561463   561 raft_consensus.cc:2272] T 30cb0580c4514b41b650a6b2d5089f3a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.564739   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.565451   561 raft_consensus.cc:2243] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.566015   561 raft_consensus.cc:2272] T a2b3043eef1a4f98b328054262cfe08c P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.569645   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.570580   561 raft_consensus.cc:2243] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.571290   561 raft_consensus.cc:2272] T afd1573aba344096affc47d60971991c P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.575155   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.576386   561 raft_consensus.cc:2243] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.577608   561 raft_consensus.cc:2272] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.581982   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.582896   561 raft_consensus.cc:2243] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.583567   561 raft_consensus.cc:2272] T 0034a6bef1bf4e119ee686581612f8c0 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.586746   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.587589   561 raft_consensus.cc:2243] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.588105   561 raft_consensus.cc:2272] T 303cfb4dc35a4ac094447486c4419179 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.591341   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.592139   561 raft_consensus.cc:2243] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.592676   561 raft_consensus.cc:2272] T 508c4d341f444ad2867dfb09b3b2a125 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.596042   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.596840   561 raft_consensus.cc:2243] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.597415   561 raft_consensus.cc:2272] T 09c1da90a8da49f6be72a8361eead7df P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.600594   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.601402   561 raft_consensus.cc:2243] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.602566   561 raft_consensus.cc:2272] T dc65da4fd7234bbf96ecc59d58baa5c1 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.605894   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.606716   561 raft_consensus.cc:2243] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.607288   561 raft_consensus.cc:2272] T acdd78885d1044a3a523c14603afcb9a P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.610608   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.611471   561 raft_consensus.cc:2243] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.612260   561 raft_consensus.cc:2272] T c1144634236d42f5b2d3656f6eb55389 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.615504   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.616288   561 raft_consensus.cc:2243] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.616807   561 raft_consensus.cc:2272] T d9b2c51994d34d189bd05462fa15c84b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.619932   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.620738   561 raft_consensus.cc:2243] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.621352   561 raft_consensus.cc:2272] T fd444f6f267d4255873e372d6e3e3a43 P b543f973837c43b19b7d17feaaca5325 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.624670   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.625542   561 raft_consensus.cc:2243] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.626129   561 raft_consensus.cc:2272] T ff6968296a8740f4b8c329c728a64b1b P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.629185   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.629714   561 raft_consensus.cc:2243] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.630141   561 raft_consensus.cc:2272] T cc6c3b7e72b044a5b0f0806e9a6301cf P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.632396   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.633018   561 raft_consensus.cc:2243] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.633822   561 raft_consensus.cc:2272] T ca89f605f7bb4bd9acc811382ac82591 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.637934   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.638578   561 raft_consensus.cc:2243] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.639073   561 raft_consensus.cc:2272] T b32a14229832492682674ac4c282404d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.641803   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.642455   561 raft_consensus.cc:2243] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.643651   561 raft_consensus.cc:2272] T bdcb0dae3adf465cbbded92399c9cb51 P b543f973837c43b19b7d17feaaca5325 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.646787   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.647608   561 raft_consensus.cc:2243] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:14.647672 12998 consensus_peers.cc:597] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.648087   561 raft_consensus.cc:2272] T 56b78cc7f12d4c3a81aed3af33111913 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.650733   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.651504   561 raft_consensus.cc:2243] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.652043   561 raft_consensus.cc:2272] T b0eea117b500470093718b8583fa5e96 P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.655382   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.656142   561 raft_consensus.cc:2243] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.656678   561 raft_consensus.cc:2272] T a0e539dd685147a899e29016abb08e1b P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.657262 12850 consensus_peers.cc:597] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.659933   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.660880   561 raft_consensus.cc:2243] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.661412   561 raft_consensus.cc:2272] T 8d24c0921438489bb6c87bd7b1b0654d P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.664428   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.665184   561 raft_consensus.cc:2243] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.665789   561 raft_consensus.cc:2272] T 224f359a7c884a7890b05f3098408d31 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.666359 12998 consensus_peers.cc:597] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.667197 12998 consensus_peers.cc:597] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.669389   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.670166   561 raft_consensus.cc:2243] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.670645   561 raft_consensus.cc:2272] T c4c88c0aa80544daa7bae9c5ac62b0fa P b543f973837c43b19b7d17feaaca5325 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.673794   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:14.674439   561 raft_consensus.cc:2243] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.675745   561 raft_consensus.cc:2272] T f7237e470a3048a8878e2d0d49208ee1 P b543f973837c43b19b7d17feaaca5325 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.679035 12924 consensus_peers.cc:597] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.692890 12850 consensus_peers.cc:597] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.703270 12924 consensus_peers.cc:597] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.707223 12924 consensus_peers.cc:597] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.747134 12924 consensus_peers.cc:597] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.748553 12924 consensus_peers.cc:597] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.764715   561 tablet_server.cc:196] TabletServer@127.0.140.65:40461 shutdown complete.
W20260709 12:28:14.790303 12924 consensus_peers.cc:597] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.819615 12850 consensus_peers.cc:597] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.860718   561 tablet_server.cc:179] TabletServer@127.0.140.66:0 shutting down...
W20260709 12:28:14.879581 12996 consensus_peers.cc:597] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.896507 12922 consensus_peers.cc:597] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.900104 12922 consensus_peers.cc:597] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.908313 12924 consensus_peers.cc:597] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.913983 12996 consensus_peers.cc:597] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.934835 12924 consensus_peers.cc:597] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.936095 12922 consensus_peers.cc:597] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:14.947252 12998 consensus_peers.cc:597] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.954622   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:14.956262   561 tablet_replica.cc:333] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:14.957424   561 raft_consensus.cc:2243] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:14.958365 12850 consensus_peers.cc:597] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.958529   561 raft_consensus.cc:2272] T dc65da4fd7234bbf96ecc59d58baa5c1 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.961030 12996 consensus_peers.cc:597] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.963070   561 tablet_replica.cc:333] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:14.963960   561 raft_consensus.cc:2243] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.964571   561 raft_consensus.cc:2272] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.965653 12996 consensus_peers.cc:597] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.967305   561 tablet_replica.cc:333] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:14.968133   561 raft_consensus.cc:2243] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.968662   561 raft_consensus.cc:2272] T bf54d438996844a1b6028dabd88f727e P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.971274   561 tablet_replica.cc:333] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:14.971997   561 raft_consensus.cc:2243] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.972503   561 raft_consensus.cc:2272] T ed10b80d23ae40019ba46ed7087ca05a P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.976850   561 tablet_replica.cc:333] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:14.977746   561 raft_consensus.cc:2243] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:14.978406   561 raft_consensus.cc:2272] T 7d28ffbbc352420bb35bfab9e1847e67 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:14.981820 12924 consensus_peers.cc:597] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Couldn't send request to peer b543f973837c43b19b7d17feaaca5325. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:14.982053   561 tablet_replica.cc:333] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:14.983075   561 raft_consensus.cc:2243] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.984037   561 raft_consensus.cc:2272] T 423d51d99d504d708ad4af4d083eba48 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.986673   561 tablet_replica.cc:333] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:14.987444   561 raft_consensus.cc:2243] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.988364   561 raft_consensus.cc:2272] T 068448e5fa294d199668e58a170cbee3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.990999   561 tablet_replica.cc:333] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:14.991765   561 raft_consensus.cc:2243] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:14.993090   561 raft_consensus.cc:2272] T 0034a6bef1bf4e119ee686581612f8c0 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:14.997217   561 tablet_replica.cc:333] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:14.998287   561 raft_consensus.cc:2243] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:28:14.999886 12922 consensus_peers.cc:597] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.000733   561 raft_consensus.cc:2272] T c1144634236d42f5b2d3656f6eb55389 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.005285   561 tablet_replica.cc:333] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.006283   561 raft_consensus.cc:2243] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.006896   561 raft_consensus.cc:2272] T d9b2c51994d34d189bd05462fa15c84b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.010447   561 tablet_replica.cc:333] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.011412   561 raft_consensus.cc:2243] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.012439   561 raft_consensus.cc:2272] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.015547   561 tablet_replica.cc:333] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.016297   561 raft_consensus.cc:2243] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.016808   561 raft_consensus.cc:2272] T b0eea117b500470093718b8583fa5e96 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.020499   561 tablet_replica.cc:333] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.021510   561 raft_consensus.cc:2243] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.022173   561 raft_consensus.cc:2272] T b32a14229832492682674ac4c282404d P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.024221 12996 consensus_peers.cc:597] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.027719   561 tablet_replica.cc:333] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
W20260709 12:28:15.028570 12996 consensus_peers.cc:597] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.028827   561 raft_consensus.cc:2243] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.029732   561 raft_consensus.cc:2272] T 8d24c0921438489bb6c87bd7b1b0654d P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.032961   561 tablet_replica.cc:333] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.033802   561 raft_consensus.cc:2243] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.034456   561 raft_consensus.cc:2272] T 0d4e24d741fa47ce960e848db5967200 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.038200   561 tablet_replica.cc:333] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.039018   561 raft_consensus.cc:2243] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:15.039300 12922 consensus_peers.cc:597] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.039774   561 raft_consensus.cc:2272] T bdcb0dae3adf465cbbded92399c9cb51 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.043879   561 tablet_replica.cc:333] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.044916   561 raft_consensus.cc:2243] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.045647   561 raft_consensus.cc:2272] T 81eacfb947b741a2a11b5b645c586003 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.049566   561 tablet_replica.cc:333] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.050858   561 raft_consensus.cc:2243] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.051767   561 raft_consensus.cc:2272] T f7237e470a3048a8878e2d0d49208ee1 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.054312 12996 consensus_peers.cc:597] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.055519   561 tablet_replica.cc:333] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.056394   561 raft_consensus.cc:2243] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.057428   561 raft_consensus.cc:2272] T 3ca75f6a5016474a844f5532614ebc75 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.061250   561 tablet_replica.cc:333] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.062314   561 raft_consensus.cc:2243] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.063055   561 raft_consensus.cc:2272] T a2b3043eef1a4f98b328054262cfe08c P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.066726   561 tablet_replica.cc:333] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.067664   561 raft_consensus.cc:2243] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.068941   561 raft_consensus.cc:2272] T 4f810e611ff745b2a7fa598c418b93db P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.073010   561 tablet_replica.cc:333] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.073948   561 raft_consensus.cc:2243] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.074612   561 raft_consensus.cc:2272] T cc6c3b7e72b044a5b0f0806e9a6301cf P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.078253   561 tablet_replica.cc:333] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.079243   561 raft_consensus.cc:2243] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.080878   561 raft_consensus.cc:2272] T ca89f605f7bb4bd9acc811382ac82591 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.083489 12922 consensus_peers.cc:597] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.084470   561 tablet_replica.cc:333] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.085598   561 raft_consensus.cc:2243] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.086350   561 raft_consensus.cc:2272] T ef0476290f24476eae1cbb574a059780 P 8a995dc08b2a4497b71db084a2641e6f [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.087396 12922 consensus_peers.cc:597] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.089970   561 tablet_replica.cc:333] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.091112   561 raft_consensus.cc:2243] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.092363   561 raft_consensus.cc:2272] T 25fae6bafbf34e888a111d96f9fe73d4 P 8a995dc08b2a4497b71db084a2641e6f [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.093253 12922 consensus_peers.cc:597] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.095954   561 tablet_replica.cc:333] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.096923   561 raft_consensus.cc:2243] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.097510   561 raft_consensus.cc:2272] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.100737   561 tablet_replica.cc:333] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.101661   561 raft_consensus.cc:2243] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.102272   561 raft_consensus.cc:2272] T 35361d4194194047849e55975d5a91b9 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.105429   561 tablet_replica.cc:333] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.106212   561 raft_consensus.cc:2243] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.106768   561 raft_consensus.cc:2272] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.109977   561 tablet_replica.cc:333] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.110774   561 raft_consensus.cc:2243] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.112234   561 raft_consensus.cc:2272] T 070b536983ee48e1853040e1652619e2 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.115538   561 tablet_replica.cc:333] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.116456   561 raft_consensus.cc:2243] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.117045   561 raft_consensus.cc:2272] T 6c22979583614eb8b61240b2fa2424e9 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.116938 12922 consensus_peers.cc:597] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.120879   561 tablet_replica.cc:333] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.121783   561 raft_consensus.cc:2243] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.122393   561 raft_consensus.cc:2272] T 82ca562394a94f40ae84d9607fb885f4 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.125025   561 tablet_replica.cc:333] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.125705   561 raft_consensus.cc:2243] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.126310   561 raft_consensus.cc:2272] T f9b4619b6db04b159a1d568ba006bdb4 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.129738   561 tablet_replica.cc:333] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.130568   561 raft_consensus.cc:2243] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.131819   561 raft_consensus.cc:2272] T d12b72a433e1403ab5c6d70ccad35e0f P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.135365   561 tablet_replica.cc:333] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.136299   561 raft_consensus.cc:2243] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.137590   561 raft_consensus.cc:2272] T 34ac366ceb574dd18638a5dcbde9b2de P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.140839   561 tablet_replica.cc:333] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.141518   561 raft_consensus.cc:2243] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.142180   561 raft_consensus.cc:2272] T 30cb0580c4514b41b650a6b2d5089f3a P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.145336   561 tablet_replica.cc:333] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.147315   561 raft_consensus.cc:2243] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.147962   561 raft_consensus.cc:2272] T fc164dcd3f514a00a186655a9a7b2215 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.153182   561 tablet_replica.cc:333] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.154031   561 raft_consensus.cc:2243] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.154704   561 raft_consensus.cc:2272] T 81ab261b0dc04756a70faccbf14a33d6 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.157981   561 tablet_replica.cc:333] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.158862   561 raft_consensus.cc:2243] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.159468   561 raft_consensus.cc:2272] T f40e0b80665048e09e181f1a8ea7509a P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.162453   561 tablet_replica.cc:333] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.163332   561 raft_consensus.cc:2243] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.163928   561 raft_consensus.cc:2272] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.167003   561 tablet_replica.cc:333] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
W20260709 12:28:15.167032 12996 consensus_peers.cc:597] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.167893   561 raft_consensus.cc:2243] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.168453   561 raft_consensus.cc:2272] T e643b9d35a9f4515abe259109274d191 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.171221   561 tablet_replica.cc:333] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.172073   561 raft_consensus.cc:2243] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.173143   561 raft_consensus.cc:2272] T eae9dd87d90f4d18b0d36431646603e3 P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.175765   561 tablet_replica.cc:333] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.176337   561 raft_consensus.cc:2243] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.176877   561 raft_consensus.cc:2272] T a0e539dd685147a899e29016abb08e1b P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.179401   561 tablet_replica.cc:333] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.179939   561 raft_consensus.cc:2243] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.180846   561 raft_consensus.cc:2272] T 224f359a7c884a7890b05f3098408d31 P 8a995dc08b2a4497b71db084a2641e6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.183178   561 tablet_replica.cc:333] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f: stopping tablet replica
I20260709 12:28:15.183704   561 raft_consensus.cc:2243] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.184680   561 raft_consensus.cc:2272] T 735482fa1588451b93a1735236b1859b P 8a995dc08b2a4497b71db084a2641e6f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.233937 12922 consensus_peers.cc:597] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:15.272569 12996 consensus_peers.cc:597] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:38975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.278010   561 tablet_server.cc:196] TabletServer@127.0.140.66:0 shutdown complete.
W20260709 12:28:15.298359 12922 consensus_peers.cc:597] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 -> Peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Couldn't send request to peer 8a995dc08b2a4497b71db084a2641e6f. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:38975: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.391453   561 tablet_server.cc:179] TabletServer@127.0.140.67:0 shutting down...
W20260709 12:28:15.472952 12999 consensus_peers.cc:597] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.477579 14054 raft_consensus.cc:493] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.478296 14054 raft_consensus.cc:515] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:15.479447 14055 raft_consensus.cc:493] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.480221 14055 raft_consensus.cc:515] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:15.480988 14054 leader_election.cc:290] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:28:15.484258 13027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38871c668f414df593e0f6d35b1feab7" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
W20260709 12:28:15.487957 12922 leader_election.cc:336] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Network error: Client connection negotiation failed: client connection to 127.0.140.66:38975: connect: Connection refused (error 111)
W20260709 12:28:15.489408 12924 leader_election.cc:336] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.489871 12924 leader_election.cc:304] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.490583 14054 raft_consensus.cc:2749] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:28:15.500573 12924 leader_election.cc:336] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.501068 12924 leader_election.cc:304] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.503033 14029 raft_consensus.cc:2749] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.484434 14055 leader_election.cc:290] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:28:15.512625 12999 consensus_peers.cc:597] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.515223   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:15.517246   561 tablet_replica.cc:333] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.519496   561 raft_consensus.cc:2243] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.520709   561 raft_consensus.cc:2272] T bf54d438996844a1b6028dabd88f727e P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.523661 12999 consensus_peers.cc:597] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.523731   561 tablet_replica.cc:333] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.525395   561 raft_consensus.cc:2243] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.526043   561 raft_consensus.cc:2272] T 068448e5fa294d199668e58a170cbee3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.529278   561 tablet_replica.cc:333] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.530149   561 raft_consensus.cc:2243] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.530761   561 raft_consensus.cc:2272] T ff6968296a8740f4b8c329c728a64b1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.533792   561 tablet_replica.cc:333] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.534432   561 raft_consensus.cc:2243] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.534992   561 raft_consensus.cc:2272] T fd444f6f267d4255873e372d6e3e3a43 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.537658   561 tablet_replica.cc:333] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.538442   561 raft_consensus.cc:2243] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.539038   561 raft_consensus.cc:2272] T ed10b80d23ae40019ba46ed7087ca05a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.541905   561 tablet_replica.cc:333] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.542510   561 raft_consensus.cc:2243] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.543079   561 raft_consensus.cc:2272] T 38871c668f414df593e0f6d35b1feab7 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.545208   561 tablet_replica.cc:333] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.545974   561 raft_consensus.cc:2243] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.546952   561 raft_consensus.cc:2272] T d9b2c51994d34d189bd05462fa15c84b P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.549140   561 tablet_replica.cc:333] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.549834   561 raft_consensus.cc:2243] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.550446   561 raft_consensus.cc:2272] T 9cf46416458c4a9398e8de1c6b8ccb5d P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.554067   561 tablet_replica.cc:333] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.554889   561 raft_consensus.cc:2243] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
W20260709 12:28:15.555183 12999 consensus_peers.cc:597] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.556025   561 raft_consensus.cc:2272] T b32a14229832492682674ac4c282404d P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.558931   561 tablet_replica.cc:333] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.559590   561 raft_consensus.cc:2243] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.560086   561 raft_consensus.cc:2272] T 9e1ba3f1907244abbd44d141e09ff65d P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.562750   561 tablet_replica.cc:333] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.563580   561 raft_consensus.cc:2243] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.564193   561 raft_consensus.cc:2272] T e2cbb9487e7d4b9a90f2a4aa06956329 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.567128   561 tablet_replica.cc:333] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.567938   561 raft_consensus.cc:2243] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.568598   561 raft_consensus.cc:2272] T af4bf74fe4b6486a83af6b41bb30f373 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.569444 14054 raft_consensus.cc:493] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.569999 14054 raft_consensus.cc:515] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:15.572307 14054 leader_election.cc:290] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 18cf4829b3bc4fafb3ef7f1b98f853fe (127.0.140.68:32909), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:28:15.571848   561 tablet_replica.cc:333] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.573088 13032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d74f8401a4854eb2b4fc3bde97c43ff8" candidate_uuid: "8012bb9a8b274890bcaad0d192cd30d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" is_pre_election: true
I20260709 12:28:15.574060   561 raft_consensus.cc:2243] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.574608   561 raft_consensus.cc:2272] T 303cfb4dc35a4ac094447486c4419179 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.578001   561 tablet_replica.cc:333] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.578850   561 raft_consensus.cc:2243] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.579960   561 raft_consensus.cc:2272] T 508c4d341f444ad2867dfb09b3b2a125 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.582360 12924 leader_election.cc:336] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.582872 12924 leader_election.cc:304] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8012bb9a8b274890bcaad0d192cd30d5; no voters: 18cf4829b3bc4fafb3ef7f1b98f853fe, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.583083   561 tablet_replica.cc:333] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.583614 14054 raft_consensus.cc:2749] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.583977   561 raft_consensus.cc:2243] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.584995   561 raft_consensus.cc:2272] T c4c88c0aa80544daa7bae9c5ac62b0fa P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.588233   561 tablet_replica.cc:333] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.588958   561 raft_consensus.cc:2243] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.589509   561 raft_consensus.cc:2272] T bdcb0dae3adf465cbbded92399c9cb51 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.592567   561 tablet_replica.cc:333] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.593351   561 raft_consensus.cc:2243] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.594316   561 raft_consensus.cc:2272] T 81eacfb947b741a2a11b5b645c586003 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.597195   561 tablet_replica.cc:333] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.598062   561 raft_consensus.cc:2243] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.598634   561 raft_consensus.cc:2272] T f7237e470a3048a8878e2d0d49208ee1 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.600776   561 tablet_replica.cc:333] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.601279   561 raft_consensus.cc:2243] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.601678   561 raft_consensus.cc:2272] T 3ca75f6a5016474a844f5532614ebc75 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.603185 13992 raft_consensus.cc:493] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.604049   561 tablet_replica.cc:333] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.604031 13992 raft_consensus.cc:515] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:15.604871   561 raft_consensus.cc:2243] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.605865 14058 raft_consensus.cc:493] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.606034   561 raft_consensus.cc:2272] T afd1573aba344096affc47d60971991c P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.606554 14058 raft_consensus.cc:515] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:15.607337 13992 leader_election.cc:290] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:28:15.609265 14058 leader_election.cc:290] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
I20260709 12:28:15.609472   561 tablet_replica.cc:333] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.610400   561 raft_consensus.cc:2243] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.611032   561 raft_consensus.cc:2272] T 4f810e611ff745b2a7fa598c418b93db P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.611749 12999 consensus_peers.cc:597] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:15.612681 12998 leader_election.cc:336] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.614060   561 tablet_replica.cc:333] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.615005   561 raft_consensus.cc:2243] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.615846   561 raft_consensus.cc:2272] T acdd78885d1044a3a523c14603afcb9a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.616122 12996 leader_election.cc:336] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Network error: Client connection negotiation failed: client connection to 127.0.140.66:38975: connect: Connection refused (error 111)
W20260709 12:28:15.616429 12998 leader_election.cc:336] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.616580 12996 leader_election.cc:304] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.617506 14058 raft_consensus.cc:2749] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:28:15.617679 12996 leader_election.cc:336] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Network error: Client connection negotiation failed: client connection to 127.0.140.66:38975: connect: Connection refused (error 111)
I20260709 12:28:15.618171 12996 leader_election.cc:304] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.618736   561 tablet_replica.cc:333] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.618901 14058 raft_consensus.cc:2749] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.619694   561 raft_consensus.cc:2243] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.620239   561 raft_consensus.cc:2272] T d12b72a433e1403ab5c6d70ccad35e0f P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.622395   561 tablet_replica.cc:333] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.623044   561 raft_consensus.cc:2243] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.623517   561 raft_consensus.cc:2272] T 09c1da90a8da49f6be72a8361eead7df P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.625545   561 tablet_replica.cc:333] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.626219   561 raft_consensus.cc:2243] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.626688   561 raft_consensus.cc:2272] T 070b536983ee48e1853040e1652619e2 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.628902   561 tablet_replica.cc:333] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.629518   561 raft_consensus.cc:2243] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.629988   561 raft_consensus.cc:2272] T 8e3f59c4b36b4b49925eaa8d5a4bbf8c P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.632784   561 tablet_replica.cc:333] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.633606   561 raft_consensus.cc:2243] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.634234   561 raft_consensus.cc:2272] T 7d28ffbbc352420bb35bfab9e1847e67 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.636479   561 tablet_replica.cc:333] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.637080   561 raft_consensus.cc:2243] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.637769   561 raft_consensus.cc:2272] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.640115   561 tablet_replica.cc:333] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.640666   561 raft_consensus.cc:2243] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.641491   561 raft_consensus.cc:2272] T 82ca562394a94f40ae84d9607fb885f4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.643968   561 tablet_replica.cc:333] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.644582   561 raft_consensus.cc:2243] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.644747 14058 raft_consensus.cc:493] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.645540 14058 raft_consensus.cc:515] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:15.646026   561 raft_consensus.cc:2272] T f9b4619b6db04b159a1d568ba006bdb4 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.649830   561 tablet_replica.cc:333] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.650748   561 raft_consensus.cc:2243] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.651194 14058 leader_election.cc:290] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
I20260709 12:28:15.651371   561 raft_consensus.cc:2272] T 34ac366ceb574dd18638a5dcbde9b2de P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.653475 12999 leader_election.cc:336] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:28:15.657286 12998 leader_election.cc:336] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.657811 12998 leader_election.cc:304] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.657310   561 tablet_replica.cc:333] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.658591 14058 raft_consensus.cc:2749] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.659451   561 raft_consensus.cc:2243] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.660112   561 raft_consensus.cc:2272] T d74f8401a4854eb2b4fc3bde97c43ff8 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.663447   561 tablet_replica.cc:333] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.664306   561 raft_consensus.cc:2243] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.665361   561 raft_consensus.cc:2272] T 6c22979583614eb8b61240b2fa2424e9 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.667899   561 tablet_replica.cc:333] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.668525   561 raft_consensus.cc:2243] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.669066   561 raft_consensus.cc:2272] T 56b78cc7f12d4c3a81aed3af33111913 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.671716   561 tablet_replica.cc:333] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.672564   561 raft_consensus.cc:2243] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.673161   561 raft_consensus.cc:2272] T f40e0b80665048e09e181f1a8ea7509a P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.676215   561 tablet_replica.cc:333] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.677060   561 raft_consensus.cc:2243] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.678102   561 raft_consensus.cc:2272] T 2c6179fd5410440a89ea4ca9c5ef3944 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.680740   561 tablet_replica.cc:333] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.681331   561 raft_consensus.cc:2243] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.681792   561 raft_consensus.cc:2272] T e643b9d35a9f4515abe259109274d191 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.684294   561 tablet_replica.cc:333] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.684929   561 raft_consensus.cc:2243] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.685388   561 raft_consensus.cc:2272] T eae9dd87d90f4d18b0d36431646603e3 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.687772   561 tablet_replica.cc:333] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.688387   561 raft_consensus.cc:2243] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.688894   561 raft_consensus.cc:2272] T 224f359a7c884a7890b05f3098408d31 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.691319   561 tablet_replica.cc:333] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.691910   561 raft_consensus.cc:2243] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.692780   561 raft_consensus.cc:2272] T a0e539dd685147a899e29016abb08e1b P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.695005   561 tablet_replica.cc:333] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.695638   561 raft_consensus.cc:2243] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.696668   561 raft_consensus.cc:2272] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.699523   561 tablet_replica.cc:333] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
W20260709 12:28:15.699864 12999 consensus_peers.cc:597] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.700300   561 raft_consensus.cc:2243] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.701210   561 raft_consensus.cc:2272] T fc164dcd3f514a00a186655a9a7b2215 P 8012bb9a8b274890bcaad0d192cd30d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.703588   561 tablet_replica.cc:333] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
I20260709 12:28:15.704203   561 raft_consensus.cc:2243] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.704766   561 raft_consensus.cc:2272] T cfad63c176a643f3bf8bba8f61523f87 P 8012bb9a8b274890bcaad0d192cd30d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.707362   561 tablet_replica.cc:333] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5: stopping tablet replica
W20260709 12:28:15.707409 12999 consensus_peers.cc:597] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.707988   561 raft_consensus.cc:2243] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.708566   561 raft_consensus.cc:2272] T ef0476290f24476eae1cbb574a059780 P 8012bb9a8b274890bcaad0d192cd30d5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.725900 14058 raft_consensus.cc:493] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.726511 14058 raft_consensus.cc:515] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:15.729003 14058 leader_election.cc:290] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:28:15.732651 12999 leader_election.cc:336] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:28:15.734524 12998 leader_election.cc:336] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.735057 12998 leader_election.cc:304] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.735826 14058 raft_consensus.cc:2749] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.739063 14058 raft_consensus.cc:493] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.739729 14058 raft_consensus.cc:515] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:15.742179 14058 leader_election.cc:290] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:28:15.746258 12999 leader_election.cc:336] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:28:15.747730 12998 leader_election.cc:336] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.748250 12998 leader_election.cc:304] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.749087 14058 raft_consensus.cc:2749] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.799343   561 tablet_server.cc:196] TabletServer@127.0.140.67:0 shutdown complete.
W20260709 12:28:15.800830 12999 consensus_peers.cc:597] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:32861: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.800834 14058 raft_consensus.cc:493] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
W20260709 12:28:15.801819 12999 consensus_peers.cc:597] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:32861: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.801971 14058 raft_consensus.cc:515] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:15.804533 14058 leader_election.cc:290] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:28:15.809324 12998 leader_election.cc:336] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
W20260709 12:28:15.810051 12996 leader_election.cc:336] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Network error: Client connection negotiation failed: client connection to 127.0.140.66:38975: connect: Connection refused (error 111)
I20260709 12:28:15.810612 12996 leader_election.cc:304] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.811484 14058 raft_consensus.cc:2749] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:28:15.859735 12999 consensus_peers.cc:597] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe -> Peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Couldn't send request to peer 8012bb9a8b274890bcaad0d192cd30d5. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.67:32861: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:15.865617 14058 raft_consensus.cc:493] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.866456 14058 raft_consensus.cc:515] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } } peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } }
I20260709 12:28:15.868956 14058 leader_election.cc:290] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461), 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975)
W20260709 12:28:15.874624 12996 leader_election.cc:336] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Network error: Client connection negotiation failed: client connection to 127.0.140.66:38975: connect: Connection refused (error 111)
W20260709 12:28:15.875675 12998 leader_election.cc:336] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.875772 14058 raft_consensus.cc:493] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.876351 12998 leader_election.cc:304] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8a995dc08b2a4497b71db084a2641e6f, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.877172 14048 raft_consensus.cc:2749] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.877082 14058 raft_consensus.cc:515] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
I20260709 12:28:15.880158 14058 leader_election.cc:290] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:28:15.884337 12999 leader_election.cc:336] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Network error: Client connection negotiation failed: client connection to 127.0.140.67:32861: connect: Connection refused (error 111)
I20260709 12:28:15.885233 14058 raft_consensus.cc:493] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b543f973837c43b19b7d17feaaca5325)
I20260709 12:28:15.885989 14058 raft_consensus.cc:515] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } } peers { permanent_uuid: "b543f973837c43b19b7d17feaaca5325" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 40461 } }
W20260709 12:28:15.886129 12998 leader_election.cc:336] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.886831 12998 leader_election.cc:304] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.887740 14048 raft_consensus.cc:2749] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.888422 14058 leader_election.cc:290] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861), b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461)
W20260709 12:28:15.895964 12998 leader_election.cc:336] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b543f973837c43b19b7d17feaaca5325 (127.0.140.65:40461): Network error: Client connection negotiation failed: client connection to 127.0.140.65:40461: connect: Connection refused (error 111)
I20260709 12:28:15.896322   561 tablet_server.cc:179] TabletServer@127.0.140.68:0 shutting down...
W20260709 12:28:15.896564 12999 leader_election.cc:336] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Network error: Client connection negotiation failed: client connection to 127.0.140.67:32861: connect: Connection refused (error 111)
I20260709 12:28:15.898882 12999 leader_election.cc:304] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, b543f973837c43b19b7d17feaaca5325
I20260709 12:28:15.900585 14058 raft_consensus.cc:2749] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.950364   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:15.951512   561 tablet_replica.cc:333] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.952626   561 raft_consensus.cc:2243] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.953357   561 raft_consensus.cc:2272] T bf54d438996844a1b6028dabd88f727e P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.956591   561 tablet_replica.cc:333] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.957337   561 raft_consensus.cc:2243] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.957855   561 raft_consensus.cc:2272] T 423d51d99d504d708ad4af4d083eba48 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.961187   561 tablet_replica.cc:333] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.962272   561 raft_consensus.cc:2243] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.963558   561 raft_consensus.cc:2272] T ed10b80d23ae40019ba46ed7087ca05a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.966995   561 tablet_replica.cc:333] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.967712   561 raft_consensus.cc:2243] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.968914   561 raft_consensus.cc:2272] T ff6968296a8740f4b8c329c728a64b1b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.969625 14058 raft_consensus.cc:493] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8a995dc08b2a4497b71db084a2641e6f)
I20260709 12:28:15.970427 14058 raft_consensus.cc:515] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a995dc08b2a4497b71db084a2641e6f" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 38975 } } peers { permanent_uuid: "8012bb9a8b274890bcaad0d192cd30d5" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 32861 } } peers { permanent_uuid: "18cf4829b3bc4fafb3ef7f1b98f853fe" member_type: VOTER last_known_addr { host: "127.0.140.68" port: 32909 } }
I20260709 12:28:15.971743   561 tablet_replica.cc:333] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.972530   561 raft_consensus.cc:2243] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:15.972975 14058 leader_election.cc:290] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975), 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861)
I20260709 12:28:15.973526   561 raft_consensus.cc:2272] T fd444f6f267d4255873e372d6e3e3a43 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.976814   561 tablet_replica.cc:333] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.977800   561 raft_consensus.cc:2243] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:15.978291 12999 leader_election.cc:336] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8012bb9a8b274890bcaad0d192cd30d5 (127.0.140.67:32861): Network error: Client connection negotiation failed: client connection to 127.0.140.67:32861: connect: Connection refused (error 111)
I20260709 12:28:15.978381   561 raft_consensus.cc:2272] T 0034a6bef1bf4e119ee686581612f8c0 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:15.979521 12996 leader_election.cc:336] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8a995dc08b2a4497b71db084a2641e6f (127.0.140.66:38975): Network error: Client connection negotiation failed: client connection to 127.0.140.66:38975: connect: Connection refused (error 111)
I20260709 12:28:15.980026 12996 leader_election.cc:304] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 18cf4829b3bc4fafb3ef7f1b98f853fe; no voters: 8012bb9a8b274890bcaad0d192cd30d5, 8a995dc08b2a4497b71db084a2641e6f
I20260709 12:28:15.980937 14058 raft_consensus.cc:2749] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:28:15.983086   561 tablet_replica.cc:333] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.984094   561 raft_consensus.cc:2243] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.984782   561 raft_consensus.cc:2272] T 38871c668f414df593e0f6d35b1feab7 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.987429   561 tablet_replica.cc:333] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.988111   561 raft_consensus.cc:2243] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.988569   561 raft_consensus.cc:2272] T c1144634236d42f5b2d3656f6eb55389 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.991541   561 tablet_replica.cc:333] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.992472   561 raft_consensus.cc:2243] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.993112   561 raft_consensus.cc:2272] T 9cf46416458c4a9398e8de1c6b8ccb5d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:15.996520   561 tablet_replica.cc:333] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:15.997319   561 raft_consensus.cc:2243] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:15.997872   561 raft_consensus.cc:2272] T 81eacfb947b741a2a11b5b645c586003 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.000855   561 tablet_replica.cc:333] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.001829   561 raft_consensus.cc:2243] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.003247   561 raft_consensus.cc:2272] T af4bf74fe4b6486a83af6b41bb30f373 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.005942   561 tablet_replica.cc:333] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.006660   561 raft_consensus.cc:2243] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.007687   561 raft_consensus.cc:2272] T 8d24c0921438489bb6c87bd7b1b0654d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.011108   561 tablet_replica.cc:333] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.012182   561 raft_consensus.cc:2243] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.013247   561 raft_consensus.cc:2272] T 303cfb4dc35a4ac094447486c4419179 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.016238   561 tablet_replica.cc:333] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.016954   561 raft_consensus.cc:2243] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.017541   561 raft_consensus.cc:2272] T 508c4d341f444ad2867dfb09b3b2a125 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.020648   561 tablet_replica.cc:333] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.021376   561 raft_consensus.cc:2243] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.021971   561 raft_consensus.cc:2272] T 0d4e24d741fa47ce960e848db5967200 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.024501   561 tablet_replica.cc:333] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.025169   561 raft_consensus.cc:2243] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.025645   561 raft_consensus.cc:2272] T 34ac366ceb574dd18638a5dcbde9b2de P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.028539   561 tablet_replica.cc:333] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.029145   561 raft_consensus.cc:2243] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.029623   561 raft_consensus.cc:2272] T 3ca75f6a5016474a844f5532614ebc75 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.032527   561 tablet_replica.cc:333] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.033375   561 raft_consensus.cc:2243] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.034233   561 raft_consensus.cc:2272] T afd1573aba344096affc47d60971991c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.037539   561 tablet_replica.cc:333] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.038295   561 raft_consensus.cc:2243] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.039392   561 raft_consensus.cc:2272] T a2b3043eef1a4f98b328054262cfe08c P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.042160   561 tablet_replica.cc:333] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.042757   561 raft_consensus.cc:2243] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.043351   561 raft_consensus.cc:2272] T 4f810e611ff745b2a7fa598c418b93db P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.045737   561 tablet_replica.cc:333] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.046370   561 raft_consensus.cc:2243] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.046876   561 raft_consensus.cc:2272] T d74f8401a4854eb2b4fc3bde97c43ff8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.049407   561 tablet_replica.cc:333] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.049974   561 raft_consensus.cc:2243] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.050391   561 raft_consensus.cc:2272] T dc65da4fd7234bbf96ecc59d58baa5c1 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.052812   561 tablet_replica.cc:333] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.053391   561 raft_consensus.cc:2243] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.053813   561 raft_consensus.cc:2272] T acdd78885d1044a3a523c14603afcb9a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.056033   561 tablet_replica.cc:333] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.056595   561 raft_consensus.cc:2243] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.057034   561 raft_consensus.cc:2272] T 9e1ba3f1907244abbd44d141e09ff65d P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.059813   561 tablet_replica.cc:333] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.060465   561 raft_consensus.cc:2243] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.060950   561 raft_consensus.cc:2272] T 35361d4194194047849e55975d5a91b9 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.063598   561 tablet_replica.cc:333] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.064224   561 raft_consensus.cc:2243] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.065337   561 raft_consensus.cc:2272] T cc6c3b7e72b044a5b0f0806e9a6301cf P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.067813   561 tablet_replica.cc:333] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.068408   561 raft_consensus.cc:2243] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.068858   561 raft_consensus.cc:2272] T ca89f605f7bb4bd9acc811382ac82591 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.071206   561 tablet_replica.cc:333] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.071870   561 raft_consensus.cc:2243] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.072358   561 raft_consensus.cc:2272] T 25fae6bafbf34e888a111d96f9fe73d4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.075021   561 tablet_replica.cc:333] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.075687   561 raft_consensus.cc:2243] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.076192   561 raft_consensus.cc:2272] T d12b72a433e1403ab5c6d70ccad35e0f P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.078562   561 tablet_replica.cc:333] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.079344   561 raft_consensus.cc:2243] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.080200   561 raft_consensus.cc:2272] T 09c1da90a8da49f6be72a8361eead7df P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.082679   561 tablet_replica.cc:333] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.083395   561 raft_consensus.cc:2243] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.083915   561 raft_consensus.cc:2272] T 070b536983ee48e1853040e1652619e2 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.086243   561 tablet_replica.cc:333] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.086982   561 raft_consensus.cc:2243] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.087899   561 raft_consensus.cc:2272] T 7d28ffbbc352420bb35bfab9e1847e67 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.090276   561 tablet_replica.cc:333] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.090981   561 raft_consensus.cc:2243] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.091466   561 raft_consensus.cc:2272] T c4eccc8971ad4aacb553dc8e1bb00cd8 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.093717   561 tablet_replica.cc:333] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.094357   561 raft_consensus.cc:2243] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.094902   561 raft_consensus.cc:2272] T 8bc4a9064e4143f7a4f1ce3a3a4bb860 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.097203   561 tablet_replica.cc:333] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.097817   561 raft_consensus.cc:2243] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.098870   561 raft_consensus.cc:2272] T 30cb0580c4514b41b650a6b2d5089f3a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.101182   561 tablet_replica.cc:333] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.102011   561 raft_consensus.cc:2243] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.102536   561 raft_consensus.cc:2272] T cfad63c176a643f3bf8bba8f61523f87 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.105062   561 tablet_replica.cc:333] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.105659   561 raft_consensus.cc:2243] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.106168   561 raft_consensus.cc:2272] T 81ab261b0dc04756a70faccbf14a33d6 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.108470   561 tablet_replica.cc:333] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.109045   561 raft_consensus.cc:2243] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.109995   561 raft_consensus.cc:2272] T b0eea117b500470093718b8583fa5e96 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.112347   561 tablet_replica.cc:333] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.112915   561 raft_consensus.cc:2243] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.113806   561 raft_consensus.cc:2272] T 56b78cc7f12d4c3a81aed3af33111913 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.116434   561 tablet_replica.cc:333] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.116986   561 raft_consensus.cc:2243] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.117877   561 raft_consensus.cc:2272] T f40e0b80665048e09e181f1a8ea7509a P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.120592   561 tablet_replica.cc:333] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.121223   561 raft_consensus.cc:2243] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.122241   561 raft_consensus.cc:2272] T e643b9d35a9f4515abe259109274d191 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.124830   561 tablet_replica.cc:333] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.125429   561 raft_consensus.cc:2243] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.125941   561 raft_consensus.cc:2272] T eae9dd87d90f4d18b0d36431646603e3 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.128549   561 tablet_replica.cc:333] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.129221   561 raft_consensus.cc:2243] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.129776   561 raft_consensus.cc:2272] T f9b4619b6db04b159a1d568ba006bdb4 P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.132769   561 tablet_replica.cc:333] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe: stopping tablet replica
I20260709 12:28:16.133394   561 raft_consensus.cc:2243] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:16.133883   561 raft_consensus.cc:2272] T 735482fa1588451b93a1735236b1859b P 18cf4829b3bc4fafb3ef7f1b98f853fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.213815   561 tablet_server.cc:196] TabletServer@127.0.140.68:0 shutdown complete.
I20260709 12:28:16.283707   561 master.cc:562] Master@127.0.140.126:45447 shutting down...
I20260709 12:28:16.322762   561 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:16.323676   561 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:16.324074   561 tablet_replica.cc:333] T 00000000000000000000000000000000 P 8de3ba5eb28a4b36a9640e3eb6cb4663: stopping tablet replica
I20260709 12:28:16.373788   561 master.cc:584] Master@127.0.140.126:45447 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (42297 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 12:28:16.475641   561 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.140.126:46745
I20260709 12:28:16.477155   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:16.485090 14061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:16.485291 14064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:16.485124 14062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:16.487267   561 server_base.cc:1061] running on GCE node
I20260709 12:28:16.488606   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:16.488809   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:16.489004   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600096488989 us; error 0 us; skew 500 ppm
I20260709 12:28:16.489610   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:16.493098   561 webserver.cc:533] Webserver started at http://127.0.140.126:42453/ using document root <none> and password file <none>
I20260709 12:28:16.493772   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:16.494014   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:16.494371   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:16.495782   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/master-0-root/instance:
uuid: "bff96fb285a644399a4fe632a1eb2c4b"
format_stamp: "Formatted at 2026-07-09 12:28:16 on dist-test-slave-7mcn"
I20260709 12:28:16.501622   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:16.505944 14069 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:16.507054   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.002s
I20260709 12:28:16.507514   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/master-0-root
uuid: "bff96fb285a644399a4fe632a1eb2c4b"
format_stamp: "Formatted at 2026-07-09 12:28:16 on dist-test-slave-7mcn"
I20260709 12:28:16.507859   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:16.520453   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:16.521777   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:16.569409   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.126:46745
I20260709 12:28:16.569586 14120 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.126:46745 every 8 connection(s)
I20260709 12:28:16.574327 14121 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:16.589185 14121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b: Bootstrap starting.
I20260709 12:28:16.596777 14121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:16.602630 14121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b: No bootstrap required, opened a new log
I20260709 12:28:16.605368 14121 raft_consensus.cc:359] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff96fb285a644399a4fe632a1eb2c4b" member_type: VOTER }
I20260709 12:28:16.605821 14121 raft_consensus.cc:385] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:16.606108 14121 raft_consensus.cc:740] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bff96fb285a644399a4fe632a1eb2c4b, State: Initialized, Role: FOLLOWER
I20260709 12:28:16.606921 14121 consensus_queue.cc:260] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff96fb285a644399a4fe632a1eb2c4b" member_type: VOTER }
I20260709 12:28:16.607422 14121 raft_consensus.cc:399] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:28:16.607651 14121 raft_consensus.cc:493] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:28:16.607937 14121 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:16.613962 14121 raft_consensus.cc:515] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff96fb285a644399a4fe632a1eb2c4b" member_type: VOTER }
I20260709 12:28:16.614595 14121 leader_election.cc:304] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bff96fb285a644399a4fe632a1eb2c4b; no voters: 
I20260709 12:28:16.616035 14121 leader_election.cc:290] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:28:16.616576 14124 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:16.618494 14124 raft_consensus.cc:697] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 1 LEADER]: Becoming Leader. State: Replica: bff96fb285a644399a4fe632a1eb2c4b, State: Running, Role: LEADER
I20260709 12:28:16.619364 14124 consensus_queue.cc:237] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff96fb285a644399a4fe632a1eb2c4b" member_type: VOTER }
I20260709 12:28:16.619504 14121 sys_catalog.cc:565] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:28:16.625571 14126 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bff96fb285a644399a4fe632a1eb2c4b. Latest consensus state: current_term: 1 leader_uuid: "bff96fb285a644399a4fe632a1eb2c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff96fb285a644399a4fe632a1eb2c4b" member_type: VOTER } }
I20260709 12:28:16.625676 14125 sys_catalog.cc:455] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bff96fb285a644399a4fe632a1eb2c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bff96fb285a644399a4fe632a1eb2c4b" member_type: VOTER } }
I20260709 12:28:16.626340 14126 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:16.626363 14125 sys_catalog.cc:458] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:16.632761 14132 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:28:16.639268 14132 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:28:16.640900   561 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:28:16.649638 14132 catalog_manager.cc:1383] Generated new cluster ID: a733c2f672f24303beff36c2efe20cc3
I20260709 12:28:16.650043 14132 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:28:16.686182 14132 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:28:16.687886 14132 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:28:16.706809 14132 catalog_manager.cc:6070] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b: Generated new TSK 0
I20260709 12:28:16.707500 14132 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:28:16.772830   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:16.780244 14142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:16.781764 14143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:16.783591 14145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:16.783931   561 server_base.cc:1061] running on GCE node
I20260709 12:28:16.785226   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:16.785475   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:16.785684   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600096785664 us; error 0 us; skew 500 ppm
I20260709 12:28:16.786320   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:16.789280   561 webserver.cc:533] Webserver started at http://127.0.140.65:39867/ using document root <none> and password file <none>
I20260709 12:28:16.789868   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:16.790104   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:16.790421   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:16.791738   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-0-root/instance:
uuid: "e767df82fbf4455395ffd6fa8d992d59"
format_stamp: "Formatted at 2026-07-09 12:28:16 on dist-test-slave-7mcn"
I20260709 12:28:16.797420   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.002s
I20260709 12:28:16.801810 14150 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:16.803144   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:28:16.803591   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "e767df82fbf4455395ffd6fa8d992d59"
format_stamp: "Formatted at 2026-07-09 12:28:16 on dist-test-slave-7mcn"
I20260709 12:28:16.803999   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:16.820744   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:16.822273   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:16.824148   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:16.827023   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:16.827335   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:16.827611   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:16.827832   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:16.879997   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:43923
I20260709 12:28:16.880162 14212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:43923 every 8 connection(s)
I20260709 12:28:16.885419   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:16.894529 14217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:16.895033 14218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:16.898518   561 server_base.cc:1061] running on GCE node
W20260709 12:28:16.898748 14220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:16.900303   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:16.900511   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:16.900781   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600096900753 us; error 0 us; skew 500 ppm
I20260709 12:28:16.901405   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:16.904697   561 webserver.cc:533] Webserver started at http://127.0.140.66:38535/ using document root <none> and password file <none>
I20260709 12:28:16.905148 14213 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46745
I20260709 12:28:16.905360   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:16.905635   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:16.905668 14213 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:16.906216   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:16.906819 14213 heartbeater.cc:507] Master 127.0.140.126:46745 requested a full tablet report, sending...
I20260709 12:28:16.907809   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-1-root/instance:
uuid: "661ba84d83184824bdf0dfc8dc3a5036"
format_stamp: "Formatted at 2026-07-09 12:28:16 on dist-test-slave-7mcn"
I20260709 12:28:16.909718 14086 ts_manager.cc:194] Registered new tserver with Master: e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923)
I20260709 12:28:16.912089 14086 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51558
I20260709 12:28:16.913256   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:16.917919 14225 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:16.919081   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:28:16.919392   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-1-root
uuid: "661ba84d83184824bdf0dfc8dc3a5036"
format_stamp: "Formatted at 2026-07-09 12:28:16 on dist-test-slave-7mcn"
I20260709 12:28:16.919696   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:16.943629   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:16.945040   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:16.946738   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:16.949353   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:16.949555   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:16.949860   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:16.950042   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:17.009488   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.66:33877
I20260709 12:28:17.009658 14287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.66:33877 every 8 connection(s)
I20260709 12:28:17.015647   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:17.025810 14291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:17.027228 14292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:17.030308   561 server_base.cc:1061] running on GCE node
W20260709 12:28:17.031854 14295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:17.033272   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:17.033576   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:17.033808   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600097033784 us; error 0 us; skew 500 ppm
I20260709 12:28:17.034646   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:17.036437 14288 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46745
I20260709 12:28:17.036950 14288 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:17.037951 14288 heartbeater.cc:507] Master 127.0.140.126:46745 requested a full tablet report, sending...
I20260709 12:28:17.038653   561 webserver.cc:533] Webserver started at http://127.0.140.67:39469/ using document root <none> and password file <none>
I20260709 12:28:17.039408   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:17.039587   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:17.039995   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:17.041002 14086 ts_manager.cc:194] Registered new tserver with Master: 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:17.041227   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-2-root/instance:
uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
format_stamp: "Formatted at 2026-07-09 12:28:17 on dist-test-slave-7mcn"
I20260709 12:28:17.042762 14086 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51574
I20260709 12:28:17.047240   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:28:17.052430 14299 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:17.053767   561 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.001s
I20260709 12:28:17.054149   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-2-root
uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
format_stamp: "Formatted at 2026-07-09 12:28:17 on dist-test-slave-7mcn"
I20260709 12:28:17.054484   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:17.083220   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:17.084535   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:17.086225   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:17.088833   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:17.089078   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:17.089350   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:17.089534   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:17.140676   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.67:38711
I20260709 12:28:17.140817 14361 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.67:38711 every 8 connection(s)
I20260709 12:28:17.157509 14362 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46745
I20260709 12:28:17.158141 14362 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:17.159122 14362 heartbeater.cc:507] Master 127.0.140.126:46745 requested a full tablet report, sending...
I20260709 12:28:17.161608 14086 ts_manager.cc:194] Registered new tserver with Master: 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:17.162245   561 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017801139s
I20260709 12:28:17.163401 14086 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51580
I20260709 12:28:17.194769 14086 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51596:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:17.198087 14086 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:17.297655 14178 tablet_service.cc:1511] Processing CreateTablet for tablet d2566a924ce24d5688df3bb577c471ad (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:28:17.299304 14178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2566a924ce24d5688df3bb577c471ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.299647 14176 tablet_service.cc:1511] Processing CreateTablet for tablet 772c8428cc824192a9e43f11e1956b62 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:28:17.301518 14176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 772c8428cc824192a9e43f11e1956b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.303959 14177 tablet_service.cc:1511] Processing CreateTablet for tablet 4065fbe85a54437b9198d8e44423227f (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:28:17.305605 14177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4065fbe85a54437b9198d8e44423227f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.312578 14175 tablet_service.cc:1511] Processing CreateTablet for tablet fbf7e8af7d2f437ebbea676e59903938 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:28:17.313511 14252 tablet_service.cc:1511] Processing CreateTablet for tablet 4065fbe85a54437b9198d8e44423227f (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:28:17.314420 14175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbf7e8af7d2f437ebbea676e59903938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.315084 14253 tablet_service.cc:1511] Processing CreateTablet for tablet d2566a924ce24d5688df3bb577c471ad (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:28:17.315653 14173 tablet_service.cc:1511] Processing CreateTablet for tablet f5bd9df3750f4a3589fd9d5adfeb6adb (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:28:17.316073 14250 tablet_service.cc:1511] Processing CreateTablet for tablet fbf7e8af7d2f437ebbea676e59903938 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:28:17.316608 14253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2566a924ce24d5688df3bb577c471ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.317508 14173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5bd9df3750f4a3589fd9d5adfeb6adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.320328 14174 tablet_service.cc:1511] Processing CreateTablet for tablet 66e2f6774eae4e3e94ed4d2194d0d3e7 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:28:17.324045 14247 tablet_service.cc:1511] Processing CreateTablet for tablet b6131f7efca94584a3892ade802c0781 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:28:17.325438 14174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66e2f6774eae4e3e94ed4d2194d0d3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.326716 14327 tablet_service.cc:1511] Processing CreateTablet for tablet d2566a924ce24d5688df3bb577c471ad (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:28:17.328713 14327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2566a924ce24d5688df3bb577c471ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.331251 14246 tablet_service.cc:1511] Processing CreateTablet for tablet 680e16c044cb4adca91a1a5a1a072516 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:28:17.331866 14326 tablet_service.cc:1511] Processing CreateTablet for tablet 4065fbe85a54437b9198d8e44423227f (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:28:17.333508 14326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4065fbe85a54437b9198d8e44423227f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.336146 14325 tablet_service.cc:1511] Processing CreateTablet for tablet 772c8428cc824192a9e43f11e1956b62 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:28:17.337836 14325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 772c8428cc824192a9e43f11e1956b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.342161 14251 tablet_service.cc:1511] Processing CreateTablet for tablet 772c8428cc824192a9e43f11e1956b62 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:28:17.344370 14324 tablet_service.cc:1511] Processing CreateTablet for tablet fbf7e8af7d2f437ebbea676e59903938 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:28:17.345875 14324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbf7e8af7d2f437ebbea676e59903938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.320417 14248 tablet_service.cc:1511] Processing CreateTablet for tablet f5bd9df3750f4a3589fd9d5adfeb6adb (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:28:17.318843 14249 tablet_service.cc:1511] Processing CreateTablet for tablet 66e2f6774eae4e3e94ed4d2194d0d3e7 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:28:17.317559 14250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbf7e8af7d2f437ebbea676e59903938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.352150 14245 tablet_service.cc:1511] Processing CreateTablet for tablet 89967a24adc644b1a86747eb3d6e3384 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:28:17.352275 14323 tablet_service.cc:1511] Processing CreateTablet for tablet 66e2f6774eae4e3e94ed4d2194d0d3e7 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:28:17.356993 14323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66e2f6774eae4e3e94ed4d2194d0d3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.360258 14321 tablet_service.cc:1511] Processing CreateTablet for tablet b6131f7efca94584a3892ade802c0781 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:28:17.362258 14171 tablet_service.cc:1511] Processing CreateTablet for tablet 680e16c044cb4adca91a1a5a1a072516 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:28:17.363932 14171 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680e16c044cb4adca91a1a5a1a072516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.364727 14320 tablet_service.cc:1511] Processing CreateTablet for tablet 680e16c044cb4adca91a1a5a1a072516 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:28:17.365821 14319 tablet_service.cc:1511] Processing CreateTablet for tablet 89967a24adc644b1a86747eb3d6e3384 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:28:17.368391 14172 tablet_service.cc:1511] Processing CreateTablet for tablet b6131f7efca94584a3892ade802c0781 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:28:17.352150 14322 tablet_service.cc:1511] Processing CreateTablet for tablet f5bd9df3750f4a3589fd9d5adfeb6adb (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:28:17.374100 14170 tablet_service.cc:1511] Processing CreateTablet for tablet 89967a24adc644b1a86747eb3d6e3384 (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:28:17.370659 14172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6131f7efca94584a3892ade802c0781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.377372 14170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89967a24adc644b1a86747eb3d6e3384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.354583 14245 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89967a24adc644b1a86747eb3d6e3384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.383201 14247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6131f7efca94584a3892ade802c0781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.384223 14248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5bd9df3750f4a3589fd9d5adfeb6adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.385197 14252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4065fbe85a54437b9198d8e44423227f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.387339 14246 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680e16c044cb4adca91a1a5a1a072516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.391185 14249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66e2f6774eae4e3e94ed4d2194d0d3e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.391394 14321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6131f7efca94584a3892ade802c0781. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.394348 14319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89967a24adc644b1a86747eb3d6e3384. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.396770 14251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 772c8428cc824192a9e43f11e1956b62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.399171 14320 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 680e16c044cb4adca91a1a5a1a072516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.403694 14322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5bd9df3750f4a3589fd9d5adfeb6adb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.456614 14169 tablet_service.cc:1511] Processing CreateTablet for tablet 90dd16b1f50e479cbe0a84f08d753d6f (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:28:17.459471 14169 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90dd16b1f50e479cbe0a84f08d753d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.468104 14244 tablet_service.cc:1511] Processing CreateTablet for tablet 90dd16b1f50e479cbe0a84f08d753d6f (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:28:17.469827 14244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90dd16b1f50e479cbe0a84f08d753d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.479779 14380 tablet_bootstrap.cc:492] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:17.529738 14318 tablet_service.cc:1511] Processing CreateTablet for tablet 90dd16b1f50e479cbe0a84f08d753d6f (DEFAULT_TABLE table=test-workload [id=8283396c2cac49b8bb6801cbb0d8ef9d]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:28:17.531524 14318 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90dd16b1f50e479cbe0a84f08d753d6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:17.530236 14381 tablet_bootstrap.cc:492] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:17.563845 14380 tablet_bootstrap.cc:654] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.598755 14382 tablet_bootstrap.cc:492] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:17.603142 14381 tablet_bootstrap.cc:654] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.607894 14382 tablet_bootstrap.cc:654] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.608038 14380 tablet_bootstrap.cc:492] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:17.616305 14380 ts_tablet_manager.cc:1403] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.137s	user 0.018s	sys 0.030s
I20260709 12:28:17.617831 14381 tablet_bootstrap.cc:492] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:17.618438 14381 ts_tablet_manager.cc:1403] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.089s	user 0.015s	sys 0.008s
I20260709 12:28:17.620203 14382 tablet_bootstrap.cc:492] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
I20260709 12:28:17.620083 14380 raft_consensus.cc:359] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.620936 14382 ts_tablet_manager.cc:1403] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.022s	user 0.002s	sys 0.013s
I20260709 12:28:17.621116 14380 raft_consensus.cc:385] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.621483 14380 raft_consensus.cc:740] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.621359 14381 raft_consensus.cc:359] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.622443 14381 raft_consensus.cc:385] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.622817 14381 raft_consensus.cc:740] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.622632 14380 consensus_queue.cc:260] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.623764 14381 consensus_queue.cc:260] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.624298 14382 raft_consensus.cc:359] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.625237 14382 raft_consensus.cc:385] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.625532 14382 raft_consensus.cc:740] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.626320 14382 consensus_queue.cc:260] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.629297 14213 heartbeater.cc:499] Master 127.0.140.126:46745 was elected leader, sending a full tablet report...
I20260709 12:28:17.629844 14380 ts_tablet_manager.cc:1434] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.013s	user 0.012s	sys 0.000s
I20260709 12:28:17.631127 14380 tablet_bootstrap.cc:492] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:17.631783 14362 heartbeater.cc:499] Master 127.0.140.126:46745 was elected leader, sending a full tablet report...
W20260709 12:28:17.632476 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:17.634713 14382 ts_tablet_manager.cc:1434] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.001s
I20260709 12:28:17.635296 14288 heartbeater.cc:499] Master 127.0.140.126:46745 was elected leader, sending a full tablet report...
W20260709 12:28:17.636539 14214 tablet.cc:2406] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:28:17.638516 14382 tablet_bootstrap.cc:492] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:17.639846 14380 tablet_bootstrap.cc:654] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.649094 14382 tablet_bootstrap.cc:654] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.651773 14381 ts_tablet_manager.cc:1434] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.033s	user 0.007s	sys 0.004s
I20260709 12:28:17.653386 14381 tablet_bootstrap.cc:492] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:17.653920 14380 tablet_bootstrap.cc:492] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:17.654748 14380 ts_tablet_manager.cc:1403] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.024s	user 0.010s	sys 0.008s
I20260709 12:28:17.658733 14380 raft_consensus.cc:359] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.659587 14380 raft_consensus.cc:385] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.659919 14380 raft_consensus.cc:740] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.660780 14380 consensus_queue.cc:260] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.663218 14380 ts_tablet_manager.cc:1434] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.004s
I20260709 12:28:17.664327 14380 tablet_bootstrap.cc:492] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:17.664402 14381 tablet_bootstrap.cc:654] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.672801 14382 tablet_bootstrap.cc:492] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
I20260709 12:28:17.673285 14380 tablet_bootstrap.cc:654] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.673703 14382 ts_tablet_manager.cc:1403] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.035s	user 0.018s	sys 0.014s
I20260709 12:28:17.677963 14382 raft_consensus.cc:359] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.673897 14381 tablet_bootstrap.cc:492] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:17.679522 14381 ts_tablet_manager.cc:1403] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.006s
I20260709 12:28:17.679872 14382 raft_consensus.cc:385] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.680266 14382 raft_consensus.cc:740] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.681456 14382 consensus_queue.cc:260] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.683943 14381 raft_consensus.cc:359] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.685137 14381 raft_consensus.cc:385] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.685895 14381 raft_consensus.cc:740] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.685149 14382 ts_tablet_manager.cc:1434] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.006s
I20260709 12:28:17.688011 14382 tablet_bootstrap.cc:492] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:17.688611 14381 consensus_queue.cc:260] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.700529 14381 ts_tablet_manager.cc:1434] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.021s	user 0.004s	sys 0.004s
I20260709 12:28:17.701936 14381 tablet_bootstrap.cc:492] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:17.703019 14380 tablet_bootstrap.cc:492] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:17.703832 14380 ts_tablet_manager.cc:1403] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.040s	user 0.018s	sys 0.016s
I20260709 12:28:17.707576 14380 raft_consensus.cc:359] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.708295 14380 raft_consensus.cc:385] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.708608 14380 raft_consensus.cc:740] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.708846 14381 tablet_bootstrap.cc:654] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.709589 14380 consensus_queue.cc:260] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.711593 14382 tablet_bootstrap.cc:654] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.712296 14380 ts_tablet_manager.cc:1434] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.003s
I20260709 12:28:17.713441 14380 tablet_bootstrap.cc:492] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:17.720389 14380 tablet_bootstrap.cc:654] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.733407 14380 tablet_bootstrap.cc:492] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:17.733421 14382 tablet_bootstrap.cc:492] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
I20260709 12:28:17.734020 14380 ts_tablet_manager.cc:1403] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.004s
I20260709 12:28:17.734632 14382 ts_tablet_manager.cc:1403] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.047s	user 0.023s	sys 0.007s
I20260709 12:28:17.735110 14381 tablet_bootstrap.cc:492] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:17.735719 14381 ts_tablet_manager.cc:1403] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.034s	user 0.016s	sys 0.011s
I20260709 12:28:17.737491 14380 raft_consensus.cc:359] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.737614 14382 raft_consensus.cc:359] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.738427 14380 raft_consensus.cc:385] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.738605 14382 raft_consensus.cc:385] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.739085 14382 raft_consensus.cc:740] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.738901 14380 raft_consensus.cc:740] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.738850 14381 raft_consensus.cc:359] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.740191 14381 raft_consensus.cc:385] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.740522 14381 raft_consensus.cc:740] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.740689 14380 consensus_queue.cc:260] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.741765 14381 consensus_queue.cc:260] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.743391 14380 ts_tablet_manager.cc:1434] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.003s
I20260709 12:28:17.744028 14381 ts_tablet_manager.cc:1434] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 12:28:17.744562 14380 tablet_bootstrap.cc:492] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:17.745090 14381 tablet_bootstrap.cc:492] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:17.740470 14382 consensus_queue.cc:260] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.748303 14382 ts_tablet_manager.cc:1434] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:28:17.749361 14382 tablet_bootstrap.cc:492] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:17.751962 14388 raft_consensus.cc:493] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.752648 14388 raft_consensus.cc:515] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.755674 14380 tablet_bootstrap.cc:654] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.762845 14381 tablet_bootstrap.cc:654] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.755553 14382 tablet_bootstrap.cc:654] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.776618 14388 leader_election.cc:290] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:17.787906 14381 tablet_bootstrap.cc:492] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:17.788805 14381 ts_tablet_manager.cc:1403] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.044s	user 0.012s	sys 0.008s
I20260709 12:28:17.793078 14381 raft_consensus.cc:359] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.794135 14381 raft_consensus.cc:385] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.794541 14381 raft_consensus.cc:740] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.795539 14381 consensus_queue.cc:260] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.797159 14380 tablet_bootstrap.cc:492] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:17.797768 14380 ts_tablet_manager.cc:1403] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.053s	user 0.021s	sys 0.024s
I20260709 12:28:17.797927 14381 ts_tablet_manager.cc:1434] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:28:17.800015 14382 tablet_bootstrap.cc:492] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
I20260709 12:28:17.800586 14382 ts_tablet_manager.cc:1403] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.051s	user 0.010s	sys 0.010s
I20260709 12:28:17.800798 14188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59" is_pre_election: true
I20260709 12:28:17.801537 14188 raft_consensus.cc:2468] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 0.
I20260709 12:28:17.803212 14380 raft_consensus.cc:359] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.804029 14380 raft_consensus.cc:385] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.803701 14382 raft_consensus.cc:359] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.804350 14380 raft_consensus.cc:740] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.804587 14382 raft_consensus.cc:385] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.804953 14382 raft_consensus.cc:740] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.804648 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:17.805347 14380 consensus_queue.cc:260] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.806217 14381 tablet_bootstrap.cc:492] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:17.806677 14382 consensus_queue.cc:260] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.808251 14380 ts_tablet_manager.cc:1434] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.010s	user 0.002s	sys 0.006s
I20260709 12:28:17.809161 14382 ts_tablet_manager.cc:1434] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.006s
I20260709 12:28:17.810192 14380 tablet_bootstrap.cc:492] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:17.812493 14382 tablet_bootstrap.cc:492] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:17.814908 14381 tablet_bootstrap.cc:654] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.816465 14302 leader_election.cc:304] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 
W20260709 12:28:17.817692 14302 leader_election.cc:343] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:17.819628 14380 tablet_bootstrap.cc:654] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.822611 14382 tablet_bootstrap.cc:654] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.824074 14388 raft_consensus.cc:2804] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:17.825177 14388 raft_consensus.cc:493] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:17.825724 14388 raft_consensus.cc:3060] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:17.830672 14381 tablet_bootstrap.cc:492] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:17.831497 14381 ts_tablet_manager.cc:1403] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.026s	user 0.010s	sys 0.013s
I20260709 12:28:17.832942 14380 tablet_bootstrap.cc:492] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:17.833664 14380 ts_tablet_manager.cc:1403] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.024s	user 0.014s	sys 0.002s
I20260709 12:28:17.834928 14382 tablet_bootstrap.cc:492] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
I20260709 12:28:17.835590 14382 ts_tablet_manager.cc:1403] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.023s	user 0.015s	sys 0.002s
I20260709 12:28:17.835691 14381 raft_consensus.cc:359] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.836746 14381 raft_consensus.cc:385] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.837112 14381 raft_consensus.cc:740] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.837270 14380 raft_consensus.cc:359] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.838362 14380 raft_consensus.cc:385] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.838097 14381 consensus_queue.cc:260] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.838821 14380 raft_consensus.cc:740] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.839092 14382 raft_consensus.cc:359] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.840150 14382 raft_consensus.cc:385] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.840077 14380 consensus_queue.cc:260] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.840651 14382 raft_consensus.cc:740] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.841647 14381 ts_tablet_manager.cc:1434] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.001s
I20260709 12:28:17.843118 14380 ts_tablet_manager.cc:1434] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.003s
I20260709 12:28:17.841717 14382 consensus_queue.cc:260] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.844441 14380 tablet_bootstrap.cc:492] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:17.844529 14381 tablet_bootstrap.cc:492] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:17.852907 14380 tablet_bootstrap.cc:654] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.852988 14381 tablet_bootstrap.cc:654] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.855333 14382 ts_tablet_manager.cc:1434] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.001s
I20260709 12:28:17.854508 14388 raft_consensus.cc:515] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.861171 14382 tablet_bootstrap.cc:492] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:17.867303 14386 raft_consensus.cc:493] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.868134 14386 raft_consensus.cc:515] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.868700 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
W20260709 12:28:17.871701 14302 leader_election.cc:343] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:17.868705 14188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:17.873121 14188 raft_consensus.cc:3060] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:17.875187 14382 tablet_bootstrap.cc:654] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.882967 14388 leader_election.cc:290] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:17.890964 14386 leader_election.cc:290] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:17.901255 14188 raft_consensus.cc:2468] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 1.
I20260709 12:28:17.907392 14302 leader_election.cc:304] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:17.908401 14381 tablet_bootstrap.cc:492] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:17.909152 14381 ts_tablet_manager.cc:1403] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.065s	user 0.019s	sys 0.012s
I20260709 12:28:17.908459 14388 raft_consensus.cc:2804] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:17.912074 14388 raft_consensus.cc:697] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:17.912644 14386 raft_consensus.cc:493] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.913075 14387 raft_consensus.cc:493] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.912936 14381 raft_consensus.cc:359] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.913280 14386 raft_consensus.cc:515] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.913877 14387 raft_consensus.cc:515] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.914590 14381 raft_consensus.cc:385] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.915382 14381 raft_consensus.cc:740] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.913159 14388 consensus_queue.cc:237] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.916698 14381 consensus_queue.cc:260] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.917327 14386 leader_election.cc:290] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:17.924103 14382 tablet_bootstrap.cc:492] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
I20260709 12:28:17.924856 14382 ts_tablet_manager.cc:1403] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.064s	user 0.021s	sys 0.006s
I20260709 12:28:17.928280 14382 raft_consensus.cc:359] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.929291 14387 leader_election.cc:290] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:17.929551 14382 raft_consensus.cc:385] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.930140 14382 raft_consensus.cc:740] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.931578 14382 consensus_queue.cc:260] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.935372 14382 ts_tablet_manager.cc:1434] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.004s
I20260709 12:28:17.937001 14382 tablet_bootstrap.cc:492] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:17.940469 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:17.941577 14263 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
I20260709 12:28:17.943302 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:17.944221 14262 raft_consensus.cc:2468] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
I20260709 12:28:17.946481 14382 tablet_bootstrap.cc:654] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:17.946376 14153 leader_election.cc:304] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:17.947947 14153 leader_election.cc:304] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:17.949724 14386 raft_consensus.cc:2804] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:17.950233 14386 raft_consensus.cc:493] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:17.950721 14386 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:17.952265 14416 raft_consensus.cc:2804] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:17.952997 14416 raft_consensus.cc:493] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:17.953823 14380 tablet_bootstrap.cc:492] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:17.953857 14416 raft_consensus.cc:3060] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:17.961520 14386 raft_consensus.cc:515] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.954546 14380 ts_tablet_manager.cc:1403] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.110s	user 0.030s	sys 0.028s
I20260709 12:28:17.972478 14404 raft_consensus.cc:493] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:17.973176 14404 raft_consensus.cc:515] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.975560 14404 leader_election.cc:290] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:17.977540 14188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59" is_pre_election: true
I20260709 12:28:17.977926 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:17.977571 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:17.980185 14386 leader_election.cc:290] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Requested vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:17.980314 14337 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
I20260709 12:28:17.980868 14386 raft_consensus.cc:493] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:17.979594 14302 leader_election.cc:343] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:17.981315 14386 raft_consensus.cc:515] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.977993 14336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:17.981410 14382 tablet_bootstrap.cc:492] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
W20260709 12:28:17.982172 14302 leader_election.cc:343] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:17.982821 14336 raft_consensus.cc:2468] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
I20260709 12:28:17.983834 14302 leader_election.cc:304] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a; no voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:17.982123 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:17.985724 14335 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:17.989841 14416 raft_consensus.cc:515] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.983834 14382 ts_tablet_manager.cc:1403] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.047s	user 0.015s	sys 0.005s
I20260709 12:28:17.991171 14404 raft_consensus.cc:2749] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:17.992792 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:17.993559 14262 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:17.994473 14382 raft_consensus.cc:359] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.995396 14382 raft_consensus.cc:385] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:17.995713 14382 raft_consensus.cc:740] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:17.995903 14084 catalog_manager.cc:5697] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 1 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:17.996624 14382 consensus_queue.cc:260] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.002185 14418 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.003319 14262 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.003993 14382 ts_tablet_manager.cc:1434] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.004s
I20260709 12:28:18.003343 14418 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:17.997870 14387 raft_consensus.cc:493] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.005693 14387 raft_consensus.cc:515] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.006505 14386 leader_election.cc:290] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.007393 14416 leader_election.cc:290] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Requested vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.007733 14336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:18.008678 14387 leader_election.cc:290] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.004897 14153 leader_election.cc:304] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
W20260709 12:28:18.010428 14154 leader_election.cc:343] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.010715 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.011348 14336 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:18.012712 14336 raft_consensus.cc:3060] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.013015 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:18.013680 14263 raft_consensus.cc:3060] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.014178 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:18.015082 14416 raft_consensus.cc:2804] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.011435 14262 raft_consensus.cc:2468] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
W20260709 12:28:18.016491 14154 leader_election.cc:343] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.011371 14335 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.017562 14416 raft_consensus.cc:697] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Becoming Leader. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Running, Role: LEADER
I20260709 12:28:18.018607 14416 consensus_queue.cc:237] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.021055 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:18.021834 14263 raft_consensus.cc:2468] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.024089 14336 raft_consensus.cc:2468] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.024438 14153 leader_election.cc:304] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:18.005343 14382 tablet_bootstrap.cc:492] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:18.025645 14386 raft_consensus.cc:2804] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.025837 14153 leader_election.cc:304] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:18.026307 14386 raft_consensus.cc:493] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.026897 14386 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:28:18.027849 14229 leader_election.cc:343] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.030050 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6131f7efca94584a3892ade802c0781" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:18.031669 14335 raft_consensus.cc:2468] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 0.
I20260709 12:28:18.031852 14380 raft_consensus.cc:359] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.032948 14380 raft_consensus.cc:385] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.033303 14380 raft_consensus.cc:740] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.033492 14229 leader_election.cc:304] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
I20260709 12:28:18.017236 14418 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.035228 14382 tablet_bootstrap.cc:654] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.034955 14387 raft_consensus.cc:2804] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.036295 14387 raft_consensus.cc:493] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.036867 14387 raft_consensus.cc:3060] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.035251 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.039829 14263 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
I20260709 12:28:18.040539 14420 raft_consensus.cc:2804] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.042270 14420 raft_consensus.cc:697] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Becoming Leader. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Running, Role: LEADER
I20260709 12:28:18.043546 14420 consensus_queue.cc:237] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.044332 14386 raft_consensus.cc:515] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.041208 14381 ts_tablet_manager.cc:1434] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.132s	user 0.006s	sys 0.062s
I20260709 12:28:18.034116 14380 consensus_queue.cc:260] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.048615 14188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6131f7efca94584a3892ade802c0781" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59" is_pre_election: true
I20260709 12:28:18.050159 14187 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59" is_pre_election: true
I20260709 12:28:18.051192 14187 raft_consensus.cc:2393] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 661ba84d83184824bdf0dfc8dc3a5036 in current term 1: Already voted for candidate e767df82fbf4455395ffd6fa8d992d59 in this term.
W20260709 12:28:18.053124 14228 leader_election.cc:343] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.054451 14228 leader_election.cc:304] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036; no voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:18.057008 14412 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.066725 14381 tablet_bootstrap.cc:492] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:18.094655 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
W20260709 12:28:18.095857 14153 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.66:33877, user_credentials={real_user=slave}} blocked reactor thread for 70100.9us
W20260709 12:28:18.096933 14154 leader_election.cc:343] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.098371 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:18.100484 14380 ts_tablet_manager.cc:1434] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.137s	user 0.007s	sys 0.011s
I20260709 12:28:18.100981 14387 raft_consensus.cc:515] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.101795 14380 tablet_bootstrap.cc:492] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:18.102113 14412 raft_consensus.cc:2749] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.101536 14386 leader_election.cc:290] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Requested vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.104344 14382 tablet_bootstrap.cc:492] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
I20260709 12:28:18.104903 14382 ts_tablet_manager.cc:1403] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.100s	user 0.023s	sys 0.005s
I20260709 12:28:18.105603 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6131f7efca94584a3892ade802c0781" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:18.106503 14335 raft_consensus.cc:3060] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.107264 14187 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6131f7efca94584a3892ade802c0781" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:18.108359 14153 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:18.110199 14416 raft_consensus.cc:2804] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.110605 14416 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:28:18.110954 14228 leader_election.cc:343] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.110981 14416 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.113883 14387 leader_election.cc:290] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.117466 14380 tablet_bootstrap.cc:654] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.117544 14335 raft_consensus.cc:2468] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 1.
I20260709 12:28:18.122124 14382 raft_consensus.cc:359] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.123143 14382 raft_consensus.cc:385] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.122900 14229 leader_election.cc:304] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:18.123509 14382 raft_consensus.cc:740] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.124639 14382 consensus_queue.cc:260] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.117545 14381 tablet_bootstrap.cc:654] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.125514 14085 catalog_manager.cc:5697] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "e767df82fbf4455395ffd6fa8d992d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.127079 14382 ts_tablet_manager.cc:1434] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.022s	user 0.010s	sys 0.000s
I20260709 12:28:18.128700 14382 tablet_bootstrap.cc:492] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:18.132602 14387 raft_consensus.cc:2804] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.133880 14416 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.135203 14380 tablet_bootstrap.cc:492] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:18.135782 14380 ts_tablet_manager.cc:1403] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.007s
I20260709 12:28:18.136571 14416 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Requested vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.137511 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:18.138176 14262 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.146539 14382 tablet_bootstrap.cc:654] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.138123 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
W20260709 12:28:18.149005 14154 leader_election.cc:343] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.153223 14262 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.155409 14153 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:18.155941 14263 raft_consensus.cc:2468] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.156525 14416 raft_consensus.cc:2804] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.157124 14416 raft_consensus.cc:697] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Becoming Leader. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Running, Role: LEADER
I20260709 12:28:18.158084 14416 consensus_queue.cc:237] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.159044 14380 raft_consensus.cc:359] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.159778 14380 raft_consensus.cc:385] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.159886 14387 raft_consensus.cc:697] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:18.161448 14085 catalog_manager.cc:5697] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "e767df82fbf4455395ffd6fa8d992d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.161674 14387 consensus_queue.cc:237] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.166679 14381 tablet_bootstrap.cc:492] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:18.160066 14380 raft_consensus.cc:740] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.169083 14153 leader_election.cc:304] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:18.169404 14380 consensus_queue.cc:260] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.167521 14381 ts_tablet_manager.cc:1403] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.101s	user 0.011s	sys 0.014s
I20260709 12:28:18.170863 14388 raft_consensus.cc:493] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.171734 14420 raft_consensus.cc:2804] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.171957 14388 raft_consensus.cc:515] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.172511 14420 raft_consensus.cc:697] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Becoming Leader. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Running, Role: LEADER
I20260709 12:28:18.173700 14380 ts_tablet_manager.cc:1434] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.038s	user 0.006s	sys 0.003s
I20260709 12:28:18.175197 14380 tablet_bootstrap.cc:492] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:18.176199 14187 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59" is_pre_election: true
I20260709 12:28:18.173779 14420 consensus_queue.cc:237] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
W20260709 12:28:18.180843 14302 leader_election.cc:343] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.182133 14382 tablet_bootstrap.cc:492] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
I20260709 12:28:18.182744 14382 ts_tablet_manager.cc:1403] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.054s	user 0.019s	sys 0.002s
I20260709 12:28:18.183501 14388 leader_election.cc:290] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:18.185577 14382 raft_consensus.cc:359] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.186419 14382 raft_consensus.cc:385] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.186280 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.187203 14382 raft_consensus.cc:740] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.188357 14382 consensus_queue.cc:260] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
W20260709 12:28:18.188926 14302 leader_election.cc:343] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.189713 14404 raft_consensus.cc:493] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.189893 14302 leader_election.cc:304] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a; no voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:18.190346 14404 raft_consensus.cc:515] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.193728 14404 leader_election.cc:290] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:18.194381 14404 raft_consensus.cc:2749] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.197448 14382 ts_tablet_manager.cc:1434] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:28:18.199465 14382 tablet_bootstrap.cc:492] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a: Bootstrap starting.
I20260709 12:28:18.200671 14081 catalog_manager.cc:5697] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 0 to 1, leader changed from <none> to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.205399 14187 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59" is_pre_election: true
I20260709 12:28:18.206344 14187 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 0.
I20260709 12:28:18.206693 14382 tablet_bootstrap.cc:654] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.200771 14381 raft_consensus.cc:359] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.210017 14381 raft_consensus.cc:385] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.210405 14381 raft_consensus.cc:740] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.211453 14381 consensus_queue.cc:260] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.214411 14381 ts_tablet_manager.cc:1434] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:28:18.215708 14381 tablet_bootstrap.cc:492] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:18.217133 14382 tablet_bootstrap.cc:492] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a: No bootstrap required, opened a new log
I20260709 12:28:18.217767 14382 ts_tablet_manager.cc:1403] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent bootstrapping tablet: real 0.019s	user 0.006s	sys 0.009s
I20260709 12:28:18.218902 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.219735 14263 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 0.
I20260709 12:28:18.221201 14302 leader_election.cc:304] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
I20260709 12:28:18.221369 14382 raft_consensus.cc:359] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.223423 14382 raft_consensus.cc:385] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.225245 14382 raft_consensus.cc:740] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.224071 14433 raft_consensus.cc:2804] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.226866 14380 tablet_bootstrap.cc:654] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.227144 14433 raft_consensus.cc:493] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.225632 14420 raft_consensus.cc:493] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.227533 14382 consensus_queue.cc:260] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.228075 14420 raft_consensus.cc:515] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.223917 14381 tablet_bootstrap.cc:654] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.227522 14433 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.231834 14420 leader_election.cc:290] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.225818 14085 catalog_manager.cc:5697] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "e767df82fbf4455395ffd6fa8d992d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.233726 14382 ts_tablet_manager.cc:1434] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.002s
I20260709 12:28:18.235073 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90dd16b1f50e479cbe0a84f08d753d6f" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:18.239199 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90dd16b1f50e479cbe0a84f08d753d6f" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.239296 14335 raft_consensus.cc:2468] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
W20260709 12:28:18.241669 14153 leader_election.cc:343] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.243127 14154 leader_election.cc:304] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:18.244396 14420 raft_consensus.cc:2804] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.244869 14420 raft_consensus.cc:493] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.245262 14420 raft_consensus.cc:3060] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.243299 14085 catalog_manager.cc:5697] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "e767df82fbf4455395ffd6fa8d992d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.249480 14380 tablet_bootstrap.cc:492] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:18.250167 14380 ts_tablet_manager.cc:1403] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.075s	user 0.008s	sys 0.013s
I20260709 12:28:18.246484 14433 raft_consensus.cc:515] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.254117 14187 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:18.255672 14187 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.260198 14433 leader_election.cc:290] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:18.261065 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:18.261866 14263 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.264882 14416 raft_consensus.cc:493] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.265554 14416 raft_consensus.cc:515] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.263531 14420 raft_consensus.cc:515] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.267643 14416 leader_election.cc:290] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.269503 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:18.270292 14335 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
I20260709 12:28:18.270377 14380 raft_consensus.cc:359] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.271900 14380 raft_consensus.cc:385] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.272320 14380 raft_consensus.cc:740] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.273934 14261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90dd16b1f50e479cbe0a84f08d753d6f" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:18.269805 14420 leader_election.cc:290] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Requested vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.275283 14381 tablet_bootstrap.cc:492] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:18.272375 14263 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 1.
I20260709 12:28:18.273125 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.275957 14381 ts_tablet_manager.cc:1403] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.061s	user 0.024s	sys 0.004s
I20260709 12:28:18.276459 14262 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
I20260709 12:28:18.275116 14380 consensus_queue.cc:260] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.277158 14187 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 1.
I20260709 12:28:18.277832 14302 leader_election.cc:304] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
I20260709 12:28:18.279034 14381 raft_consensus.cc:359] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.279824 14381 raft_consensus.cc:385] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:28:18.279953 14153 leader_election.cc:343] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:18.280125 14381 raft_consensus.cc:740] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.280903 14433 raft_consensus.cc:2804] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.280871 14381 consensus_queue.cc:260] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.281526 14433 raft_consensus.cc:697] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:18.282408 14433 consensus_queue.cc:237] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.283936 14380 ts_tablet_manager.cc:1434] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.029s	user 0.008s	sys 0.003s
I20260709 12:28:18.285320 14380 tablet_bootstrap.cc:492] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:18.285538 14381 ts_tablet_manager.cc:1434] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.001s
I20260709 12:28:18.286760 14381 tablet_bootstrap.cc:492] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:18.289812 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90dd16b1f50e479cbe0a84f08d753d6f" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:18.290517 14335 raft_consensus.cc:3060] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.275992 14154 leader_election.cc:304] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:18.292863 14420 raft_consensus.cc:2804] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.293260 14420 raft_consensus.cc:493] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.293625 14420 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.294991 14380 tablet_bootstrap.cc:654] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.295329 14381 tablet_bootstrap.cc:654] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.307029 14381 tablet_bootstrap.cc:492] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:18.307554 14381 ts_tablet_manager.cc:1403] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.021s	user 0.005s	sys 0.012s
I20260709 12:28:18.310488 14381 raft_consensus.cc:359] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.311223 14381 raft_consensus.cc:385] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.311506 14381 raft_consensus.cc:740] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.312353 14381 consensus_queue.cc:260] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.316314 14380 tablet_bootstrap.cc:492] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: No bootstrap required, opened a new log
I20260709 12:28:18.317020 14380 ts_tablet_manager.cc:1403] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.032s	user 0.013s	sys 0.014s
I20260709 12:28:18.314708 14381 ts_tablet_manager.cc:1434] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:28:18.318509 14381 tablet_bootstrap.cc:492] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036: Bootstrap starting.
I20260709 12:28:18.319530 14388 consensus_queue.cc:1048] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:18.320212 14380 raft_consensus.cc:359] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.321020 14380 raft_consensus.cc:385] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.321528 14380 raft_consensus.cc:740] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.322283 14380 consensus_queue.cc:260] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.324468 14380 ts_tablet_manager.cc:1434] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:28:18.318159 14420 raft_consensus.cc:515] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.327396 14084 catalog_manager.cc:5697] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 1 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:18.359994 14381 tablet_bootstrap.cc:654] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:18.362327 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:18.363044 14262 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.334563 14335 raft_consensus.cc:2468] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.364892 14154 leader_election.cc:304] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:18.366266 14386 raft_consensus.cc:2804] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.366883 14386 raft_consensus.cc:697] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Becoming Leader. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Running, Role: LEADER
I20260709 12:28:18.367956 14386 consensus_queue.cc:237] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.361527 14420 leader_election.cc:290] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Requested vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.372164 14262 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.362918 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:18.373023 14337 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.338682 14416 raft_consensus.cc:493] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.375675 14416 raft_consensus.cc:515] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.378818 14416 leader_election.cc:290] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.379748 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.380107 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:18.380676 14263 raft_consensus.cc:2468] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
I20260709 12:28:18.380968 14335 raft_consensus.cc:2468] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 0.
I20260709 12:28:18.381958 14381 tablet_bootstrap.cc:492] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036: No bootstrap required, opened a new log
I20260709 12:28:18.382288 14153 leader_election.cc:304] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:18.382583 14381 ts_tablet_manager.cc:1403] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent bootstrapping tablet: real 0.064s	user 0.009s	sys 0.004s
I20260709 12:28:18.384373 14416 raft_consensus.cc:2804] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:18.384783 14416 raft_consensus.cc:493] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:18.385094 14416 raft_consensus.cc:3060] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.385876 14381 raft_consensus.cc:359] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.386636 14381 raft_consensus.cc:385] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:18.386924 14381 raft_consensus.cc:740] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Initialized, Role: FOLLOWER
I20260709 12:28:18.387864 14381 consensus_queue.cc:260] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.389605 14263 raft_consensus.cc:3060] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.390477 14381 ts_tablet_manager.cc:1434] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:28:18.395553 14153 leader_election.cc:304] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:18.397277 14420 raft_consensus.cc:2804] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.397951 14420 raft_consensus.cc:697] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Becoming Leader. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Running, Role: LEADER
I20260709 12:28:18.399124 14420 consensus_queue.cc:237] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.402081 14337 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.407891 14416 raft_consensus.cc:515] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.410885 14416 leader_election.cc:290] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Requested vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.414824 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:18.415475 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:18.415764 14262 raft_consensus.cc:3060] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.416297 14337 raft_consensus.cc:3060] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.422982 14433 consensus_queue.cc:1048] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:18.419912 14085 catalog_manager.cc:5697] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "e767df82fbf4455395ffd6fa8d992d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.427610 14337 raft_consensus.cc:2468] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.428063 14262 raft_consensus.cc:2468] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e767df82fbf4455395ffd6fa8d992d59 in term 1.
I20260709 12:28:18.429014 14154 leader_election.cc:304] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:18.430024 14420 raft_consensus.cc:2804] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:18.430553 14420 raft_consensus.cc:697] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Becoming Leader. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Running, Role: LEADER
I20260709 12:28:18.431483 14420 consensus_queue.cc:237] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.449256 14084 catalog_manager.cc:5697] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "e767df82fbf4455395ffd6fa8d992d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.451241 14084 catalog_manager.cc:5697] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "e767df82fbf4455395ffd6fa8d992d59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:18.464970 14420 consensus_queue.cc:1048] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:18.479099 14416 consensus_queue.cc:1048] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:18.497051 14388 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.497871 14388 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.500103 14388 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:18.501400 14187 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59" is_pre_election: true
I20260709 12:28:18.501583 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.502498 14263 raft_consensus.cc:2393] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 03a6fb5ac22045e4b35e11b41f53bb9a in current term 1: Already voted for candidate e767df82fbf4455395ffd6fa8d992d59 in this term.
I20260709 12:28:18.503629 14302 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a; no voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:18.504473 14388 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.510983 14388 raft_consensus.cc:2749] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.513252 14416 raft_consensus.cc:493] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.513926 14416 raft_consensus.cc:515] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.515962 14416 leader_election.cc:290] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.517037 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6131f7efca94584a3892ade802c0781" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.517445 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6131f7efca94584a3892ade802c0781" candidate_uuid: "e767df82fbf4455395ffd6fa8d992d59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:18.518481 14337 raft_consensus.cc:2393] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e767df82fbf4455395ffd6fa8d992d59 in current term 1: Already voted for candidate 661ba84d83184824bdf0dfc8dc3a5036 in this term.
I20260709 12:28:18.519670 14154 leader_election.cc:304] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e767df82fbf4455395ffd6fa8d992d59; no voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:18.520455 14416 raft_consensus.cc:3060] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.529042 14416 raft_consensus.cc:2749] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.573338 14416 consensus_queue.cc:1048] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:18.588848 14416 consensus_queue.cc:1048] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:18.590562 14416 consensus_queue.cc:1048] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:18.593097 14388 raft_consensus.cc:493] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.593767 14388 raft_consensus.cc:515] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.598959 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:18.600234 14262 raft_consensus.cc:2393] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 03a6fb5ac22045e4b35e11b41f53bb9a in current term 1: Already voted for candidate e767df82fbf4455395ffd6fa8d992d59 in this term.
I20260709 12:28:18.600320 14388 leader_election.cc:290] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:18.602600 14187 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e767df82fbf4455395ffd6fa8d992d59" is_pre_election: true
I20260709 12:28:18.604631 14302 leader_election.cc:304] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a; no voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:18.605994 14388 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.613986 14420 consensus_queue.cc:1048] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:18.615953 14388 raft_consensus.cc:2749] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.637564 14420 consensus_queue.cc:1048] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:18.643788 14420 consensus_queue.cc:1048] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:18.689807 14387 consensus_queue.cc:1048] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:18.705493 14460 consensus_queue.cc:1048] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:18.721689   561 ts_manager.cc:295] Set tserver state for e767df82fbf4455395ffd6fa8d992d59 to MAINTENANCE_MODE
I20260709 12:28:18.722641   561 tablet_server.cc:179] TabletServer@127.0.140.65:0 shutting down...
I20260709 12:28:18.752971 14460 raft_consensus.cc:493] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:18.753773 14460 raft_consensus.cc:515] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:18.755033   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:18.756453   561 tablet_replica.cc:333] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
I20260709 12:28:18.756902 14460 leader_election.cc:290] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:18.757642   561 raft_consensus.cc:2243] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:18.758539   561 raft_consensus.cc:2272] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.759168 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90dd16b1f50e479cbe0a84f08d753d6f" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:18.760269 14337 raft_consensus.cc:2393] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 661ba84d83184824bdf0dfc8dc3a5036 in current term 1: Already voted for candidate e767df82fbf4455395ffd6fa8d992d59 in this term.
W20260709 12:28:18.761552 14228 leader_election.cc:336] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:28:18.761893   561 tablet_replica.cc:333] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
I20260709 12:28:18.761973 14228 leader_election.cc:304] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036; no voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:18.762869   561 raft_consensus.cc:2243] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:18.762972 14460 raft_consensus.cc:3060] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:18.763880   561 pending_rounds.cc:70] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Trying to abort 1 pending ops.
I20260709 12:28:18.764199   561 pending_rounds.cc:77] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305626003083112448 op_type: NO_OP noop_request { }
I20260709 12:28:18.764626   561 raft_consensus.cc:2889] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:18.765003   561 raft_consensus.cc:2272] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.768501   561 tablet_replica.cc:333] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
I20260709 12:28:18.769388   561 raft_consensus.cc:2243] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:18.769892   561 raft_consensus.cc:2272] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.771484 14460 raft_consensus.cc:2749] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:18.772449   561 tablet_replica.cc:333] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
I20260709 12:28:18.773080   561 raft_consensus.cc:2243] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:18.772949 14388 consensus_queue.cc:1048] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:18.774224   561 raft_consensus.cc:2272] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.776798   561 tablet_replica.cc:333] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
I20260709 12:28:18.777532   561 raft_consensus.cc:2243] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:18.778395   561 pending_rounds.cc:70] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Trying to abort 1 pending ops.
I20260709 12:28:18.778730   561 pending_rounds.cc:77] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305626003190321152 op_type: NO_OP noop_request { }
I20260709 12:28:18.779145   561 raft_consensus.cc:2889] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:18.779451   561 raft_consensus.cc:2272] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.783562   561 tablet_replica.cc:333] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
I20260709 12:28:18.784569   561 raft_consensus.cc:2243] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:18.785710   561 raft_consensus.cc:2272] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.789604   561 tablet_replica.cc:333] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
I20260709 12:28:18.790612   561 raft_consensus.cc:2243] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:18.792112   561 raft_consensus.cc:2272] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.796104   561 tablet_replica.cc:333] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
W20260709 12:28:18.796471 14302 consensus_peers.cc:597] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:18.797214   561 raft_consensus.cc:2243] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:18.798266   561 raft_consensus.cc:2272] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.801149   561 tablet_replica.cc:333] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
I20260709 12:28:18.801695   561 raft_consensus.cc:2243] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:18.802352   561 pending_rounds.cc:70] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Trying to abort 1 pending ops.
I20260709 12:28:18.802536   561 pending_rounds.cc:77] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7305626002927001600 op_type: NO_OP noop_request { }
I20260709 12:28:18.802873   561 raft_consensus.cc:2889] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:18.803148   561 raft_consensus.cc:2272] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.805788   561 tablet_replica.cc:333] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: stopping tablet replica
I20260709 12:28:18.806432   561 raft_consensus.cc:2243] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:18.806939   561 raft_consensus.cc:2272] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:18.844094   561 tablet_server.cc:196] TabletServer@127.0.140.65:0 shutdown complete.
W20260709 12:28:18.938350 14302 consensus_peers.cc:597] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:19.115649 14228 consensus_peers.cc:597] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:19.664695 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:28:19.709164 14228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111) [suppressed 160 similar messages]
I20260709 12:28:19.972599 14460 raft_consensus.cc:493] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:19.973270 14460 raft_consensus.cc:515] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:19.975557 14460 leader_election.cc:290] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:19.977249 14388 raft_consensus.cc:493] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:19.977092 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:19.978101 14337 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 1.
I20260709 12:28:19.977908 14388 raft_consensus.cc:515] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:19.979701 14229 leader_election.cc:304] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
I20260709 12:28:19.980244 14388 leader_election.cc:290] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
W20260709 12:28:19.980320 14228 leader_election.cc:336] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:19.980779 14460 raft_consensus.cc:2804] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:19.981331 14460 raft_consensus.cc:493] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:19.981709 14460 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.982107 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:19.982892 14263 raft_consensus.cc:2468] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 1.
I20260709 12:28:19.984288 14302 leader_election.cc:304] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
I20260709 12:28:19.985163 14388 raft_consensus.cc:2804] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:19.985781 14388 raft_consensus.cc:493] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:19.986301 14388 raft_consensus.cc:3060] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:19.988390 14302 leader_election.cc:336] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:19.989547 14460 raft_consensus.cc:515] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:19.992146 14460 leader_election.cc:290] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:19.993599 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:19.994446 14337 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:19.994874 14388 raft_consensus.cc:515] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:19.997782 14388 leader_election.cc:290] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
W20260709 12:28:19.999035 14228 leader_election.cc:336] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.000577 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:20.001449 14263 raft_consensus.cc:3060] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:20.003578 14302 leader_election.cc:336] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.005802 14337 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 2.
I20260709 12:28:20.007895 14229 leader_election.cc:304] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.009264 14460 raft_consensus.cc:2804] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:20.009629 14263 raft_consensus.cc:2468] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 2.
I20260709 12:28:20.010442 14460 raft_consensus.cc:697] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:20.011440 14302 leader_election.cc:304] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.011677 14460 consensus_queue.cc:237] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.012748 14388 raft_consensus.cc:2804] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:20.014008 14388 raft_consensus.cc:697] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:20.014990 14388 consensus_queue.cc:237] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.024814 14085 catalog_manager.cc:5697] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 1 to 2, leader changed from e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:20.026706 14081 catalog_manager.cc:5697] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 1 to 2, leader changed from e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 2 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:20.065095 14460 raft_consensus.cc:493] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.065780 14460 raft_consensus.cc:515] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.068248 14460 leader_election.cc:290] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:20.069954 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:20.070921 14337 raft_consensus.cc:2468] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 1.
I20260709 12:28:20.072099 14229 leader_election.cc:304] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
W20260709 12:28:20.072849 14228 leader_election.cc:336] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.072865 14460 raft_consensus.cc:2804] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:20.073447 14460 raft_consensus.cc:493] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:20.073734 14460 raft_consensus.cc:3060] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.078912 14460 raft_consensus.cc:515] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.080667 14460 leader_election.cc:290] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:20.081986 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:20.082823 14337 raft_consensus.cc:3060] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.084722 14460 raft_consensus.cc:493] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.085367 14460 raft_consensus.cc:515] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
W20260709 12:28:20.085446 14228 leader_election.cc:336] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.087083 14460 leader_election.cc:290] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:20.088341 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
W20260709 12:28:20.091115 14228 leader_election.cc:336] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.091179 14337 raft_consensus.cc:2468] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 2.
I20260709 12:28:20.091560 14228 leader_election.cc:304] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036; no voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.092533 14460 raft_consensus.cc:2749] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:20.092607 14229 leader_election.cc:304] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.093509 14460 raft_consensus.cc:2804] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:20.093864 14388 raft_consensus.cc:493] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.094074 14460 raft_consensus.cc:697] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:20.094446 14388 raft_consensus.cc:515] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.095183 14460 consensus_queue.cc:237] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.096498 14388 leader_election.cc:290] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:20.098209 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:20.098977 14263 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 1.
I20260709 12:28:20.100770 14302 leader_election.cc:304] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
I20260709 12:28:20.101652 14388 raft_consensus.cc:2804] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:28:20.101740 14302 leader_election.cc:336] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.102087 14388 raft_consensus.cc:493] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.102481 14388 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.106263 14081 catalog_manager.cc:5697] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 1 to 2, leader changed from e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:20.110015 14388 raft_consensus.cc:515] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.112325 14388 leader_election.cc:290] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:20.113317 14460 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.113963 14460 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.114521 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:20.115415 14263 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.118224 14460 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
W20260709 12:28:20.120302 14302 leader_election.cc:336] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.120694 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:20.121477 14388 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.122098 14388 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
W20260709 12:28:20.122349 14228 leader_election.cc:336] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.123162 14337 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 1.
W20260709 12:28:20.130381 14302 leader_election.cc:336] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.125336 14229 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.126657 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:20.126529 14263 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 2.
I20260709 12:28:20.124449 14388 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:20.132803 14262 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 1.
I20260709 12:28:20.133658 14302 leader_election.cc:304] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.134037 14460 raft_consensus.cc:2804] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:20.134670 14388 raft_consensus.cc:2804] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:20.134900 14460 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.135282 14302 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.135356 14460 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.135423 14388 raft_consensus.cc:697] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:20.136415 14471 raft_consensus.cc:2804] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:20.136860 14471 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting leader election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.137216 14471 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.137076 14388 consensus_queue.cc:237] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.144001 14460 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.147239 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:20.147452 14471 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.148094 14460 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:20.147891 14081 catalog_manager.cc:5697] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 1 to 2, leader changed from e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 2 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:20.149160 14337 raft_consensus.cc:2393] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 661ba84d83184824bdf0dfc8dc3a5036 in current term 2: Already voted for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in this term.
W20260709 12:28:20.150547 14228 leader_election.cc:336] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.150949 14388 raft_consensus.cc:493] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.152099 14229 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036; no voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.152237 14471 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:20.153365 14460 raft_consensus.cc:2749] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:28:20.151484 14388 raft_consensus.cc:515] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.155033 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:20.156049 14262 raft_consensus.cc:2393] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 03a6fb5ac22045e4b35e11b41f53bb9a in current term 2: Already voted for candidate 661ba84d83184824bdf0dfc8dc3a5036 in this term.
I20260709 12:28:20.157487 14388 leader_election.cc:290] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
W20260709 12:28:20.158667 14302 leader_election.cc:336] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.159557 14302 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a; no voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.160554 14388 raft_consensus.cc:2749] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:28:20.160969 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
W20260709 12:28:20.162206 14302 leader_election.cc:336] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.163081 14302 leader_election.cc:304] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a; no voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.163761 14460 raft_consensus.cc:493] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.163806 14388 raft_consensus.cc:2749] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:28:20.164549 14460 raft_consensus.cc:515] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.166850 14460 leader_election.cc:290] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:20.168614 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" is_pre_election: true
I20260709 12:28:20.169539 14337 raft_consensus.cc:2468] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 1.
I20260709 12:28:20.170717 14229 leader_election.cc:304] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
W20260709 12:28:20.171437 14228 leader_election.cc:336] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.171643 14460 raft_consensus.cc:2804] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:20.172163 14460 raft_consensus.cc:493] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e767df82fbf4455395ffd6fa8d992d59)
I20260709 12:28:20.172497 14460 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.179231 14460 raft_consensus.cc:515] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.181419 14460 leader_election.cc:290] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:20.182927 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:20.183630 14337 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:20.185158 14228 leader_election.cc:336] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.190341 14337 raft_consensus.cc:2468] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 2.
I20260709 12:28:20.191867 14229 leader_election.cc:304] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.192770 14460 raft_consensus.cc:2804] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:20.193324 14460 raft_consensus.cc:697] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:20.194209 14460 consensus_queue.cc:237] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.204756 14081 catalog_manager.cc:5697] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 1 to 2, leader changed from e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:20.220070 14388 raft_consensus.cc:493] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:20.220782 14388 raft_consensus.cc:515] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.223450 14388 leader_election.cc:290] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:20.225281 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90dd16b1f50e479cbe0a84f08d753d6f" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:20.226117 14262 raft_consensus.cc:2468] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 1.
I20260709 12:28:20.227324 14302 leader_election.cc:304] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
W20260709 12:28:20.228062 14302 leader_election.cc:336] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.228080 14388 raft_consensus.cc:2804] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:20.228653 14388 raft_consensus.cc:493] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:20.229000 14388 raft_consensus.cc:3060] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:20.235993 14388 raft_consensus.cc:515] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.238103 14388 leader_election.cc:290] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:20.240305 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90dd16b1f50e479cbe0a84f08d753d6f" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:20.240974 14262 raft_consensus.cc:3060] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:28:20.242681 14302 leader_election.cc:336] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:20.249083 14262 raft_consensus.cc:2468] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 2.
I20260709 12:28:20.250743 14302 leader_election.cc:304] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:20.251777 14388 raft_consensus.cc:2804] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:20.252334 14388 raft_consensus.cc:697] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:20.253259 14388 consensus_queue.cc:237] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:20.261883 14081 catalog_manager.cc:5697] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 1 to 2, leader changed from e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 2 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:20.397517 14262 raft_consensus.cc:1275] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:20.399286 14388 consensus_queue.cc:1048] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:20.410107 14337 raft_consensus.cc:1275] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260709 12:28:20.412503 14228 consensus_peers.cc:597] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:20.412875 14460 consensus_queue.cc:1048] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:28:20.423369 14302 consensus_peers.cc:597] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:20.602537 14337 raft_consensus.cc:1275] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20260709 12:28:20.603888 14228 consensus_peers.cc:597] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:20.604444 14487 consensus_queue.cc:1048] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:28:20.666255 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:20.676231 14337 raft_consensus.cc:1275] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:20.678081 14487 consensus_queue.cc:1048] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:28:20.694844 14228 consensus_peers.cc:597] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:20.713497 14262 raft_consensus.cc:1275] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:20.715416 14388 consensus_queue.cc:1048] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:28:20.733922 14302 consensus_peers.cc:597] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:20.734632 14302 consensus_peers.cc:597] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:20.868896 14262 raft_consensus.cc:1275] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:28:20.870546 14471 consensus_queue.cc:1048] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:28:21.025830 14302 consensus_peers.cc:597] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:21.240995 14302 consensus_peers.cc:597] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:21.667730 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:28:21.798077 14228 consensus_peers.cc:597] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:22.043370 14491 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:22.043876 14491 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:22.045753 14491 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:22.047663 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036" is_pre_election: true
I20260709 12:28:22.048538 14263 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 2.
I20260709 12:28:22.049883 14302 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
I20260709 12:28:22.050647 14491 raft_consensus.cc:2804] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:28:22.050725 14302 leader_election.cc:336] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:22.051146 14491 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:22.051566 14491 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:22.057855 14491 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:22.059777 14491 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:22.061651 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:22.062399 14263 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:28:22.064391 14302 leader_election.cc:336] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111)
I20260709 12:28:22.068656 14263 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:22.070061 14302 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: e767df82fbf4455395ffd6fa8d992d59
I20260709 12:28:22.070986 14491 raft_consensus.cc:2804] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:22.072069 14491 raft_consensus.cc:697] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:22.073058 14491 consensus_queue.cc:237] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:22.081318 14081 catalog_manager.cc:5697] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 1 to 3, leader changed from e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 3 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
W20260709 12:28:22.415591 14228 consensus_peers.cc:597] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:28:22.489555 14263 raft_consensus.cc:1275] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:22.491035 14491 consensus_queue.cc:1048] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:28:22.523898 14302 consensus_peers.cc:597] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:22.669245 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:28:23.123731 14302 consensus_peers.cc:597] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:23.156414 14302 consensus_peers.cc:597] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:23.285259 14228 consensus_peers.cc:597] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:23.340696 14302 consensus_peers.cc:597] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:23.472177 14228 consensus_peers.cc:597] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:28:23.486135 14302 consensus_peers.cc:597] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:28:23.670863 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:23.872558   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:23.881654 14507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:23.883816 14508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:23.886605 14510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:23.887809   561 server_base.cc:1061] running on GCE node
I20260709 12:28:23.889130   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:23.889410   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:23.889621   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600103889605 us; error 0 us; skew 500 ppm
I20260709 12:28:23.890298   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:23.893690   561 webserver.cc:533] Webserver started at http://127.0.140.65:39867/ using document root <none> and password file <none>
I20260709 12:28:23.894397   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:23.894652   561 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:28:23.901530   561 fs_manager.cc:714] Time spent opening directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:23.907670 14515 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:23.908623   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:28:23.909039   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "e767df82fbf4455395ffd6fa8d992d59"
format_stamp: "Formatted at 2026-07-09 12:28:16 on dist-test-slave-7mcn"
I20260709 12:28:23.909634   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260709 12:28:23.918231 14228 consensus_peers.cc:597] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:23.926092   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:23.927912   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:23.929714   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:23.934367 14522 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
W20260709 12:28:23.953976 14302 consensus_peers.cc:597] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:28:24.009076   561 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 12:28:24.009604   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.076s	user 0.003s	sys 0.000s
I20260709 12:28:24.010001   561 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260709 12:28:24.017334 14522 tablet_bootstrap.cc:492] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.036909 14522 tablet_bootstrap.cc:492] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:28:24.038110 14522 tablet_bootstrap.cc:492] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.038908 14522 ts_tablet_manager.cc:1403] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.022s	user 0.018s	sys 0.003s
I20260709 12:28:24.042320 14522 raft_consensus.cc:359] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.043792 14522 raft_consensus.cc:740] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.044875 14522 consensus_queue.cc:260] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.048305 14522 ts_tablet_manager.cc:1434] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:28:24.049358 14522 tablet_bootstrap.cc:492] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.067349 14522 tablet_bootstrap.cc:492] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:24.068389 14522 tablet_bootstrap.cc:492] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.069061 14522 ts_tablet_manager.cc:1403] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.020s	user 0.017s	sys 0.000s
I20260709 12:28:24.071805 14522 raft_consensus.cc:359] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.072597 14522 raft_consensus.cc:740] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.073426 14522 consensus_queue.cc:260] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.075795 14522 ts_tablet_manager.cc:1434] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:28:24.076646 14522 tablet_bootstrap.cc:492] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.083990   561 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 12:28:24.084419   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.074s	user 0.073s	sys 0.000s
I20260709 12:28:24.095547 14522 tablet_bootstrap.cc:492] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:24.096674 14522 tablet_bootstrap.cc:492] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.097651 14522 ts_tablet_manager.cc:1403] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.000s
I20260709 12:28:24.100664 14522 raft_consensus.cc:359] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.101369 14522 raft_consensus.cc:740] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.102347 14522 consensus_queue.cc:260] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.104609 14522 ts_tablet_manager.cc:1434] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:28:24.105443 14522 tablet_bootstrap.cc:492] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.125479 14522 tablet_bootstrap.cc:492] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:24.126503 14522 tablet_bootstrap.cc:492] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.127383 14522 ts_tablet_manager.cc:1403] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.022s	user 0.020s	sys 0.000s
I20260709 12:28:24.130311 14522 raft_consensus.cc:359] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.131140 14522 raft_consensus.cc:740] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.132086 14522 consensus_queue.cc:260] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.134667 14522 ts_tablet_manager.cc:1434] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:28:24.135802 14522 tablet_bootstrap.cc:492] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.155049 14522 tablet_bootstrap.cc:492] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:24.156219 14522 tablet_bootstrap.cc:492] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.157160 14522 ts_tablet_manager.cc:1403] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.022s	user 0.020s	sys 0.000s
I20260709 12:28:24.160266 14522 raft_consensus.cc:359] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.160959 14522 raft_consensus.cc:740] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.161837 14522 consensus_queue.cc:260] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.164095 14522 ts_tablet_manager.cc:1434] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.004s
I20260709 12:28:24.164227   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:43923
I20260709 12:28:24.164376 14584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:43923 every 8 connection(s)
I20260709 12:28:24.165040 14522 tablet_bootstrap.cc:492] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.170686   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:24.191699 14522 tablet_bootstrap.cc:492] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:24.193284 14522 tablet_bootstrap.cc:492] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.194401 14522 ts_tablet_manager.cc:1403] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.030s	user 0.021s	sys 0.004s
I20260709 12:28:24.198370 14522 raft_consensus.cc:359] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.199422 14522 raft_consensus.cc:740] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.201153 14522 consensus_queue.cc:260] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.204685 14522 ts_tablet_manager.cc:1434] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260709 12:28:24.206168 14522 tablet_bootstrap.cc:492] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.208720 14585 heartbeater.cc:344] Connected to a master server at 127.0.140.126:46745
I20260709 12:28:24.209434 14585 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:24.210664 14585 heartbeater.cc:507] Master 127.0.140.126:46745 requested a full tablet report, sending...
I20260709 12:28:24.219422 14081 ts_manager.cc:194] Re-registered known tserver with Master: e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923)
I20260709 12:28:24.223730 14263 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
mode: GRACEFUL
new_leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:48754
I20260709 12:28:24.224758 14263 raft_consensus.cc:606] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Received request to transfer leadership to TS 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:24.225909 14522 tablet_bootstrap.cc:492] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:28:24.227269 14522 tablet_bootstrap.cc:492] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.228034 14522 ts_tablet_manager.cc:1403] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.007s
I20260709 12:28:24.231168 14522 raft_consensus.cc:359] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.232075 14522 raft_consensus.cc:740] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.232872 14522 consensus_queue.cc:260] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.234953 14081 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38466
I20260709 12:28:24.235234 14522 ts_tablet_manager.cc:1434] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.004s
I20260709 12:28:24.236487 14522 tablet_bootstrap.cc:492] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.240196 14585 heartbeater.cc:499] Master 127.0.140.126:46745 was elected leader, sending a full tablet report...
I20260709 12:28:24.245193 14335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
mode: GRACEFUL
new_leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:59264
I20260709 12:28:24.245934 14335 raft_consensus.cc:606] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Received request to transfer leadership to TS 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:24.249732 14335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
mode: GRACEFUL
new_leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:59264
I20260709 12:28:24.250559 14335 raft_consensus.cc:606] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Received request to transfer leadership to TS 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:24.252424   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:24.253135   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:24.253963   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:24.256227 14522 tablet_bootstrap.cc:492] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:24.262302 14522 tablet_bootstrap.cc:492] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.263286 14522 ts_tablet_manager.cc:1403] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.027s	user 0.013s	sys 0.004s
I20260709 12:28:24.264345 14559 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.265765 14558 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.267213 14522 raft_consensus.cc:359] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.268127 14522 raft_consensus.cc:740] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.269212 14522 consensus_queue.cc:260] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.272601 14522 ts_tablet_manager.cc:1434] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:28:24.273736 14522 tablet_bootstrap.cc:492] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.292222 14557 raft_consensus.cc:3060] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.304546 14522 tablet_bootstrap.cc:492] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20260709 12:28:24.305282 14557 pending_rounds.cc:85] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Aborting all ops after (but not including) 0
I20260709 12:28:24.305729 14557 pending_rounds.cc:107] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:28:24.305970 14522 tablet_bootstrap.cc:492] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.306135 14557 raft_consensus.cc:2889] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:24.306969 14522 ts_tablet_manager.cc:1403] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.033s	user 0.026s	sys 0.000s
I20260709 12:28:24.310612 14522 raft_consensus.cc:359] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.311471 14522 raft_consensus.cc:740] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.312451 14522 consensus_queue.cc:260] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.314692 14522 ts_tablet_manager.cc:1434] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:28:24.315622 14522 tablet_bootstrap.cc:492] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap starting.
I20260709 12:28:24.330862 14522 tablet_bootstrap.cc:492] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:28:24.331907 14522 tablet_bootstrap.cc:492] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: Bootstrap complete.
I20260709 12:28:24.332616 14522 ts_tablet_manager.cc:1403] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.003s
I20260709 12:28:24.335017 14522 raft_consensus.cc:359] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.335633 14522 raft_consensus.cc:740] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e767df82fbf4455395ffd6fa8d992d59, State: Initialized, Role: FOLLOWER
I20260709 12:28:24.336364 14522 consensus_queue.cc:260] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.338168 14522 ts_tablet_manager.cc:1434] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:28:24.380676 14603 raft_consensus.cc:993] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036: : Instructing follower 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:24.381392 14494 raft_consensus.cc:1081] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Signalling peer 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:24.383404 14335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:45170
I20260709 12:28:24.384240 14335 raft_consensus.cc:493] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:24.384909 14335 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.398025 14335 raft_consensus.cc:515] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.400216 14335 leader_election.cc:290] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:24.401575 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:24.402380 14262 raft_consensus.cc:3055] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:24.402822 14262 raft_consensus.cc:740] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:24.401237 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:24.403712 14557 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.403836 14262 consensus_queue.cc:260] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.405529 14262 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.413740 14262 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:24.414245 14557 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:24.414975 14302 leader_election.cc:304] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
I20260709 12:28:24.416065 14503 raft_consensus.cc:2804] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:24.417272 14503 raft_consensus.cc:697] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:24.418526 14503 consensus_queue.cc:237] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.427734 14081 catalog_manager.cc:5697] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 2 to 3, leader changed from 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 3 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:24.451857 14557 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.489557 14557 raft_consensus.cc:3060] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.498492 14557 pending_rounds.cc:85] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Aborting all ops after (but not including) 0
I20260709 12:28:24.498847 14557 pending_rounds.cc:107] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:28:24.499171 14557 raft_consensus.cc:2889] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:24.522872 14557 raft_consensus.cc:3060] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:24.562304 14503 raft_consensus.cc:993] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a: : Instructing follower 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:24.563005 14605 raft_consensus.cc:1081] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Signalling peer 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:24.565037 14262 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:50884
I20260709 12:28:24.565553 14262 raft_consensus.cc:493] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:24.565887 14262 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.572515 14262 raft_consensus.cc:515] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.575500 14262 leader_election.cc:290] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 3 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:24.576782 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:24.577301 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:24.577546 14335 raft_consensus.cc:3055] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:24.577946 14557 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.577965 14335 raft_consensus.cc:740] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:24.579368 14335 consensus_queue.cc:260] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.580957 14335 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.587234 14557 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 3.
I20260709 12:28:24.588845 14228 leader_election.cc:304] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:24.589745 14335 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 3.
I20260709 12:28:24.590139 14494 raft_consensus.cc:2804] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:24.591053 14494 raft_consensus.cc:697] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:24.592334 14494 consensus_queue.cc:237] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.600876 14081 catalog_manager.cc:5697] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 2 to 3, leader changed from 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:24.672026 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:24.801815 14616 raft_consensus.cc:993] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a: : Instructing follower 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:24.802438 14605 raft_consensus.cc:1081] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Signalling peer 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:24.804225 14262 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:50884
I20260709 12:28:24.804795 14262 raft_consensus.cc:493] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:24.805094 14262 raft_consensus.cc:3060] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.810684 14262 raft_consensus.cc:515] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.812669 14262 leader_election.cc:290] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 3 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:24.813818 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:24.814047 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbf7e8af7d2f437ebbea676e59903938" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:24.814630 14557 raft_consensus.cc:3060] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.814752 14335 raft_consensus.cc:3055] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:24.815270 14335 raft_consensus.cc:740] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:24.816200 14335 consensus_queue.cc:260] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.817337 14335 raft_consensus.cc:3060] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:24.821761 14557 raft_consensus.cc:2468] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 3.
I20260709 12:28:24.823405 14228 leader_election.cc:304] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:24.823710 14335 raft_consensus.cc:2468] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 3.
I20260709 12:28:24.824599 14615 raft_consensus.cc:2804] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:24.825198 14615 raft_consensus.cc:697] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:24.826117 14615 consensus_queue.cc:237] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:24.833693 14557 raft_consensus.cc:1275] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:24.835198 14605 consensus_queue.cc:1048] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:24.834496 14081 catalog_manager.cc:5697] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 2 to 3, leader changed from 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 3 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:24.835940 14605 consensus_queue.cc:1243] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Peer e767df82fbf4455395ffd6fa8d992d59 log is divergent from this leader: its last log entry 1.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260709 12:28:24.838738 14557 pending_rounds.cc:85] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Aborting all ops after (but not including) 0
I20260709 12:28:24.839251 14557 pending_rounds.cc:107] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:28:24.839614 14557 raft_consensus.cc:2889] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:28:24.850821 14262 raft_consensus.cc:1275] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:24.852770 14503 consensus_queue.cc:1048] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:25.097553 14335 raft_consensus.cc:1275] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:25.099159 14603 consensus_queue.cc:1048] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:25.110193 14557 raft_consensus.cc:1275] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:25.112496 14615 consensus_queue.cc:1048] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:25.269165   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:25.274219 14557 raft_consensus.cc:1275] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:25.276010 14494 consensus_queue.cc:1048] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:25.278323 14262 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
mode: GRACEFUL
new_leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:48754
I20260709 12:28:25.278963 14262 raft_consensus.cc:606] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Received request to transfer leadership to TS 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:25.282553 14335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
mode: GRACEFUL
new_leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:59264
I20260709 12:28:25.283362 14335 raft_consensus.cc:606] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Received request to transfer leadership to TS 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:25.287035 14335 raft_consensus.cc:1275] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:25.288550 14603 consensus_queue.cc:1048] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:25.290253 14262 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
mode: GRACEFUL
new_leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:48754
I20260709 12:28:25.291105 14262 raft_consensus.cc:606] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 LEADER]: Received request to transfer leadership to TS 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:25.292742   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:25.293582   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:25.294549   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:25.572902 14615 raft_consensus.cc:993] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036: : Instructing follower 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:25.573621 14603 raft_consensus.cc:1081] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 LEADER]: Signalling peer 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:25.575420 14335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:45170
I20260709 12:28:25.575932 14335 raft_consensus.cc:493] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:25.576221 14335 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:25.581426 14335 raft_consensus.cc:515] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:25.583115 14335 leader_election.cc:290] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 4 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:25.584412 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:25.584652 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "89967a24adc644b1a86747eb3d6e3384" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:25.585002 14557 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:25.585528 14262 raft_consensus.cc:3055] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:28:25.585893 14262 raft_consensus.cc:740] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:25.586755 14262 consensus_queue.cc:260] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:25.588088 14262 raft_consensus.cc:3060] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:25.590480 14557 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 4.
I20260709 12:28:25.591841 14302 leader_election.cc:304] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:25.593006 14616 raft_consensus.cc:2804] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:28:25.593628 14616 raft_consensus.cc:697] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:25.594627 14616 consensus_queue.cc:237] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:25.596530 14262 raft_consensus.cc:2468] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 4.
I20260709 12:28:25.602746 14081 catalog_manager.cc:5697] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 3 to 4, leader changed from 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 4 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:25.607031 14615 raft_consensus.cc:993] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036: : Instructing follower 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:25.607690 14603 raft_consensus.cc:1081] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Signalling peer 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:25.609828 14335 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:45170
I20260709 12:28:25.610468 14335 raft_consensus.cc:493] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:25.610882 14335 raft_consensus.cc:3060] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:25.618305 14335 raft_consensus.cc:515] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:25.620151 14335 leader_election.cc:290] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:25.621353 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:25.621737 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4065fbe85a54437b9198d8e44423227f" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:25.621961 14557 raft_consensus.cc:3060] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:25.622385 14262 raft_consensus.cc:3055] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:25.622668 14262 raft_consensus.cc:740] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:25.623953 14262 consensus_queue.cc:260] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:25.625685 14262 raft_consensus.cc:3060] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:25.628561 14557 raft_consensus.cc:2468] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:25.629791 14302 leader_election.cc:304] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:25.630595 14616 raft_consensus.cc:2804] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:25.631367 14616 raft_consensus.cc:697] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:25.632632 14616 consensus_queue.cc:237] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:25.634757 14262 raft_consensus.cc:2468] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:25.641752 14081 catalog_manager.cc:5697] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 2 to 3, leader changed from 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 3 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
W20260709 12:28:25.673369 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:25.716560 14503 raft_consensus.cc:993] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a: : Instructing follower 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:25.717206 14616 raft_consensus.cc:1081] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Signalling peer 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:25.719152 14262 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:50884
I20260709 12:28:25.719744 14262 raft_consensus.cc:493] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:25.720073 14262 raft_consensus.cc:3060] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:25.725435 14262 raft_consensus.cc:515] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:25.727078 14262 leader_election.cc:290] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:25.728121 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:25.728575 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:25.728850 14557 raft_consensus.cc:3060] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:25.729197 14335 raft_consensus.cc:3055] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:25.729529 14335 raft_consensus.cc:740] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:25.730515 14335 consensus_queue.cc:260] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:25.732013 14335 raft_consensus.cc:3060] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:25.738261 14557 raft_consensus.cc:2468] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 2.
I20260709 12:28:25.739859 14335 raft_consensus.cc:2468] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 2.
I20260709 12:28:25.739914 14228 leader_election.cc:304] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:25.741133 14603 raft_consensus.cc:2804] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:25.741911 14603 raft_consensus.cc:697] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:25.743148 14603 consensus_queue.cc:237] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:25.751151 14081 catalog_manager.cc:5697] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 1 to 2, leader changed from 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:26.090227 14262 raft_consensus.cc:1275] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:28:26.092290 14616 consensus_queue.cc:1048] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:28:26.102522 14557 raft_consensus.cc:1275] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:28:26.104820 14503 consensus_queue.cc:1048] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:28:26.157589 14557 raft_consensus.cc:1275] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:26.157734 14558 raft_consensus.cc:1275] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:26.159168 14603 consensus_queue.cc:1048] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:26.159403 14503 consensus_queue.cc:1048] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:26.177384 14335 raft_consensus.cc:1275] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:26.179862 14262 raft_consensus.cc:1275] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:26.182098 14603 consensus_queue.cc:1048] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:26.182381 14636 consensus_queue.cc:1048] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:26.309088   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:26.318039 14335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
mode: GRACEFUL
new_leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:59264
I20260709 12:28:26.318632 14335 raft_consensus.cc:606] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Received request to transfer leadership to TS 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:26.322063 14262 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
mode: GRACEFUL
new_leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:48754
I20260709 12:28:26.322670 14262 raft_consensus.cc:606] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Received request to transfer leadership to TS 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:26.325860 14335 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
mode: GRACEFUL
new_leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:59264
I20260709 12:28:26.326455 14335 raft_consensus.cc:606] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Received request to transfer leadership to TS 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:26.327893   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:26.328512   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:26.329169   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:26.502898 14503 raft_consensus.cc:993] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a: : Instructing follower 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:26.503563 14636 raft_consensus.cc:1081] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Signalling peer 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:26.505508 14262 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:50884
I20260709 12:28:26.506065 14262 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:26.506367 14262 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:26.512521 14262 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:26.514412 14262 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 4 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:26.515481 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:26.515810 14335 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:26.516180 14557 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:26.516578 14335 raft_consensus.cc:3055] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:28:26.516916 14335 raft_consensus.cc:740] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:26.517767 14335 consensus_queue.cc:260] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:26.519021 14335 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:26.522125 14557 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 4.
I20260709 12:28:26.523502 14228 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:26.524308 14603 raft_consensus.cc:2804] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:28:26.524751 14603 raft_consensus.cc:697] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:26.525714 14603 consensus_queue.cc:237] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:26.526338 14335 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 4.
I20260709 12:28:26.534860 14084 catalog_manager.cc:5697] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 3 to 4, leader changed from 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 4 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:26.674530 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:26.754302 14615 raft_consensus.cc:993] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036: : Instructing follower 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:26.755177 14643 raft_consensus.cc:1081] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Signalling peer 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:26.757333 14337 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:45170
I20260709 12:28:26.757939 14337 raft_consensus.cc:493] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:26.758296 14337 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:26.765488 14337 raft_consensus.cc:515] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:26.767494 14337 leader_election.cc:290] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:26.768553 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:26.769264 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:26.769268 14557 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:26.769987 14262 raft_consensus.cc:3055] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:26.770342 14262 raft_consensus.cc:740] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:26.771202 14262 consensus_queue.cc:260] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:26.771931 14636 raft_consensus.cc:993] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a: : Instructing follower 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:26.772477 14262 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:26.772531 14503 raft_consensus.cc:1081] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Signalling peer 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:26.774730 14263 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:50884
I20260709 12:28:26.775301 14263 raft_consensus.cc:493] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:26.775656 14263 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:26.776491 14557 raft_consensus.cc:2468] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:26.777554 14302 leader_election.cc:304] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:26.778296 14503 raft_consensus.cc:2804] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:26.778756 14503 raft_consensus.cc:697] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:26.779790 14262 raft_consensus.cc:2468] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:26.780194 14503 consensus_queue.cc:237] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:26.782976 14263 raft_consensus.cc:515] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:26.786284 14263 leader_election.cc:290] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:26.786375 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:26.787811 14557 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:26.789273 14084 catalog_manager.cc:5697] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 2 to 3, leader changed from 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 3 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:26.786522 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:26.791524 14337 raft_consensus.cc:3055] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:26.791841 14337 raft_consensus.cc:740] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:26.792675 14337 consensus_queue.cc:260] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:26.794095 14337 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:26.796085 14557 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 2.
I20260709 12:28:26.798013 14228 leader_election.cc:304] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:26.799206 14643 raft_consensus.cc:2804] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:26.799750 14643 raft_consensus.cc:697] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:26.800894 14643 consensus_queue.cc:237] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:26.801779 14337 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 2.
I20260709 12:28:26.810590 14084 catalog_manager.cc:5697] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 1 to 2, leader changed from 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:27.079787 14337 raft_consensus.cc:1275] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:28:27.081620 14643 consensus_queue.cc:1048] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:27.094274 14557 raft_consensus.cc:1275] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:28:27.096959 14615 consensus_queue.cc:1048] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:27.281101 14557 raft_consensus.cc:1275] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:27.282550 14615 consensus_queue.cc:1048] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:27.294941 14337 raft_consensus.cc:1275] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:27.296684 14615 consensus_queue.cc:1048] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:27.297605 14263 raft_consensus.cc:1275] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:27.299314 14503 consensus_queue.cc:1048] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:27.320305 14557 raft_consensus.cc:1275] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:27.322350 14657 consensus_queue.cc:1048] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:27.336063   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:27.345677 14263 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
mode: GRACEFUL
new_leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:48754
I20260709 12:28:27.346318 14263 raft_consensus.cc:606] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Received request to transfer leadership to TS 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:27.349165 14263 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
mode: GRACEFUL
new_leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:48754
I20260709 12:28:27.349829 14263 raft_consensus.cc:606] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Received request to transfer leadership to TS 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:27.352698 14337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
mode: GRACEFUL
new_leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:59264
I20260709 12:28:27.353361 14337 raft_consensus.cc:606] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Received request to transfer leadership to TS 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:27.354983   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:27.355538   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:27.356240   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:27.605794 14615 raft_consensus.cc:993] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036: : Instructing follower 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:27.606334 14615 raft_consensus.cc:1081] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Signalling peer 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:27.608229 14337 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:45170
I20260709 12:28:27.608771 14337 raft_consensus.cc:493] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:27.609081 14337 raft_consensus.cc:3060] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:27.615384 14337 raft_consensus.cc:515] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:27.617103 14337 leader_election.cc:290] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:27.618279 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:27.618700 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66e2f6774eae4e3e94ed4d2194d0d3e7" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:27.619305 14557 raft_consensus.cc:3060] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:27.619346 14263 raft_consensus.cc:3055] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:27.619967 14263 raft_consensus.cc:740] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:27.620971 14263 consensus_queue.cc:260] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:27.622519 14263 raft_consensus.cc:3060] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:27.628490 14557 raft_consensus.cc:2468] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:27.630147 14302 leader_election.cc:304] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:27.630667 14263 raft_consensus.cc:2468] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:27.631340 14657 raft_consensus.cc:2804] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:27.632102 14657 raft_consensus.cc:697] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:27.632863 14657 consensus_queue.cc:237] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:27.642719 14085 catalog_manager.cc:5697] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 2 to 3, leader changed from 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 3 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
W20260709 12:28:27.675650 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:27.717196 14648 raft_consensus.cc:993] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036: : Instructing follower 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:27.717862 14615 raft_consensus.cc:1081] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Signalling peer 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:27.719870 14337 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:45170
I20260709 12:28:27.720499 14337 raft_consensus.cc:493] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:27.720846 14337 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:27.726543 14337 raft_consensus.cc:515] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:27.728286 14337 leader_election.cc:290] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:27.729452 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:27.730377 14263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:27.730924 14263 raft_consensus.cc:3055] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:27.730970 14557 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:27.731280 14263 raft_consensus.cc:740] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:27.732283 14263 consensus_queue.cc:260] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:27.733675 14263 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:27.739138 14557 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:27.740307 14302 leader_election.cc:304] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:27.740581 14263 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 3.
I20260709 12:28:27.741076 14503 raft_consensus.cc:2804] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:27.741652 14503 raft_consensus.cc:697] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:27.742470 14503 consensus_queue.cc:237] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:27.749763 14085 catalog_manager.cc:5697] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 2 to 3, leader changed from 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 3 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:27.838773 14657 raft_consensus.cc:993] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a: : Instructing follower 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:27.839627 14503 raft_consensus.cc:1081] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Signalling peer 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:27.841491 14262 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:50884
I20260709 12:28:27.842087 14262 raft_consensus.cc:493] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:27.842491 14262 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:27.847500 14262 raft_consensus.cc:515] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:27.849067 14262 leader_election.cc:290] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 4 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:27.850405 14557 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:27.850660 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2566a924ce24d5688df3bb577c471ad" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:27.851037 14557 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:27.851488 14337 raft_consensus.cc:3055] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:28:27.851796 14337 raft_consensus.cc:740] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:27.852641 14337 consensus_queue.cc:260] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:27.854055 14337 raft_consensus.cc:3060] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:27.857219 14557 raft_consensus.cc:2468] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 4.
I20260709 12:28:27.858559 14228 leader_election.cc:304] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:27.859544 14615 raft_consensus.cc:2804] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:28:27.860157 14615 raft_consensus.cc:697] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:27.860925 14615 consensus_queue.cc:237] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:27.861146 14337 raft_consensus.cc:2468] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 4.
I20260709 12:28:27.868163 14085 catalog_manager.cc:5697] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 3 to 4, leader changed from 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 4 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:28.147889 14262 raft_consensus.cc:1275] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:28.149641 14503 consensus_queue.cc:1048] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:28.161793 14557 raft_consensus.cc:1275] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:28.163717 14663 consensus_queue.cc:1048] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:28.230361 14557 raft_consensus.cc:1275] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:28.232163 14657 consensus_queue.cc:1048] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:28.245249 14262 raft_consensus.cc:1275] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:28.247113 14670 consensus_queue.cc:1048] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:28.257669 14337 raft_consensus.cc:1275] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:28:28.259232 14615 consensus_queue.cc:1048] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:28:28.270431 14557 raft_consensus.cc:1275] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:28:28.271983 14648 consensus_queue.cc:1048] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:28:28.362828   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:28.371142 14337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
mode: GRACEFUL
new_leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:59264
I20260709 12:28:28.371835 14337 raft_consensus.cc:606] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Received request to transfer leadership to TS 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:28.375372 14337 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
mode: GRACEFUL
new_leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:59264
I20260709 12:28:28.376087 14337 raft_consensus.cc:606] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Received request to transfer leadership to TS 661ba84d83184824bdf0dfc8dc3a5036
I20260709 12:28:28.378955 14262 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
mode: GRACEFUL
new_leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:48754
I20260709 12:28:28.379547 14262 raft_consensus.cc:606] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Received request to transfer leadership to TS 03a6fb5ac22045e4b35e11b41f53bb9a
I20260709 12:28:28.381070   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:28:28.381896   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:28.382659   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:28:28.676836 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:28:28.701241 14648 raft_consensus.cc:993] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036: : Instructing follower 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:28.701820 14615 raft_consensus.cc:1081] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Signalling peer 03a6fb5ac22045e4b35e11b41f53bb9a to start an election
I20260709 12:28:28.703392 14337 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb"
dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
 from {username='slave'} at 127.0.0.1:45170
I20260709 12:28:28.703889 14337 raft_consensus.cc:493] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:28.704136 14337 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:28:28.709873 14337 raft_consensus.cc:515] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:28.711388 14337 leader_election.cc:290] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 5 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877)
I20260709 12:28:28.712603 14558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:28.712905 14262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5bd9df3750f4a3589fd9d5adfeb6adb" candidate_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
I20260709 12:28:28.713423 14558 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:28:28.713538 14262 raft_consensus.cc:3055] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:28:28.713990 14262 raft_consensus.cc:740] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:28.714666 14262 consensus_queue.cc:260] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:28.715636 14262 raft_consensus.cc:3060] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:28:28.720299 14262 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 5.
I20260709 12:28:28.720574 14558 raft_consensus.cc:2468] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03a6fb5ac22045e4b35e11b41f53bb9a in term 5.
I20260709 12:28:28.721683 14302 leader_election.cc:304] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03a6fb5ac22045e4b35e11b41f53bb9a, 661ba84d83184824bdf0dfc8dc3a5036; no voters: 
I20260709 12:28:28.722479 14663 raft_consensus.cc:2804] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:28:28.722967 14663 raft_consensus.cc:697] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 5 LEADER]: Becoming Leader. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:28.723604 14663 consensus_queue.cc:237] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:28.731988 14081 catalog_manager.cc:5697] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a reported cstate change: term changed from 4 to 5, leader changed from 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66) to 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67). New cstate: current_term: 5 leader_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:28.755241 14670 raft_consensus.cc:993] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a: : Instructing follower 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:28.755786 14663 raft_consensus.cc:1081] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Signalling peer 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:28.757694 14262 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:50884
I20260709 12:28:28.758273 14262 raft_consensus.cc:493] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:28.758585 14262 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:28.764046 14262 raft_consensus.cc:515] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:28.765779 14262 leader_election.cc:290] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 4 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:28.767102 14558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:28.767232 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "680e16c044cb4adca91a1a5a1a072516" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:28.768242 14558 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:28.768364 14337 raft_consensus.cc:3055] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:28:28.768986 14337 raft_consensus.cc:740] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:28.769831 14337 consensus_queue.cc:260] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:28.771171 14337 raft_consensus.cc:3060] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:28.777652 14558 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 4.
I20260709 12:28:28.779477 14337 raft_consensus.cc:2468] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 4.
I20260709 12:28:28.779541 14228 leader_election.cc:304] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:28.780795 14615 raft_consensus.cc:2804] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:28:28.781339 14615 raft_consensus.cc:697] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:28.781347 14503 raft_consensus.cc:993] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a: : Instructing follower 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:28.782243 14663 raft_consensus.cc:1081] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Signalling peer 661ba84d83184824bdf0dfc8dc3a5036 to start an election
I20260709 12:28:28.782341 14615 consensus_queue.cc:237] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:28.784262 14262 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62"
dest_uuid: "661ba84d83184824bdf0dfc8dc3a5036"
 from {username='slave'} at 127.0.0.1:50884
I20260709 12:28:28.784886 14262 raft_consensus.cc:493] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:28.785284 14262 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:28.791983 14081 catalog_manager.cc:5697] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 3 to 4, leader changed from 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 4 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:28.792824 14262 raft_consensus.cc:515] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:28.795001 14262 leader_election.cc:290] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 4 election: Requested vote from peers e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923), 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67:38711)
I20260709 12:28:28.796223 14337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a"
I20260709 12:28:28.796195 14558 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "772c8428cc824192a9e43f11e1956b62" candidate_uuid: "661ba84d83184824bdf0dfc8dc3a5036" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e767df82fbf4455395ffd6fa8d992d59"
I20260709 12:28:28.797209 14337 raft_consensus.cc:3055] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:28:28.797290 14558 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:28.797667 14337 raft_consensus.cc:740] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 03a6fb5ac22045e4b35e11b41f53bb9a, State: Running, Role: LEADER
I20260709 12:28:28.798615 14337 consensus_queue.cc:260] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:28.800021 14337 raft_consensus.cc:3060] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:28:28.807765 14337 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 4.
I20260709 12:28:28.807765 14558 raft_consensus.cc:2468] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 661ba84d83184824bdf0dfc8dc3a5036 in term 4.
I20260709 12:28:28.813369 14228 leader_election.cc:304] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 661ba84d83184824bdf0dfc8dc3a5036, e767df82fbf4455395ffd6fa8d992d59; no voters: 
I20260709 12:28:28.814195 14648 raft_consensus.cc:2804] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:28:28.814740 14648 raft_consensus.cc:697] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Becoming Leader. State: Replica: 661ba84d83184824bdf0dfc8dc3a5036, State: Running, Role: LEADER
I20260709 12:28:28.815512 14648 consensus_queue.cc:237] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } }
I20260709 12:28:28.822124 14081 catalog_manager.cc:5697] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 reported cstate change: term changed from 3 to 4, leader changed from 03a6fb5ac22045e4b35e11b41f53bb9a (127.0.140.67) to 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66). New cstate: current_term: 4 leader_uuid: "661ba84d83184824bdf0dfc8dc3a5036" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:29.222230 14558 raft_consensus.cc:1275] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 5 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:28:29.223668 14663 consensus_queue.cc:1048] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:28:29.235214 14262 raft_consensus.cc:1275] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 5 FOLLOWER]: Refusing update from remote peer 03a6fb5ac22045e4b35e11b41f53bb9a: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:28:29.238245 14670 consensus_queue.cc:1048] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "661ba84d83184824bdf0dfc8dc3a5036" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 33877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:28:29.295132 14337 raft_consensus.cc:1275] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:28:29.295132 14558 raft_consensus.cc:1275] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:28:29.297096 14615 consensus_queue.cc:1048] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:28:29.297936 14648 consensus_queue.cc:1048] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:28:29.328058 14337 raft_consensus.cc:1275] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:28:29.329646 14677 consensus_queue.cc:1048] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03a6fb5ac22045e4b35e11b41f53bb9a" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 38711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:29.341295 14558 raft_consensus.cc:1275] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Refusing update from remote peer 661ba84d83184824bdf0dfc8dc3a5036: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:28:29.343588 14648 consensus_queue.cc:1048] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e767df82fbf4455395ffd6fa8d992d59" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:29.389683   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:29.418748   561 tablet_server.cc:179] TabletServer@127.0.140.65:43923 shutting down...
I20260709 12:28:29.442476   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:29.443413   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.444096   561 raft_consensus.cc:2243] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.444657   561 raft_consensus.cc:2272] T 772c8428cc824192a9e43f11e1956b62 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.447569   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.448221   561 raft_consensus.cc:2243] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.448724   561 raft_consensus.cc:2272] T 680e16c044cb4adca91a1a5a1a072516 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.451592   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.452303   561 raft_consensus.cc:2243] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.452750   561 raft_consensus.cc:2272] T fbf7e8af7d2f437ebbea676e59903938 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.454937   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.455482   561 raft_consensus.cc:2243] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.455901   561 raft_consensus.cc:2272] T 89967a24adc644b1a86747eb3d6e3384 P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.458429   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.459157   561 raft_consensus.cc:2243] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.459610   561 raft_consensus.cc:2272] T b6131f7efca94584a3892ade802c0781 P e767df82fbf4455395ffd6fa8d992d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.461881   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.462467   561 raft_consensus.cc:2243] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.463011   561 raft_consensus.cc:2272] T 90dd16b1f50e479cbe0a84f08d753d6f P e767df82fbf4455395ffd6fa8d992d59 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:29.464390 14302 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 97 similar messages]
I20260709 12:28:29.465823   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.466563   561 raft_consensus.cc:2243] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.467268   561 raft_consensus.cc:2272] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:29.468189 14302 consensus_peers.cc:597] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:29.469862   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.470506   561 raft_consensus.cc:2243] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.471058   561 raft_consensus.cc:2272] T f5bd9df3750f4a3589fd9d5adfeb6adb P e767df82fbf4455395ffd6fa8d992d59 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.473680   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.474207   561 raft_consensus.cc:2243] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.474643   561 raft_consensus.cc:2272] T d2566a924ce24d5688df3bb577c471ad P e767df82fbf4455395ffd6fa8d992d59 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.476702   561 tablet_replica.cc:333] stopping tablet replica
I20260709 12:28:29.477171   561 raft_consensus.cc:2243] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.477566   561 raft_consensus.cc:2272] T 4065fbe85a54437b9198d8e44423227f P e767df82fbf4455395ffd6fa8d992d59 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.509913   561 tablet_server.cc:196] TabletServer@127.0.140.65:43923 shutdown complete.
W20260709 12:28:29.513826 14228 consensus_peers.cc:597] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 -> Peer e767df82fbf4455395ffd6fa8d992d59 (127.0.140.65:43923): Couldn't send request to peer e767df82fbf4455395ffd6fa8d992d59. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:29.535053   561 tablet_server.cc:179] TabletServer@127.0.140.66:0 shutting down...
I20260709 12:28:29.566907   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:29.567713   561 tablet_replica.cc:333] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.568550   561 raft_consensus.cc:2243] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.569116   561 raft_consensus.cc:2272] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.572017   561 tablet_replica.cc:333] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.572799   561 raft_consensus.cc:2243] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.573360   561 raft_consensus.cc:2272] T 90dd16b1f50e479cbe0a84f08d753d6f P 661ba84d83184824bdf0dfc8dc3a5036 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.576032   561 tablet_replica.cc:333] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.576658   561 raft_consensus.cc:2243] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.577607   561 raft_consensus.cc:2272] T 680e16c044cb4adca91a1a5a1a072516 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.580729   561 tablet_replica.cc:333] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.581372   561 raft_consensus.cc:2243] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.582248   561 raft_consensus.cc:2272] T 772c8428cc824192a9e43f11e1956b62 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.585821   561 tablet_replica.cc:333] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.586674   561 raft_consensus.cc:2243] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.587266   561 raft_consensus.cc:2272] T f5bd9df3750f4a3589fd9d5adfeb6adb P 661ba84d83184824bdf0dfc8dc3a5036 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.590695   561 tablet_replica.cc:333] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.591806   561 raft_consensus.cc:2243] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.593168   561 raft_consensus.cc:2272] T fbf7e8af7d2f437ebbea676e59903938 P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.597011   561 tablet_replica.cc:333] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.597735   561 raft_consensus.cc:2243] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.598244   561 raft_consensus.cc:2272] T 89967a24adc644b1a86747eb3d6e3384 P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.600943   561 tablet_replica.cc:333] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.601536   561 raft_consensus.cc:2243] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.602005   561 raft_consensus.cc:2272] T 4065fbe85a54437b9198d8e44423227f P 661ba84d83184824bdf0dfc8dc3a5036 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.604436   561 tablet_replica.cc:333] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.604960   561 raft_consensus.cc:2243] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.605886   561 raft_consensus.cc:2272] T d2566a924ce24d5688df3bb577c471ad P 661ba84d83184824bdf0dfc8dc3a5036 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.608428   561 tablet_replica.cc:333] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036: stopping tablet replica
I20260709 12:28:29.609009   561 raft_consensus.cc:2243] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.610029   561 raft_consensus.cc:2272] T b6131f7efca94584a3892ade802c0781 P 661ba84d83184824bdf0dfc8dc3a5036 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.646464   561 tablet_server.cc:196] TabletServer@127.0.140.66:0 shutdown complete.
I20260709 12:28:29.677100   561 tablet_server.cc:179] TabletServer@127.0.140.67:0 shutting down...
W20260709 12:28:29.678176 14134 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:28:29.683379 14302 consensus_peers.cc:597] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a -> Peer 661ba84d83184824bdf0dfc8dc3a5036 (127.0.140.66:33877): Couldn't send request to peer 661ba84d83184824bdf0dfc8dc3a5036. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:33877: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:29.712450   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:29.713241   561 tablet_replica.cc:333] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.714287   561 raft_consensus.cc:2243] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.715544   561 raft_consensus.cc:2272] T f5bd9df3750f4a3589fd9d5adfeb6adb P 03a6fb5ac22045e4b35e11b41f53bb9a [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.718219   561 tablet_replica.cc:333] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.718837   561 raft_consensus.cc:2243] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.719252   561 raft_consensus.cc:2272] T d2566a924ce24d5688df3bb577c471ad P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.721333   561 tablet_replica.cc:333] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.721909   561 raft_consensus.cc:2243] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.722651   561 raft_consensus.cc:2272] T 4065fbe85a54437b9198d8e44423227f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.724792   561 tablet_replica.cc:333] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.725304   561 raft_consensus.cc:2243] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.725674   561 raft_consensus.cc:2272] T 772c8428cc824192a9e43f11e1956b62 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.728901   561 tablet_replica.cc:333] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.729532   561 raft_consensus.cc:2243] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.729979   561 raft_consensus.cc:2272] T b6131f7efca94584a3892ade802c0781 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.732288   561 tablet_replica.cc:333] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.732812   561 raft_consensus.cc:2243] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.733208   561 raft_consensus.cc:2272] T fbf7e8af7d2f437ebbea676e59903938 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.735519   561 tablet_replica.cc:333] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.736104   561 raft_consensus.cc:2243] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.736900   561 raft_consensus.cc:2272] T 89967a24adc644b1a86747eb3d6e3384 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.739086   561 tablet_replica.cc:333] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.739650   561 raft_consensus.cc:2243] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:29.740082   561 raft_consensus.cc:2272] T 680e16c044cb4adca91a1a5a1a072516 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.743033   561 tablet_replica.cc:333] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.743597   561 raft_consensus.cc:2243] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.744414   561 raft_consensus.cc:2272] T 90dd16b1f50e479cbe0a84f08d753d6f P 03a6fb5ac22045e4b35e11b41f53bb9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.746507   561 tablet_replica.cc:333] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a: stopping tablet replica
I20260709 12:28:29.747066   561 raft_consensus.cc:2243] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.747882   561 raft_consensus.cc:2272] T 66e2f6774eae4e3e94ed4d2194d0d3e7 P 03a6fb5ac22045e4b35e11b41f53bb9a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.781631   561 tablet_server.cc:196] TabletServer@127.0.140.67:0 shutdown complete.
I20260709 12:28:29.809294   561 master.cc:562] Master@127.0.140.126:46745 shutting down...
I20260709 12:28:29.840816   561 raft_consensus.cc:2243] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:29.841574   561 raft_consensus.cc:2272] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:29.841922   561 tablet_replica.cc:333] T 00000000000000000000000000000000 P bff96fb285a644399a4fe632a1eb2c4b: stopping tablet replica
I20260709 12:28:29.867767   561 master.cc:584] Master@127.0.140.126:46745 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (13443 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 12:28:29.918885   561 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.140.126:38779
I20260709 12:28:29.920082   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:29.927259 14695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:29.927186 14694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:29.929157 14697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:29.929314   561 server_base.cc:1061] running on GCE node
I20260709 12:28:29.930562   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:29.930795   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:29.931144   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600109931091 us; error 0 us; skew 500 ppm
I20260709 12:28:29.931854   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:29.935034   561 webserver.cc:533] Webserver started at http://127.0.140.126:35897/ using document root <none> and password file <none>
I20260709 12:28:29.935652   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:29.935870   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:29.936206   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:29.937623   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/master-0-root/instance:
uuid: "13e24fee2c514a638a58981f934f9e67"
format_stamp: "Formatted at 2026-07-09 12:28:29 on dist-test-slave-7mcn"
I20260709 12:28:29.943115   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20260709 12:28:29.947438 14702 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:29.948359   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.002s
I20260709 12:28:29.948712   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/master-0-root
uuid: "13e24fee2c514a638a58981f934f9e67"
format_stamp: "Formatted at 2026-07-09 12:28:29 on dist-test-slave-7mcn"
I20260709 12:28:29.949026   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:29.977100   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:29.978642   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:30.028636   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.126:38779
I20260709 12:28:30.028782 14753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.126:38779 every 8 connection(s)
I20260709 12:28:30.033687 14754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:30.049427 14754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67: Bootstrap starting.
I20260709 12:28:30.056808 14754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:30.062525 14754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67: No bootstrap required, opened a new log
I20260709 12:28:30.065407 14754 raft_consensus.cc:359] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e24fee2c514a638a58981f934f9e67" member_type: VOTER }
I20260709 12:28:30.066138 14754 raft_consensus.cc:385] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:30.066461 14754 raft_consensus.cc:740] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13e24fee2c514a638a58981f934f9e67, State: Initialized, Role: FOLLOWER
I20260709 12:28:30.067360 14754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e24fee2c514a638a58981f934f9e67" member_type: VOTER }
I20260709 12:28:30.068109 14754 raft_consensus.cc:399] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:28:30.068434 14754 raft_consensus.cc:493] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:28:30.068892 14754 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:30.075713 14754 raft_consensus.cc:515] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e24fee2c514a638a58981f934f9e67" member_type: VOTER }
I20260709 12:28:30.076447 14754 leader_election.cc:304] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13e24fee2c514a638a58981f934f9e67; no voters: 
I20260709 12:28:30.077865 14754 leader_election.cc:290] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:28:30.078398 14757 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:30.080010 14757 raft_consensus.cc:697] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 1 LEADER]: Becoming Leader. State: Replica: 13e24fee2c514a638a58981f934f9e67, State: Running, Role: LEADER
I20260709 12:28:30.080729 14757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e24fee2c514a638a58981f934f9e67" member_type: VOTER }
I20260709 12:28:30.081473 14754 sys_catalog.cc:565] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:28:30.085863 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13e24fee2c514a638a58981f934f9e67. Latest consensus state: current_term: 1 leader_uuid: "13e24fee2c514a638a58981f934f9e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e24fee2c514a638a58981f934f9e67" member_type: VOTER } }
I20260709 12:28:30.086539 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:30.086340 14758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13e24fee2c514a638a58981f934f9e67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13e24fee2c514a638a58981f934f9e67" member_type: VOTER } }
I20260709 12:28:30.087493 14758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:30.095707 14763 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:28:30.100906 14763 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:28:30.104862   561 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:28:30.110772 14763 catalog_manager.cc:1383] Generated new cluster ID: 2c66f583b670404e8d4be607daa675da
I20260709 12:28:30.111184 14763 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:28:30.149727 14763 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:28:30.151414 14763 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:28:30.169060 14763 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67: Generated new TSK 0
I20260709 12:28:30.169736 14763 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:28:30.236866   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:30.245152 14775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:30.246670 14776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:30.249771 14778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:30.249836   561 server_base.cc:1061] running on GCE node
I20260709 12:28:30.251636   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:30.251937   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:30.252158   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600110252135 us; error 0 us; skew 500 ppm
I20260709 12:28:30.252859   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:30.256481   561 webserver.cc:533] Webserver started at http://127.0.140.65:38275/ using document root <none> and password file <none>
I20260709 12:28:30.257143   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:30.257395   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:30.257718   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:30.259179   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-0-root/instance:
uuid: "4a4b6102eb3f479783d0cfae19c6fa49"
format_stamp: "Formatted at 2026-07-09 12:28:30 on dist-test-slave-7mcn"
I20260709 12:28:30.264817   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:30.268920 14783 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:30.269865   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:28:30.270238   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "4a4b6102eb3f479783d0cfae19c6fa49"
format_stamp: "Formatted at 2026-07-09 12:28:30 on dist-test-slave-7mcn"
I20260709 12:28:30.270576   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:30.292989   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:30.294569   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:30.296516   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:30.299528   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:30.299809   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:30.300098   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:30.300284   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:30.353283   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:44535
I20260709 12:28:30.353395 14845 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:44535 every 8 connection(s)
I20260709 12:28:30.359771   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:30.369577 14850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:30.371805 14851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:30.374756 14846 heartbeater.cc:344] Connected to a master server at 127.0.140.126:38779
I20260709 12:28:30.375319 14846 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:30.376178 14846 heartbeater.cc:507] Master 127.0.140.126:38779 requested a full tablet report, sending...
I20260709 12:28:30.380254 14719 ts_manager.cc:194] Registered new tserver with Master: 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65:44535)
W20260709 12:28:30.381721 14853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:30.381779   561 server_base.cc:1061] running on GCE node
I20260709 12:28:30.383205 14719 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53296
I20260709 12:28:30.383508   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:30.383862   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:30.384078   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600110384049 us; error 0 us; skew 500 ppm
I20260709 12:28:30.384919   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:30.387931   561 webserver.cc:533] Webserver started at http://127.0.140.66:35609/ using document root <none> and password file <none>
I20260709 12:28:30.388449   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:30.388677   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:30.388973   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:30.390094   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-1-root/instance:
uuid: "6794c11484634742a8728fbcdc8c8cec"
format_stamp: "Formatted at 2026-07-09 12:28:30 on dist-test-slave-7mcn"
I20260709 12:28:30.395255   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:28:30.399107 14858 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:30.400177   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:28:30.400619   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-1-root
uuid: "6794c11484634742a8728fbcdc8c8cec"
format_stamp: "Formatted at 2026-07-09 12:28:30 on dist-test-slave-7mcn"
I20260709 12:28:30.400993   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:30.427414   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:30.428712   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:30.430299   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:30.433070   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:30.433334   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:30.433594   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:30.433768   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:30.479293   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.66:46021
I20260709 12:28:30.479521 14920 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.66:46021 every 8 connection(s)
I20260709 12:28:30.484479   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:30.493431 14924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:30.495608 14925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:30.498000 14927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:30.498153   561 server_base.cc:1061] running on GCE node
I20260709 12:28:30.499617   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:30.500037   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:30.499758 14921 heartbeater.cc:344] Connected to a master server at 127.0.140.126:38779
I20260709 12:28:30.500397   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600110500380 us; error 0 us; skew 500 ppm
I20260709 12:28:30.500689 14921 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:30.501166   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:30.501711 14921 heartbeater.cc:507] Master 127.0.140.126:38779 requested a full tablet report, sending...
I20260709 12:28:30.504490   561 webserver.cc:533] Webserver started at http://127.0.140.67:38987/ using document root <none> and password file <none>
I20260709 12:28:30.504837 14719 ts_manager.cc:194] Registered new tserver with Master: 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021)
I20260709 12:28:30.505280   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:30.505604   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:30.505937   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:30.506749 14719 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53308
I20260709 12:28:30.507520   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-2-root/instance:
uuid: "b661494dcef8498b9af969b7772a0d62"
format_stamp: "Formatted at 2026-07-09 12:28:30 on dist-test-slave-7mcn"
I20260709 12:28:30.512770   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.001s
I20260709 12:28:30.517027 14932 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:30.517987   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 12:28:30.518376   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-2-root
uuid: "b661494dcef8498b9af969b7772a0d62"
format_stamp: "Formatted at 2026-07-09 12:28:30 on dist-test-slave-7mcn"
I20260709 12:28:30.518687   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599624140507-561-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:30.540187   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:30.541918   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:30.543788   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:30.546403   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:30.546612   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:30.547006   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:30.547194   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:30.599014   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.67:45671
I20260709 12:28:30.599146 14994 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.67:45671 every 8 connection(s)
I20260709 12:28:30.615341 14995 heartbeater.cc:344] Connected to a master server at 127.0.140.126:38779
I20260709 12:28:30.616080 14995 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:30.617075 14995 heartbeater.cc:507] Master 127.0.140.126:38779 requested a full tablet report, sending...
I20260709 12:28:30.619477 14719 ts_manager.cc:194] Registered new tserver with Master: b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:30.620255   561 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017575517s
I20260709 12:28:30.621251 14719 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53316
I20260709 12:28:30.652088 14719 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53330:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:30.655056 14719 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:30.714450 14886 tablet_service.cc:1511] Processing CreateTablet for tablet db68371d71cd4be59f981f26ebf0aced (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b8038f154bbe4c9997084669ce502701]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:30.716385 14886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db68371d71cd4be59f981f26ebf0aced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:30.717492 14885 tablet_service.cc:1511] Processing CreateTablet for tablet 308d27089579460195356087edb20840 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b8038f154bbe4c9997084669ce502701]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:30.719254 14885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308d27089579460195356087edb20840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:30.728056 14960 tablet_service.cc:1511] Processing CreateTablet for tablet db68371d71cd4be59f981f26ebf0aced (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b8038f154bbe4c9997084669ce502701]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:30.730645 14960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db68371d71cd4be59f981f26ebf0aced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:30.734170 14959 tablet_service.cc:1511] Processing CreateTablet for tablet 308d27089579460195356087edb20840 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b8038f154bbe4c9997084669ce502701]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:30.736442 14959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308d27089579460195356087edb20840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:30.746299 14811 tablet_service.cc:1511] Processing CreateTablet for tablet db68371d71cd4be59f981f26ebf0aced (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b8038f154bbe4c9997084669ce502701]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:30.748245 14810 tablet_service.cc:1511] Processing CreateTablet for tablet 308d27089579460195356087edb20840 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=b8038f154bbe4c9997084669ce502701]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:30.749212 14811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db68371d71cd4be59f981f26ebf0aced. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:30.750090 14810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308d27089579460195356087edb20840. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:30.770102 15010 tablet_bootstrap.cc:492] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec: Bootstrap starting.
I20260709 12:28:30.781169 15010 tablet_bootstrap.cc:654] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:30.785283 15012 tablet_bootstrap.cc:492] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49: Bootstrap starting.
I20260709 12:28:30.785835 15011 tablet_bootstrap.cc:492] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62: Bootstrap starting.
I20260709 12:28:30.790094 15010 tablet_bootstrap.cc:492] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec: No bootstrap required, opened a new log
I20260709 12:28:30.790891 15010 ts_tablet_manager.cc:1403] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.004s
I20260709 12:28:30.793726 15011 tablet_bootstrap.cc:654] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:30.794556 15010 raft_consensus.cc:359] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.795477 15012 tablet_bootstrap.cc:654] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:30.795522 15010 raft_consensus.cc:385] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:30.796048 15010 raft_consensus.cc:740] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6794c11484634742a8728fbcdc8c8cec, State: Initialized, Role: FOLLOWER
I20260709 12:28:30.796970 15010 consensus_queue.cc:260] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.800189 14921 heartbeater.cc:499] Master 127.0.140.126:38779 was elected leader, sending a full tablet report...
I20260709 12:28:30.800558 15010 ts_tablet_manager.cc:1434] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.007s
I20260709 12:28:30.802011 15010 tablet_bootstrap.cc:492] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec: Bootstrap starting.
I20260709 12:28:30.810988 15011 tablet_bootstrap.cc:492] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62: No bootstrap required, opened a new log
I20260709 12:28:30.811734 15011 ts_tablet_manager.cc:1403] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62: Time spent bootstrapping tablet: real 0.029s	user 0.014s	sys 0.010s
I20260709 12:28:30.811754 15010 tablet_bootstrap.cc:654] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:30.815109 15012 tablet_bootstrap.cc:492] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49: No bootstrap required, opened a new log
I20260709 12:28:30.815742 15012 ts_tablet_manager.cc:1403] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.010s
I20260709 12:28:30.815719 15011 raft_consensus.cc:359] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.816727 15011 raft_consensus.cc:385] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:30.817122 15011 raft_consensus.cc:740] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661494dcef8498b9af969b7772a0d62, State: Initialized, Role: FOLLOWER
I20260709 12:28:30.818070 15011 consensus_queue.cc:260] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.821326 15012 raft_consensus.cc:359] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.822676 15012 raft_consensus.cc:385] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:30.823148 15012 raft_consensus.cc:740] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4b6102eb3f479783d0cfae19c6fa49, State: Initialized, Role: FOLLOWER
I20260709 12:28:30.824510 15012 consensus_queue.cc:260] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.825809 14995 heartbeater.cc:499] Master 127.0.140.126:38779 was elected leader, sending a full tablet report...
I20260709 12:28:30.827374 15015 raft_consensus.cc:493] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:30.827950 15010 tablet_bootstrap.cc:492] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec: No bootstrap required, opened a new log
I20260709 12:28:30.828189 15012 ts_tablet_manager.cc:1434] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.001s
I20260709 12:28:30.828797 15010 ts_tablet_manager.cc:1403] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec: Time spent bootstrapping tablet: real 0.027s	user 0.013s	sys 0.011s
I20260709 12:28:30.828038 15015 raft_consensus.cc:515] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.830181 15012 tablet_bootstrap.cc:492] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49: Bootstrap starting.
I20260709 12:28:30.831162 14846 heartbeater.cc:499] Master 127.0.140.126:38779 was elected leader, sending a full tablet report...
I20260709 12:28:30.836074 15010 raft_consensus.cc:359] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.837057 15010 raft_consensus.cc:385] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:30.837421 15010 raft_consensus.cc:740] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6794c11484634742a8728fbcdc8c8cec, State: Initialized, Role: FOLLOWER
I20260709 12:28:30.838313 15012 tablet_bootstrap.cc:654] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:30.838428 15010 consensus_queue.cc:260] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.848667 14821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "6794c11484634742a8728fbcdc8c8cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" is_pre_election: true
I20260709 12:28:30.849949 15011 ts_tablet_manager.cc:1434] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62: Time spent starting tablet: real 0.038s	user 0.009s	sys 0.005s
W20260709 12:28:30.851583 14862 leader_election.cc:343] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65:44535): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:30.856400 15015 leader_election.cc:290] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65:44535), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:30.871773 15011 tablet_bootstrap.cc:492] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62: Bootstrap starting.
I20260709 12:28:30.880071 15018 raft_consensus.cc:493] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:30.880950 15018 raft_consensus.cc:515] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.883399 15011 tablet_bootstrap.cc:654] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:30.892136 15018 leader_election.cc:290] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021), 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65:44535)
I20260709 12:28:30.899273 15019 raft_consensus.cc:493] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:30.900173 15019 raft_consensus.cc:515] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.902585 15011 tablet_bootstrap.cc:492] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62: No bootstrap required, opened a new log
I20260709 12:28:30.903393 15011 ts_tablet_manager.cc:1403] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62: Time spent bootstrapping tablet: real 0.032s	user 0.018s	sys 0.007s
I20260709 12:28:30.913633 15019 leader_election.cc:290] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:30.927672 15011 raft_consensus.cc:359] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.929883 15011 raft_consensus.cc:385] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:30.935459 14896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "b661494dcef8498b9af969b7772a0d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6794c11484634742a8728fbcdc8c8cec" is_pre_election: true
I20260709 12:28:30.936566 14896 raft_consensus.cc:2468] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b661494dcef8498b9af969b7772a0d62 in term 0.
I20260709 12:28:30.936578 14970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "6794c11484634742a8728fbcdc8c8cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62" is_pre_election: true
I20260709 12:28:30.937634 14970 raft_consensus.cc:2468] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6794c11484634742a8728fbcdc8c8cec in term 0.
I20260709 12:28:30.938385 14936 leader_election.cc:304] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6794c11484634742a8728fbcdc8c8cec, b661494dcef8498b9af969b7772a0d62; no voters: 
I20260709 12:28:30.939338 14861 leader_election.cc:304] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6794c11484634742a8728fbcdc8c8cec, b661494dcef8498b9af969b7772a0d62; no voters: 4a4b6102eb3f479783d0cfae19c6fa49
I20260709 12:28:30.940960 15018 raft_consensus.cc:2804] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:30.941424 15018 raft_consensus.cc:493] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:30.941828 15018 raft_consensus.cc:3060] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:30.942984 15015 raft_consensus.cc:2804] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:30.943445 15015 raft_consensus.cc:493] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:30.930423 15011 raft_consensus.cc:740] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661494dcef8498b9af969b7772a0d62, State: Initialized, Role: FOLLOWER
I20260709 12:28:30.943832 15015 raft_consensus.cc:3060] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:30.947358 15010 ts_tablet_manager.cc:1434] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec: Time spent starting tablet: real 0.118s	user 0.045s	sys 0.046s
I20260709 12:28:30.947937 15012 tablet_bootstrap.cc:492] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49: No bootstrap required, opened a new log
I20260709 12:28:30.948597 15012 ts_tablet_manager.cc:1403] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49: Time spent bootstrapping tablet: real 0.119s	user 0.040s	sys 0.043s
I20260709 12:28:30.948403 15011 consensus_queue.cc:260] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.953097 15012 raft_consensus.cc:359] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.955377 15012 raft_consensus.cc:385] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:30.955773 15012 raft_consensus.cc:740] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4b6102eb3f479783d0cfae19c6fa49, State: Initialized, Role: FOLLOWER
I20260709 12:28:30.956745 15012 consensus_queue.cc:260] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.958300 14821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "b661494dcef8498b9af969b7772a0d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" is_pre_election: true
I20260709 12:28:30.959095 14821 raft_consensus.cc:2468] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b661494dcef8498b9af969b7772a0d62 in term 0.
I20260709 12:28:30.961969 15012 ts_tablet_manager.cc:1434] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.001s
I20260709 12:28:30.964076 15020 raft_consensus.cc:493] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:30.964105 14896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "308d27089579460195356087edb20840" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6794c11484634742a8728fbcdc8c8cec" is_pre_election: true
I20260709 12:28:30.971331 15015 raft_consensus.cc:515] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.975106 14821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "6794c11484634742a8728fbcdc8c8cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4b6102eb3f479783d0cfae19c6fa49"
I20260709 12:28:30.975375 15011 ts_tablet_manager.cc:1434] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62: Time spent starting tablet: real 0.072s	user 0.015s	sys 0.008s
I20260709 12:28:30.975836 14821 raft_consensus.cc:3060] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:30.964756 15020 raft_consensus.cc:515] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.976130 15018 raft_consensus.cc:515] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:30.979833 14896 raft_consensus.cc:2468] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 0.
I20260709 12:28:30.981602 15020 leader_election.cc:290] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65:44535), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:30.982743 14895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "b661494dcef8498b9af969b7772a0d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6794c11484634742a8728fbcdc8c8cec"
I20260709 12:28:30.983789 14787 leader_election.cc:304] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4b6102eb3f479783d0cfae19c6fa49, 6794c11484634742a8728fbcdc8c8cec; no voters: 
I20260709 12:28:30.984184 14970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "308d27089579460195356087edb20840" candidate_uuid: "6794c11484634742a8728fbcdc8c8cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62" is_pre_election: true
I20260709 12:28:30.984771 15019 raft_consensus.cc:2804] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:30.985073 14970 raft_consensus.cc:2468] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6794c11484634742a8728fbcdc8c8cec in term 0.
I20260709 12:28:30.985276 15019 raft_consensus.cc:493] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:30.986959 15019 raft_consensus.cc:3060] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:30.988977 15018 leader_election.cc:290] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [CANDIDATE]: Term 1 election: Requested vote from peers 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021), 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65:44535)
I20260709 12:28:30.989413 14861 leader_election.cc:304] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6794c11484634742a8728fbcdc8c8cec, b661494dcef8498b9af969b7772a0d62; no voters: 
I20260709 12:28:30.987182 14820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "308d27089579460195356087edb20840" candidate_uuid: "6794c11484634742a8728fbcdc8c8cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" is_pre_election: true
I20260709 12:28:30.990325 15020 raft_consensus.cc:2804] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:30.983786 14895 raft_consensus.cc:2393] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b661494dcef8498b9af969b7772a0d62 in current term 1: Already voted for candidate 6794c11484634742a8728fbcdc8c8cec in this term.
I20260709 12:28:30.991101 15020 raft_consensus.cc:493] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:30.991572 15020 raft_consensus.cc:3060] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:30.990252 14819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "b661494dcef8498b9af969b7772a0d62" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4b6102eb3f479783d0cfae19c6fa49"
I20260709 12:28:30.993922 14936 leader_election.cc:304] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b661494dcef8498b9af969b7772a0d62; no voters: 4a4b6102eb3f479783d0cfae19c6fa49, 6794c11484634742a8728fbcdc8c8cec
I20260709 12:28:30.994264 14970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "308d27089579460195356087edb20840" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62" is_pre_election: true
I20260709 12:28:30.995474 15018 raft_consensus.cc:2749] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:28:30.995589 14970 raft_consensus.cc:2468] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 0.
I20260709 12:28:30.999500 15015 leader_election.cc:290] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [CANDIDATE]: Term 1 election: Requested vote from peers 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65:44535), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:31.000226 14821 raft_consensus.cc:2468] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6794c11484634742a8728fbcdc8c8cec in term 1.
I20260709 12:28:31.001030 14970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "6794c11484634742a8728fbcdc8c8cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62"
I20260709 12:28:31.001111 15020 raft_consensus.cc:515] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.001458 14862 leader_election.cc:304] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4b6102eb3f479783d0cfae19c6fa49, 6794c11484634742a8728fbcdc8c8cec; no voters: 
I20260709 12:28:31.002197 14970 raft_consensus.cc:2393] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6794c11484634742a8728fbcdc8c8cec in current term 1: Already voted for candidate b661494dcef8498b9af969b7772a0d62 in this term.
I20260709 12:28:31.002599 15015 raft_consensus.cc:2804] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:31.004514 15020 leader_election.cc:290] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [CANDIDATE]: Term 1 election: Requested vote from peers 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65:44535), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:31.005142 14970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "308d27089579460195356087edb20840" candidate_uuid: "6794c11484634742a8728fbcdc8c8cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62"
I20260709 12:28:31.005836 14970 raft_consensus.cc:3060] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:31.006143 15019 raft_consensus.cc:515] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.006860 14821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "308d27089579460195356087edb20840" candidate_uuid: "6794c11484634742a8728fbcdc8c8cec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a4b6102eb3f479783d0cfae19c6fa49"
I20260709 12:28:31.007928 14820 raft_consensus.cc:2393] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6794c11484634742a8728fbcdc8c8cec in current term 1: Already voted for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in this term.
I20260709 12:28:31.008499 15015 raft_consensus.cc:697] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 1 LEADER]: Becoming Leader. State: Replica: 6794c11484634742a8728fbcdc8c8cec, State: Running, Role: LEADER
I20260709 12:28:31.009708 15015 consensus_queue.cc:237] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.011962 14895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "308d27089579460195356087edb20840" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6794c11484634742a8728fbcdc8c8cec"
I20260709 12:28:31.012828 14895 raft_consensus.cc:2393] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4a4b6102eb3f479783d0cfae19c6fa49 in current term 1: Already voted for candidate 6794c11484634742a8728fbcdc8c8cec in this term.
I20260709 12:28:31.015442 14969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "308d27089579460195356087edb20840" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62"
I20260709 12:28:31.015865 14970 raft_consensus.cc:2468] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6794c11484634742a8728fbcdc8c8cec in term 1.
I20260709 12:28:31.017354 14861 leader_election.cc:304] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6794c11484634742a8728fbcdc8c8cec, b661494dcef8498b9af969b7772a0d62; no voters: 4a4b6102eb3f479783d0cfae19c6fa49
I20260709 12:28:31.017884 14786 leader_election.cc:304] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4a4b6102eb3f479783d0cfae19c6fa49; no voters: 6794c11484634742a8728fbcdc8c8cec, b661494dcef8498b9af969b7772a0d62
I20260709 12:28:31.018814 15029 raft_consensus.cc:2804] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:31.018028 15019 leader_election.cc:290] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:31.019596 15029 raft_consensus.cc:697] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 1 LEADER]: Becoming Leader. State: Replica: 6794c11484634742a8728fbcdc8c8cec, State: Running, Role: LEADER
I20260709 12:28:31.020390 15019 raft_consensus.cc:2749] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:28:31.020748 15029 consensus_queue.cc:237] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.028254 14717 catalog_manager.cc:5697] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6794c11484634742a8728fbcdc8c8cec (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6794c11484634742a8728fbcdc8c8cec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:31.044441 14717 catalog_manager.cc:5697] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec reported cstate change: term changed from 0 to 1, leader changed from <none> to 6794c11484634742a8728fbcdc8c8cec (127.0.140.66). New cstate: current_term: 1 leader_uuid: "6794c11484634742a8728fbcdc8c8cec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:31.155902 14717 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53338:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:31.158437 14717 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:31.188923 14811 tablet_service.cc:1511] Processing CreateTablet for tablet fd0a32fd622546eebefa2a3f2244720d (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=e1fad5b740254b268b8b158f7f7c2550]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:31.190631 14811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0a32fd622546eebefa2a3f2244720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:31.190284 14886 tablet_service.cc:1511] Processing CreateTablet for tablet fd0a32fd622546eebefa2a3f2244720d (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=e1fad5b740254b268b8b158f7f7c2550]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:31.191926 14886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0a32fd622546eebefa2a3f2244720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:31.191568 14960 tablet_service.cc:1511] Processing CreateTablet for tablet fd0a32fd622546eebefa2a3f2244720d (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=e1fad5b740254b268b8b158f7f7c2550]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:28:31.193158 14960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0a32fd622546eebefa2a3f2244720d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:31.196406 14959 tablet_service.cc:1511] Processing CreateTablet for tablet 3da8bfa779654f3f9aef2b92b19a7ff0 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=e1fad5b740254b268b8b158f7f7c2550]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:31.198067 14959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da8bfa779654f3f9aef2b92b19a7ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:31.199172 14885 tablet_service.cc:1511] Processing CreateTablet for tablet 3da8bfa779654f3f9aef2b92b19a7ff0 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=e1fad5b740254b268b8b158f7f7c2550]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:31.200711 14885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da8bfa779654f3f9aef2b92b19a7ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:31.203984 14810 tablet_service.cc:1511] Processing CreateTablet for tablet 3da8bfa779654f3f9aef2b92b19a7ff0 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=e1fad5b740254b268b8b158f7f7c2550]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:28:31.205601 14810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da8bfa779654f3f9aef2b92b19a7ff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:31.210813 15011 tablet_bootstrap.cc:492] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62: Bootstrap starting.
I20260709 12:28:31.213647 15010 tablet_bootstrap.cc:492] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec: Bootstrap starting.
I20260709 12:28:31.217814 15011 tablet_bootstrap.cc:654] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:31.220868 15010 tablet_bootstrap.cc:654] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:31.233052 15011 tablet_bootstrap.cc:492] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62: No bootstrap required, opened a new log
I20260709 12:28:31.233805 15011 ts_tablet_manager.cc:1403] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.008s
I20260709 12:28:31.237054 15012 tablet_bootstrap.cc:492] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49: Bootstrap starting.
I20260709 12:28:31.237705 15011 raft_consensus.cc:359] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.238574 15011 raft_consensus.cc:385] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:31.238899 15010 tablet_bootstrap.cc:492] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec: No bootstrap required, opened a new log
I20260709 12:28:31.239557 15010 ts_tablet_manager.cc:1403] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.010s
I20260709 12:28:31.238929 15011 raft_consensus.cc:740] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661494dcef8498b9af969b7772a0d62, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.243924 15011 consensus_queue.cc:260] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.246174 15012 tablet_bootstrap.cc:654] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:31.245893 15010 raft_consensus.cc:359] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.246758 15010 raft_consensus.cc:385] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:31.247138 15010 raft_consensus.cc:740] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6794c11484634742a8728fbcdc8c8cec, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.248088 15010 consensus_queue.cc:260] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.250429 15010 ts_tablet_manager.cc:1434] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec: Time spent starting tablet: real 0.011s	user 0.002s	sys 0.004s
I20260709 12:28:31.252374 15011 ts_tablet_manager.cc:1434] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.008s
I20260709 12:28:31.253798 15011 tablet_bootstrap.cc:492] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62: Bootstrap starting.
I20260709 12:28:31.257442 15010 tablet_bootstrap.cc:492] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec: Bootstrap starting.
I20260709 12:28:31.260855 15011 tablet_bootstrap.cc:654] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:31.265173 15010 tablet_bootstrap.cc:654] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:31.269556 15011 tablet_bootstrap.cc:492] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62: No bootstrap required, opened a new log
I20260709 12:28:31.270102 15011 ts_tablet_manager.cc:1403] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62: Time spent bootstrapping tablet: real 0.017s	user 0.014s	sys 0.000s
I20260709 12:28:31.270273 15012 tablet_bootstrap.cc:492] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49: No bootstrap required, opened a new log
I20260709 12:28:31.271018 15012 ts_tablet_manager.cc:1403] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.007s
I20260709 12:28:31.272729 15010 tablet_bootstrap.cc:492] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec: No bootstrap required, opened a new log
I20260709 12:28:31.272852 15011 raft_consensus.cc:359] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.273772 15010 ts_tablet_manager.cc:1403] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.001s
I20260709 12:28:31.273965 15011 raft_consensus.cc:385] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:31.274343 15011 raft_consensus.cc:740] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b661494dcef8498b9af969b7772a0d62, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.274497 15012 raft_consensus.cc:359] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.275570 15012 raft_consensus.cc:385] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:31.275161 15011 consensus_queue.cc:260] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.275897 15012 raft_consensus.cc:740] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4b6102eb3f479783d0cfae19c6fa49, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.276928 15012 consensus_queue.cc:260] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.277377 15010 raft_consensus.cc:359] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.278129 15010 raft_consensus.cc:385] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:31.278427 15010 raft_consensus.cc:740] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6794c11484634742a8728fbcdc8c8cec, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.279613 15012 ts_tablet_manager.cc:1434] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:28:31.279439 15010 consensus_queue.cc:260] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.280974 15012 tablet_bootstrap.cc:492] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49: Bootstrap starting.
I20260709 12:28:31.285382 15011 ts_tablet_manager.cc:1434] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 12:28:31.288084 15010 ts_tablet_manager.cc:1434] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:28:31.290580 15012 tablet_bootstrap.cc:654] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:31.300096 15012 tablet_bootstrap.cc:492] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49: No bootstrap required, opened a new log
I20260709 12:28:31.300566 15012 ts_tablet_manager.cc:1403] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.000s
I20260709 12:28:31.303050 15012 raft_consensus.cc:359] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.303653 15012 raft_consensus.cc:385] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:31.303907 15012 raft_consensus.cc:740] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a4b6102eb3f479783d0cfae19c6fa49, State: Initialized, Role: FOLLOWER
I20260709 12:28:31.304564 15012 consensus_queue.cc:260] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.306553 15012 ts_tablet_manager.cc:1434] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.001s
I20260709 12:28:31.309105 15031 raft_consensus.cc:493] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.309769 15031 raft_consensus.cc:515] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.312955 15031 leader_election.cc:290] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:31.313076 14895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da8bfa779654f3f9aef2b92b19a7ff0" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6794c11484634742a8728fbcdc8c8cec" is_pre_election: true
I20260709 12:28:31.313319 14970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da8bfa779654f3f9aef2b92b19a7ff0" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62" is_pre_election: true
I20260709 12:28:31.313962 14895 raft_consensus.cc:2468] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 0.
I20260709 12:28:31.314162 14970 raft_consensus.cc:2468] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 0.
I20260709 12:28:31.315395 14787 leader_election.cc:304] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4b6102eb3f479783d0cfae19c6fa49, 6794c11484634742a8728fbcdc8c8cec; no voters: 
I20260709 12:28:31.316084 15031 raft_consensus.cc:493] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:31.316411 15019 raft_consensus.cc:2804] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:31.316860 15019 raft_consensus.cc:493] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:31.316704 15031 raft_consensus.cc:515] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.317278 15019 raft_consensus.cc:3060] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:31.319921 15031 leader_election.cc:290] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:31.320588 14970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0a32fd622546eebefa2a3f2244720d" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62" is_pre_election: true
I20260709 12:28:31.320777 14895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0a32fd622546eebefa2a3f2244720d" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6794c11484634742a8728fbcdc8c8cec" is_pre_election: true
I20260709 12:28:31.321473 14970 raft_consensus.cc:2468] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 0.
I20260709 12:28:31.321589 14895 raft_consensus.cc:2468] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 0.
I20260709 12:28:31.322891 14786 leader_election.cc:304] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4b6102eb3f479783d0cfae19c6fa49, b661494dcef8498b9af969b7772a0d62; no voters: 
I20260709 12:28:31.323860 15031 raft_consensus.cc:2804] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:31.324321 15031 raft_consensus.cc:493] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:31.324685 15031 raft_consensus.cc:3060] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:31.325954 15019 raft_consensus.cc:515] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.328081 15019 leader_election.cc:290] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:31.329068 14895 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da8bfa779654f3f9aef2b92b19a7ff0" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6794c11484634742a8728fbcdc8c8cec"
I20260709 12:28:31.329377 14970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3da8bfa779654f3f9aef2b92b19a7ff0" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62"
I20260709 12:28:31.329761 14895 raft_consensus.cc:3060] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:31.330134 14970 raft_consensus.cc:3060] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:31.335502 15031 raft_consensus.cc:515] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.338076 14895 raft_consensus.cc:2468] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 1.
I20260709 12:28:31.339259 14970 raft_consensus.cc:2468] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 1.
I20260709 12:28:31.339587 14787 leader_election.cc:304] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4b6102eb3f479783d0cfae19c6fa49, 6794c11484634742a8728fbcdc8c8cec; no voters: 
I20260709 12:28:31.339758 14969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0a32fd622546eebefa2a3f2244720d" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b661494dcef8498b9af969b7772a0d62"
I20260709 12:28:31.339174 15031 leader_election.cc:290] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 election: Requested vote from peers 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:31.339231 14896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0a32fd622546eebefa2a3f2244720d" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6794c11484634742a8728fbcdc8c8cec"
I20260709 12:28:31.341293 14969 raft_consensus.cc:3060] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:31.341617 14896 raft_consensus.cc:3060] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:31.342774 15019 raft_consensus.cc:2804] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:31.343478 15019 raft_consensus.cc:697] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 4a4b6102eb3f479783d0cfae19c6fa49, State: Running, Role: LEADER
I20260709 12:28:31.344537 15019 consensus_queue.cc:237] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.347956 14969 raft_consensus.cc:2468] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 1.
I20260709 12:28:31.349225 14786 leader_election.cc:304] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4b6102eb3f479783d0cfae19c6fa49, b661494dcef8498b9af969b7772a0d62; no voters: 
I20260709 12:28:31.350194 15031 raft_consensus.cc:2804] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:31.352135 14896 raft_consensus.cc:2468] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 1.
I20260709 12:28:31.352901 15031 raft_consensus.cc:697] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 LEADER]: Becoming Leader. State: Replica: 4a4b6102eb3f479783d0cfae19c6fa49, State: Running, Role: LEADER
I20260709 12:28:31.353940 15031 consensus_queue.cc:237] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.356439 14716 catalog_manager.cc:5697] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:31.371912 14716 catalog_manager.cc:5697] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:31.397872 15029 consensus_queue.cc:1048] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:31.414899 14896 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced"
dest_uuid: "6794c11484634742a8728fbcdc8c8cec"
mode: GRACEFUL
new_leader_uuid: "4a4b6102eb3f479783d0cfae19c6fa49"
 from {username='slave'} at 127.0.0.1:53432
I20260709 12:28:31.414455 15020 consensus_queue.cc:1048] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:31.415647 14896 raft_consensus.cc:606] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 1 LEADER]: Received request to transfer leadership to TS 4a4b6102eb3f479783d0cfae19c6fa49
I20260709 12:28:31.427796 15029 raft_consensus.cc:993] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec: : Instructing follower 4a4b6102eb3f479783d0cfae19c6fa49 to start an election
I20260709 12:28:31.428426 15020 raft_consensus.cc:1081] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 1 LEADER]: Signalling peer 4a4b6102eb3f479783d0cfae19c6fa49 to start an election
I20260709 12:28:31.430243 14820 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced"
dest_uuid: "4a4b6102eb3f479783d0cfae19c6fa49"
 from {username='slave'} at 127.0.0.1:35382
I20260709 12:28:31.430773 14820 raft_consensus.cc:493] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:31.431155 14820 raft_consensus.cc:3060] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.437599 14820 raft_consensus.cc:515] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.440122 14820 leader_election.cc:290] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 2 election: Requested vote from peers 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021), b661494dcef8498b9af969b7772a0d62 (127.0.140.67:45671)
I20260709 12:28:31.440483 14896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6794c11484634742a8728fbcdc8c8cec"
I20260709 12:28:31.440863 14969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db68371d71cd4be59f981f26ebf0aced" candidate_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b661494dcef8498b9af969b7772a0d62"
I20260709 12:28:31.441705 14969 raft_consensus.cc:3060] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.441183 14896 raft_consensus.cc:3055] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:31.442217 14896 raft_consensus.cc:740] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6794c11484634742a8728fbcdc8c8cec, State: Running, Role: LEADER
I20260709 12:28:31.443244 14896 consensus_queue.cc:260] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.444653 14896 raft_consensus.cc:3060] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.450922 14969 raft_consensus.cc:2468] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 2.
I20260709 12:28:31.451625 14896 raft_consensus.cc:2468] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4a4b6102eb3f479783d0cfae19c6fa49 in term 2.
I20260709 12:28:31.452343 14786 leader_election.cc:304] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a4b6102eb3f479783d0cfae19c6fa49, b661494dcef8498b9af969b7772a0d62; no voters: 
I20260709 12:28:31.453310 15031 raft_consensus.cc:2804] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:31.453763 15031 raft_consensus.cc:697] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 2 LEADER]: Becoming Leader. State: Replica: 4a4b6102eb3f479783d0cfae19c6fa49, State: Running, Role: LEADER
I20260709 12:28:31.454563 15031 consensus_queue.cc:237] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.461841 14718 catalog_manager.cc:5697] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 reported cstate change: term changed from 1 to 2, leader changed from 6794c11484634742a8728fbcdc8c8cec (127.0.140.66) to 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:31.491876   561 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 12:28:31.494110   561 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 12:28:31.494599   561 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 12:28:31.508888 14820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fd0a32fd622546eebefa2a3f2244720d"
dest_uuid: "4a4b6102eb3f479783d0cfae19c6fa49"
mode: GRACEFUL
new_leader_uuid: "b661494dcef8498b9af969b7772a0d62"
 from {username='slave'} at 127.0.0.1:35408
I20260709 12:28:31.509578 14820 raft_consensus.cc:606] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 LEADER]: Received request to transfer leadership to TS b661494dcef8498b9af969b7772a0d62
I20260709 12:28:31.511163   561 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 12:28:31.517259 15020 consensus_queue.cc:1048] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:31.532945 15029 consensus_queue.cc:1048] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [LEADER]: Connected to new peer: Peer: permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:31.774387 15031 consensus_queue.cc:1048] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:31.778899 15031 consensus_queue.cc:1048] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:31.783557 15045 raft_consensus.cc:993] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49: : Instructing follower b661494dcef8498b9af969b7772a0d62 to start an election
I20260709 12:28:31.784096 15045 raft_consensus.cc:1081] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 LEADER]: Signalling peer b661494dcef8498b9af969b7772a0d62 to start an election
I20260709 12:28:31.789309 14969 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fd0a32fd622546eebefa2a3f2244720d"
dest_uuid: "b661494dcef8498b9af969b7772a0d62"
 from {username='slave'} at 127.0.0.1:38048
I20260709 12:28:31.790086 14969 raft_consensus.cc:493] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:31.790508 14969 raft_consensus.cc:3060] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.799751 14969 raft_consensus.cc:515] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.802170 14969 leader_election.cc:290] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [CANDIDATE]: Term 2 election: Requested vote from peers 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65:44535), 6794c11484634742a8728fbcdc8c8cec (127.0.140.66:46021)
I20260709 12:28:31.803285 14820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0a32fd622546eebefa2a3f2244720d" candidate_uuid: "b661494dcef8498b9af969b7772a0d62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4a4b6102eb3f479783d0cfae19c6fa49"
I20260709 12:28:31.803745 14896 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd0a32fd622546eebefa2a3f2244720d" candidate_uuid: "b661494dcef8498b9af969b7772a0d62" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "6794c11484634742a8728fbcdc8c8cec"
I20260709 12:28:31.804054 14820 raft_consensus.cc:3055] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:31.804524 14820 raft_consensus.cc:740] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4a4b6102eb3f479783d0cfae19c6fa49, State: Running, Role: LEADER
I20260709 12:28:31.804559 14896 raft_consensus.cc:3060] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.805423 14820 consensus_queue.cc:260] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.806686 14820 raft_consensus.cc:3060] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:31.811335 14896 raft_consensus.cc:2468] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b661494dcef8498b9af969b7772a0d62 in term 2.
I20260709 12:28:31.812674 14936 leader_election.cc:304] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6794c11484634742a8728fbcdc8c8cec, b661494dcef8498b9af969b7772a0d62; no voters: 
I20260709 12:28:31.813021 14820 raft_consensus.cc:2468] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b661494dcef8498b9af969b7772a0d62 in term 2.
I20260709 12:28:31.813710 15018 raft_consensus.cc:2804] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:31.815047 15018 raft_consensus.cc:697] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 2 LEADER]: Becoming Leader. State: Replica: b661494dcef8498b9af969b7772a0d62, State: Running, Role: LEADER
I20260709 12:28:31.815889 15018 consensus_queue.cc:237] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } }
I20260709 12:28:31.822263 14716 catalog_manager.cc:5697] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 reported cstate change: term changed from 1 to 2, leader changed from 4a4b6102eb3f479783d0cfae19c6fa49 (127.0.140.65) to b661494dcef8498b9af969b7772a0d62 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "b661494dcef8498b9af969b7772a0d62" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a4b6102eb3f479783d0cfae19c6fa49" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 44535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:31.843351 15031 consensus_queue.cc:1048] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6794c11484634742a8728fbcdc8c8cec" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 46021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:31.857702 15061 consensus_queue.cc:1048] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b661494dcef8498b9af969b7772a0d62" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 45671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:31.890662   561 tablet_server.cc:179] TabletServer@127.0.140.65:0 shutting down...
I20260709 12:28:31.917105   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:31.917858   561 tablet_replica.cc:333] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49: stopping tablet replica
I20260709 12:28:31.918526   561 raft_consensus.cc:2243] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:31.918996   561 raft_consensus.cc:2272] T fd0a32fd622546eebefa2a3f2244720d P 4a4b6102eb3f479783d0cfae19c6fa49 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:31.921545   561 tablet_replica.cc:333] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49: stopping tablet replica
I20260709 12:28:31.922098   561 raft_consensus.cc:2243] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:31.923027   561 raft_consensus.cc:2272] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:31.925343   561 tablet_replica.cc:333] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49: stopping tablet replica
I20260709 12:28:31.925890   561 raft_consensus.cc:2243] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:31.926522   561 pending_rounds.cc:70] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49: Trying to abort 1 pending ops.
I20260709 12:28:31.926751   561 pending_rounds.cc:77] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7305626056526163968 op_type: NO_OP noop_request { }
I20260709 12:28:31.927156   561 raft_consensus.cc:2889] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:31.927449   561 raft_consensus.cc:2272] T db68371d71cd4be59f981f26ebf0aced P 4a4b6102eb3f479783d0cfae19c6fa49 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:31.929955   561 tablet_replica.cc:333] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49: stopping tablet replica
I20260709 12:28:31.930488   561 raft_consensus.cc:2243] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:31.931010   561 raft_consensus.cc:2272] T 308d27089579460195356087edb20840 P 4a4b6102eb3f479783d0cfae19c6fa49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:31.947237   561 tablet_server.cc:196] TabletServer@127.0.140.65:0 shutdown complete.
I20260709 12:28:31.960978   561 tablet_server.cc:179] TabletServer@127.0.140.66:0 shutting down...
I20260709 12:28:31.989429   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:31.990411   561 tablet_replica.cc:333] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec: stopping tablet replica
I20260709 12:28:31.991204   561 raft_consensus.cc:2243] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:31.991791   561 raft_consensus.cc:2272] T fd0a32fd622546eebefa2a3f2244720d P 6794c11484634742a8728fbcdc8c8cec [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:31.995399   561 tablet_replica.cc:333] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec: stopping tablet replica
I20260709 12:28:31.996017   561 raft_consensus.cc:2243] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:31.996474   561 raft_consensus.cc:2272] T 3da8bfa779654f3f9aef2b92b19a7ff0 P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:31.999619   561 tablet_replica.cc:333] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec: stopping tablet replica
I20260709 12:28:32.000213   561 raft_consensus.cc:2243] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:32.001183   561 raft_consensus.cc:2272] T 308d27089579460195356087edb20840 P 6794c11484634742a8728fbcdc8c8cec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:32.004565   561 tablet_replica.cc:333] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec: stopping tablet replica
I20260709 12:28:32.005118   561 raft_consensus.cc:2243] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:32.005548   561 raft_consensus.cc:2272] T db68371d71cd4be59f981f26ebf0aced P 6794c11484634742a8728fbcdc8c8cec [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:32.032577   561 tablet_server.cc:196] TabletServer@127.0.140.66:0 shutdown complete.
I20260709 12:28:32.046415   561 tablet_server.cc:179] TabletServer@127.0.140.67:0 shutting down...
I20260709 12:28:32.071348   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:32.072151   561 tablet_replica.cc:333] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62: stopping tablet replica
I20260709 12:28:32.072825   561 raft_consensus.cc:2243] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:32.073618   561 pending_rounds.cc:70] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62: Trying to abort 1 pending ops.
I20260709 12:28:32.073853   561 pending_rounds.cc:77] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7305626058005114880 op_type: NO_OP noop_request { }
I20260709 12:28:32.074154   561 raft_consensus.cc:2889] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:28:32.074517   561 raft_consensus.cc:2272] T fd0a32fd622546eebefa2a3f2244720d P b661494dcef8498b9af969b7772a0d62 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:32.077920   561 tablet_replica.cc:333] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62: stopping tablet replica
I20260709 12:28:32.078672   561 raft_consensus.cc:2243] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:32.079180   561 raft_consensus.cc:2272] T 3da8bfa779654f3f9aef2b92b19a7ff0 P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:32.081861   561 tablet_replica.cc:333] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62: stopping tablet replica
I20260709 12:28:32.082373   561 raft_consensus.cc:2243] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:32.082858   561 raft_consensus.cc:2272] T 308d27089579460195356087edb20840 P b661494dcef8498b9af969b7772a0d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:32.086252   561 tablet_replica.cc:333] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62: stopping tablet replica
I20260709 12:28:32.086930   561 raft_consensus.cc:2243] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:32.087368   561 raft_consensus.cc:2272] T db68371d71cd4be59f981f26ebf0aced P b661494dcef8498b9af969b7772a0d62 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:32.113469   561 tablet_server.cc:196] TabletServer@127.0.140.67:0 shutdown complete.
I20260709 12:28:32.126907   561 master.cc:562] Master@127.0.140.126:38779 shutting down...
I20260709 12:28:32.146932   561 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:32.147567   561 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:32.147935   561 tablet_replica.cc:333] T 00000000000000000000000000000000 P 13e24fee2c514a638a58981f934f9e67: stopping tablet replica
I20260709 12:28:32.169986   561 master.cc:584] Master@127.0.140.126:38779 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (2283 ms)
[----------] 6 tests from LeaderRebalancerTest (487888 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 12:28:32.203460   561 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.140.126:37653
I20260709 12:28:32.204764   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:32.210631 15063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:32.210891 15064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:32.212414 15066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:32.214039   561 server_base.cc:1061] running on GCE node
I20260709 12:28:32.215376   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:32.215685   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:32.215880   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600112215845 us; error 0 us; skew 500 ppm
I20260709 12:28:32.216591   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:32.219584   561 webserver.cc:533] Webserver started at http://127.0.140.126:42519/ using document root <none> and password file <none>
I20260709 12:28:32.220126   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:32.220322   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:32.220647   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:32.221793   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/master-0-root/instance:
uuid: "ca3a059e0be54f9898675800d2b70be7"
format_stamp: "Formatted at 2026-07-09 12:28:32 on dist-test-slave-7mcn"
I20260709 12:28:32.227085   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.004s
I20260709 12:28:32.231593 15071 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:32.232709   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:28:32.233199   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/master-0-root
uuid: "ca3a059e0be54f9898675800d2b70be7"
format_stamp: "Formatted at 2026-07-09 12:28:32 on dist-test-slave-7mcn"
I20260709 12:28:32.233491   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:32.251072   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:32.252223   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:32.294093   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.126:37653
I20260709 12:28:32.294270 15122 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.126:37653 every 8 connection(s)
I20260709 12:28:32.298739 15123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:32.312479 15123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7: Bootstrap starting.
I20260709 12:28:32.317924 15123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:32.322992 15123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7: No bootstrap required, opened a new log
I20260709 12:28:32.325466 15123 raft_consensus.cc:359] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3a059e0be54f9898675800d2b70be7" member_type: VOTER }
I20260709 12:28:32.325891 15123 raft_consensus.cc:385] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:32.326064 15123 raft_consensus.cc:740] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3a059e0be54f9898675800d2b70be7, State: Initialized, Role: FOLLOWER
I20260709 12:28:32.326620 15123 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3a059e0be54f9898675800d2b70be7" member_type: VOTER }
I20260709 12:28:32.327098 15123 raft_consensus.cc:399] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:28:32.327303 15123 raft_consensus.cc:493] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:28:32.327528 15123 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:32.333487 15123 raft_consensus.cc:515] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3a059e0be54f9898675800d2b70be7" member_type: VOTER }
I20260709 12:28:32.334055 15123 leader_election.cc:304] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca3a059e0be54f9898675800d2b70be7; no voters: 
I20260709 12:28:32.335649 15123 leader_election.cc:290] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:28:32.336079 15126 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:32.337859 15126 raft_consensus.cc:697] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 1 LEADER]: Becoming Leader. State: Replica: ca3a059e0be54f9898675800d2b70be7, State: Running, Role: LEADER
I20260709 12:28:32.338582 15126 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3a059e0be54f9898675800d2b70be7" member_type: VOTER }
I20260709 12:28:32.339105 15123 sys_catalog.cc:565] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:28:32.341852 15127 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca3a059e0be54f9898675800d2b70be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3a059e0be54f9898675800d2b70be7" member_type: VOTER } }
I20260709 12:28:32.341866 15128 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca3a059e0be54f9898675800d2b70be7. Latest consensus state: current_term: 1 leader_uuid: "ca3a059e0be54f9898675800d2b70be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca3a059e0be54f9898675800d2b70be7" member_type: VOTER } }
I20260709 12:28:32.342763 15127 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:32.342926 15128 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:32.348256 15131 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:28:32.353924 15131 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:28:32.357368   561 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:28:32.364408 15131 catalog_manager.cc:1383] Generated new cluster ID: 5759bb7397fb49979049968006c2910d
I20260709 12:28:32.364765 15131 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:28:32.407013 15131 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:28:32.408573 15131 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:28:32.423134 15131 catalog_manager.cc:6070] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7: Generated new TSK 0
I20260709 12:28:32.423877 15131 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:28:32.491413   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:32.498164 15144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:32.499596 15145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:32.501084 15147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:32.502236   561 server_base.cc:1061] running on GCE node
I20260709 12:28:32.503238   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:32.503448   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:32.503588   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600112503576 us; error 0 us; skew 500 ppm
I20260709 12:28:32.504098   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:32.507184   561 webserver.cc:533] Webserver started at http://127.0.140.65:44421/ using document root <none> and password file <none>
I20260709 12:28:32.507673   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:32.507854   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:32.508107   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:32.509220   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-0-root/instance:
uuid: "96664db3a4e8450a8d96fd457960bba6"
format_stamp: "Formatted at 2026-07-09 12:28:32 on dist-test-slave-7mcn"
I20260709 12:28:32.514320   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.002s
I20260709 12:28:32.518383 15152 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:32.519462   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:28:32.519853   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "96664db3a4e8450a8d96fd457960bba6"
format_stamp: "Formatted at 2026-07-09 12:28:32 on dist-test-slave-7mcn"
I20260709 12:28:32.520155   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:32.542497   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:32.543813   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:32.545442   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:32.548041   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:32.548271   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:32.548527   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:32.548702   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:32.595005   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:39481
I20260709 12:28:32.595170 15214 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:39481 every 8 connection(s)
I20260709 12:28:32.599892   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:32.608253 15219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:32.610463 15220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:32.615015 15222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:32.617411 15215 heartbeater.cc:344] Connected to a master server at 127.0.140.126:37653
I20260709 12:28:32.618054 15215 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:32.618206   561 server_base.cc:1061] running on GCE node
I20260709 12:28:32.619246 15215 heartbeater.cc:507] Master 127.0.140.126:37653 requested a full tablet report, sending...
I20260709 12:28:32.619746   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:32.620144   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:32.620343   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600112620325 us; error 0 us; skew 500 ppm
I20260709 12:28:32.621071   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:32.622073 15088 ts_manager.cc:194] Registered new tserver with Master: 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481)
I20260709 12:28:32.624305   561 webserver.cc:533] Webserver started at http://127.0.140.66:40187/ using document root <none> and password file <none>
I20260709 12:28:32.624886 15088 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53766
I20260709 12:28:32.625036   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:32.625332   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:32.625675   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:32.626935   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-1-root/instance:
uuid: "5a360f0ebb5c4bac86e43907cd673062"
format_stamp: "Formatted at 2026-07-09 12:28:32 on dist-test-slave-7mcn"
I20260709 12:28:32.631896   561 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:28:32.636579 15227 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:32.637976   561 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:28:32.638422   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-1-root
uuid: "5a360f0ebb5c4bac86e43907cd673062"
format_stamp: "Formatted at 2026-07-09 12:28:32 on dist-test-slave-7mcn"
I20260709 12:28:32.638991   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:32.668792   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:32.669938   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:32.671717   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:32.674170   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:32.674415   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:32.674679   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:32.674873   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.001s
I20260709 12:28:32.722623   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.66:42199
I20260709 12:28:32.722874 15289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.66:42199 every 8 connection(s)
I20260709 12:28:32.727646   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:32.736632 15293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:32.736886 15294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:32.741302   561 server_base.cc:1061] running on GCE node
W20260709 12:28:32.741339 15297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:32.742936   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:32.743188   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:32.743391   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600112743373 us; error 0 us; skew 500 ppm
I20260709 12:28:32.744091   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:32.745580 15290 heartbeater.cc:344] Connected to a master server at 127.0.140.126:37653
I20260709 12:28:32.746047 15290 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:32.746905 15290 heartbeater.cc:507] Master 127.0.140.126:37653 requested a full tablet report, sending...
I20260709 12:28:32.747517   561 webserver.cc:533] Webserver started at http://127.0.140.67:35347/ using document root <none> and password file <none>
I20260709 12:28:32.748482   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:32.748688   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:32.748991   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:32.749759 15088 ts_manager.cc:194] Registered new tserver with Master: 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:32.750762   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-2-root/instance:
uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
format_stamp: "Formatted at 2026-07-09 12:28:32 on dist-test-slave-7mcn"
I20260709 12:28:32.752322 15088 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53774
I20260709 12:28:32.757555   561 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:28:32.762111 15301 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:32.763628   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20260709 12:28:32.764008   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-2-root
uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
format_stamp: "Formatted at 2026-07-09 12:28:32 on dist-test-slave-7mcn"
I20260709 12:28:32.764403   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599624140507-561-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:32.783826   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:32.785202   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:32.787003   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:32.789832   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:32.790056   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:32.790342   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:32.790495   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:32.844003   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.67:34945
I20260709 12:28:32.844240 15363 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.67:34945 every 8 connection(s)
I20260709 12:28:32.860321 15364 heartbeater.cc:344] Connected to a master server at 127.0.140.126:37653
I20260709 12:28:32.860858 15364 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:32.861677 15364 heartbeater.cc:507] Master 127.0.140.126:37653 requested a full tablet report, sending...
I20260709 12:28:32.864404 15088 ts_manager.cc:194] Registered new tserver with Master: 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:32.864548   561 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016907797s
I20260709 12:28:32.866703 15088 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:53786
I20260709 12:28:32.898447 15088 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53788:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:32.901693 15088 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:32.990116 15180 tablet_service.cc:1511] Processing CreateTablet for tablet b7b1b7149d02468ab8902f6725e815d6 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:32.991951 15180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b1b7149d02468ab8902f6725e815d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:32.994105 15175 tablet_service.cc:1511] Processing CreateTablet for tablet 9b894d51dffb436797541081590814ad (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:32.994891 15255 tablet_service.cc:1511] Processing CreateTablet for tablet b7b1b7149d02468ab8902f6725e815d6 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:32.996132 15175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b894d51dffb436797541081590814ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:32.996510 15255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b1b7149d02468ab8902f6725e815d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:32.996182 15177 tablet_service.cc:1511] Processing CreateTablet for tablet d417fe864273410fb5740d055cb7e9c9 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:32.998224 15177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d417fe864273410fb5740d055cb7e9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.001808 15178 tablet_service.cc:1511] Processing CreateTablet for tablet 8ccb4ca0924e4f7f955623ff7004e70b (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:32.994187 15176 tablet_service.cc:1511] Processing CreateTablet for tablet 411b646e3a71433a817ca20eab98d884 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:33.003767 15178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ccb4ca0924e4f7f955623ff7004e70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.004701 15176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 411b646e3a71433a817ca20eab98d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.006295 15179 tablet_service.cc:1511] Processing CreateTablet for tablet 38d8237b9ab9493eb41e584ffde4f070 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:33.008041 15179 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38d8237b9ab9493eb41e584ffde4f070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.012238 15254 tablet_service.cc:1511] Processing CreateTablet for tablet 38d8237b9ab9493eb41e584ffde4f070 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:33.014122 15254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38d8237b9ab9493eb41e584ffde4f070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.016709 15174 tablet_service.cc:1511] Processing CreateTablet for tablet 7a717a5ca8f84dbb9e36a0f73e90d720 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:33.018743 15174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a717a5ca8f84dbb9e36a0f73e90d720. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.023115 15253 tablet_service.cc:1511] Processing CreateTablet for tablet 8ccb4ca0924e4f7f955623ff7004e70b (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:33.025017 15253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ccb4ca0924e4f7f955623ff7004e70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.027331 15173 tablet_service.cc:1511] Processing CreateTablet for tablet 48a887aef66e4290bfed409cfa9483b7 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:33.029716 15173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a887aef66e4290bfed409cfa9483b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.036798 15252 tablet_service.cc:1511] Processing CreateTablet for tablet d417fe864273410fb5740d055cb7e9c9 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:33.038655 15252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d417fe864273410fb5740d055cb7e9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.039597 15172 tablet_service.cc:1511] Processing CreateTablet for tablet 94397791de6a4c489337a8bd7803f231 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:33.041231 15172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94397791de6a4c489337a8bd7803f231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.052407 15251 tablet_service.cc:1511] Processing CreateTablet for tablet 411b646e3a71433a817ca20eab98d884 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:33.054374 15251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 411b646e3a71433a817ca20eab98d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.064692 15250 tablet_service.cc:1511] Processing CreateTablet for tablet 9b894d51dffb436797541081590814ad (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:33.066390 15250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b894d51dffb436797541081590814ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.083432 15249 tablet_service.cc:1511] Processing CreateTablet for tablet 7a717a5ca8f84dbb9e36a0f73e90d720 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:33.085016 15249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a717a5ca8f84dbb9e36a0f73e90d720. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.092216 15329 tablet_service.cc:1511] Processing CreateTablet for tablet b7b1b7149d02468ab8902f6725e815d6 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:33.094031 15329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7b1b7149d02468ab8902f6725e815d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.094117 15325 tablet_service.cc:1511] Processing CreateTablet for tablet 411b646e3a71433a817ca20eab98d884 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:33.095779 15325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 411b646e3a71433a817ca20eab98d884. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.095997 15328 tablet_service.cc:1511] Processing CreateTablet for tablet 38d8237b9ab9493eb41e584ffde4f070 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:33.097440 15328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38d8237b9ab9493eb41e584ffde4f070. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.099666 15248 tablet_service.cc:1511] Processing CreateTablet for tablet 48a887aef66e4290bfed409cfa9483b7 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:33.100065 15324 tablet_service.cc:1511] Processing CreateTablet for tablet 9b894d51dffb436797541081590814ad (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:33.101245 15326 tablet_service.cc:1511] Processing CreateTablet for tablet d417fe864273410fb5740d055cb7e9c9 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:33.102439 15247 tablet_service.cc:1511] Processing CreateTablet for tablet 94397791de6a4c489337a8bd7803f231 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:33.100065 15327 tablet_service.cc:1511] Processing CreateTablet for tablet 8ccb4ca0924e4f7f955623ff7004e70b (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:33.103179 15323 tablet_service.cc:1511] Processing CreateTablet for tablet 7a717a5ca8f84dbb9e36a0f73e90d720 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:33.101290 15248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a887aef66e4290bfed409cfa9483b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.105111 15247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94397791de6a4c489337a8bd7803f231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.101714 15324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b894d51dffb436797541081590814ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.105468 15322 tablet_service.cc:1511] Processing CreateTablet for tablet 48a887aef66e4290bfed409cfa9483b7 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:33.108002 15321 tablet_service.cc:1511] Processing CreateTablet for tablet 94397791de6a4c489337a8bd7803f231 (DEFAULT_TABLE table=test-workload [id=dcbcc3e91c4545c69b585f720f6e98cf]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:33.109592 15322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a887aef66e4290bfed409cfa9483b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.115285 15327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ccb4ca0924e4f7f955623ff7004e70b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.116077 15323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a717a5ca8f84dbb9e36a0f73e90d720. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.117877 15326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d417fe864273410fb5740d055cb7e9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.119382 15321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94397791de6a4c489337a8bd7803f231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:33.128034 15379 tablet_bootstrap.cc:492] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:33.182049 15381 tablet_bootstrap.cc:492] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:33.205330 15379 tablet_bootstrap.cc:654] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.217936 15381 tablet_bootstrap.cc:654] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.244872 15383 tablet_bootstrap.cc:492] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:33.246485 15379 tablet_bootstrap.cc:492] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:33.247192 15379 ts_tablet_manager.cc:1403] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.120s	user 0.012s	sys 0.022s
I20260709 12:28:33.250242 15379 raft_consensus.cc:359] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.251031 15379 raft_consensus.cc:385] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.328838 15383 tablet_bootstrap.cc:654] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.251411 15379 raft_consensus.cc:740] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.337615 15379 consensus_queue.cc:260] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.343055 15381 tablet_bootstrap.cc:492] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:33.344364 15381 ts_tablet_manager.cc:1403] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.163s	user 0.020s	sys 0.017s
I20260709 12:28:33.347375 15383 tablet_bootstrap.cc:492] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:33.348397 15383 ts_tablet_manager.cc:1403] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.104s	user 0.002s	sys 0.018s
W20260709 12:28:33.353963 15136 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:33.354997 15215 heartbeater.cc:499] Master 127.0.140.126:37653 was elected leader, sending a full tablet report...
I20260709 12:28:33.355273 15383 raft_consensus.cc:359] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.355815 15381 raft_consensus.cc:359] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.356455 15383 raft_consensus.cc:385] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.356812 15381 raft_consensus.cc:385] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.356920 15383 raft_consensus.cc:740] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.357286 15381 raft_consensus.cc:740] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.358174 15381 consensus_queue.cc:260] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.357934 15383 consensus_queue.cc:260] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.361768 15379 ts_tablet_manager.cc:1434] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.114s	user 0.010s	sys 0.003s
I20260709 12:28:33.364331 15379 tablet_bootstrap.cc:492] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:33.375902 15379 tablet_bootstrap.cc:654] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.381929 15383 ts_tablet_manager.cc:1434] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.031s	user 0.012s	sys 0.014s
I20260709 12:28:33.381814 15364 heartbeater.cc:499] Master 127.0.140.126:37653 was elected leader, sending a full tablet report...
I20260709 12:28:33.382004 15381 ts_tablet_manager.cc:1434] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.037s	user 0.020s	sys 0.012s
I20260709 12:28:33.384567 15290 heartbeater.cc:499] Master 127.0.140.126:37653 was elected leader, sending a full tablet report...
I20260709 12:28:33.395960 15383 tablet_bootstrap.cc:492] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:33.402575 15381 tablet_bootstrap.cc:492] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:33.404997 15379 tablet_bootstrap.cc:492] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:33.405776 15379 ts_tablet_manager.cc:1403] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.042s	user 0.022s	sys 0.011s
I20260709 12:28:33.406013 15383 tablet_bootstrap.cc:654] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.409937 15379 raft_consensus.cc:359] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.411257 15379 raft_consensus.cc:385] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.411731 15379 raft_consensus.cc:740] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.413020 15381 tablet_bootstrap.cc:654] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.413698 15379 consensus_queue.cc:260] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.417768 15379 ts_tablet_manager.cc:1434] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.009s
I20260709 12:28:33.419970 15379 tablet_bootstrap.cc:492] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:33.425583 15383 tablet_bootstrap.cc:492] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:33.426537 15383 ts_tablet_manager.cc:1403] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.037s	user 0.016s	sys 0.009s
I20260709 12:28:33.428913 15379 tablet_bootstrap.cc:654] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.430181 15381 tablet_bootstrap.cc:492] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:33.431039 15381 ts_tablet_manager.cc:1403] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.029s	user 0.017s	sys 0.007s
I20260709 12:28:33.432049 15383 raft_consensus.cc:359] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.433036 15383 raft_consensus.cc:385] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.433403 15383 raft_consensus.cc:740] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.434654 15383 consensus_queue.cc:260] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.435848 15381 raft_consensus.cc:359] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.436775 15381 raft_consensus.cc:385] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.437170 15381 raft_consensus.cc:740] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.438194 15383 ts_tablet_manager.cc:1434] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:28:33.438203 15381 consensus_queue.cc:260] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.439929 15383 tablet_bootstrap.cc:492] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:33.441872 15381 ts_tablet_manager.cc:1434] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.000s
I20260709 12:28:33.443360 15381 tablet_bootstrap.cc:492] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:33.450220 15383 tablet_bootstrap.cc:654] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.452557 15381 tablet_bootstrap.cc:654] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.460178 15379 tablet_bootstrap.cc:492] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:33.460785 15379 ts_tablet_manager.cc:1403] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.041s	user 0.015s	sys 0.016s
I20260709 12:28:33.464399 15379 raft_consensus.cc:359] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.465209 15379 raft_consensus.cc:385] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.465549 15379 raft_consensus.cc:740] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.466557 15379 consensus_queue.cc:260] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.469571 15379 ts_tablet_manager.cc:1434] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.003s
I20260709 12:28:33.471139 15381 tablet_bootstrap.cc:492] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:33.471163 15379 tablet_bootstrap.cc:492] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:33.471920 15381 ts_tablet_manager.cc:1403] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.029s	user 0.007s	sys 0.017s
I20260709 12:28:33.472235 15383 tablet_bootstrap.cc:492] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:33.472925 15383 ts_tablet_manager.cc:1403] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.033s	user 0.006s	sys 0.017s
I20260709 12:28:33.475976 15381 raft_consensus.cc:359] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.476883 15381 raft_consensus.cc:385] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.476397 15383 raft_consensus.cc:359] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.477243 15381 raft_consensus.cc:740] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.477542 15383 raft_consensus.cc:385] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.478015 15383 raft_consensus.cc:740] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.478531 15381 consensus_queue.cc:260] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.479115 15383 consensus_queue.cc:260] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.481182 15381 ts_tablet_manager.cc:1434] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.009s	user 0.002s	sys 0.006s
I20260709 12:28:33.482251 15379 tablet_bootstrap.cc:654] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.482566 15381 tablet_bootstrap.cc:492] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:33.485190 15383 ts_tablet_manager.cc:1434] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:28:33.486663 15383 tablet_bootstrap.cc:492] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:33.490888 15387 raft_consensus.cc:493] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.491643 15387 raft_consensus.cc:515] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.501264 15388 raft_consensus.cc:493] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.502161 15388 raft_consensus.cc:515] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.502803 15381 tablet_bootstrap.cc:654] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.504961 15387 leader_election.cc:290] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.505774 15388 leader_election.cc:290] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:33.527077 15383 tablet_bootstrap.cc:654] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.545212 15381 tablet_bootstrap.cc:492] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:33.545814 15381 ts_tablet_manager.cc:1403] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.064s	user 0.009s	sys 0.021s
I20260709 12:28:33.545928 15379 tablet_bootstrap.cc:492] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:33.546710 15379 ts_tablet_manager.cc:1403] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.076s	user 0.017s	sys 0.018s
I20260709 12:28:33.547034 15388 raft_consensus.cc:493] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.547680 15388 raft_consensus.cc:515] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.549293 15381 raft_consensus.cc:359] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.550222 15381 raft_consensus.cc:385] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.550527 15381 raft_consensus.cc:740] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.551692 15388 leader_election.cc:290] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:33.553970 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:33.555315 15190 raft_consensus.cc:2468] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:33.551565 15381 consensus_queue.cc:260] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.559669 15381 ts_tablet_manager.cc:1434] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.013s	user 0.004s	sys 0.006s
I20260709 12:28:33.557821 15229 leader_election.cc:304] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:33.569561 15381 tablet_bootstrap.cc:492] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:33.572712 15387 raft_consensus.cc:2804] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:33.573343 15387 raft_consensus.cc:493] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.573750 15387 raft_consensus.cc:3060] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.578169 15381 tablet_bootstrap.cc:654] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.583276 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:33.556704 15379 raft_consensus.cc:359] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.584849 15379 raft_consensus.cc:385] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.585249 15379 raft_consensus.cc:740] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.585304 15189 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:33.586004 15383 tablet_bootstrap.cc:492] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:33.586549 15383 ts_tablet_manager.cc:1403] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.100s	user 0.017s	sys 0.025s
I20260709 12:28:33.586290 15379 consensus_queue.cc:260] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
W20260709 12:28:33.589293 15303 leader_election.cc:343] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.589645 15379 ts_tablet_manager.cc:1434] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.043s	user 0.005s	sys 0.002s
W20260709 12:28:33.590175 15303 leader_election.cc:343] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.591205 15379 tablet_bootstrap.cc:492] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:33.596601 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
W20260709 12:28:33.598613 15229 leader_election.cc:343] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.601847 15379 tablet_bootstrap.cc:654] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.601312 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:33.602327 15264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
W20260709 12:28:33.605662 15302 leader_election.cc:343] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.606148 15302 leader_election.cc:304] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e; no voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:33.607081 15388 raft_consensus.cc:2749] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:28:33.607406 15302 leader_election.cc:343] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.607921 15302 leader_election.cc:304] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e; no voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:33.607795 15383 raft_consensus.cc:359] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.608594 15383 raft_consensus.cc:385] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.608760 15388 raft_consensus.cc:2749] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:33.608908 15383 raft_consensus.cc:740] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.610031 15383 consensus_queue.cc:260] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.610387 15387 raft_consensus.cc:515] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.613474 15387 leader_election.cc:290] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.613732 15383 ts_tablet_manager.cc:1434] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260709 12:28:33.614265 15189 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:33.614945 15189 raft_consensus.cc:3060] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.615065 15383 tablet_bootstrap.cc:492] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:33.616829 15381 tablet_bootstrap.cc:492] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:33.617487 15381 ts_tablet_manager.cc:1403] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.048s	user 0.015s	sys 0.000s
I20260709 12:28:33.621193 15381 raft_consensus.cc:359] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.622032 15381 raft_consensus.cc:385] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.622357 15381 raft_consensus.cc:740] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.622520 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:33.623504 15383 tablet_bootstrap.cc:654] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.623404 15381 consensus_queue.cc:260] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
W20260709 12:28:33.624523 15229 leader_election.cc:343] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.625203 15387 raft_consensus.cc:493] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.625813 15387 raft_consensus.cc:515] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.629154 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:33.630007 15387 leader_election.cc:290] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.630115 15190 raft_consensus.cc:2468] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:33.633064 15379 tablet_bootstrap.cc:492] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:33.633306 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:33.633770 15379 ts_tablet_manager.cc:1403] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.043s	user 0.012s	sys 0.008s
W20260709 12:28:33.635185 15229 leader_election.cc:343] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.636019 15229 leader_election.cc:304] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:33.640916 15189 raft_consensus.cc:2468] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:33.639398 15379 raft_consensus.cc:359] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.641881 15379 raft_consensus.cc:385] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.642252 15379 raft_consensus.cc:740] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.642549 15383 tablet_bootstrap.cc:492] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:33.639720 15387 raft_consensus.cc:2804] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:33.642913 15229 leader_election.cc:304] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:33.642899 15381 ts_tablet_manager.cc:1434] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.025s	user 0.011s	sys 0.008s
I20260709 12:28:33.643327 15383 ts_tablet_manager.cc:1403] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.029s	user 0.017s	sys 0.003s
I20260709 12:28:33.643411 15379 consensus_queue.cc:260] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.649529 15379 ts_tablet_manager.cc:1434] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.015s	user 0.010s	sys 0.002s
I20260709 12:28:33.643564 15387 raft_consensus.cc:493] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.650488 15387 raft_consensus.cc:3060] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.650558 15383 raft_consensus.cc:359] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.651696 15383 raft_consensus.cc:385] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.652032 15383 raft_consensus.cc:740] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.652935 15383 consensus_queue.cc:260] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.653959 15381 tablet_bootstrap.cc:492] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:33.644398 15407 raft_consensus.cc:2804] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:33.659051 15379 tablet_bootstrap.cc:492] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:33.663259 15383 ts_tablet_manager.cc:1434] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.003s
I20260709 12:28:33.665014 15407 raft_consensus.cc:697] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:33.666730 15381 tablet_bootstrap.cc:654] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.667325 15383 tablet_bootstrap.cc:492] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:33.666217 15407 consensus_queue.cc:237] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.675449 15383 tablet_bootstrap.cc:654] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.678493 15381 tablet_bootstrap.cc:492] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:33.679183 15381 ts_tablet_manager.cc:1403] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.010s
I20260709 12:28:33.681121 15383 tablet_bootstrap.cc:492] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:33.680961 15409 raft_consensus.cc:493] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.685832 15383 ts_tablet_manager.cc:1403] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.019s	user 0.006s	sys 0.007s
I20260709 12:28:33.685667 15409 raft_consensus.cc:515] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.688349 15409 leader_election.cc:290] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.684034 15379 tablet_bootstrap.cc:654] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.682852 15387 raft_consensus.cc:515] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.689080 15084 catalog_manager.cc:5697] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:33.689898 15383 raft_consensus.cc:359] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.690948 15383 raft_consensus.cc:385] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.691033 15387 leader_election.cc:290] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.691370 15383 raft_consensus.cc:740] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.692648 15383 consensus_queue.cc:260] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.693487 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:33.694188 15338 raft_consensus.cc:3060] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.694301 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b894d51dffb436797541081590814ad" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:33.695469 15383 ts_tablet_manager.cc:1434] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.003s
I20260709 12:28:33.695899 15189 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b894d51dffb436797541081590814ad" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
W20260709 12:28:33.696095 15229 leader_election.cc:343] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.696779 15383 tablet_bootstrap.cc:492] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:33.697422 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
W20260709 12:28:33.697594 15229 leader_election.cc:343] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.698129 15229 leader_election.cc:304] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:33.698132 15190 raft_consensus.cc:3060] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.689188 15381 raft_consensus.cc:359] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.702555 15381 raft_consensus.cc:385] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.702904 15381 raft_consensus.cc:740] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.703971 15381 consensus_queue.cc:260] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.704735 15379 tablet_bootstrap.cc:492] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:33.705190 15409 raft_consensus.cc:2749] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:33.707077 15379 ts_tablet_manager.cc:1403] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.048s	user 0.016s	sys 0.002s
I20260709 12:28:33.705343 15383 tablet_bootstrap.cc:654] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.708112 15386 raft_consensus.cc:493] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.706578 15381 ts_tablet_manager.cc:1434] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.027s	user 0.001s	sys 0.004s
I20260709 12:28:33.709828 15388 raft_consensus.cc:493] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.710336 15388 raft_consensus.cc:515] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.706599 15338 raft_consensus.cc:2468] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:33.710546 15379 raft_consensus.cc:359] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.711580 15379 raft_consensus.cc:385] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.711951 15379 raft_consensus.cc:740] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.710739 15381 tablet_bootstrap.cc:492] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:33.713184 15379 consensus_queue.cc:260] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.713174 15409 raft_consensus.cc:493] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.715276 15409 raft_consensus.cc:515] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.717003 15388 leader_election.cc:290] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:33.717904 15409 leader_election.cc:290] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.718652 15379 ts_tablet_manager.cc:1434] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.011s	user 0.008s	sys 0.000s
I20260709 12:28:33.717139 15417 raft_consensus.cc:493] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.720014 15379 tablet_bootstrap.cc:492] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:33.723723 15383 tablet_bootstrap.cc:492] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:33.724001 15189 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:33.724977 15189 raft_consensus.cc:2468] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:33.726012 15264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "411b646e3a71433a817ca20eab98d884" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:33.727756 15264 raft_consensus.cc:2468] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 0.
I20260709 12:28:33.708876 15386 raft_consensus.cc:515] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.729118 15381 tablet_bootstrap.cc:654] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.730545 15379 tablet_bootstrap.cc:654] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.731540 15189 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "411b646e3a71433a817ca20eab98d884" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
W20260709 12:28:33.734720 15303 leader_election.cc:343] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.738729 15386 leader_election.cc:290] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.724341 15383 ts_tablet_manager.cc:1403] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.028s	user 0.014s	sys 0.004s
I20260709 12:28:33.726375 15229 leader_election.cc:304] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062; no voters: 
I20260709 12:28:33.748068 15387 raft_consensus.cc:2804] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:33.749279 15387 raft_consensus.cc:697] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:33.754518 15387 consensus_queue.cc:237] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.756120 15379 tablet_bootstrap.cc:492] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:33.756835 15379 ts_tablet_manager.cc:1403] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.037s	user 0.016s	sys 0.005s
I20260709 12:28:33.740533 15417 raft_consensus.cc:515] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.760012 15379 raft_consensus.cc:359] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.760828 15379 raft_consensus.cc:385] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.761057 15417 leader_election.cc:290] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.742871 15302 leader_election.cc:304] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062; no voters: 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:33.758482 15190 raft_consensus.cc:2468] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:33.762461 15388 raft_consensus.cc:2804] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:33.763767 15388 raft_consensus.cc:493] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.765106 15229 leader_election.cc:304] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:33.765581 15388 raft_consensus.cc:3060] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.751387 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:33.774463 15383 raft_consensus.cc:359] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.780151 15383 raft_consensus.cc:385] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.780660 15383 raft_consensus.cc:740] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.767210 15387 raft_consensus.cc:2804] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:33.786424 15387 raft_consensus.cc:493] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.787012 15387 raft_consensus.cc:3060] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.761147 15379 raft_consensus.cc:740] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.781948 15383 consensus_queue.cc:260] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.805516 15381 tablet_bootstrap.cc:492] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:33.806187 15381 ts_tablet_manager.cc:1403] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.096s	user 0.016s	sys 0.001s
I20260709 12:28:33.805140 15388 raft_consensus.cc:515] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.810012 15381 raft_consensus.cc:359] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.810858 15381 raft_consensus.cc:385] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.811201 15381 raft_consensus.cc:740] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.812332 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "411b646e3a71433a817ca20eab98d884" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:33.820677 15417 raft_consensus.cc:493] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:28:33.814843 15229 leader_election.cc:343] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.821635 15417 raft_consensus.cc:515] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.824311 15264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "411b646e3a71433a817ca20eab98d884" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:33.825464 15264 raft_consensus.cc:3060] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.826227 15417 leader_election.cc:290] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.815027 15379 consensus_queue.cc:260] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.826624 15381 consensus_queue.cc:260] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.825850 15387 raft_consensus.cc:515] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.814386 15388 leader_election.cc:290] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:33.831864 15388 raft_consensus.cc:493] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.832408 15388 raft_consensus.cc:515] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.829286 15190 raft_consensus.cc:3060] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.833307 15379 ts_tablet_manager.cc:1434] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.076s	user 0.005s	sys 0.005s
I20260709 12:28:33.834842 15379 tablet_bootstrap.cc:492] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:33.833279 15383 ts_tablet_manager.cc:1434] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.090s	user 0.013s	sys 0.009s
I20260709 12:28:33.835590 15189 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94397791de6a4c489337a8bd7803f231" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:33.836364 15189 raft_consensus.cc:2468] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 0.
I20260709 12:28:33.839555 15383 tablet_bootstrap.cc:492] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:33.844130 15379 tablet_bootstrap.cc:654] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.845901 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94397791de6a4c489337a8bd7803f231" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:33.846014 15263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38d8237b9ab9493eb41e584ffde4f070" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:33.847026 15263 raft_consensus.cc:2468] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 0.
I20260709 12:28:33.846129 15381 ts_tablet_manager.cc:1434] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.040s	user 0.004s	sys 0.007s
I20260709 12:28:33.852609 15263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b894d51dffb436797541081590814ad" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:33.853747 15263 raft_consensus.cc:2468] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 0.
W20260709 12:28:33.855207 15153 leader_election.cc:343] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.856312 15425 raft_consensus.cc:493] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.856575 15153 leader_election.cc:304] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:33.857084 15425 raft_consensus.cc:515] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.859205 15303 leader_election.cc:304] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:33.860471 15424 raft_consensus.cc:2804] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:33.860924 15424 raft_consensus.cc:493] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.861002 15381 tablet_bootstrap.cc:492] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:33.861299 15424 raft_consensus.cc:3060] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.862136 15153 leader_election.cc:304] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:33.863099 15386 raft_consensus.cc:2804] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:33.863278 15417 raft_consensus.cc:2804] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:33.863848 15386 raft_consensus.cc:493] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.863994 15417 raft_consensus.cc:493] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.864535 15386 raft_consensus.cc:3060] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.864765 15417 raft_consensus.cc:3060] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.868625 15263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94397791de6a4c489337a8bd7803f231" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:33.863015 15388 leader_election.cc:290] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
W20260709 12:28:33.870677 15302 leader_election.cc:343] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.872498 15189 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:33.855634 15190 raft_consensus.cc:2468] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 1.
I20260709 12:28:33.876502 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:33.877228 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:33.877449 15188 raft_consensus.cc:3060] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.878355 15387 leader_election.cc:290] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.879923 15264 raft_consensus.cc:2468] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 1.
W20260709 12:28:33.881253 15229 leader_election.cc:343] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.873714 15189 raft_consensus.cc:2393] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in current term 1: Already voted for candidate 5a360f0ebb5c4bac86e43907cd673062 in this term.
I20260709 12:28:33.883369 15302 leader_election.cc:304] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062; no voters: 
I20260709 12:28:33.884795 15422 raft_consensus.cc:2804] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:33.894934 15425 leader_election.cc:290] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:33.901499 15381 tablet_bootstrap.cc:654] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.890939 15424 raft_consensus.cc:515] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.890748 15386 raft_consensus.cc:515] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.907389 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94397791de6a4c489337a8bd7803f231" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:33.907855 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38d8237b9ab9493eb41e584ffde4f070" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:33.924757 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38d8237b9ab9493eb41e584ffde4f070" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:33.925292 15338 raft_consensus.cc:2393] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 96664db3a4e8450a8d96fd457960bba6 in current term 1: Already voted for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in this term.
I20260709 12:28:33.909078 15383 tablet_bootstrap.cc:654] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:33.908548 15337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b894d51dffb436797541081590814ad" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:33.926882 15386 leader_election.cc:290] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.926560 15085 catalog_manager.cc:5697] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:33.928486 15154 leader_election.cc:343] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.929768 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "38d8237b9ab9493eb41e584ffde4f070" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:33.930033 15154 leader_election.cc:304] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:33.911943 15379 tablet_bootstrap.cc:492] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:33.931902 15379 ts_tablet_manager.cc:1403] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.097s	user 0.028s	sys 0.011s
I20260709 12:28:33.933444 15381 tablet_bootstrap.cc:492] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:33.934170 15381 ts_tablet_manager.cc:1403] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.073s	user 0.012s	sys 0.011s
I20260709 12:28:33.935175 15386 raft_consensus.cc:3060] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.913084 15417 raft_consensus.cc:515] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.938715 15379 raft_consensus.cc:359] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.939528 15262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b894d51dffb436797541081590814ad" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:33.940390 15262 raft_consensus.cc:3060] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.940481 15417 leader_election.cc:290] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.911335 15424 leader_election.cc:290] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:33.925314 15339 raft_consensus.cc:2468] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 0.
I20260709 12:28:33.916782 15188 raft_consensus.cc:2468] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:33.913093 15263 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94397791de6a4c489337a8bd7803f231" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:33.925311 15265 raft_consensus.cc:3060] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.911705 15264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:33.944423 15381 raft_consensus.cc:359] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.946273 15381 raft_consensus.cc:385] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.946537 15381 raft_consensus.cc:740] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
W20260709 12:28:33.947597 15302 leader_election.cc:343] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.947376 15381 consensus_queue.cc:260] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.906307 15422 raft_consensus.cc:697] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Becoming Leader. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:33.948151 15379 raft_consensus.cc:385] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:33.948619 15386 raft_consensus.cc:2749] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:33.950492 15379 raft_consensus.cc:740] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:33.950875 15381 ts_tablet_manager.cc:1434] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.016s	user 0.010s	sys 0.000s
I20260709 12:28:33.951408 15302 leader_election.cc:304] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e; no voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:33.951706 15379 consensus_queue.cc:260] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.952394 15381 tablet_bootstrap.cc:492] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:33.952920 15424 raft_consensus.cc:3060] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.947945 15229 leader_election.cc:304] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:33.955823 15386 raft_consensus.cc:493] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:33.956461 15386 raft_consensus.cc:515] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.956826 15387 raft_consensus.cc:2804] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:33.957556 15387 raft_consensus.cc:697] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:33.954629 15379 ts_tablet_manager.cc:1434] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260709 12:28:33.964152 15422 consensus_queue.cc:237] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.969736 15379 tablet_bootstrap.cc:492] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:33.971488 15387 consensus_queue.cc:237] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:33.975828 15264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:33.915176 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94397791de6a4c489337a8bd7803f231" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
W20260709 12:28:33.979300 15153 leader_election.cc:343] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.944232 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b894d51dffb436797541081590814ad" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
W20260709 12:28:33.983462 15154 leader_election.cc:343] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:33.985930 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:33.986775 15339 raft_consensus.cc:2468] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 0.
I20260709 12:28:33.988531 15154 leader_election.cc:304] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:33.989600 15417 raft_consensus.cc:2804] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:33.990046 15417 raft_consensus.cc:493] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:33.990403 15417 raft_consensus.cc:3060] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:33.992722 15265 raft_consensus.cc:2468] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 1.
I20260709 12:28:33.994024 15153 leader_election.cc:304] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:33.994714 15386 leader_election.cc:290] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:33.995163 15416 raft_consensus.cc:2804] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:34.003545 15417 raft_consensus.cc:515] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.006335 15379 tablet_bootstrap.cc:654] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:34.007637 15416 raft_consensus.cc:697] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:34.010643 15424 raft_consensus.cc:2749] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:34.011941 15416 consensus_queue.cc:237] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.010764 15262 raft_consensus.cc:2468] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 1.
I20260709 12:28:34.015213 15381 tablet_bootstrap.cc:654] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:34.016218 15153 leader_election.cc:304] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:34.017547 15434 raft_consensus.cc:2804] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:34.018299 15434 raft_consensus.cc:697] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:34.019495 15434 consensus_queue.cc:237] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.020483 15190 raft_consensus.cc:2468] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 1.
I20260709 12:28:34.024765 15303 leader_election.cc:304] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:34.030719 15379 tablet_bootstrap.cc:492] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:34.030311 15262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:34.031594 15379 ts_tablet_manager.cc:1403] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.070s	user 0.021s	sys 0.002s
I20260709 12:28:34.034972 15379 raft_consensus.cc:359] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.035811 15379 raft_consensus.cc:385] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:34.036418 15379 raft_consensus.cc:740] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:34.037775 15422 raft_consensus.cc:2804] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:34.037428 15379 consensus_queue.cc:260] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.038398 15422 raft_consensus.cc:697] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Becoming Leader. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:34.038486 15086 catalog_manager.cc:5697] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
W20260709 12:28:34.042035 15153 leader_election.cc:343] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:34.044024 15417 leader_election.cc:290] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.037345 15085 catalog_manager.cc:5697] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67). New cstate: current_term: 1 leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:34.045017 15383 tablet_bootstrap.cc:492] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:34.047775 15383 ts_tablet_manager.cc:1403] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.209s	user 0.028s	sys 0.018s
I20260709 12:28:34.046106 15379 ts_tablet_manager.cc:1434] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:28:34.047797 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:34.052968 15338 raft_consensus.cc:3060] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:34.039992 15422 consensus_queue.cc:237] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.073282 15383 raft_consensus.cc:359] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.074589 15383 raft_consensus.cc:385] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:34.074995 15383 raft_consensus.cc:740] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:34.076202 15383 consensus_queue.cc:260] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.082212 15381 tablet_bootstrap.cc:492] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:34.083164 15381 ts_tablet_manager.cc:1403] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.131s	user 0.024s	sys 0.015s
I20260709 12:28:34.084911 15338 raft_consensus.cc:2468] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 1.
I20260709 12:28:34.086504 15154 leader_election.cc:304] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:34.087788 15434 raft_consensus.cc:2804] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:34.088373 15434 raft_consensus.cc:697] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:34.089579 15383 ts_tablet_manager.cc:1434] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.041s	user 0.011s	sys 0.000s
I20260709 12:28:34.090487 15383 tablet_bootstrap.cc:492] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:34.089866 15434 consensus_queue.cc:237] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.094455 15381 raft_consensus.cc:359] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.095368 15381 raft_consensus.cc:385] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:34.095659 15381 raft_consensus.cc:740] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:34.096479 15381 consensus_queue.cc:260] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.098798 15381 ts_tablet_manager.cc:1434] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20260709 12:28:34.102536 15383 tablet_bootstrap.cc:654] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:34.107295 15387 raft_consensus.cc:493] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.108147 15387 raft_consensus.cc:515] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.112375 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:34.113353 15190 raft_consensus.cc:2468] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:34.113942 15086 catalog_manager.cc:5697] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67). New cstate: current_term: 1 leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:34.118216 15387 leader_election.cc:290] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.118983 15229 leader_election.cc:304] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:34.119769 15387 raft_consensus.cc:2804] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:34.120205 15387 raft_consensus.cc:493] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:34.120609 15387 raft_consensus.cc:3060] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:34.121284 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:34.122195 15338 raft_consensus.cc:2468] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:34.132055 15387 raft_consensus.cc:515] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.133139 15084 catalog_manager.cc:5697] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.135025 15409 consensus_queue.cc:1048] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.135779 15387 leader_election.cc:290] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.136111 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:34.136822 15190 raft_consensus.cc:3060] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:34.139113 15084 catalog_manager.cc:5697] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.142349 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:34.143114 15338 raft_consensus.cc:3060] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:34.147377 15383 tablet_bootstrap.cc:492] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:34.147969 15383 ts_tablet_manager.cc:1403] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.058s	user 0.010s	sys 0.017s
I20260709 12:28:34.152127 15190 raft_consensus.cc:2468] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:34.153002 15383 raft_consensus.cc:359] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.153928 15383 raft_consensus.cc:385] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:34.154256 15383 raft_consensus.cc:740] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:34.155028 15383 consensus_queue.cc:260] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.157227 15383 ts_tablet_manager.cc:1434] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:28:34.157603 15229 leader_election.cc:304] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:34.158547 15387 raft_consensus.cc:2804] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:34.159222 15387 raft_consensus.cc:697] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:34.159025 15409 consensus_queue.cc:1048] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.161299 15387 consensus_queue.cc:237] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.161646 15338 raft_consensus.cc:2468] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:34.176826 15409 consensus_queue.cc:1048] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.185426 15086 catalog_manager.cc:5697] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.203457 15387 consensus_queue.cc:1048] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.206399 15084 catalog_manager.cc:5697] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.256314 15422 raft_consensus.cc:493] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.256836 15422 raft_consensus.cc:515] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.258683 15422 leader_election.cc:290] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481)
I20260709 12:28:34.260036 15262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:34.260133 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:34.261232 15188 raft_consensus.cc:2393] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in current term 1: Already voted for candidate 5a360f0ebb5c4bac86e43907cd673062 in this term.
I20260709 12:28:34.262427 15303 leader_election.cc:304] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e; no voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:34.263298 15422 raft_consensus.cc:3060] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:34.269367 15422 raft_consensus.cc:2749] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:34.304010 15446 raft_consensus.cc:493] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.304620 15446 raft_consensus.cc:515] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.306342 15446 leader_election.cc:290] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.307766 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94397791de6a4c489337a8bd7803f231" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:34.308107 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "94397791de6a4c489337a8bd7803f231" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:34.308923 15188 raft_consensus.cc:2393] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a360f0ebb5c4bac86e43907cd673062 in current term 1: Already voted for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in this term.
I20260709 12:28:34.310045 15229 leader_election.cc:304] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:34.310892 15446 raft_consensus.cc:3060] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:34.316375 15446 raft_consensus.cc:2749] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:34.408180 15434 consensus_queue.cc:1048] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:34.408728 15446 consensus_queue.cc:1048] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.424396 15338 raft_consensus.cc:3060] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:34.430455 15407 raft_consensus.cc:493] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:34.431208 15407 raft_consensus.cc:515] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.433091 15387 consensus_queue.cc:1048] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.433889 15407 leader_election.cc:290] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.430183 15422 consensus_queue.cc:1048] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.436517 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:34.436633 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:34.437644 15339 raft_consensus.cc:2393] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a360f0ebb5c4bac86e43907cd673062 in current term 1: Already voted for candidate 96664db3a4e8450a8d96fd457960bba6 in this term.
I20260709 12:28:34.443487 15229 leader_election.cc:304] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:34.443652 15434 consensus_queue.cc:1048] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.444475 15446 raft_consensus.cc:3060] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:34.443492 15422 consensus_queue.cc:1048] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:28:34.463403 15446 raft_consensus.cc:2749] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:34.490326 15434 consensus_queue.cc:1048] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.494829 15339 raft_consensus.cc:3060] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:34.510385 15417 consensus_queue.cc:1048] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:28:34.519490 15302 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.66:42199, user_credentials={real_user=slave}} blocked reactor thread for 47644.3us
I20260709 12:28:34.520236 15417 consensus_queue.cc:1048] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.555286 15416 consensus_queue.cc:1048] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:34.570400 15265 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
mode: GRACEFUL
new_leader_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:32876
I20260709 12:28:34.571246 15265 raft_consensus.cc:606] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Received request to transfer leadership to TS 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:34.587822 15265 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
mode: GRACEFUL
new_leader_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:32876
I20260709 12:28:34.588407 15265 raft_consensus.cc:606] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Received request to transfer leadership to TS 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:34.594545 15265 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
mode: GRACEFUL
new_leader_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:32876
I20260709 12:28:34.595259 15265 raft_consensus.cc:606] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Received request to transfer leadership to TS 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:34.613693 15461 consensus_queue.cc:1048] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:34.615254 15338 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "411b646e3a71433a817ca20eab98d884"
dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
mode: GRACEFUL
new_leader_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:53714
I20260709 12:28:34.615967 15338 raft_consensus.cc:606] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Received request to transfer leadership to TS 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:34.620167 15461 consensus_queue.cc:1048] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:34.657394 15190 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9b894d51dffb436797541081590814ad"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
mode: GRACEFUL
new_leader_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:44518
I20260709 12:28:34.658078 15190 raft_consensus.cc:606] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Received request to transfer leadership to TS 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:34.667189 15190 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
mode: GRACEFUL
new_leader_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:44518
I20260709 12:28:34.667872 15190 raft_consensus.cc:606] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Received request to transfer leadership to TS 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:34.676981 15387 raft_consensus.cc:993] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062: : Instructing follower 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:34.677546 15446 raft_consensus.cc:1081] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Signalling peer 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:34.679519 15190 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:44504
I20260709 12:28:34.680140 15190 raft_consensus.cc:493] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:34.680497 15190 raft_consensus.cc:3060] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.688191 15190 raft_consensus.cc:515] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.690577 15190 leader_election.cc:290] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.691444 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:34.691809 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:34.692315 15265 raft_consensus.cc:3055] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:34.692396 15338 raft_consensus.cc:3060] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.692770 15265 raft_consensus.cc:740] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:34.693694 15265 consensus_queue.cc:260] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.695070 15265 raft_consensus.cc:3060] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.700398 15338 raft_consensus.cc:2468] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:34.702167 15154 leader_election.cc:304] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:34.702417 15265 raft_consensus.cc:2468] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:34.703234 15466 raft_consensus.cc:2804] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:34.703768 15466 raft_consensus.cc:697] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:34.704695 15466 consensus_queue.cc:237] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.713065 15086 catalog_manager.cc:5697] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 1 to 2, leader changed from 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66) to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.745190 15446 consensus_queue.cc:1048] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:34.757454 15387 consensus_queue.cc:1048] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:34.773393 15387 raft_consensus.cc:993] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062: : Instructing follower 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:34.773993 15446 raft_consensus.cc:1081] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Signalling peer 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:34.775954 15190 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:44504
I20260709 12:28:34.776614 15190 raft_consensus.cc:493] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:34.777014 15190 raft_consensus.cc:3060] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.783771 15190 raft_consensus.cc:515] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.785775 15190 leader_election.cc:290] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.786845 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:34.787150 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ccb4ca0924e4f7f955623ff7004e70b" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:34.787741 15265 raft_consensus.cc:3055] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:34.787892 15338 raft_consensus.cc:3060] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.788192 15265 raft_consensus.cc:740] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:34.789172 15265 consensus_queue.cc:260] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.790306 15265 raft_consensus.cc:3060] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.794373 15338 raft_consensus.cc:2468] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:34.795749 15154 leader_election.cc:304] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:34.796576 15466 raft_consensus.cc:2804] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:34.796484 15265 raft_consensus.cc:2468] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:34.797490 15466 raft_consensus.cc:697] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:34.798470 15466 consensus_queue.cc:237] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.806537 15085 catalog_manager.cc:5697] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 1 to 2, leader changed from 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66) to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.841262 15387 raft_consensus.cc:993] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062: : Instructing follower 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:34.841816 15387 raft_consensus.cc:1081] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Signalling peer 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:34.843681 15190 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:44504
I20260709 12:28:34.844309 15190 raft_consensus.cc:493] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:34.844726 15190 raft_consensus.cc:3060] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.850329 15190 raft_consensus.cc:515] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.852083 15190 leader_election.cc:290] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.853349 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:34.853351 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b7b1b7149d02468ab8902f6725e815d6" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:34.854245 15265 raft_consensus.cc:3055] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:34.854298 15338 raft_consensus.cc:3060] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.854655 15265 raft_consensus.cc:740] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:34.855777 15265 consensus_queue.cc:260] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.857038 15265 raft_consensus.cc:3060] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.860368 15338 raft_consensus.cc:2468] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:34.861804 15154 leader_election.cc:304] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:34.862907 15466 raft_consensus.cc:2804] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:34.863474 15466 raft_consensus.cc:697] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:34.864364 15466 consensus_queue.cc:237] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.864564 15265 raft_consensus.cc:2468] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:34.870383 15416 raft_consensus.cc:993] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6: : Instructing follower 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:34.871042 15466 raft_consensus.cc:1081] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Signalling peer 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:34.872920 15265 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9b894d51dffb436797541081590814ad"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:32862
I20260709 12:28:34.873538 15265 raft_consensus.cc:493] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:34.873931 15265 raft_consensus.cc:3060] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.873163 15085 catalog_manager.cc:5697] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 1 to 2, leader changed from 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66) to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.881559 15265 raft_consensus.cc:515] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.883944 15265 leader_election.cc:290] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.885313 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b894d51dffb436797541081590814ad" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:34.885118 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b894d51dffb436797541081590814ad" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:34.886137 15338 raft_consensus.cc:3060] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.886209 15190 raft_consensus.cc:3055] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:34.886718 15190 raft_consensus.cc:740] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:34.887850 15190 consensus_queue.cc:260] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.889309 15190 raft_consensus.cc:3060] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.894147 15338 raft_consensus.cc:2468] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:34.895874 15229 leader_election.cc:304] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062; no voters: 
I20260709 12:28:34.896785 15190 raft_consensus.cc:2468] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:34.896865 15387 raft_consensus.cc:2804] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:34.897733 15387 raft_consensus.cc:697] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:34.898587 15387 consensus_queue.cc:237] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.906626 15085 catalog_manager.cc:5697] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 1 to 2, leader changed from 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65) to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.925351 15460 raft_consensus.cc:993] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e: : Instructing follower 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:34.925933 15424 raft_consensus.cc:1081] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Signalling peer 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:34.927798 15265 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "411b646e3a71433a817ca20eab98d884"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:32846
I20260709 12:28:34.928368 15265 raft_consensus.cc:493] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:34.928726 15265 raft_consensus.cc:3060] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.934635 15265 raft_consensus.cc:515] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.936630 15265 leader_election.cc:290] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.937844 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "411b646e3a71433a817ca20eab98d884" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:34.937904 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "411b646e3a71433a817ca20eab98d884" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:34.939055 15338 raft_consensus.cc:3055] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:34.939059 15190 raft_consensus.cc:3060] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.939637 15338 raft_consensus.cc:740] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:34.940730 15338 consensus_queue.cc:260] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.942591 15338 raft_consensus.cc:3060] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.949009 15190 raft_consensus.cc:2468] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:34.950703 15229 leader_election.cc:304] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:34.951500 15338 raft_consensus.cc:2468] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:34.951649 15387 raft_consensus.cc:2804] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:34.952458 15387 raft_consensus.cc:697] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:34.953665 15387 consensus_queue.cc:237] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.961388 15085 catalog_manager.cc:5697] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 1 to 2, leader changed from 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67) to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:34.963896 15416 raft_consensus.cc:993] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6: : Instructing follower 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:34.964596 15466 raft_consensus.cc:1081] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Signalling peer 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:34.966495 15265 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:32862
I20260709 12:28:34.967199 15265 raft_consensus.cc:493] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:34.967602 15265 raft_consensus.cc:3060] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.975503 15265 raft_consensus.cc:515] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.977475 15265 leader_election.cc:290] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:34.978555 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:34.978832 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48a887aef66e4290bfed409cfa9483b7" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:34.979432 15190 raft_consensus.cc:3055] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:34.979458 15338 raft_consensus.cc:3060] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.979995 15190 raft_consensus.cc:740] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:34.980894 15190 consensus_queue.cc:260] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.982367 15190 raft_consensus.cc:3060] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:34.988011 15338 raft_consensus.cc:2468] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:34.989511 15229 leader_election.cc:304] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062; no voters: 
I20260709 12:28:34.989759 15190 raft_consensus.cc:2468] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:34.990448 15387 raft_consensus.cc:2804] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:34.991046 15387 raft_consensus.cc:697] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:34.991896 15387 consensus_queue.cc:237] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:34.999562 15085 catalog_manager.cc:5697] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 1 to 2, leader changed from 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65) to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:35.146533 15338 raft_consensus.cc:1275] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.148321 15466 consensus_queue.cc:1048] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.164721 15265 raft_consensus.cc:1275] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.167341 15466 consensus_queue.cc:1048] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.200791 15338 raft_consensus.cc:1275] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.202538 15466 consensus_queue.cc:1048] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.217206 15265 raft_consensus.cc:1275] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.219017 15479 consensus_queue.cc:1048] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.344848 15338 raft_consensus.cc:1275] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.346558 15479 consensus_queue.cc:1048] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.356046 15265 raft_consensus.cc:1275] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.358724 15416 consensus_queue.cc:1048] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.454259 15190 raft_consensus.cc:1275] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.455786 15387 consensus_queue.cc:1048] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.465386 15338 raft_consensus.cc:1275] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.467551 15407 consensus_queue.cc:1048] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.472520 15339 raft_consensus.cc:1275] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.474174 15407 consensus_queue.cc:1048] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:35.478878 15190 raft_consensus.cc:1275] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.480461 15407 consensus_queue.cc:1048] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.525353 15338 raft_consensus.cc:1275] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.527118 15407 consensus_queue.cc:1048] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:35.536505 15190 raft_consensus.cc:1275] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:35.538146 15407 consensus_queue.cc:1048] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:37.707875 15086 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53796:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:37.710466 15086 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:37.773244 15174 tablet_service.cc:1511] Processing CreateTablet for tablet 50b4eda044ac4aacbeda58806815994d (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:37.775106 15174 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50b4eda044ac4aacbeda58806815994d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.774581 15248 tablet_service.cc:1511] Processing CreateTablet for tablet 50b4eda044ac4aacbeda58806815994d (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:37.776084 15324 tablet_service.cc:1511] Processing CreateTablet for tablet 50b4eda044ac4aacbeda58806815994d (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:37.776952 15248 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50b4eda044ac4aacbeda58806815994d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.777415 15173 tablet_service.cc:1511] Processing CreateTablet for tablet 74e9eb1cd7b040e2b60bd58eae90eee2 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:37.779003 15173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74e9eb1cd7b040e2b60bd58eae90eee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.780390 15327 tablet_service.cc:1511] Processing CreateTablet for tablet 74e9eb1cd7b040e2b60bd58eae90eee2 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:37.777422 15324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50b4eda044ac4aacbeda58806815994d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.783905 15247 tablet_service.cc:1511] Processing CreateTablet for tablet 74e9eb1cd7b040e2b60bd58eae90eee2 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:37.785391 15247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74e9eb1cd7b040e2b60bd58eae90eee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.787685 15327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74e9eb1cd7b040e2b60bd58eae90eee2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.788932 15329 tablet_service.cc:1511] Processing CreateTablet for tablet f174bfb665b14afa8e228b1a24a8e228 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:37.793190 15326 tablet_service.cc:1511] Processing CreateTablet for tablet a7c6fff46060428a87dfcba972f0c09c (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:37.782578 15176 tablet_service.cc:1511] Processing CreateTablet for tablet f174bfb665b14afa8e228b1a24a8e228 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:37.797084 15254 tablet_service.cc:1511] Processing CreateTablet for tablet f174bfb665b14afa8e228b1a24a8e228 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:37.797573 15176 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f174bfb665b14afa8e228b1a24a8e228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.800011 15322 tablet_service.cc:1511] Processing CreateTablet for tablet f4388c97e8bb4c02bc184d84eda3a4f5 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:37.798677 15254 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f174bfb665b14afa8e228b1a24a8e228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.790669 15329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f174bfb665b14afa8e228b1a24a8e228. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.802377 15180 tablet_service.cc:1511] Processing CreateTablet for tablet a7c6fff46060428a87dfcba972f0c09c (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:37.803790 15249 tablet_service.cc:1511] Processing CreateTablet for tablet a7c6fff46060428a87dfcba972f0c09c (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:37.804157 15175 tablet_service.cc:1511] Processing CreateTablet for tablet f4388c97e8bb4c02bc184d84eda3a4f5 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:37.805440 15249 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c6fff46060428a87dfcba972f0c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.806093 15172 tablet_service.cc:1511] Processing CreateTablet for tablet 9c80ee6244c24d1dbf9c0443f19a6691 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:37.804239 15180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c6fff46060428a87dfcba972f0c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.808158 15250 tablet_service.cc:1511] Processing CreateTablet for tablet f4388c97e8bb4c02bc184d84eda3a4f5 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:37.809648 15250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4388c97e8bb4c02bc184d84eda3a4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.811290 15326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7c6fff46060428a87dfcba972f0c09c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.812827 15172 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c80ee6244c24d1dbf9c0443f19a6691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.819020 15322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4388c97e8bb4c02bc184d84eda3a4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.821183 15175 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4388c97e8bb4c02bc184d84eda3a4f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.823909 15252 tablet_service.cc:1511] Processing CreateTablet for tablet 9c80ee6244c24d1dbf9c0443f19a6691 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:37.825474 15252 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c80ee6244c24d1dbf9c0443f19a6691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.841605 15328 tablet_service.cc:1511] Processing CreateTablet for tablet 9c80ee6244c24d1dbf9c0443f19a6691 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:37.841605 15179 tablet_service.cc:1511] Processing CreateTablet for tablet 1e140d24292d43759a9ab5efc412b0fb (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:37.843812 15179 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e140d24292d43759a9ab5efc412b0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.846908 15328 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c80ee6244c24d1dbf9c0443f19a6691. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.855146 15251 tablet_service.cc:1511] Processing CreateTablet for tablet 1e140d24292d43759a9ab5efc412b0fb (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:37.857990 15325 tablet_service.cc:1511] Processing CreateTablet for tablet 1e140d24292d43759a9ab5efc412b0fb (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:37.859443 15325 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e140d24292d43759a9ab5efc412b0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.864111 15177 tablet_service.cc:1511] Processing CreateTablet for tablet 770b082647c54ccd9d673f69b6a6b2f9 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:37.865604 15177 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770b082647c54ccd9d673f69b6a6b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.867887 15253 tablet_service.cc:1511] Processing CreateTablet for tablet 770b082647c54ccd9d673f69b6a6b2f9 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:37.874493 15251 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e140d24292d43759a9ab5efc412b0fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.876294 15178 tablet_service.cc:1511] Processing CreateTablet for tablet fee1c7b2ffcd4350bccae1b4d489a17a (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:37.877729 15178 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee1c7b2ffcd4350bccae1b4d489a17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.882004 15253 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770b082647c54ccd9d673f69b6a6b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.883682 15323 tablet_service.cc:1511] Processing CreateTablet for tablet 770b082647c54ccd9d673f69b6a6b2f9 (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:37.886993 15321 tablet_service.cc:1511] Processing CreateTablet for tablet fee1c7b2ffcd4350bccae1b4d489a17a (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:37.888603 15321 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee1c7b2ffcd4350bccae1b4d489a17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.895394 15323 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 770b082647c54ccd9d673f69b6a6b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.882158 15255 tablet_service.cc:1511] Processing CreateTablet for tablet fee1c7b2ffcd4350bccae1b4d489a17a (DEFAULT_TABLE table=soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:37.899878 15255 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee1c7b2ffcd4350bccae1b4d489a17a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:37.967672 15498 tablet_bootstrap.cc:492] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:37.988690 15499 tablet_bootstrap.cc:492] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:37.996814 15498 tablet_bootstrap.cc:654] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.006549 15499 tablet_bootstrap.cc:654] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.021343 15497 tablet_bootstrap.cc:492] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:38.028501 15499 tablet_bootstrap.cc:492] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:38.029220 15499 ts_tablet_manager.cc:1403] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.041s	user 0.010s	sys 0.006s
I20260709 12:28:38.032799 15499 raft_consensus.cc:359] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.033643 15499 raft_consensus.cc:385] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.033991 15499 raft_consensus.cc:740] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.034865 15499 consensus_queue.cc:260] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.037498 15499 ts_tablet_manager.cc:1434] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:28:38.038695 15499 tablet_bootstrap.cc:492] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:38.047163 15497 tablet_bootstrap.cc:654] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.052861 15499 tablet_bootstrap.cc:654] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.057452 15498 tablet_bootstrap.cc:492] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:38.057858 15497 tablet_bootstrap.cc:492] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:38.058139 15498 ts_tablet_manager.cc:1403] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.091s	user 0.015s	sys 0.008s
I20260709 12:28:38.058527 15497 ts_tablet_manager.cc:1403] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.038s	user 0.006s	sys 0.013s
I20260709 12:28:38.059644 15499 tablet_bootstrap.cc:492] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:38.060266 15499 ts_tablet_manager.cc:1403] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.003s
I20260709 12:28:38.062143 15497 raft_consensus.cc:359] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.062690 15499 raft_consensus.cc:359] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.063366 15497 raft_consensus.cc:385] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.062330 15498 raft_consensus.cc:359] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.063722 15499 raft_consensus.cc:385] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.063926 15497 raft_consensus.cc:740] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.064077 15498 raft_consensus.cc:385] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.064177 15499 raft_consensus.cc:740] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.064821 15498 raft_consensus.cc:740] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.065425 15499 consensus_queue.cc:260] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.065702 15498 consensus_queue.cc:260] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.068493 15498 ts_tablet_manager.cc:1434] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:28:38.068493 15499 ts_tablet_manager.cc:1434] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.005s
I20260709 12:28:38.069736 15498 tablet_bootstrap.cc:492] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:38.069737 15499 tablet_bootstrap.cc:492] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:38.065536 15497 consensus_queue.cc:260] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.077772 15498 tablet_bootstrap.cc:654] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.077772 15499 tablet_bootstrap.cc:654] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.080751 15497 ts_tablet_manager.cc:1434] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260709 12:28:38.082306 15497 tablet_bootstrap.cc:492] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:38.090560 15497 tablet_bootstrap.cc:654] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.135849 15498 tablet_bootstrap.cc:492] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:38.135864 15499 tablet_bootstrap.cc:492] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:38.136504 15498 ts_tablet_manager.cc:1403] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.067s	user 0.018s	sys 0.045s
I20260709 12:28:38.136504 15499 ts_tablet_manager.cc:1403] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.067s	user 0.025s	sys 0.032s
I20260709 12:28:38.138020 15497 tablet_bootstrap.cc:492] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:38.138729 15497 ts_tablet_manager.cc:1403] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.057s	user 0.008s	sys 0.006s
I20260709 12:28:38.140290 15498 raft_consensus.cc:359] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.140579 15499 raft_consensus.cc:359] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.141322 15498 raft_consensus.cc:385] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.141527 15499 raft_consensus.cc:385] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.141680 15498 raft_consensus.cc:740] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.141927 15499 raft_consensus.cc:740] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.142539 15497 raft_consensus.cc:359] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.142551 15498 consensus_queue.cc:260] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.142727 15499 consensus_queue.cc:260] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.143714 15497 raft_consensus.cc:385] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.145586 15498 ts_tablet_manager.cc:1434] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:28:38.146296 15499 ts_tablet_manager.cc:1434] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.001s
I20260709 12:28:38.147346 15498 tablet_bootstrap.cc:492] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:38.147657 15499 tablet_bootstrap.cc:492] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:38.144292 15497 raft_consensus.cc:740] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.154743 15497 consensus_queue.cc:260] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.155445 15498 tablet_bootstrap.cc:654] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.157841 15497 ts_tablet_manager.cc:1434] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:28:38.159279 15497 tablet_bootstrap.cc:492] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:38.160915 15499 tablet_bootstrap.cc:654] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.165817 15416 raft_consensus.cc:493] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.166491 15416 raft_consensus.cc:515] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.167977 15498 tablet_bootstrap.cc:492] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:38.168815 15498 ts_tablet_manager.cc:1403] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.005s
I20260709 12:28:38.170678 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c6fff46060428a87dfcba972f0c09c" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:38.171561 15339 raft_consensus.cc:2468] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 0.
I20260709 12:28:38.172435 15497 tablet_bootstrap.cc:654] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.172569 15498 raft_consensus.cc:359] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.173496 15154 leader_election.cc:304] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.173858 15498 raft_consensus.cc:385] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.174196 15498 raft_consensus.cc:740] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.172730 15416 leader_election.cc:290] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.177585 15416 raft_consensus.cc:2804] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.178041 15416 raft_consensus.cc:493] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.178560 15416 raft_consensus.cc:3060] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.177891 15498 consensus_queue.cc:260] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.180264 15498 ts_tablet_manager.cc:1434] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:28:38.183183 15499 tablet_bootstrap.cc:492] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:38.184108 15499 ts_tablet_manager.cc:1403] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.037s	user 0.013s	sys 0.011s
I20260709 12:28:38.186031 15498 tablet_bootstrap.cc:492] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:38.188123 15499 raft_consensus.cc:359] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.189415 15499 raft_consensus.cc:385] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.189038 15416 raft_consensus.cc:515] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.189961 15499 raft_consensus.cc:740] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.195591 15416 leader_election.cc:290] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.191052 15499 consensus_queue.cc:260] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.196095 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c6fff46060428a87dfcba972f0c09c" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:38.211104 15498 tablet_bootstrap.cc:654] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
W20260709 12:28:38.217259 15154 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:34945, user_credentials={real_user=slave}} blocked reactor thread for 43879.8us
I20260709 12:28:38.216519 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c6fff46060428a87dfcba972f0c09c" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:38.217743 15499 ts_tablet_manager.cc:1434] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.033s	user 0.011s	sys 0.001s
I20260709 12:28:38.221220 15499 tablet_bootstrap.cc:492] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
W20260709 12:28:38.222573 15153 leader_election.cc:343] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:28:38.223763 15153 leader_election.cc:343] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:38.234143 15497 tablet_bootstrap.cc:492] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:38.235128 15497 ts_tablet_manager.cc:1403] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.076s	user 0.015s	sys 0.007s
I20260709 12:28:38.235881 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c6fff46060428a87dfcba972f0c09c" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:38.236960 15339 raft_consensus.cc:3060] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.239228 15497 raft_consensus.cc:359] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.241521 15497 raft_consensus.cc:385] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.241973 15497 raft_consensus.cc:740] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.242986 15497 consensus_queue.cc:260] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.247283 15498 tablet_bootstrap.cc:492] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:38.241078 15499 tablet_bootstrap.cc:654] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.247915 15498 ts_tablet_manager.cc:1403] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.062s	user 0.019s	sys 0.010s
I20260709 12:28:38.251345 15498 raft_consensus.cc:359] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.251930 15497 ts_tablet_manager.cc:1434] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.016s	user 0.009s	sys 0.000s
I20260709 12:28:38.252156 15498 raft_consensus.cc:385] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.252759 15498 raft_consensus.cc:740] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.253698 15497 tablet_bootstrap.cc:492] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:38.253539 15498 consensus_queue.cc:260] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.254038 15339 raft_consensus.cc:2468] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 1.
I20260709 12:28:38.255524 15154 leader_election.cc:304] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:38.256413 15511 raft_consensus.cc:2804] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.256938 15511 raft_consensus.cc:697] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:38.257822 15511 consensus_queue.cc:237] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.261637 15498 ts_tablet_manager.cc:1434] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.007s
I20260709 12:28:38.262993 15498 tablet_bootstrap.cc:492] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:38.263250 15485 raft_consensus.cc:493] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.263794 15485 raft_consensus.cc:515] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.267431 15485 leader_election.cc:290] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.269441 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f174bfb665b14afa8e228b1a24a8e228" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:38.267956 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f174bfb665b14afa8e228b1a24a8e228" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:38.272046 15339 raft_consensus.cc:2468] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:38.272571 15188 raft_consensus.cc:2468] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:38.274267 15229 leader_election.cc:304] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062; no voters: 
I20260709 12:28:38.271361 15497 tablet_bootstrap.cc:654] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.275890 15485 raft_consensus.cc:2804] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.276352 15485 raft_consensus.cc:493] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.276952 15485 raft_consensus.cc:3060] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.279809 15085 catalog_manager.cc:5697] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:38.286515 15424 raft_consensus.cc:493] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.287453 15424 raft_consensus.cc:515] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.292111 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4388c97e8bb4c02bc184d84eda3a4f5" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:38.293061 15188 raft_consensus.cc:2468] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 0.
I20260709 12:28:38.294875 15303 leader_election.cc:304] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.298532 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4388c97e8bb4c02bc184d84eda3a4f5" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
W20260709 12:28:38.300711 15302 leader_election.cc:343] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:38.301585 15424 leader_election.cc:290] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:38.302343 15424 raft_consensus.cc:2804] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.302853 15424 raft_consensus.cc:493] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.303313 15424 raft_consensus.cc:3060] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.305982 15499 tablet_bootstrap.cc:492] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:38.306674 15499 ts_tablet_manager.cc:1403] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.086s	user 0.026s	sys 0.016s
I20260709 12:28:38.311251 15499 raft_consensus.cc:359] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.312347 15499 raft_consensus.cc:385] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.312757 15499 raft_consensus.cc:740] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.315714 15499 consensus_queue.cc:260] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.319459 15485 raft_consensus.cc:515] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.320264 15499 ts_tablet_manager.cc:1434] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.001s
I20260709 12:28:38.322288 15499 tablet_bootstrap.cc:492] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:38.322433 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f174bfb665b14afa8e228b1a24a8e228" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:38.323280 15188 raft_consensus.cc:3060] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.312821 15498 tablet_bootstrap.cc:654] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.336282 15188 raft_consensus.cc:2468] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:38.337953 15229 leader_election.cc:304] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.339373 15506 raft_consensus.cc:2804] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.343839 15499 tablet_bootstrap.cc:654] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.357589 15499 tablet_bootstrap.cc:492] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:38.358290 15499 ts_tablet_manager.cc:1403] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.036s	user 0.012s	sys 0.011s
I20260709 12:28:38.363399 15485 leader_election.cc:290] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.363045 15499 raft_consensus.cc:359] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.364224 15499 raft_consensus.cc:385] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.364614 15499 raft_consensus.cc:740] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.365510 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f174bfb665b14afa8e228b1a24a8e228" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:38.365723 15499 consensus_queue.cc:260] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.366729 15339 raft_consensus.cc:3060] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.369661 15499 ts_tablet_manager.cc:1434] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.006s
I20260709 12:28:38.373626 15506 raft_consensus.cc:697] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:38.375304 15499 tablet_bootstrap.cc:492] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:38.375133 15506 consensus_queue.cc:237] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.337182 15424 raft_consensus.cc:515] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.381120 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4388c97e8bb4c02bc184d84eda3a4f5" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:38.381901 15188 raft_consensus.cc:3060] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.383076 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4388c97e8bb4c02bc184d84eda3a4f5" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:38.384138 15265 raft_consensus.cc:3060] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.386900 15499 tablet_bootstrap.cc:654] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.390869 15424 leader_election.cc:290] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:38.398990 15188 raft_consensus.cc:2468] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 1.
I20260709 12:28:38.401335 15303 leader_election.cc:304] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.401335 15086 catalog_manager.cc:5697] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:38.403680 15424 raft_consensus.cc:2804] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.404404 15424 raft_consensus.cc:697] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Becoming Leader. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:38.405339 15424 consensus_queue.cc:237] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.406289 15498 tablet_bootstrap.cc:492] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:38.406925 15498 ts_tablet_manager.cc:1403] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.144s	user 0.016s	sys 0.018s
I20260709 12:28:38.407807 15265 raft_consensus.cc:2468] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 1.
I20260709 12:28:38.411306 15497 tablet_bootstrap.cc:492] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:38.410121 15498 raft_consensus.cc:359] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.411981 15498 raft_consensus.cc:385] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.412050 15497 ts_tablet_manager.cc:1403] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.159s	user 0.052s	sys 0.051s
I20260709 12:28:38.412427 15498 raft_consensus.cc:740] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.413410 15339 raft_consensus.cc:2468] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
W20260709 12:28:38.415807 15136 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:38.417515 15497 raft_consensus.cc:359] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.418030 15498 consensus_queue.cc:260] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.422766 15497 raft_consensus.cc:385] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.423223 15497 raft_consensus.cc:740] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.424058 15497 consensus_queue.cc:260] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.427598 15498 ts_tablet_manager.cc:1434] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.020s	user 0.008s	sys 0.000s
I20260709 12:28:38.426592 15087 catalog_manager.cc:5697] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67). New cstate: current_term: 1 leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:38.428795 15498 tablet_bootstrap.cc:492] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:38.419795 15499 tablet_bootstrap.cc:492] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:38.430532 15485 raft_consensus.cc:493] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.431336 15499 ts_tablet_manager.cc:1403] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.056s	user 0.021s	sys 0.009s
I20260709 12:28:38.431257 15485 raft_consensus.cc:515] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.433724 15485 leader_election.cc:290] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.435745 15499 raft_consensus.cc:359] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.436741 15499 raft_consensus.cc:385] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.437083 15498 tablet_bootstrap.cc:654] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.437062 15499 raft_consensus.cc:740] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.437757 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c6fff46060428a87dfcba972f0c09c" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:38.438863 15338 raft_consensus.cc:2393] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a360f0ebb5c4bac86e43907cd673062 in current term 1: Already voted for candidate 96664db3a4e8450a8d96fd457960bba6 in this term.
I20260709 12:28:38.438879 15499 consensus_queue.cc:260] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.437112 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7c6fff46060428a87dfcba972f0c09c" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:38.442109 15506 raft_consensus.cc:493] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.442260 15499 ts_tablet_manager.cc:1434] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.010s	user 0.009s	sys 0.001s
I20260709 12:28:38.442746 15506 raft_consensus.cc:515] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.443755 15499 tablet_bootstrap.cc:492] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:38.443574 15229 leader_election.cc:304] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:38.445253 15521 raft_consensus.cc:3060] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.447388 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:38.448297 15188 raft_consensus.cc:2468] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:38.450306 15229 leader_election.cc:304] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.450644 15498 tablet_bootstrap.cc:492] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:38.451426 15498 ts_tablet_manager.cc:1403] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.023s	user 0.015s	sys 0.006s
I20260709 12:28:38.451510 15485 raft_consensus.cc:2804] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.452179 15485 raft_consensus.cc:493] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.452634 15485 raft_consensus.cc:3060] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.454324 15499 tablet_bootstrap.cc:654] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.455152 15498 raft_consensus.cc:359] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.456067 15498 raft_consensus.cc:385] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.456396 15498 raft_consensus.cc:740] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.457267 15498 consensus_queue.cc:260] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.459612 15497 ts_tablet_manager.cc:1434] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.047s	user 0.003s	sys 0.005s
I20260709 12:28:38.460280 15498 ts_tablet_manager.cc:1434] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.001s
I20260709 12:28:38.461153 15497 tablet_bootstrap.cc:492] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:38.461416 15498 tablet_bootstrap.cc:492] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:38.470602 15497 tablet_bootstrap.cc:654] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.471038 15506 leader_election.cc:290] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.471325 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
W20260709 12:28:38.473592 15229 leader_election.cc:343] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:38.479310 15499 tablet_bootstrap.cc:492] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:38.479434 15485 raft_consensus.cc:515] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.480135 15499 ts_tablet_manager.cc:1403] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.037s	user 0.010s	sys 0.011s
I20260709 12:28:38.482003 15485 leader_election.cc:290] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.483819 15498 tablet_bootstrap.cc:654] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.485241 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:38.486027 15188 raft_consensus.cc:3060] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.485580 15499 raft_consensus.cc:359] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.486594 15499 raft_consensus.cc:385] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.486963 15499 raft_consensus.cc:740] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.487780 15499 consensus_queue.cc:260] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.490327 15499 ts_tablet_manager.cc:1434] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260709 12:28:38.491475 15499 tablet_bootstrap.cc:492] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062: Bootstrap starting.
I20260709 12:28:38.495153 15510 raft_consensus.cc:493] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.496068 15510 raft_consensus.cc:515] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.501585 15499 tablet_bootstrap.cc:654] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.504628 15522 raft_consensus.cc:493] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.505329 15522 raft_consensus.cc:515] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.507536 15498 tablet_bootstrap.cc:492] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:38.507652 15521 raft_consensus.cc:2749] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:38.508208 15498 ts_tablet_manager.cc:1403] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.047s	user 0.013s	sys 0.014s
I20260709 12:28:38.509272 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
W20260709 12:28:38.511387 15229 leader_election.cc:343] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:38.511811 15498 raft_consensus.cc:359] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.513991 15522 leader_election.cc:290] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.514256 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:38.515211 15265 raft_consensus.cc:2468] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 0.
I20260709 12:28:38.516927 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:38.517715 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:38.518471 15339 raft_consensus.cc:2468] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:38.519788 15153 leader_election.cc:304] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.520467 15511 raft_consensus.cc:2804] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.520831 15511 raft_consensus.cc:493] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.521591 15510 leader_election.cc:290] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
W20260709 12:28:38.522539 15229 leader_election.cc:343] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:38.521159 15511 raft_consensus.cc:3060] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.523307 15229 leader_election.cc:304] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062; no voters: 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:38.529341 15188 raft_consensus.cc:2468] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:38.530699 15229 leader_election.cc:304] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:38.531122 15522 raft_consensus.cc:2804] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.532027 15522 raft_consensus.cc:493] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.532393 15522 raft_consensus.cc:3060] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.531211 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:38.535087 15521 raft_consensus.cc:2804] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.534487 15499 tablet_bootstrap.cc:492] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062: No bootstrap required, opened a new log
I20260709 12:28:38.536051 15499 ts_tablet_manager.cc:1403] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062: Time spent bootstrapping tablet: real 0.045s	user 0.022s	sys 0.017s
W20260709 12:28:38.536417 15154 leader_election.cc:343] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:38.536747 15521 raft_consensus.cc:697] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:38.513201 15498 raft_consensus.cc:385] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.537834 15498 raft_consensus.cc:740] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.537834 15521 consensus_queue.cc:237] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.540400 15498 consensus_queue.cc:260] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.543388 15498 ts_tablet_manager.cc:1434] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.035s	user 0.007s	sys 0.003s
I20260709 12:28:38.544704 15498 tablet_bootstrap.cc:492] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:38.548530 15511 raft_consensus.cc:515] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.551409 15499 raft_consensus.cc:359] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.552297 15497 tablet_bootstrap.cc:492] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:38.552446 15499 raft_consensus.cc:385] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.552981 15497 ts_tablet_manager.cc:1403] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.092s	user 0.020s	sys 0.000s
I20260709 12:28:38.553057 15499 raft_consensus.cc:740] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.552485 15522 raft_consensus.cc:515] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.556869 15497 raft_consensus.cc:359] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.557673 15497 raft_consensus.cc:385] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.557960 15497 raft_consensus.cc:740] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.559423 15499 consensus_queue.cc:260] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.560688 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:38.561344 15265 raft_consensus.cc:3060] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.564800 15498 tablet_bootstrap.cc:654] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.561321 15499 ts_tablet_manager.cc:1434] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.001s
I20260709 12:28:38.568305 15265 raft_consensus.cc:2468] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 1.
I20260709 12:28:38.569185 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
W20260709 12:28:38.571023 15229 leader_election.cc:343] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:38.571213 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:38.572044 15339 raft_consensus.cc:3060] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.573697 15086 catalog_manager.cc:5697] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:38.578176 15522 leader_election.cc:290] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.579707 15339 raft_consensus.cc:2468] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:38.581117 15229 leader_election.cc:304] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062; no voters: 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:38.582393 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:38.583123 15522 raft_consensus.cc:2804] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:28:38.583915 15154 leader_election.cc:343] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:38.584378 15522 raft_consensus.cc:697] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:38.586505 15511 leader_election.cc:290] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.585873 15522 consensus_queue.cc:237] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.587917 15497 consensus_queue.cc:260] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.588676 15153 leader_election.cc:304] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:38.589526 15511 raft_consensus.cc:2804] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.590205 15511 raft_consensus.cc:697] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:38.591058 15511 consensus_queue.cc:237] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.595615 15497 ts_tablet_manager.cc:1434] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.042s	user 0.007s	sys 0.003s
I20260709 12:28:38.596868 15497 tablet_bootstrap.cc:492] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:38.604604 15497 tablet_bootstrap.cc:654] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.605973 15498 tablet_bootstrap.cc:492] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:38.606623 15498 ts_tablet_manager.cc:1403] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.062s	user 0.029s	sys 0.016s
I20260709 12:28:38.605844 15083 catalog_manager.cc:5697] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:38.610240 15498 raft_consensus.cc:359] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.611056 15498 raft_consensus.cc:385] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.611317 15498 raft_consensus.cc:740] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.612130 15498 consensus_queue.cc:260] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.615033 15498 ts_tablet_manager.cc:1434] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:28:38.616870 15498 tablet_bootstrap.cc:492] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6: Bootstrap starting.
I20260709 12:28:38.624354 15498 tablet_bootstrap.cc:654] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.631479 15497 tablet_bootstrap.cc:492] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:38.632088 15497 ts_tablet_manager.cc:1403] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.035s	user 0.027s	sys 0.004s
I20260709 12:28:38.632704 15083 catalog_manager.cc:5697] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:38.636307 15497 raft_consensus.cc:359] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.637672 15497 raft_consensus.cc:385] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.637969 15497 raft_consensus.cc:740] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.640314 15497 consensus_queue.cc:260] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.641880 15498 tablet_bootstrap.cc:492] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6: No bootstrap required, opened a new log
I20260709 12:28:38.642580 15498 ts_tablet_manager.cc:1403] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.008s
I20260709 12:28:38.642815 15497 ts_tablet_manager.cc:1434] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:28:38.644155 15497 tablet_bootstrap.cc:492] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:38.645845 15498 raft_consensus.cc:359] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.646961 15498 raft_consensus.cc:385] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.647289 15498 raft_consensus.cc:740] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.648267 15498 consensus_queue.cc:260] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.650831 15497 tablet_bootstrap.cc:654] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.650966 15498 ts_tablet_manager.cc:1434] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.007s
I20260709 12:28:38.650928 15510 consensus_queue.cc:1048] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.668713 15497 tablet_bootstrap.cc:492] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:38.669399 15497 ts_tablet_manager.cc:1403] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.008s
I20260709 12:28:38.674486 15497 raft_consensus.cc:359] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.675374 15497 raft_consensus.cc:385] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.675727 15497 raft_consensus.cc:740] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.676774 15497 consensus_queue.cc:260] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.679396 15497 ts_tablet_manager.cc:1434] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.003s
I20260709 12:28:38.681499 15497 tablet_bootstrap.cc:492] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:38.688586 15497 tablet_bootstrap.cc:654] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.692312 15416 consensus_queue.cc:1048] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.707492 15497 tablet_bootstrap.cc:492] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:38.708055 15497 ts_tablet_manager.cc:1403] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.013s
I20260709 12:28:38.711115 15497 raft_consensus.cc:359] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.711745 15497 raft_consensus.cc:385] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.712091 15497 raft_consensus.cc:740] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.712832 15497 consensus_queue.cc:260] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.715247 15497 ts_tablet_manager.cc:1434] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
I20260709 12:28:38.716655 15497 tablet_bootstrap.cc:492] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e: Bootstrap starting.
I20260709 12:28:38.723690 15497 tablet_bootstrap.cc:654] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:38.729866 15497 tablet_bootstrap.cc:492] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e: No bootstrap required, opened a new log
I20260709 12:28:38.730464 15497 ts_tablet_manager.cc:1403] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.004s
I20260709 12:28:38.733243 15497 raft_consensus.cc:359] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.734035 15497 raft_consensus.cc:385] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:38.734313 15497 raft_consensus.cc:740] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Initialized, Role: FOLLOWER
I20260709 12:28:38.735164 15497 consensus_queue.cc:260] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.738066 15497 ts_tablet_manager.cc:1434] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:28:38.768922 15510 raft_consensus.cc:493] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.769658 15510 raft_consensus.cc:515] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.771991 15510 leader_election.cc:290] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.773010 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee1c7b2ffcd4350bccae1b4d489a17a" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:38.773468 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee1c7b2ffcd4350bccae1b4d489a17a" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:38.773730 15265 raft_consensus.cc:2468] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 0.
I20260709 12:28:38.774297 15339 raft_consensus.cc:2468] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 0.
I20260709 12:28:38.775103 15153 leader_election.cc:304] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.776221 15510 raft_consensus.cc:2804] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.776641 15510 raft_consensus.cc:493] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.776942 15510 raft_consensus.cc:3060] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.783114 15510 raft_consensus.cc:515] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.784044 15424 raft_consensus.cc:493] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.784689 15424 raft_consensus.cc:515] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.785411 15510 leader_election.cc:290] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.786412 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee1c7b2ffcd4350bccae1b4d489a17a" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:38.786618 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee1c7b2ffcd4350bccae1b4d489a17a" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:38.787014 15424 leader_election.cc:290] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:38.787768 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e140d24292d43759a9ab5efc412b0fb" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:38.787882 15424 raft_consensus.cc:493] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.787827 15339 raft_consensus.cc:3060] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.788861 15188 raft_consensus.cc:2468] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 0.
I20260709 12:28:38.789084 15262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e140d24292d43759a9ab5efc412b0fb" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:38.789495 15424 raft_consensus.cc:515] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.789919 15262 raft_consensus.cc:2468] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 0.
I20260709 12:28:38.790247 15303 leader_election.cc:304] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.791311 15518 raft_consensus.cc:2804] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.791738 15518 raft_consensus.cc:493] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.792091 15518 raft_consensus.cc:3060] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.792917 15424 leader_election.cc:290] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:38.787099 15265 raft_consensus.cc:3060] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.793605 15262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:38.796022 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:38.796804 15188 raft_consensus.cc:2393] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in current term 1: Already voted for candidate 5a360f0ebb5c4bac86e43907cd673062 in this term.
I20260709 12:28:38.796737 15339 raft_consensus.cc:2468] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 1.
I20260709 12:28:38.798892 15154 leader_election.cc:304] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.799419 15303 leader_election.cc:304] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e; no voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:38.799877 15510 raft_consensus.cc:2804] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.800406 15510 raft_consensus.cc:697] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:38.800732 15424 raft_consensus.cc:3060] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.802186 15510 consensus_queue.cc:237] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.803284 15518 raft_consensus.cc:515] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.807466 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e140d24292d43759a9ab5efc412b0fb" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:38.807770 15518 leader_election.cc:290] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:38.808159 15190 raft_consensus.cc:3060] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.808705 15518 consensus_queue.cc:1048] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.809233 15265 raft_consensus.cc:2468] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 1.
I20260709 12:28:38.812623 15262 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e140d24292d43759a9ab5efc412b0fb" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:38.813293 15262 raft_consensus.cc:3060] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.814098 15424 raft_consensus.cc:2749] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:38.813747 15522 raft_consensus.cc:493] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.814715 15522 raft_consensus.cc:515] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.817849 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770b082647c54ccd9d673f69b6a6b2f9" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:38.818583 15188 raft_consensus.cc:2468] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:38.818696 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770b082647c54ccd9d673f69b6a6b2f9" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:38.819528 15339 raft_consensus.cc:2468] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 0.
I20260709 12:28:38.820226 15229 leader_election.cc:304] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.820685 15522 leader_election.cc:290] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.821408 15522 raft_consensus.cc:2804] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:38.821817 15522 raft_consensus.cc:493] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:38.822199 15522 raft_consensus.cc:3060] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.822610 15190 raft_consensus.cc:2468] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 1.
I20260709 12:28:38.824002 15303 leader_election.cc:304] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.824872 15262 raft_consensus.cc:2468] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 1.
I20260709 12:28:38.824965 15424 raft_consensus.cc:2804] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.825856 15424 raft_consensus.cc:697] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Becoming Leader. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:38.826817 15424 consensus_queue.cc:237] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.834960 15086 catalog_manager.cc:5697] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 1 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:38.839260 15547 consensus_queue.cc:1048] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.840015 15082 catalog_manager.cc:5697] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67). New cstate: current_term: 1 leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:38.841601 15522 raft_consensus.cc:515] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.844878 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770b082647c54ccd9d673f69b6a6b2f9" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:38.845554 15188 raft_consensus.cc:3060] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.852298 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770b082647c54ccd9d673f69b6a6b2f9" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:38.853060 15339 raft_consensus.cc:3060] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:38.854830 15188 raft_consensus.cc:2468] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:38.856367 15229 leader_election.cc:304] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:38.857367 15521 raft_consensus.cc:2804] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:38.857910 15521 raft_consensus.cc:697] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:38.858902 15521 consensus_queue.cc:237] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.860172 15522 leader_election.cc:290] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 1 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.867447 15339 raft_consensus.cc:2468] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 1.
I20260709 12:28:38.874663 15086 catalog_manager.cc:5697] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 1 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:38.896566 15265 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
mode: GRACEFUL
new_leader_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:32876
I20260709 12:28:38.897133 15265 raft_consensus.cc:606] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Received request to transfer leadership to TS 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:38.904121 15265 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
mode: GRACEFUL
new_leader_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:32876
I20260709 12:28:38.904706 15265 raft_consensus.cc:606] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Received request to transfer leadership to TS 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:38.911578 15265 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f174bfb665b14afa8e228b1a24a8e228"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
mode: GRACEFUL
new_leader_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:32876
I20260709 12:28:38.912350 15265 raft_consensus.cc:606] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Received request to transfer leadership to TS 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:38.923523 15339 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f4388c97e8bb4c02bc184d84eda3a4f5"
dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
mode: GRACEFUL
new_leader_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:53714
I20260709 12:28:38.924190 15339 raft_consensus.cc:606] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Received request to transfer leadership to TS 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:38.930743 15190 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
mode: GRACEFUL
new_leader_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:44518
I20260709 12:28:38.931550 15190 raft_consensus.cc:606] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Received request to transfer leadership to TS 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:38.938400 15339 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1e140d24292d43759a9ab5efc412b0fb"
dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
mode: GRACEFUL
new_leader_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:53714
I20260709 12:28:38.939000 15339 raft_consensus.cc:606] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Received request to transfer leadership to TS 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:38.949169 15190 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fee1c7b2ffcd4350bccae1b4d489a17a"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
mode: GRACEFUL
new_leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:44518
I20260709 12:28:38.949805 15190 raft_consensus.cc:606] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Received request to transfer leadership to TS 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:38.973733 15521 consensus_queue.cc:1048] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.983870 15506 consensus_queue.cc:1048] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.988708 15416 raft_consensus.cc:493] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:38.990664 15416 raft_consensus.cc:515] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:38.993422 15416 leader_election.cc:290] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:38.993955 15521 consensus_queue.cc:1048] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:38.995510 15337 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" is_pre_election: true
I20260709 12:28:38.996539 15337 raft_consensus.cc:2393] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 96664db3a4e8450a8d96fd457960bba6 in current term 1: Already voted for candidate 5a360f0ebb5c4bac86e43907cd673062 in this term.
I20260709 12:28:39.000874 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:39.002933 15153 leader_election.cc:304] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:39.005187 15510 raft_consensus.cc:3060] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:39.013298 15485 raft_consensus.cc:993] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062: : Instructing follower 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:39.014021 15485 raft_consensus.cc:1081] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Signalling peer 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:39.018970 15190 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f174bfb665b14afa8e228b1a24a8e228"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:44504
I20260709 12:28:39.020946 15190 raft_consensus.cc:493] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:39.025381 15190 raft_consensus.cc:3060] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.028726 15506 consensus_queue.cc:1048] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.031742 15506 consensus_queue.cc:1048] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:39.032598 15518 raft_consensus.cc:493] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:39.033195 15518 raft_consensus.cc:515] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.037782 15189 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96664db3a4e8450a8d96fd457960bba6" is_pre_election: true
I20260709 12:28:39.038007 15518 leader_election.cc:290] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:39.040469 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a360f0ebb5c4bac86e43907cd673062" is_pre_election: true
I20260709 12:28:39.041327 15510 raft_consensus.cc:2749] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:39.041720 15265 raft_consensus.cc:2393] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in current term 1: Already voted for candidate 96664db3a4e8450a8d96fd457960bba6 in this term.
I20260709 12:28:39.045297 15302 leader_election.cc:304] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e; no voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6
I20260709 12:28:39.042316 15190 raft_consensus.cc:515] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.046224 15518 raft_consensus.cc:3060] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:39.052609 15506 raft_consensus.cc:993] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062: : Instructing follower 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:39.053162 15506 raft_consensus.cc:1081] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Signalling peer 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:39.054939 15188 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:44504
I20260709 12:28:39.055243 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f174bfb665b14afa8e228b1a24a8e228" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:39.055544 15188 raft_consensus.cc:493] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:39.055954 15188 raft_consensus.cc:3060] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.064581 15265 raft_consensus.cc:3055] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:39.055907 15190 leader_election.cc:290] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:39.068946 15265 raft_consensus.cc:740] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:39.069856 15265 consensus_queue.cc:260] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.071031 15265 raft_consensus.cc:3060] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.073544 15188 raft_consensus.cc:515] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.077414 15522 consensus_queue.cc:1048] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.078543 15416 consensus_queue.cc:1048] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.079794 15264 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:39.080425 15264 raft_consensus.cc:3055] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:39.080757 15264 raft_consensus.cc:740] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:39.081602 15264 consensus_queue.cc:260] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.083806 15264 raft_consensus.cc:3060] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.086668 15188 leader_election.cc:290] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:39.089758 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:39.090438 15339 raft_consensus.cc:3060] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.095589 15265 raft_consensus.cc:2468] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:39.097059 15153 leader_election.cc:304] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:39.098023 15416 raft_consensus.cc:2804] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:39.098214 15339 raft_consensus.cc:2468] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:39.090365 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f174bfb665b14afa8e228b1a24a8e228" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:39.098517 15416 raft_consensus.cc:697] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:39.099565 15416 consensus_queue.cc:237] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.099627 15338 raft_consensus.cc:3060] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.101095 15154 leader_election.cc:304] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:39.102034 15510 raft_consensus.cc:2804] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:39.107156 15510 raft_consensus.cc:697] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:39.108276 15510 consensus_queue.cc:237] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.112566 15518 raft_consensus.cc:2749] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:39.117442 15338 raft_consensus.cc:2468] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:39.121528 15264 raft_consensus.cc:2468] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:39.130374 15506 raft_consensus.cc:993] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062: : Instructing follower 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:39.131294 15521 raft_consensus.cc:1081] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Signalling peer 96664db3a4e8450a8d96fd457960bba6 to start an election
I20260709 12:28:39.131541 15511 raft_consensus.cc:993] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6: : Instructing follower 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:39.133369 15416 raft_consensus.cc:1081] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Signalling peer 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:39.136821 15083 catalog_manager.cc:5697] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 1 to 2, leader changed from 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66) to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.138823 15264 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:32862
I20260709 12:28:39.139585 15264 raft_consensus.cc:493] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:39.139981 15264 raft_consensus.cc:3060] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.140964 15188 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
 from {username='slave'} at 127.0.0.1:44504
I20260709 12:28:39.141680 15188 raft_consensus.cc:493] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:39.142087 15188 raft_consensus.cc:3060] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.143517 15083 catalog_manager.cc:5697] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 1 to 2, leader changed from 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66) to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.147068 15560 consensus_queue.cc:1048] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.151485 15188 raft_consensus.cc:515] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.155051 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:39.155788 15265 raft_consensus.cc:3055] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:39.156112 15265 raft_consensus.cc:740] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:39.156824 15188 leader_election.cc:290] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Requested vote from peers 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:39.156932 15265 consensus_queue.cc:260] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.158253 15265 raft_consensus.cc:3060] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.158478 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50b4eda044ac4aacbeda58806815994d" candidate_uuid: "96664db3a4e8450a8d96fd457960bba6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:39.159127 15338 raft_consensus.cc:3060] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.154986 15264 raft_consensus.cc:515] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.168180 15262 raft_consensus.cc:1240] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Rejecting Update request from peer 96664db3a4e8450a8d96fd457960bba6 for earlier term 1. Current term is 2. Ops: []
I20260709 12:28:39.168593 15338 raft_consensus.cc:2468] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:39.170274 15154 leader_election.cc:304] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:39.171469 15265 raft_consensus.cc:2468] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96664db3a4e8450a8d96fd457960bba6 in term 2.
I20260709 12:28:39.171591 15560 raft_consensus.cc:2804] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:39.172690 15510 consensus_queue.cc:1059] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.173516 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:39.173924 15510 raft_consensus.cc:3055] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:39.174299 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c80ee6244c24d1dbf9c0443f19a6691" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:39.174556 15510 raft_consensus.cc:740] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:39.174346 15264 leader_election.cc:290] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:39.176399 15560 raft_consensus.cc:697] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Becoming Leader. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:39.175726 15510 consensus_queue.cc:260] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.177537 15560 consensus_queue.cc:237] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.178337 15510 raft_consensus.cc:3060] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.196336 15086 catalog_manager.cc:5697] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 reported cstate change: term changed from 1 to 2, leader changed from 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66) to 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65). New cstate: current_term: 2 leader_uuid: "96664db3a4e8450a8d96fd457960bba6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.197309 15188 raft_consensus.cc:2468] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:39.200501 15229 leader_election.cc:304] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:39.201491 15521 raft_consensus.cc:2804] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:39.202101 15521 raft_consensus.cc:697] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:39.203013 15521 consensus_queue.cc:237] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.212047 15086 catalog_manager.cc:5697] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 1 to 2, leader changed from 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65) to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.229859 15518 consensus_queue.cc:1048] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:39.237684 15547 raft_consensus.cc:993] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e: : Instructing follower 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:39.238329 15424 raft_consensus.cc:1081] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Signalling peer 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:39.240902 15264 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1e140d24292d43759a9ab5efc412b0fb"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:32846
I20260709 12:28:39.252959 15264 raft_consensus.cc:493] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:39.253546 15264 raft_consensus.cc:3060] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.253779 15518 consensus_queue.cc:1048] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.265439 15264 raft_consensus.cc:515] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.270004 15264 leader_election.cc:290] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:39.271188 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e140d24292d43759a9ab5efc412b0fb" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:39.271821 15188 raft_consensus.cc:3060] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.272117 15339 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e140d24292d43759a9ab5efc412b0fb" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:39.272843 15339 raft_consensus.cc:3055] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:39.273159 15339 raft_consensus.cc:740] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:39.273901 15339 consensus_queue.cc:260] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.275046 15339 raft_consensus.cc:3060] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.277769 15188 raft_consensus.cc:2468] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:39.279311 15229 leader_election.cc:304] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:39.280122 15522 raft_consensus.cc:2804] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:39.280646 15522 raft_consensus.cc:697] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:39.281451 15522 consensus_queue.cc:237] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.282884 15339 raft_consensus.cc:2468] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:39.286231 15560 consensus_queue.cc:1048] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:39.291826 15086 catalog_manager.cc:5697] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 1 to 2, leader changed from 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67) to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.295303 15511 raft_consensus.cc:993] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6: : Instructing follower 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:39.295881 15511 raft_consensus.cc:1081] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Signalling peer 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:39.298381 15338 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fee1c7b2ffcd4350bccae1b4d489a17a"
dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:53698
I20260709 12:28:39.299111 15338 raft_consensus.cc:493] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:39.299501 15338 raft_consensus.cc:3060] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.301666 15522 consensus_queue.cc:1048] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.306309 15511 consensus_queue.cc:1048] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:39.313114 15338 raft_consensus.cc:515] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.314486 15339 raft_consensus.cc:1240] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Rejecting Update request from peer 96664db3a4e8450a8d96fd457960bba6 for earlier term 1. Current term is 2. Ops: []
I20260709 12:28:39.315716 15338 leader_election.cc:290] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 2 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:39.318053 15265 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee1c7b2ffcd4350bccae1b4d489a17a" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:39.319963 15560 consensus_queue.cc:1059] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.322299 15190 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee1c7b2ffcd4350bccae1b4d489a17a" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:39.324344 15506 consensus_queue.cc:1048] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:39.330418 15547 raft_consensus.cc:993] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e: : Instructing follower 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:39.330997 15547 raft_consensus.cc:1081] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Signalling peer 5a360f0ebb5c4bac86e43907cd673062 to start an election
I20260709 12:28:39.321275 15560 raft_consensus.cc:3055] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:39.332032 15560 raft_consensus.cc:740] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:39.334342 15264 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f4388c97e8bb4c02bc184d84eda3a4f5"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
 from {username='slave'} at 127.0.0.1:32846
I20260709 12:28:39.337694 15264 raft_consensus.cc:493] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:39.338231 15264 raft_consensus.cc:3060] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.334846 15560 consensus_queue.cc:260] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.344832 15560 raft_consensus.cc:3060] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.348646 15264 raft_consensus.cc:515] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.354760 15188 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4388c97e8bb4c02bc184d84eda3a4f5" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:39.355405 15188 raft_consensus.cc:3060] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.365406 15188 raft_consensus.cc:2468] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:39.366830 15229 leader_election.cc:304] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a360f0ebb5c4bac86e43907cd673062, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:39.366899 15190 raft_consensus.cc:2468] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 2.
I20260709 12:28:39.367846 15506 raft_consensus.cc:2804] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:39.368386 15506 raft_consensus.cc:697] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Becoming Leader. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:39.368777 15303 leader_election.cc:304] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 5a360f0ebb5c4bac86e43907cd673062
I20260709 12:28:39.369305 15506 consensus_queue.cc:237] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.370070 15518 raft_consensus.cc:2804] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:39.370630 15518 raft_consensus.cc:697] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 LEADER]: Becoming Leader. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:39.371222 15264 leader_election.cc:290] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [CANDIDATE]: Term 2 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945)
I20260709 12:28:39.371488 15518 consensus_queue.cc:237] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.375881 15338 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4388c97e8bb4c02bc184d84eda3a4f5" candidate_uuid: "5a360f0ebb5c4bac86e43907cd673062" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
I20260709 12:28:39.376786 15338 raft_consensus.cc:3055] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:39.377084 15338 raft_consensus.cc:740] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:39.378007 15338 consensus_queue.cc:260] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:39.379484 15338 raft_consensus.cc:3060] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.385541 15338 raft_consensus.cc:2468] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a360f0ebb5c4bac86e43907cd673062 in term 2.
I20260709 12:28:39.384732 15086 catalog_manager.cc:5697] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 reported cstate change: term changed from 1 to 2, leader changed from 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67) to 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66). New cstate: current_term: 2 leader_uuid: "5a360f0ebb5c4bac86e43907cd673062" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:39.385615 15082 catalog_manager.cc:5697] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e reported cstate change: term changed from 1 to 2, leader changed from 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65) to 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67). New cstate: current_term: 2 leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:39.499962 15338 raft_consensus.cc:1275] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.501693 15560 consensus_queue.cc:1048] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.506572 15264 raft_consensus.cc:1275] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.508579 15511 consensus_queue.cc:1048] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:39.628335 15338 raft_consensus.cc:3060] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.636919 15338 raft_consensus.cc:1275] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.638674 15522 consensus_queue.cc:1048] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:39.648867 15190 raft_consensus.cc:1275] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.650936 15522 consensus_queue.cc:1048] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.680644 15264 raft_consensus.cc:1275] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.682073 15511 consensus_queue.cc:1048] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.691197 15338 raft_consensus.cc:1275] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.693521 15510 consensus_queue.cc:1048] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.719545 15338 raft_consensus.cc:1275] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.721132 15510 consensus_queue.cc:1048] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.730242 15264 raft_consensus.cc:1275] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Refusing update from remote peer 96664db3a4e8450a8d96fd457960bba6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.736220 15510 consensus_queue.cc:1048] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:39.781607 15338 raft_consensus.cc:1275] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.783252 15522 consensus_queue.cc:1048] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.792513 15190 raft_consensus.cc:1275] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.794487 15521 consensus_queue.cc:1048] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:39.809815 15190 raft_consensus.cc:1275] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.811447 15518 consensus_queue.cc:1048] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.821151 15264 raft_consensus.cc:3060] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:39.832691 15264 raft_consensus.cc:1275] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.834399 15424 consensus_queue.cc:1048] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.935686 15190 raft_consensus.cc:1275] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.937171 15521 consensus_queue.cc:1048] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:39.952665 15338 raft_consensus.cc:1275] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Refusing update from remote peer 5a360f0ebb5c4bac86e43907cd673062: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:39.955768 15506 consensus_queue.cc:1048] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:41.954308 15086 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53796:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:28:41.978982 15086 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53796:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:28:42.021919   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
W20260709 12:28:42.164497 15229 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:39481, user_credentials={real_user=slave}} blocked reactor thread for 53545.7us
I20260709 12:28:42.224064 15612 tablet.cc:1724] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.297605 15185 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
mode: GRACEFUL
new_leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:50450
I20260709 12:28:42.298436 15185 raft_consensus.cc:606] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Received request to transfer leadership to TS 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:42.294255 15623 tablet.cc:1724] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.324399 15626 tablet.cc:1724] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:42.327603 15609 log.cc:927] Time spent T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
W20260709 12:28:42.328413 15618 log.cc:927] Time spent T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062: Append to log took a long time: real 0.075s	user 0.004s	sys 0.000s
I20260709 12:28:42.364722 15631 tablet.cc:1724] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.406199 15633 tablet.cc:1724] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.394990 15624 tablet.cc:1724] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.391363 15629 tablet.cc:1724] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:42.436237 15625 log.cc:927] Time spent T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Append to log took a long time: real 0.106s	user 0.002s	sys 0.000s
W20260709 12:28:42.455369 15291 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:1e140d24292d43759a9ab5efc412b0fb)
I20260709 12:28:42.457252 15632 tablet.cc:1724] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.473344 15635 tablet.cc:1724] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:28:42.478005 15291 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:1e140d24292d43759a9ab5efc412b0fb)
W20260709 12:28:42.479043 15291 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:1e140d24292d43759a9ab5efc412b0fb)
I20260709 12:28:42.478899 15630 tablet.cc:1724] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.516702 15642 tablet.cc:1724] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.541729 15624 tablet.cc:1724] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.543205 15632 tablet.cc:1724] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.547874 15639 tablet.cc:1724] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.548049 15612 tablet.cc:1724] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.557197 15633 tablet.cc:1724] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.563594 15623 tablet.cc:1724] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.565459 15629 tablet.cc:1724] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.565006 15631 tablet.cc:1724] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.570976 15630 tablet.cc:1724] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.574913 15603 raft_consensus.cc:993] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6: : Instructing follower 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:42.575548 15603 raft_consensus.cc:1081] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Signalling peer 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:42.577491 15338 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2"
dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:53698
I20260709 12:28:42.578193 15338 raft_consensus.cc:493] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:42.578553 15338 raft_consensus.cc:3060] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:42.582383 15626 tablet.cc:1724] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.584425 15261 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "770b082647c54ccd9d673f69b6a6b2f9"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
mode: GRACEFUL
new_leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:54708
I20260709 12:28:42.585125 15261 raft_consensus.cc:606] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Received request to transfer leadership to TS 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:42.586632   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260709 12:28:42.587337   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:42.589069 15338 raft_consensus.cc:515] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.590451 15332 raft_consensus.cc:1240] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 FOLLOWER]: Rejecting Update request from peer 96664db3a4e8450a8d96fd457960bba6 for earlier term 2. Current term is 3. Ops: []
I20260709 12:28:42.593436 15187 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:42.594172 15187 raft_consensus.cc:3055] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:42.594228 15338 leader_election.cc:290] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 3 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:42.594615 15187 raft_consensus.cc:740] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:42.596037 15187 consensus_queue.cc:260] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.597167 15187 raft_consensus.cc:3060] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:42.598508 15261 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720"
dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
mode: GRACEFUL
new_leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:54708
I20260709 12:28:42.599308 15261 raft_consensus.cc:606] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Received request to transfer leadership to TS 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:42.604911 15630 tablet.cc:1724] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.606134 15187 raft_consensus.cc:2468] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 3.
I20260709 12:28:42.608640 15632 tablet.cc:1724] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.610561 15261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "74e9eb1cd7b040e2b60bd58eae90eee2" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:42.611303 15261 raft_consensus.cc:3060] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:42.622637 15303 leader_election.cc:304] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:42.625898 15611 raft_consensus.cc:2804] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:42.626529 15611 raft_consensus.cc:697] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 LEADER]: Becoming Leader. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:42.627447 15611 consensus_queue.cc:237] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.629305 15642 tablet.cc:1724] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.630736 15629 tablet.cc:1724] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.641281 15186 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9"
dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
mode: GRACEFUL
new_leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:50450
I20260709 12:28:42.642143 15186 raft_consensus.cc:606] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Received request to transfer leadership to TS 009b6a1feaec4e1e87beb3d3d5e58e3e
I20260709 12:28:42.653110 15083 catalog_manager.cc:5697] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e reported cstate change: term changed from 2 to 3, leader changed from 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65) to 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67). New cstate: current_term: 3 leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:42.656378 15261 raft_consensus.cc:2468] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 3.
I20260709 12:28:42.657764 15083 catalog_manager.cc:5789] TS 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67:34945) does not have the latest schema for tablet 74e9eb1cd7b040e2b60bd58eae90eee2 (table soft_deleted_table [id=0205589b234641ebad99447587a0f1f5]). Expected version 1 got 0
I20260709 12:28:42.618643 15631 tablet.cc:1724] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:42.667033   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:28:42.672016   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:42.703581 15082 catalog_manager.cc:6242] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7: soft_deleted_table [id=0205589b234641ebad99447587a0f1f5] alter complete (version 1)
I20260709 12:28:42.852469 15506 raft_consensus.cc:993] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062: : Instructing follower 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:42.853055 15506 raft_consensus.cc:1081] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Signalling peer 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:42.855173 15332 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "770b082647c54ccd9d673f69b6a6b2f9"
dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:53690
I20260709 12:28:42.855868 15332 raft_consensus.cc:493] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:42.856153 15332 raft_consensus.cc:3060] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:42.861471 15606 raft_consensus.cc:993] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062: : Instructing follower 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:42.862233 15506 raft_consensus.cc:1081] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Signalling peer 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:42.864467 15338 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720"
dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:53690
I20260709 12:28:42.864988 15338 raft_consensus.cc:493] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:42.865293 15338 raft_consensus.cc:3060] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:42.865346 15332 raft_consensus.cc:515] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.868682 15332 leader_election.cc:290] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 2 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:42.868928 15187 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770b082647c54ccd9d673f69b6a6b2f9" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:42.869678 15187 raft_consensus.cc:3060] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:42.869755 15261 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "770b082647c54ccd9d673f69b6a6b2f9" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:42.870620 15261 raft_consensus.cc:3055] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:42.871028 15261 raft_consensus.cc:740] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:42.872038 15261 consensus_queue.cc:260] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.873611 15261 raft_consensus.cc:3060] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:42.873775 15338 raft_consensus.cc:515] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.876787 15338 leader_election.cc:290] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 2 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:42.877187 15187 raft_consensus.cc:2468] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 2.
I20260709 12:28:42.877665 15260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:42.877496 15186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a717a5ca8f84dbb9e36a0f73e90d720" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:42.878399 15303 leader_election.cc:304] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:42.878434 15260 raft_consensus.cc:3055] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:42.879492 15260 raft_consensus.cc:740] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a360f0ebb5c4bac86e43907cd673062, State: Running, Role: LEADER
I20260709 12:28:42.880805 15260 consensus_queue.cc:260] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.883116 15611 raft_consensus.cc:2804] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:42.883981 15611 raft_consensus.cc:697] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 LEADER]: Becoming Leader. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:42.885404 15611 consensus_queue.cc:237] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.878435 15186 raft_consensus.cc:3060] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:42.918088 15261 raft_consensus.cc:2468] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 2.
I20260709 12:28:42.918999 15260 raft_consensus.cc:3060] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:42.920624 15186 raft_consensus.cc:2468] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 2.
I20260709 12:28:42.922266 15303 leader_election.cc:304] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 96664db3a4e8450a8d96fd457960bba6; no voters: 
I20260709 12:28:42.923525 15611 raft_consensus.cc:2804] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:42.924120 15611 raft_consensus.cc:697] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 LEADER]: Becoming Leader. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:42.924983 15611 consensus_queue.cc:237] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.928285 15086 catalog_manager.cc:5697] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e reported cstate change: term changed from 1 to 2, leader changed from 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66) to 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67). New cstate: current_term: 2 leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:42.929900 15260 raft_consensus.cc:2468] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 2.
I20260709 12:28:42.945297 15086 catalog_manager.cc:5697] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e reported cstate change: term changed from 1 to 2, leader changed from 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66) to 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67). New cstate: current_term: 2 leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:42.978922 15638 raft_consensus.cc:993] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6: : Instructing follower 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:42.979478 15638 raft_consensus.cc:1081] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Signalling peer 009b6a1feaec4e1e87beb3d3d5e58e3e to start an election
I20260709 12:28:42.981338 15338 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9"
dest_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e"
 from {username='slave'} at 127.0.0.1:53698
I20260709 12:28:42.982101 15338 raft_consensus.cc:493] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:42.982501 15338 raft_consensus.cc:3060] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:42.991160 15338 raft_consensus.cc:515] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.993252 15338 leader_election.cc:290] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 3 election: Requested vote from peers 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481), 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199)
I20260709 12:28:42.994407 15260 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5a360f0ebb5c4bac86e43907cd673062"
I20260709 12:28:42.994426 15186 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d417fe864273410fb5740d055cb7e9c9" candidate_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "96664db3a4e8450a8d96fd457960bba6"
I20260709 12:28:42.995221 15260 raft_consensus.cc:3060] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:42.995424 15186 raft_consensus.cc:3055] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:42.995890 15186 raft_consensus.cc:740] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 96664db3a4e8450a8d96fd457960bba6, State: Running, Role: LEADER
I20260709 12:28:42.996871 15186 consensus_queue.cc:260] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:42.998353 15186 raft_consensus.cc:3060] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:43.005945 15260 raft_consensus.cc:2468] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 3.
I20260709 12:28:43.007320 15302 leader_election.cc:304] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009b6a1feaec4e1e87beb3d3d5e58e3e, 5a360f0ebb5c4bac86e43907cd673062; no voters: 
I20260709 12:28:43.008095 15611 raft_consensus.cc:2804] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:43.008481 15186 raft_consensus.cc:2468] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009b6a1feaec4e1e87beb3d3d5e58e3e in term 3.
I20260709 12:28:43.012602 15187 raft_consensus.cc:1275] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 3 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:28:43.015161 15424 consensus_queue.cc:1048] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:28:43.017555 15611 raft_consensus.cc:697] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 LEADER]: Becoming Leader. State: Replica: 009b6a1feaec4e1e87beb3d3d5e58e3e, State: Running, Role: LEADER
I20260709 12:28:43.018630 15611 consensus_queue.cc:237] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } }
I20260709 12:28:43.024765 15626 tablet.cc:1724] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:28:43.029486 15261 raft_consensus.cc:1275] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 3 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:28:43.031931 15086 catalog_manager.cc:5697] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e reported cstate change: term changed from 2 to 3, leader changed from 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65) to 009b6a1feaec4e1e87beb3d3d5e58e3e (127.0.140.67). New cstate: current_term: 3 leader_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009b6a1feaec4e1e87beb3d3d5e58e3e" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 34945 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:43.045631 15611 consensus_queue.cc:1048] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:28:43.068506 15626 tablet.cc:1719] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e: Skipping requested alter to schema version 1, tablet already version 1
I20260709 12:28:43.071866 15631 tablet.cc:1719] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062: Skipping requested alter to schema version 1, tablet already version 1
I20260709 12:28:43.078244 15633 tablet.cc:1719] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6: Skipping requested alter to schema version 1, tablet already version 1
I20260709 12:28:43.333927 15186 raft_consensus.cc:1275] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:28:43.335657 15611 consensus_queue.cc:1048] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:28:43.349344 15261 raft_consensus.cc:1275] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:28:43.353052 15651 consensus_queue.cc:1048] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:43.445097 15261 raft_consensus.cc:1275] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.449222 15651 consensus_queue.cc:1048] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:28:43.459044 15186 raft_consensus.cc:1275] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:43.461735 15651 consensus_queue.cc:1048] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:43.527153 15261 raft_consensus.cc:1275] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 3 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:43.530820 15651 consensus_queue.cc:1048] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a360f0ebb5c4bac86e43907cd673062" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:43.548484 15187 raft_consensus.cc:1275] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 3 FOLLOWER]: Refusing update from remote peer 009b6a1feaec4e1e87beb3d3d5e58e3e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:28:43.553838 15651 consensus_queue.cc:1048] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "96664db3a4e8450a8d96fd457960bba6" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 39481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:28:43.676535   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:43.683284   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:43.683993   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:43.690366   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:43.690994   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:44.692385   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:44.697971   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:44.698549   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:44.703261   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:44.703683   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:45.704819   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:45.710597   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:45.711300   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:45.717119   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:45.717553   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:46.718629   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:46.724136   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:46.724617   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:46.729607   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:46.730031   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:47.731319   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:47.736645   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:47.737133   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:47.742615   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:47.743168   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:48.744577   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:48.752347   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:48.752966   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:48.759514   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:48.760136   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:49.761283   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:49.766897   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:49.767398   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:49.772152   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:49.772567   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:50.773722   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:50.779628   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:50.780128   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:50.784801   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:50.785214   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:51.786392   561 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:28:51.791702   561 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:28:51.792207   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:51.797245   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:51.797698   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:52.874105   561 tablet_server.cc:179] TabletServer@127.0.140.65:0 shutting down...
I20260709 12:28:52.932694   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:52.933799   561 tablet_replica.cc:333] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.935362   561 raft_consensus.cc:2243] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:52.936683   561 raft_consensus.cc:2272] T 50b4eda044ac4aacbeda58806815994d P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.940148   561 tablet_replica.cc:333] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.941262   561 raft_consensus.cc:2243] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:52.941957   561 raft_consensus.cc:2272] T 1e140d24292d43759a9ab5efc412b0fb P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.944885   561 tablet_replica.cc:333] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.945667   561 raft_consensus.cc:2243] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:52.946398   561 raft_consensus.cc:2272] T fee1c7b2ffcd4350bccae1b4d489a17a P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.949493   561 tablet_replica.cc:333] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.950436   561 raft_consensus.cc:2243] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:52.951699   561 raft_consensus.cc:2272] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 96664db3a4e8450a8d96fd457960bba6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.955523   561 tablet_replica.cc:333] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.956210   561 raft_consensus.cc:2243] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:52.957456   561 raft_consensus.cc:2272] T f174bfb665b14afa8e228b1a24a8e228 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.961206   561 tablet_replica.cc:333] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.962167   561 raft_consensus.cc:2243] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:52.963120   561 raft_consensus.cc:2272] T f4388c97e8bb4c02bc184d84eda3a4f5 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.967589   561 tablet_replica.cc:333] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.968955   561 raft_consensus.cc:2243] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:52.969668   561 raft_consensus.cc:2272] T 9c80ee6244c24d1dbf9c0443f19a6691 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.972467   561 tablet_replica.cc:333] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.973275   561 raft_consensus.cc:2243] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:52.973899   561 raft_consensus.cc:2272] T 411b646e3a71433a817ca20eab98d884 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.977486   561 tablet_replica.cc:333] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.978222   561 raft_consensus.cc:2243] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:52.979072   561 raft_consensus.cc:2272] T 9b894d51dffb436797541081590814ad P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.982025   561 tablet_replica.cc:333] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.982749   561 raft_consensus.cc:2243] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:52.983906   561 raft_consensus.cc:2272] T a7c6fff46060428a87dfcba972f0c09c P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.986709   561 tablet_replica.cc:333] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.987514   561 raft_consensus.cc:2243] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:52.988066   561 raft_consensus.cc:2272] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.990836   561 tablet_replica.cc:333] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.991500   561 raft_consensus.cc:2243] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:52.992094   561 raft_consensus.cc:2272] T 48a887aef66e4290bfed409cfa9483b7 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:52.994935   561 tablet_replica.cc:333] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:52.995635   561 raft_consensus.cc:2243] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:52.996855   561 raft_consensus.cc:2272] T b7b1b7149d02468ab8902f6725e815d6 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:52.999851 15229 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
I20260709 12:28:53.000900   561 tablet_replica.cc:333] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:53.001859   561 raft_consensus.cc:2243] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.003358   561 raft_consensus.cc:2272] T 8ccb4ca0924e4f7f955623ff7004e70b P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:53.003970 15229 consensus_peers.cc:597] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:53.006526   561 tablet_replica.cc:333] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:53.007433   561 raft_consensus.cc:2243] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.007977   561 raft_consensus.cc:2272] T 94397791de6a4c489337a8bd7803f231 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.011889   561 tablet_replica.cc:333] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:53.012993   561 raft_consensus.cc:2243] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.013741   561 raft_consensus.cc:2272] T 770b082647c54ccd9d673f69b6a6b2f9 P 96664db3a4e8450a8d96fd457960bba6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.016820   561 tablet_replica.cc:333] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:53.017565   561 raft_consensus.cc:2243] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.018117   561 raft_consensus.cc:2272] T d417fe864273410fb5740d055cb7e9c9 P 96664db3a4e8450a8d96fd457960bba6 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.021293   561 tablet_replica.cc:333] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6: stopping tablet replica
I20260709 12:28:53.021965   561 raft_consensus.cc:2243] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.023284   561 raft_consensus.cc:2272] T 38d8237b9ab9493eb41e584ffde4f070 P 96664db3a4e8450a8d96fd457960bba6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:53.029753 15229 consensus_peers.cc:597] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:53.034695 15229 consensus_peers.cc:597] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:53.058982 15303 consensus_peers.cc:597] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:53.071249 15303 consensus_peers.cc:597] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:53.088061   561 tablet_server.cc:196] TabletServer@127.0.140.65:0 shutdown complete.
W20260709 12:28:53.106115 15303 consensus_peers.cc:597] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:39481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:53.149811   561 tablet_server.cc:179] TabletServer@127.0.140.66:0 shutting down...
W20260709 12:28:53.181656 15303 consensus_peers.cc:597] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:39481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:53.192215 15229 consensus_peers.cc:597] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:39481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:53.203989   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:53.205072   561 tablet_replica.cc:333] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.205858   561 raft_consensus.cc:2243] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.206930   561 raft_consensus.cc:2272] T 1e140d24292d43759a9ab5efc412b0fb P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.209770   561 tablet_replica.cc:333] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.210440   561 raft_consensus.cc:2243] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.211472   561 raft_consensus.cc:2272] T f4388c97e8bb4c02bc184d84eda3a4f5 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.213896   561 tablet_replica.cc:333] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.214589   561 raft_consensus.cc:2243] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.215582   561 raft_consensus.cc:2272] T a7c6fff46060428a87dfcba972f0c09c P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.217970   561 tablet_replica.cc:333] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.218593   561 raft_consensus.cc:2243] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.219239   561 raft_consensus.cc:2272] T f174bfb665b14afa8e228b1a24a8e228 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.221706   561 tablet_replica.cc:333] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.222283   561 raft_consensus.cc:2243] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.222864   561 raft_consensus.cc:2272] T fee1c7b2ffcd4350bccae1b4d489a17a P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.225131   561 tablet_replica.cc:333] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.225759   561 raft_consensus.cc:2243] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.226403   561 raft_consensus.cc:2272] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 5a360f0ebb5c4bac86e43907cd673062 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.228840   561 tablet_replica.cc:333] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.229789   561 raft_consensus.cc:2243] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.230440   561 raft_consensus.cc:2272] T 8ccb4ca0924e4f7f955623ff7004e70b P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.232859   561 tablet_replica.cc:333] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.233672   561 raft_consensus.cc:2243] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.234205   561 raft_consensus.cc:2272] T 94397791de6a4c489337a8bd7803f231 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.236780   561 tablet_replica.cc:333] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.237421   561 raft_consensus.cc:2243] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.237962   561 raft_consensus.cc:2272] T b7b1b7149d02468ab8902f6725e815d6 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.240481   561 tablet_replica.cc:333] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.241289   561 raft_consensus.cc:2243] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.242214   561 raft_consensus.cc:2272] T 48a887aef66e4290bfed409cfa9483b7 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.244704   561 tablet_replica.cc:333] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.245275   561 raft_consensus.cc:2243] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.245769   561 raft_consensus.cc:2272] T 38d8237b9ab9493eb41e584ffde4f070 P 5a360f0ebb5c4bac86e43907cd673062 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.248109   561 tablet_replica.cc:333] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.248696   561 raft_consensus.cc:2243] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.249769   561 raft_consensus.cc:2272] T 9c80ee6244c24d1dbf9c0443f19a6691 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.252801   561 tablet_replica.cc:333] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.253433   561 raft_consensus.cc:2243] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.254418   561 raft_consensus.cc:2272] T 411b646e3a71433a817ca20eab98d884 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.257099   561 tablet_replica.cc:333] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.257705   561 raft_consensus.cc:2243] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.258188   561 raft_consensus.cc:2272] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.261082   561 tablet_replica.cc:333] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.261631   561 raft_consensus.cc:2243] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.262188   561 raft_consensus.cc:2272] T 50b4eda044ac4aacbeda58806815994d P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.264842   561 tablet_replica.cc:333] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.265419   561 raft_consensus.cc:2243] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.266455   561 raft_consensus.cc:2272] T 9b894d51dffb436797541081590814ad P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.268968   561 tablet_replica.cc:333] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.269544   561 raft_consensus.cc:2243] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.270085   561 raft_consensus.cc:2272] T 770b082647c54ccd9d673f69b6a6b2f9 P 5a360f0ebb5c4bac86e43907cd673062 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.272491   561 tablet_replica.cc:333] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062: stopping tablet replica
I20260709 12:28:53.272986   561 raft_consensus.cc:2243] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.273478   561 raft_consensus.cc:2272] T d417fe864273410fb5740d055cb7e9c9 P 5a360f0ebb5c4bac86e43907cd673062 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:53.285596 15303 consensus_peers.cc:597] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:39481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:53.323385   561 tablet_server.cc:196] TabletServer@127.0.140.66:0 shutdown complete.
I20260709 12:28:53.370831   561 tablet_server.cc:179] TabletServer@127.0.140.67:0 shutting down...
W20260709 12:28:53.375262 15302 consensus_peers.cc:597] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e -> Peer 5a360f0ebb5c4bac86e43907cd673062 (127.0.140.66:42199): Couldn't send request to peer 5a360f0ebb5c4bac86e43907cd673062. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:42199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:53.413611   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:53.414837   561 tablet_replica.cc:333] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.415952   561 raft_consensus.cc:2243] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.416854   561 raft_consensus.cc:2272] T 1e140d24292d43759a9ab5efc412b0fb P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:53.419513 15303 consensus_peers.cc:597] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e -> Peer 96664db3a4e8450a8d96fd457960bba6 (127.0.140.65:39481): Couldn't send request to peer 96664db3a4e8450a8d96fd457960bba6. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:39481: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:53.420746   561 tablet_replica.cc:333] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.421525   561 raft_consensus.cc:2243] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.422516   561 raft_consensus.cc:2272] T fee1c7b2ffcd4350bccae1b4d489a17a P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.425240   561 tablet_replica.cc:333] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.425872   561 raft_consensus.cc:2243] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.426885   561 raft_consensus.cc:2272] T 74e9eb1cd7b040e2b60bd58eae90eee2 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.429518   561 tablet_replica.cc:333] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.430285   561 raft_consensus.cc:2243] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.430841   561 raft_consensus.cc:2272] T 50b4eda044ac4aacbeda58806815994d P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.433367   561 tablet_replica.cc:333] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.433996   561 raft_consensus.cc:2243] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.434522   561 raft_consensus.cc:2272] T f174bfb665b14afa8e228b1a24a8e228 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.437300   561 tablet_replica.cc:333] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.437958   561 raft_consensus.cc:2243] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.438494   561 raft_consensus.cc:2272] T f4388c97e8bb4c02bc184d84eda3a4f5 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.441187   561 tablet_replica.cc:333] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.441816   561 raft_consensus.cc:2243] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.463459   561 raft_consensus.cc:2272] T 9b894d51dffb436797541081590814ad P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.469210   561 tablet_replica.cc:333] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.470985   561 raft_consensus.cc:2243] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.473264   561 raft_consensus.cc:2272] T 94397791de6a4c489337a8bd7803f231 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.477205   561 tablet_replica.cc:333] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.478169   561 raft_consensus.cc:2243] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.479080   561 raft_consensus.cc:2272] T 8ccb4ca0924e4f7f955623ff7004e70b P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.482584   561 tablet_replica.cc:333] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.483599   561 raft_consensus.cc:2243] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.485314   561 raft_consensus.cc:2272] T 770b082647c54ccd9d673f69b6a6b2f9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.489440   561 tablet_replica.cc:333] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.490350   561 raft_consensus.cc:2243] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.492417   561 raft_consensus.cc:2272] T d417fe864273410fb5740d055cb7e9c9 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.495657   561 tablet_replica.cc:333] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.496495   561 raft_consensus.cc:2243] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.497315   561 raft_consensus.cc:2272] T 9c80ee6244c24d1dbf9c0443f19a6691 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.500701   561 tablet_replica.cc:333] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.501543   561 raft_consensus.cc:2243] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.502305   561 raft_consensus.cc:2272] T 411b646e3a71433a817ca20eab98d884 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.505687   561 tablet_replica.cc:333] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.506459   561 raft_consensus.cc:2243] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.507351   561 raft_consensus.cc:2272] T a7c6fff46060428a87dfcba972f0c09c P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.510836   561 tablet_replica.cc:333] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.511511   561 raft_consensus.cc:2243] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.512318   561 raft_consensus.cc:2272] T 48a887aef66e4290bfed409cfa9483b7 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.515829   561 tablet_replica.cc:333] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.516611   561 raft_consensus.cc:2243] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.517360   561 raft_consensus.cc:2272] T b7b1b7149d02468ab8902f6725e815d6 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.520624   561 tablet_replica.cc:333] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.521456   561 raft_consensus.cc:2243] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.523386   561 raft_consensus.cc:2272] T 7a717a5ca8f84dbb9e36a0f73e90d720 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.526655   561 tablet_replica.cc:333] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e: stopping tablet replica
I20260709 12:28:53.527412   561 raft_consensus.cc:2243] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:53.528156   561 raft_consensus.cc:2272] T 38d8237b9ab9493eb41e584ffde4f070 P 009b6a1feaec4e1e87beb3d3d5e58e3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.594165   561 tablet_server.cc:196] TabletServer@127.0.140.67:0 shutdown complete.
I20260709 12:28:53.662922   561 master.cc:562] Master@127.0.140.126:37653 shutting down...
I20260709 12:28:53.702602   561 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:53.703630   561 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:53.704103   561 tablet_replica.cc:333] T 00000000000000000000000000000000 P ca3a059e0be54f9898675800d2b70be7: stopping tablet replica
I20260709 12:28:53.732648   561 master.cc:584] Master@127.0.140.126:37653 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (21603 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 12:28:53.807240   561 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.140.126:37015
I20260709 12:28:53.808645   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:53.815677 15685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:53.816090 15686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:53.817765 15688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:53.818423   561 server_base.cc:1061] running on GCE node
I20260709 12:28:53.819705   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:53.819957   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:53.820129   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600133820110 us; error 0 us; skew 500 ppm
I20260709 12:28:53.820748   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:53.823848   561 webserver.cc:533] Webserver started at http://127.0.140.126:33005/ using document root <none> and password file <none>
I20260709 12:28:53.824476   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:53.824702   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:53.825006   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:53.826674   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/master-0-root/instance:
uuid: "f84b7681764f482aadad4c622874defe"
format_stamp: "Formatted at 2026-07-09 12:28:53 on dist-test-slave-7mcn"
I20260709 12:28:53.832333   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:28:53.836712 15693 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:53.837807   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:28:53.838088   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/master-0-root
uuid: "f84b7681764f482aadad4c622874defe"
format_stamp: "Formatted at 2026-07-09 12:28:53 on dist-test-slave-7mcn"
I20260709 12:28:53.838356   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:53.867646   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:53.868953   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:53.915517   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.126:37015
I20260709 12:28:53.915704 15744 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.126:37015 every 8 connection(s)
I20260709 12:28:53.920385 15745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:53.937268 15745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe: Bootstrap starting.
I20260709 12:28:53.943531 15745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:53.949107 15745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe: No bootstrap required, opened a new log
I20260709 12:28:53.952250 15745 raft_consensus.cc:359] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84b7681764f482aadad4c622874defe" member_type: VOTER }
I20260709 12:28:53.952859 15745 raft_consensus.cc:385] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:53.953137 15745 raft_consensus.cc:740] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f84b7681764f482aadad4c622874defe, State: Initialized, Role: FOLLOWER
I20260709 12:28:53.954106 15745 consensus_queue.cc:260] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84b7681764f482aadad4c622874defe" member_type: VOTER }
I20260709 12:28:53.954732 15745 raft_consensus.cc:399] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:28:53.955060 15745 raft_consensus.cc:493] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:28:53.955436 15745 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:53.965096 15745 raft_consensus.cc:515] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84b7681764f482aadad4c622874defe" member_type: VOTER }
I20260709 12:28:53.966246 15745 leader_election.cc:304] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f84b7681764f482aadad4c622874defe; no voters: 
I20260709 12:28:53.968335 15745 leader_election.cc:290] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:28:53.968915 15748 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:53.971472 15748 raft_consensus.cc:697] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 1 LEADER]: Becoming Leader. State: Replica: f84b7681764f482aadad4c622874defe, State: Running, Role: LEADER
I20260709 12:28:53.972436 15748 consensus_queue.cc:237] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84b7681764f482aadad4c622874defe" member_type: VOTER }
I20260709 12:28:53.973157 15745 sys_catalog.cc:565] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:28:53.979912 15749 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f84b7681764f482aadad4c622874defe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84b7681764f482aadad4c622874defe" member_type: VOTER } }
I20260709 12:28:53.980649 15749 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:53.980715 15750 sys_catalog.cc:455] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [sys.catalog]: SysCatalogTable state changed. Reason: New leader f84b7681764f482aadad4c622874defe. Latest consensus state: current_term: 1 leader_uuid: "f84b7681764f482aadad4c622874defe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f84b7681764f482aadad4c622874defe" member_type: VOTER } }
I20260709 12:28:53.981856 15750 sys_catalog.cc:458] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [sys.catalog]: This master's current role is: LEADER
I20260709 12:28:53.983177 15754 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:28:53.989902 15754 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:28:53.997648   561 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:28:54.003662 15754 catalog_manager.cc:1383] Generated new cluster ID: ae2a5400769d409183598fbf5872c83b
I20260709 12:28:54.004155 15754 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:28:54.031997 15754 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:28:54.033412 15754 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:28:54.048539 15754 catalog_manager.cc:6070] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe: Generated new TSK 0
I20260709 12:28:54.049309 15754 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:28:54.070545   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:54.078410 15766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:54.080495 15767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:54.083498   561 server_base.cc:1061] running on GCE node
W20260709 12:28:54.084707 15769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:54.085858   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:54.086077   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:54.086261   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600134086245 us; error 0 us; skew 500 ppm
I20260709 12:28:54.086933   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:54.089885   561 webserver.cc:533] Webserver started at http://127.0.140.65:45795/ using document root <none> and password file <none>
I20260709 12:28:54.090579   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:54.091017   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:54.091387   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:54.092949   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-0-root/instance:
uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
format_stamp: "Formatted at 2026-07-09 12:28:54 on dist-test-slave-7mcn"
I20260709 12:28:54.098570   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:28:54.103001 15774 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:54.104126   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20260709 12:28:54.104547   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-0-root
uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
format_stamp: "Formatted at 2026-07-09 12:28:54 on dist-test-slave-7mcn"
I20260709 12:28:54.104912   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:54.124459   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:54.125944   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:54.127760   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:54.130502   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:54.130756   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:54.131101   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:54.131309   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:54.186009   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.65:43921
I20260709 12:28:54.186184 15836 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.65:43921 every 8 connection(s)
I20260709 12:28:54.192049   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:54.201618 15841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:54.203675 15842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:54.208233 15844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:54.208290   561 server_base.cc:1061] running on GCE node
I20260709 12:28:54.210440   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:54.210772   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:54.211059   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600134211031 us; error 0 us; skew 500 ppm
I20260709 12:28:54.211855   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:54.215122   561 webserver.cc:533] Webserver started at http://127.0.140.66:44787/ using document root <none> and password file <none>
I20260709 12:28:54.215183 15837 heartbeater.cc:344] Connected to a master server at 127.0.140.126:37015
I20260709 12:28:54.215757 15837 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:54.215770   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:54.216212   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:54.216528   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:54.216742 15837 heartbeater.cc:507] Master 127.0.140.126:37015 requested a full tablet report, sending...
I20260709 12:28:54.218307   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-1-root/instance:
uuid: "9b36129f73bf485eb51ccf20370db33a"
format_stamp: "Formatted at 2026-07-09 12:28:54 on dist-test-slave-7mcn"
I20260709 12:28:54.219321 15710 ts_manager.cc:194] Registered new tserver with Master: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921)
I20260709 12:28:54.221289 15710 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50446
I20260709 12:28:54.224154   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:28:54.228776 15849 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:54.229925   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:28:54.230436   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-1-root
uuid: "9b36129f73bf485eb51ccf20370db33a"
format_stamp: "Formatted at 2026-07-09 12:28:54 on dist-test-slave-7mcn"
I20260709 12:28:54.230839   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:54.243767   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:54.245263   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:54.247148   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:54.249819   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:54.250074   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:54.250370   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:54.250560   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:54.303937   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.66:35631
I20260709 12:28:54.304126 15911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.66:35631 every 8 connection(s)
I20260709 12:28:54.310442   561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:28:54.320616 15915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:54.324381 15916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:28:54.325114 15918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:28:54.326437   561 server_base.cc:1061] running on GCE node
I20260709 12:28:54.327904   561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:28:54.328194   561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:28:54.328451   561 hybrid_clock.cc:648] HybridClock initialized: now 1783600134328409 us; error 0 us; skew 500 ppm
I20260709 12:28:54.329279 15912 heartbeater.cc:344] Connected to a master server at 127.0.140.126:37015
I20260709 12:28:54.329348   561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:28:54.329876 15912 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:54.330940 15912 heartbeater.cc:507] Master 127.0.140.126:37015 requested a full tablet report, sending...
I20260709 12:28:54.333114   561 webserver.cc:533] Webserver started at http://127.0.140.67:38645/ using document root <none> and password file <none>
I20260709 12:28:54.333389 15710 ts_manager.cc:194] Registered new tserver with Master: 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:54.333858   561 fs_manager.cc:362] Metadata directory not provided
I20260709 12:28:54.334163   561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:28:54.334465   561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:28:54.335222 15710 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50448
I20260709 12:28:54.336220   561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-2-root/instance:
uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
format_stamp: "Formatted at 2026-07-09 12:28:54 on dist-test-slave-7mcn"
I20260709 12:28:54.342142   561 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.001s
I20260709 12:28:54.346441 15923 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:54.347512   561 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:28:54.347857   561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-2-root
uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
format_stamp: "Formatted at 2026-07-09 12:28:54 on dist-test-slave-7mcn"
I20260709 12:28:54.348153   561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskTyHFjq/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599624140507-561-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:28:54.361459   561 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:28:54.362936   561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:28:54.364732   561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:28:54.367324   561 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:28:54.367559   561 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:54.367830   561 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:28:54.368018   561 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:28:54.421558   561 rpc_server.cc:307] RPC server started. Bound to: 127.0.140.67:35041
I20260709 12:28:54.421674 15985 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.140.67:35041 every 8 connection(s)
I20260709 12:28:54.439996 15986 heartbeater.cc:344] Connected to a master server at 127.0.140.126:37015
I20260709 12:28:54.440517 15986 heartbeater.cc:461] Registering TS with master...
I20260709 12:28:54.441505 15986 heartbeater.cc:507] Master 127.0.140.126:37015 requested a full tablet report, sending...
I20260709 12:28:54.444188 15710 ts_manager.cc:194] Registered new tserver with Master: d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:54.444842   561 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018986236s
I20260709 12:28:54.445935 15710 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:50454
I20260709 12:28:54.477506 15710 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50456:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:28:54.480957 15710 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:28:54.572115 15950 tablet_service.cc:1511] Processing CreateTablet for tablet f67bf24fa1b54ae2bf3ef7afe02cbd96 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:54.573364 15947 tablet_service.cc:1511] Processing CreateTablet for tablet 41db3c9b613f4023b493ebaa3343faf5 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:54.574036 15950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f67bf24fa1b54ae2bf3ef7afe02cbd96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.575021 15947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db3c9b613f4023b493ebaa3343faf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.574950 15948 tablet_service.cc:1511] Processing CreateTablet for tablet a95ea2d71cdf4e7c989ceea499c747aa (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:54.577211 15948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95ea2d71cdf4e7c989ceea499c747aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.576908 15949 tablet_service.cc:1511] Processing CreateTablet for tablet dc9602a9b0514fe7abc499687775488f (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:54.579559 15951 tablet_service.cc:1511] Processing CreateTablet for tablet 2e18dfee472b4c60b724b43004e356e9 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:54.584595 15949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc9602a9b0514fe7abc499687775488f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.591293 15951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e18dfee472b4c60b724b43004e356e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.592813 15876 tablet_service.cc:1511] Processing CreateTablet for tablet f67bf24fa1b54ae2bf3ef7afe02cbd96 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:54.591362 15877 tablet_service.cc:1511] Processing CreateTablet for tablet 2e18dfee472b4c60b724b43004e356e9 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:54.593159 15875 tablet_service.cc:1511] Processing CreateTablet for tablet dc9602a9b0514fe7abc499687775488f (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:54.594621 15877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e18dfee472b4c60b724b43004e356e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.595346 15874 tablet_service.cc:1511] Processing CreateTablet for tablet a95ea2d71cdf4e7c989ceea499c747aa (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:54.597038 15874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95ea2d71cdf4e7c989ceea499c747aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.600258 15873 tablet_service.cc:1511] Processing CreateTablet for tablet 41db3c9b613f4023b493ebaa3343faf5 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:54.606498 15802 tablet_service.cc:1511] Processing CreateTablet for tablet 2e18dfee472b4c60b724b43004e356e9 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:28:54.608438 15802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e18dfee472b4c60b724b43004e356e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.601683 15875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc9602a9b0514fe7abc499687775488f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.611469 15876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f67bf24fa1b54ae2bf3ef7afe02cbd96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.615306 15873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db3c9b613f4023b493ebaa3343faf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.620515 15872 tablet_service.cc:1511] Processing CreateTablet for tablet 7bbd23a4b0424576ba4cc5df1d5a7aea (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:54.622932 15872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bbd23a4b0424576ba4cc5df1d5a7aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.631215 15801 tablet_service.cc:1511] Processing CreateTablet for tablet f67bf24fa1b54ae2bf3ef7afe02cbd96 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:28:54.634972 15800 tablet_service.cc:1511] Processing CreateTablet for tablet dc9602a9b0514fe7abc499687775488f (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:28:54.633204 15801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f67bf24fa1b54ae2bf3ef7afe02cbd96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.641660 15798 tablet_service.cc:1511] Processing CreateTablet for tablet 41db3c9b613f4023b493ebaa3343faf5 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:28:54.638067 15799 tablet_service.cc:1511] Processing CreateTablet for tablet a95ea2d71cdf4e7c989ceea499c747aa (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:28:54.646602 15800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc9602a9b0514fe7abc499687775488f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.653955 15948 tablet_service.cc:1511] Processing CreateTablet for tablet 7bbd23a4b0424576ba4cc5df1d5a7aea (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:54.648659 15797 tablet_service.cc:1511] Processing CreateTablet for tablet 7bbd23a4b0424576ba4cc5df1d5a7aea (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:28:54.655283 15799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95ea2d71cdf4e7c989ceea499c747aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.655699 15948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bbd23a4b0424576ba4cc5df1d5a7aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.656412 15797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bbd23a4b0424576ba4cc5df1d5a7aea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.657514 16001 tablet_bootstrap.cc:492] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:28:54.656018 15796 tablet_service.cc:1511] Processing CreateTablet for tablet 5e42ffbebd604354925aebaa0fe7ef82 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:54.659250 15798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41db3c9b613f4023b493ebaa3343faf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.660037 15796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e42ffbebd604354925aebaa0fe7ef82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.672997 15871 tablet_service.cc:1511] Processing CreateTablet for tablet 5e42ffbebd604354925aebaa0fe7ef82 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:54.675040 15871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e42ffbebd604354925aebaa0fe7ef82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.684602 15946 tablet_service.cc:1511] Processing CreateTablet for tablet 5e42ffbebd604354925aebaa0fe7ef82 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:28:54.686295 15946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e42ffbebd604354925aebaa0fe7ef82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.691541 15795 tablet_service.cc:1511] Processing CreateTablet for tablet 22fa1d49fd24457395605b0c9943dc4a (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:54.693452 15795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fa1d49fd24457395605b0c9943dc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.705040 16001 tablet_bootstrap.cc:654] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:54.721316 15876 tablet_service.cc:1511] Processing CreateTablet for tablet 22fa1d49fd24457395605b0c9943dc4a (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:54.723035 15876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fa1d49fd24457395605b0c9943dc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.724979 15948 tablet_service.cc:1511] Processing CreateTablet for tablet 22fa1d49fd24457395605b0c9943dc4a (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:28:54.727176 15948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22fa1d49fd24457395605b0c9943dc4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.743710 16002 tablet_bootstrap.cc:492] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:28:54.751197 15794 tablet_service.cc:1511] Processing CreateTablet for tablet 869265d9a2fa4ae6b4c5b58c1c4cc1f5 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:54.753206 15794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 869265d9a2fa4ae6b4c5b58c1c4cc1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.770509 15950 tablet_service.cc:1511] Processing CreateTablet for tablet 869265d9a2fa4ae6b4c5b58c1c4cc1f5 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:54.772183 15950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 869265d9a2fa4ae6b4c5b58c1c4cc1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.783601 16002 tablet_bootstrap.cc:654] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:54.777890 15874 tablet_service.cc:1511] Processing CreateTablet for tablet 869265d9a2fa4ae6b4c5b58c1c4cc1f5 (DEFAULT_TABLE table=test-workload [id=49828ab8bf7842d4b305f20899afe769]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:28:54.792694 15874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 869265d9a2fa4ae6b4c5b58c1c4cc1f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:28:54.803776 16001 tablet_bootstrap.cc:492] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:28:54.807035 16001 ts_tablet_manager.cc:1403] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.150s	user 0.012s	sys 0.007s
I20260709 12:28:54.820642 16004 tablet_bootstrap.cc:492] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:28:54.817107 16001 raft_consensus.cc:359] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.822569 16001 raft_consensus.cc:385] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:54.822964 16001 raft_consensus.cc:740] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:28:54.830058 16004 tablet_bootstrap.cc:654] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:54.839497 16004 tablet_bootstrap.cc:492] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:28:54.839205 16001 consensus_queue.cc:260] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.840253 16004 ts_tablet_manager.cc:1403] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.020s	user 0.003s	sys 0.016s
I20260709 12:28:54.842492 16002 tablet_bootstrap.cc:492] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:28:54.843179 16002 ts_tablet_manager.cc:1403] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.100s	user 0.004s	sys 0.012s
I20260709 12:28:54.843928 16004 raft_consensus.cc:359] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.845957 15986 heartbeater.cc:499] Master 127.0.140.126:37015 was elected leader, sending a full tablet report...
I20260709 12:28:54.845952 16004 raft_consensus.cc:385] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:54.846776 16004 raft_consensus.cc:740] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:28:54.847906 16004 consensus_queue.cc:260] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.848769 16001 ts_tablet_manager.cc:1434] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.041s	user 0.007s	sys 0.001s
I20260709 12:28:54.853719 16001 tablet_bootstrap.cc:492] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:28:54.853528 16002 raft_consensus.cc:359] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.854547 16002 raft_consensus.cc:385] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:54.854952 16002 raft_consensus.cc:740] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:28:54.855800 16002 consensus_queue.cc:260] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.864959 15837 heartbeater.cc:499] Master 127.0.140.126:37015 was elected leader, sending a full tablet report...
I20260709 12:28:54.866176 16004 ts_tablet_manager.cc:1434] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.026s	user 0.014s	sys 0.000s
I20260709 12:28:54.867676 16004 tablet_bootstrap.cc:492] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:28:54.876340 16001 tablet_bootstrap.cc:654] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:54.879944 16004 tablet_bootstrap.cc:654] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:54.881525 15912 heartbeater.cc:499] Master 127.0.140.126:37015 was elected leader, sending a full tablet report...
I20260709 12:28:54.881975 16002 ts_tablet_manager.cc:1434] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.038s	user 0.017s	sys 0.011s
I20260709 12:28:54.883199 16002 tablet_bootstrap.cc:492] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:28:54.881593 16009 raft_consensus.cc:493] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.884658 16009 raft_consensus.cc:515] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.890003 16002 tablet_bootstrap.cc:654] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:54.897611 16004 tablet_bootstrap.cc:492] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:28:54.897634 16001 tablet_bootstrap.cc:492] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:28:54.898370 16004 ts_tablet_manager.cc:1403] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.031s	user 0.013s	sys 0.012s
I20260709 12:28:54.898897 16001 ts_tablet_manager.cc:1403] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.045s	user 0.015s	sys 0.005s
I20260709 12:28:54.902493 16004 raft_consensus.cc:359] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.903604 16004 raft_consensus.cc:385] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:54.904062 16004 raft_consensus.cc:740] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:28:54.905045 16004 consensus_queue.cc:260] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.903707 16001 raft_consensus.cc:359] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:54.906443 16001 raft_consensus.cc:385] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:54.908110 16004 ts_tablet_manager.cc:1434] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:28:54.913745 16004 tablet_bootstrap.cc:492] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:28:54.906754 16001 raft_consensus.cc:740] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:28:54.915311 16001 consensus_queue.cc:260] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:54.920256 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:54.921857 16004 tablet_bootstrap.cc:654] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
W20260709 12:28:54.922581 15851 leader_election.cc:343] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:54.925227 16009 leader_election.cc:290] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:54.926352 16001 ts_tablet_manager.cc:1434] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.027s	user 0.000s	sys 0.007s
W20260709 12:28:54.928169 15987 tablet.cc:2406] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:28:54.928516 16007 raft_consensus.cc:493] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.929248 16007 raft_consensus.cc:515] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.932552 16001 tablet_bootstrap.cc:492] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:28:54.935058 16007 leader_election.cc:290] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:54.932178 16002 tablet_bootstrap.cc:492] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:28:54.941570 16002 ts_tablet_manager.cc:1403] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.059s	user 0.026s	sys 0.021s
I20260709 12:28:54.943430 16008 raft_consensus.cc:493] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:54.944135 16008 raft_consensus.cc:515] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.945348 16002 raft_consensus.cc:359] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.946411 16002 raft_consensus.cc:385] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:54.946750 16002 raft_consensus.cc:740] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:28:54.947865 16002 consensus_queue.cc:260] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:54.950351 16002 ts_tablet_manager.cc:1434] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:28:54.952242 16002 tablet_bootstrap.cc:492] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:28:54.953573 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:28:54.955312 15887 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:28:54.956804 16008 leader_election.cc:290] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:54.957772 15927 leader_election.cc:304] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:54.959342 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:54.960459 15961 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 0.
I20260709 12:28:54.961782 16002 tablet_bootstrap.cc:654] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:54.962402 15853 leader_election.cc:304] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:28:54.963985 16009 raft_consensus.cc:2804] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:54.964378 16009 raft_consensus.cc:493] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:54.962482 16007 raft_consensus.cc:2804] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:54.965395 16009 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:54.978621 16001 tablet_bootstrap.cc:654] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:54.965444 16007 raft_consensus.cc:493] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:54.985023 16007 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:54.986368 16004 tablet_bootstrap.cc:492] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:28:54.987172 16004 ts_tablet_manager.cc:1403] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.074s	user 0.021s	sys 0.031s
W20260709 12:28:54.991853 15758 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:54.992956 16004 raft_consensus.cc:359] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:54.993916 16004 raft_consensus.cc:385] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:54.994196 16004 raft_consensus.cc:740] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:28:54.995290 16004 consensus_queue.cc:260] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:54.998229 16004 ts_tablet_manager.cc:1434] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.003s
I20260709 12:28:54.999567 16004 tablet_bootstrap.cc:492] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:28:55.004382 16007 raft_consensus.cc:515] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.007326 16004 tablet_bootstrap.cc:654] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.017179 16001 tablet_bootstrap.cc:492] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:28:55.018138 16001 ts_tablet_manager.cc:1403] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.090s	user 0.020s	sys 0.000s
I20260709 12:28:55.021780 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:55.027094 16009 raft_consensus.cc:515] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.031626 16007 leader_election.cc:290] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:55.035846 16001 raft_consensus.cc:359] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.037153 16001 raft_consensus.cc:385] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.037654 16001 raft_consensus.cc:740] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.040048 16008 raft_consensus.cc:493] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.040761 16008 raft_consensus.cc:515] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.043268 16008 leader_election.cc:290] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.043431 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:28:55.044279 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.042483 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:55.039072 16001 consensus_queue.cc:260] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.049003 16001 ts_tablet_manager.cc:1434] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.030s	user 0.006s	sys 0.002s
I20260709 12:28:55.049563 16004 tablet_bootstrap.cc:492] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:28:55.050253 16001 tablet_bootstrap.cc:492] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:28:55.057919 16004 ts_tablet_manager.cc:1403] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.059s	user 0.022s	sys 0.020s
I20260709 12:28:55.058974 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
W20260709 12:28:55.052008 15778 leader_election.cc:343] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.052887 15887 raft_consensus.cc:2393] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d5923b92f8db4416a3c43a90bc1cdd51 in current term 1: Already voted for candidate 9b36129f73bf485eb51ccf20370db33a in this term.
I20260709 12:28:55.062563 16009 leader_election.cc:290] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.063472 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.065183 15959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e42ffbebd604354925aebaa0fe7ef82" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
W20260709 12:28:55.067186 15778 leader_election.cc:343] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.067516 16008 raft_consensus.cc:493] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.067946 16001 tablet_bootstrap.cc:654] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
W20260709 12:28:55.067947 15778 leader_election.cc:343] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.058161 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:55.068701 15778 leader_election.cc:304] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:28:55.069154 15961 raft_consensus.cc:2393] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9b36129f73bf485eb51ccf20370db33a in current term 1: Already voted for candidate d5923b92f8db4416a3c43a90bc1cdd51 in this term.
W20260709 12:28:55.051811 15925 leader_election.cc:343] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:28:55.071138 15925 leader_election.cc:343] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.074163 16007 raft_consensus.cc:493] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.074702 16007 raft_consensus.cc:515] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.075711 16004 raft_consensus.cc:359] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.076494 16004 raft_consensus.cc:385] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.076794 16004 raft_consensus.cc:740] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.077841 16004 consensus_queue.cc:260] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.081614 16002 tablet_bootstrap.cc:492] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:28:55.082492 16002 ts_tablet_manager.cc:1403] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.131s	user 0.040s	sys 0.065s
I20260709 12:28:55.083065 16007 leader_election.cc:290] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:55.068125 16008 raft_consensus.cc:515] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:55.052390 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e42ffbebd604354925aebaa0fe7ef82" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
W20260709 12:28:55.073522 15851 leader_election.cc:343] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.090361 16008 leader_election.cc:290] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:55.091015 16008 raft_consensus.cc:2749] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.091943 15851 leader_election.cc:304] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:28:55.094676 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e18dfee472b4c60b724b43004e356e9" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.095726 16009 raft_consensus.cc:2749] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.096468 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.096881 15927 leader_election.cc:304] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:28:55.094956 16004 ts_tablet_manager.cc:1434] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.036s	user 0.017s	sys 0.003s
I20260709 12:28:55.097724 16002 raft_consensus.cc:359] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.095677 15961 raft_consensus.cc:2468] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 0.
I20260709 12:28:55.098428 16004 tablet_bootstrap.cc:492] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:28:55.098670 16002 raft_consensus.cc:385] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.099254 16002 raft_consensus.cc:740] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.098479 16007 raft_consensus.cc:2749] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.100111 16002 consensus_queue.cc:260] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.102455 16002 ts_tablet_manager.cc:1434] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
W20260709 12:28:55.105475 15925 leader_election.cc:343] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.105893 16001 tablet_bootstrap.cc:492] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:28:55.106429 16001 ts_tablet_manager.cc:1403] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.056s	user 0.011s	sys 0.016s
I20260709 12:28:55.106524 16004 tablet_bootstrap.cc:654] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.107447 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e18dfee472b4c60b724b43004e356e9" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:28:55.108532 16002 tablet_bootstrap.cc:492] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:28:55.107508 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
W20260709 12:28:55.110145 15927 leader_election.cc:343] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.110610 15927 leader_election.cc:304] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:28:55.111789 16007 raft_consensus.cc:2749] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.112668 15778 leader_election.cc:304] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
W20260709 12:28:55.113629 15778 leader_election.cc:343] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.113718 16024 raft_consensus.cc:2804] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:55.114112 15778 leader_election.cc:304] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:28:55.114439 16024 raft_consensus.cc:493] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.114975 16024 raft_consensus.cc:3060] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.115182 16022 raft_consensus.cc:2749] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.118646 16002 tablet_bootstrap.cc:654] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.118337 16001 raft_consensus.cc:359] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.119268 16001 raft_consensus.cc:385] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.119604 16001 raft_consensus.cc:740] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
W20260709 12:28:55.120325 15778 leader_election.cc:343] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.121317 16001 consensus_queue.cc:260] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.122076 16022 raft_consensus.cc:493] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.122596 16022 raft_consensus.cc:515] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.125746 16001 ts_tablet_manager.cc:1434] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:28:55.127050 16001 tablet_bootstrap.cc:492] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:28:55.126524 16024 raft_consensus.cc:515] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:55.129781 16024 leader_election.cc:290] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Requested vote from peers d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:55.133715 16004 tablet_bootstrap.cc:492] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:28:55.134405 16004 ts_tablet_manager.cc:1403] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.036s	user 0.009s	sys 0.007s
I20260709 12:28:55.138657 16001 tablet_bootstrap.cc:654] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.138386 16004 raft_consensus.cc:359] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.139777 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:28:55.140748 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.141286 16004 raft_consensus.cc:385] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.141765 16004 raft_consensus.cc:740] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.143091 16004 consensus_queue.cc:260] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.144093 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e18dfee472b4c60b724b43004e356e9" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:55.144799 15961 raft_consensus.cc:3060] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.146234 16004 ts_tablet_manager.cc:1434] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:28:55.147578 16004 tablet_bootstrap.cc:492] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:28:55.155346 16004 tablet_bootstrap.cc:654] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.156474 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e18dfee472b4c60b724b43004e356e9" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:55.158598 16001 tablet_bootstrap.cc:492] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:28:55.159559 16001 ts_tablet_manager.cc:1403] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.033s	user 0.019s	sys 0.004s
W20260709 12:28:55.163759 15778 leader_election.cc:343] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:28:55.164938 15778 leader_election.cc:343] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.127527 16022 leader_election.cc:290] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.154578 15961 raft_consensus.cc:2468] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 1.
I20260709 12:28:55.169201 16001 raft_consensus.cc:359] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.170459 16001 raft_consensus.cc:385] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.170938 16001 raft_consensus.cc:740] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.172315 16001 consensus_queue.cc:260] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.165500 15778 leader_election.cc:304] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:28:55.175046 16022 raft_consensus.cc:2749] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.175849 16001 ts_tablet_manager.cc:1434] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.016s	user 0.011s	sys 0.000s
W20260709 12:28:55.176648 15778 leader_election.cc:343] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.177712 16001 tablet_bootstrap.cc:492] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:28:55.184811 16002 tablet_bootstrap.cc:492] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:28:55.185626 16002 ts_tablet_manager.cc:1403] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.077s	user 0.035s	sys 0.038s
I20260709 12:28:55.187146 16001 tablet_bootstrap.cc:654] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.189682 16002 raft_consensus.cc:359] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.177654 15778 leader_election.cc:304] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a
I20260709 12:28:55.190690 16002 raft_consensus.cc:385] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.191159 16002 raft_consensus.cc:740] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.191329 16022 raft_consensus.cc:2804] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:55.192075 16022 raft_consensus.cc:697] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:28:55.192606 16002 consensus_queue.cc:260] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.192958 16022 consensus_queue.cc:237] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:55.197237 16004 tablet_bootstrap.cc:492] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:28:55.197906 16004 ts_tablet_manager.cc:1403] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.051s	user 0.017s	sys 0.009s
I20260709 12:28:55.200594 16004 raft_consensus.cc:359] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.201191 16004 raft_consensus.cc:385] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.201444 16004 raft_consensus.cc:740] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.202276 16004 consensus_queue.cc:260] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.205075 16004 ts_tablet_manager.cc:1434] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:28:55.206252 16004 tablet_bootstrap.cc:492] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:28:55.210146 16002 ts_tablet_manager.cc:1434] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:28:55.211450 16002 tablet_bootstrap.cc:492] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:28:55.212839 15707 catalog_manager.cc:5697] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 1 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:55.215579 16004 tablet_bootstrap.cc:654] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.218412 16002 tablet_bootstrap.cc:654] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.222262 16001 tablet_bootstrap.cc:492] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:28:55.222934 16001 ts_tablet_manager.cc:1403] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.046s	user 0.027s	sys 0.016s
I20260709 12:28:55.226405 16001 raft_consensus.cc:359] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.227275 16001 raft_consensus.cc:385] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.227587 16001 raft_consensus.cc:740] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.228473 16001 consensus_queue.cc:260] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.235385 16001 ts_tablet_manager.cc:1434] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.004s
I20260709 12:28:55.236963 16001 tablet_bootstrap.cc:492] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:28:55.237843 16002 tablet_bootstrap.cc:492] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:28:55.238479 16002 ts_tablet_manager.cc:1403] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.027s	user 0.006s	sys 0.016s
I20260709 12:28:55.241997 16002 raft_consensus.cc:359] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.242841 16002 raft_consensus.cc:385] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.243121 16002 raft_consensus.cc:740] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.244098 16001 tablet_bootstrap.cc:654] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.243916 16002 consensus_queue.cc:260] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.246570 16002 ts_tablet_manager.cc:1434] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.005s
I20260709 12:28:55.248088 16002 tablet_bootstrap.cc:492] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:28:55.254094 16004 tablet_bootstrap.cc:492] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:28:55.254829 16004 ts_tablet_manager.cc:1403] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.049s	user 0.017s	sys 0.007s
I20260709 12:28:55.257504 16002 tablet_bootstrap.cc:654] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.258715 16004 raft_consensus.cc:359] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.259675 16004 raft_consensus.cc:385] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.260004 16004 raft_consensus.cc:740] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.272605 16004 consensus_queue.cc:260] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.275745 16004 ts_tablet_manager.cc:1434] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.021s	user 0.008s	sys 0.001s
I20260709 12:28:55.276924 16004 tablet_bootstrap.cc:492] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:28:55.286744 16004 tablet_bootstrap.cc:654] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.287216 16001 tablet_bootstrap.cc:492] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:28:55.287410 16002 tablet_bootstrap.cc:492] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:28:55.288031 16001 ts_tablet_manager.cc:1403] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.051s	user 0.034s	sys 0.015s
I20260709 12:28:55.288117 16002 ts_tablet_manager.cc:1403] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.040s	user 0.028s	sys 0.010s
I20260709 12:28:55.291739 16002 raft_consensus.cc:359] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.292004 16001 raft_consensus.cc:359] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.292969 16002 raft_consensus.cc:385] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.293308 16001 raft_consensus.cc:385] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.293396 16004 tablet_bootstrap.cc:492] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:28:55.293486 16002 raft_consensus.cc:740] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.294265 16001 raft_consensus.cc:740] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.294543 16004 ts_tablet_manager.cc:1403] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.018s	user 0.006s	sys 0.009s
I20260709 12:28:55.294934 16002 consensus_queue.cc:260] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.295418 16001 consensus_queue.cc:260] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.297708 16002 ts_tablet_manager.cc:1434] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 12:28:55.298614 16001 ts_tablet_manager.cc:1434] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.001s
I20260709 12:28:55.298942 16002 tablet_bootstrap.cc:492] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:28:55.299973 16001 tablet_bootstrap.cc:492] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:28:55.300508 16004 raft_consensus.cc:359] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.301395 16004 raft_consensus.cc:385] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.301703 16004 raft_consensus.cc:740] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.302573 16004 consensus_queue.cc:260] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.305015 16004 ts_tablet_manager.cc:1434] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:28:55.305830 16007 raft_consensus.cc:493] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.306459 16007 raft_consensus.cc:515] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.307025 16002 tablet_bootstrap.cc:654] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.308868 16007 leader_election.cc:290] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:55.309453 16004 tablet_bootstrap.cc:492] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:28:55.315326 16009 raft_consensus.cc:493] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.316246 16009 raft_consensus.cc:515] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.317169 16001 tablet_bootstrap.cc:654] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.317456 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:28:55.318358 15887 raft_consensus.cc:2468] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:28:55.319911 16009 leader_election.cc:290] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.320636 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e42ffbebd604354925aebaa0fe7ef82" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.320926 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.321429 15961 raft_consensus.cc:2468] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 0.
I20260709 12:28:55.321916 15811 raft_consensus.cc:2468] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:28:55.322937 15853 leader_election.cc:304] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:55.323410 16004 tablet_bootstrap.cc:654] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.324004 16009 raft_consensus.cc:2804] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:55.324467 16009 raft_consensus.cc:493] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.324826 16009 raft_consensus.cc:3060] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.325544 15927 leader_election.cc:304] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:55.327409 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e42ffbebd604354925aebaa0fe7ef82" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.328446 15812 raft_consensus.cc:2468] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 0.
I20260709 12:28:55.329697 16007 raft_consensus.cc:2804] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:55.330312 16007 raft_consensus.cc:493] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.330757 16007 raft_consensus.cc:3060] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.357380 16009 raft_consensus.cc:515] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.361835 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e42ffbebd604354925aebaa0fe7ef82" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:55.361894 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e42ffbebd604354925aebaa0fe7ef82" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:55.363818 15961 raft_consensus.cc:3060] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.365403 16009 leader_election.cc:290] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.366897 15811 raft_consensus.cc:3060] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.397704 16007 raft_consensus.cc:515] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.405079 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:55.406059 15812 raft_consensus.cc:3060] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.417888 15811 raft_consensus.cc:2468] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 1.
I20260709 12:28:55.400305 15961 raft_consensus.cc:2468] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 1.
I20260709 12:28:55.421028 15853 leader_election.cc:304] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:55.422351 16009 raft_consensus.cc:2804] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:55.427248 16009 raft_consensus.cc:697] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:55.428458 16009 consensus_queue.cc:237] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.428591 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:55.429680 15887 raft_consensus.cc:3060] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.431732 16007 leader_election.cc:290] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:55.434508 16001 tablet_bootstrap.cc:492] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:28:55.435547 16001 ts_tablet_manager.cc:1403] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.136s	user 0.047s	sys 0.075s
I20260709 12:28:55.435621 15812 raft_consensus.cc:2468] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:28:55.437068 15925 leader_election.cc:304] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:55.438022 16004 tablet_bootstrap.cc:492] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:28:55.438148 16007 raft_consensus.cc:2804] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:55.438819 16004 ts_tablet_manager.cc:1403] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.130s	user 0.036s	sys 0.076s
I20260709 12:28:55.439494 16022 raft_consensus.cc:493] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.439327 16001 raft_consensus.cc:359] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.440194 16022 raft_consensus.cc:515] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.440500 16001 raft_consensus.cc:385] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.441023 16001 raft_consensus.cc:740] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.441883 16001 consensus_queue.cc:260] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.441923 16004 raft_consensus.cc:359] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.443037 16004 raft_consensus.cc:385] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.443360 16004 raft_consensus.cc:740] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.444106 16004 consensus_queue.cc:260] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.445842 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:28:55.446659 16042 raft_consensus.cc:493] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.447278 16042 raft_consensus.cc:515] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
W20260709 12:28:55.448122 15778 leader_election.cc:343] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.439235 15887 raft_consensus.cc:2468] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:28:55.448689 16002 tablet_bootstrap.cc:492] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:28:55.450235 16002 ts_tablet_manager.cc:1403] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.152s	user 0.028s	sys 0.056s
I20260709 12:28:55.454047 16042 leader_election.cc:290] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.446134 16004 ts_tablet_manager.cc:1434] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:28:55.458328 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.459501 15960 raft_consensus.cc:2468] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 0.
I20260709 12:28:55.454279 16002 raft_consensus.cc:359] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:55.460340 16002 raft_consensus.cc:385] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.460700 16002 raft_consensus.cc:740] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.461498 16002 consensus_queue.cc:260] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:55.461551 15853 leader_election.cc:304] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:55.462832 15708 catalog_manager.cc:5697] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 1 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:55.464449 16002 ts_tablet_manager.cc:1434] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:28:55.465598 16002 tablet_bootstrap.cc:492] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:28:55.463613 16009 raft_consensus.cc:2804] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:55.466367 16024 raft_consensus.cc:493] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.466873 16009 raft_consensus.cc:493] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.467322 16009 raft_consensus.cc:3060] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.467084 16024 raft_consensus.cc:515] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.470178 16024 leader_election.cc:290] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.442574 16022 leader_election.cc:290] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.472393 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.473147 15961 raft_consensus.cc:2468] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 0.
I20260709 12:28:55.481860 15778 leader_election.cc:304] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a
I20260709 12:28:55.483266 16022 raft_consensus.cc:2804] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:55.483862 16022 raft_consensus.cc:493] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.484259 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:28:55.484333 16022 raft_consensus.cc:3060] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.485181 15887 raft_consensus.cc:2468] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 0.
I20260709 12:28:55.485531 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.486560 15812 raft_consensus.cc:2468] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 0.
I20260709 12:28:55.487810 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.488242 15778 leader_election.cc:304] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:28:55.488914 15961 raft_consensus.cc:2468] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 0.
I20260709 12:28:55.489272 16024 raft_consensus.cc:2804] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:55.489816 16024 raft_consensus.cc:493] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.490177 16024 raft_consensus.cc:3060] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.493153 16022 raft_consensus.cc:515] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.494709 16007 raft_consensus.cc:697] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:28:55.495752 16022 leader_election.cc:290] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.498140 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:55.499091 15961 raft_consensus.cc:3060] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.499732 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:55.496191 16007 consensus_queue.cc:237] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
W20260709 12:28:55.505770 15778 leader_election.cc:343] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:28:55.506588 16002 tablet_bootstrap.cc:654] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.508492 16009 raft_consensus.cc:515] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.509262 16024 raft_consensus.cc:515] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.511703 16024 leader_election.cc:290] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.513310 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:55.514175 15887 raft_consensus.cc:3060] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.518965 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:55.520041 15960 raft_consensus.cc:3060] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.520536 16042 raft_consensus.cc:493] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.521350 16042 raft_consensus.cc:515] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } }
I20260709 12:28:55.526311 15959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e18dfee472b4c60b724b43004e356e9" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.527582 15959 raft_consensus.cc:2393] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b36129f73bf485eb51ccf20370db33a in current term 1: Already voted for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in this term.
I20260709 12:28:55.528612 16042 leader_election.cc:290] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.532160 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e18dfee472b4c60b724b43004e356e9" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.532905 16001 ts_tablet_manager.cc:1434] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.097s	user 0.029s	sys 0.052s
I20260709 12:28:55.534204 16001 tablet_bootstrap.cc:492] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:28:55.534363 15851 leader_election.cc:304] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:28:55.535945 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:55.536427 16042 raft_consensus.cc:3060] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.537313 16009 leader_election.cc:290] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.538077 16009 raft_consensus.cc:493] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.538604 16009 raft_consensus.cc:515] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.541131 16001 tablet_bootstrap.cc:654] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.542450 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.543612 15811 raft_consensus.cc:2468] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 0.
I20260709 12:28:55.545256 16044 raft_consensus.cc:493] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.545537 15959 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:55.546558 15959 raft_consensus.cc:3060] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.546110 16044 raft_consensus.cc:515] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.553131 15851 leader_election.cc:304] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:28:55.563652 15709 catalog_manager.cc:5697] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:55.536682 15812 raft_consensus.cc:3060] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.576821 15887 raft_consensus.cc:2468] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 1.
I20260709 12:28:55.580092 15778 leader_election.cc:304] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:28:55.581400 16024 raft_consensus.cc:2804] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:55.582010 16024 raft_consensus.cc:697] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:28:55.582885 16024 consensus_queue.cc:237] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.591934 15959 raft_consensus.cc:2468] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 1.
I20260709 12:28:55.595721 15958 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.602774 15958 raft_consensus.cc:2468] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 0.
I20260709 12:28:55.603204 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:28:55.596714 16009 leader_election.cc:290] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.596665 16002 tablet_bootstrap.cc:492] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:28:55.605875 16002 ts_tablet_manager.cc:1403] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.140s	user 0.025s	sys 0.035s
I20260709 12:28:55.607095 16044 leader_election.cc:290] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:55.610679 15961 raft_consensus.cc:2468] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 1.
I20260709 12:28:55.611233 16002 raft_consensus.cc:359] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.597890 15812 raft_consensus.cc:2468] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 1.
I20260709 12:28:55.599157 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.599558 16048 raft_consensus.cc:2804] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:55.600486 16042 raft_consensus.cc:2749] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.595711 15960 raft_consensus.cc:2468] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 1.
I20260709 12:28:55.614809 16001 tablet_bootstrap.cc:492] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:28:55.613780 15851 leader_election.cc:304] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:28:55.617010 16001 ts_tablet_manager.cc:1403] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.083s	user 0.038s	sys 0.039s
I20260709 12:28:55.617450 16042 raft_consensus.cc:2804] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:55.618562 16042 raft_consensus.cc:697] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:55.612675 15709 catalog_manager.cc:5697] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 1 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:55.619285 15811 raft_consensus.cc:2468] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:28:55.612596 16002 raft_consensus.cc:385] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.617482 15778 leader_election.cc:304] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a
I20260709 12:28:55.620191 16001 raft_consensus.cc:359] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.621048 16001 raft_consensus.cc:385] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.621140 16024 raft_consensus.cc:2804] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:55.621414 16001 raft_consensus.cc:740] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.622285 16024 raft_consensus.cc:697] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:28:55.623713 16024 consensus_queue.cc:237] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.625522 16042 consensus_queue.cc:237] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.625941 15925 leader_election.cc:304] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:55.616397 16048 raft_consensus.cc:493] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.627180 16044 raft_consensus.cc:2804] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:55.627646 16044 raft_consensus.cc:493] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.620193 16002 raft_consensus.cc:740] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.628080 16044 raft_consensus.cc:3060] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.622469 16001 consensus_queue.cc:260] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.631305 16048 raft_consensus.cc:3060] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.634248 16001 ts_tablet_manager.cc:1434] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:28:55.634119 16022 consensus_queue.cc:1048] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:55.635819 16002 consensus_queue.cc:260] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.638212 16002 ts_tablet_manager.cc:1434] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.032s	user 0.003s	sys 0.003s
I20260709 12:28:55.639447 16002 tablet_bootstrap.cc:492] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:28:55.639626 16022 raft_consensus.cc:493] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.640457 16022 raft_consensus.cc:515] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.643505 16022 leader_election.cc:290] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.645426 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:28:55.648022 15886 raft_consensus.cc:2393] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in current term 1: Already voted for candidate 9b36129f73bf485eb51ccf20370db33a in this term.
I20260709 12:28:55.648123 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.649328 15960 raft_consensus.cc:2393] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in current term 1: Already voted for candidate d5923b92f8db4416a3c43a90bc1cdd51 in this term.
I20260709 12:28:55.646836 16002 tablet_bootstrap.cc:654] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:28:55.651124 15778 leader_election.cc:304] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:28:55.652731 16022 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.660650 16044 raft_consensus.cc:515] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.671774 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:55.678944 16044 leader_election.cc:290] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:55.679853 16048 raft_consensus.cc:515] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.681607 15887 raft_consensus.cc:2393] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d5923b92f8db4416a3c43a90bc1cdd51 in current term 1: Already voted for candidate 9b36129f73bf485eb51ccf20370db33a in this term.
I20260709 12:28:55.683907 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:55.684964 15812 raft_consensus.cc:3060] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.685597 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:55.685992 16002 tablet_bootstrap.cc:492] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:28:55.686487 16002 ts_tablet_manager.cc:1403] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.047s	user 0.015s	sys 0.027s
I20260709 12:28:55.689462 16002 raft_consensus.cc:359] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.683897 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:55.686553 15961 raft_consensus.cc:2393] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9b36129f73bf485eb51ccf20370db33a in current term 1: Already voted for candidate d5923b92f8db4416a3c43a90bc1cdd51 in this term.
I20260709 12:28:55.690495 16048 leader_election.cc:290] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.690146 16002 raft_consensus.cc:385] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:28:55.691252 16002 raft_consensus.cc:740] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:28:55.692139 16002 consensus_queue.cc:260] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.692312 15851 leader_election.cc:304] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:28:55.693270 15812 raft_consensus.cc:2468] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:28:55.693989 16048 raft_consensus.cc:2749] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.694646 16002 ts_tablet_manager.cc:1434] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.002s
I20260709 12:28:55.694033 15706 catalog_manager.cc:5697] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 1 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:55.695911 15925 leader_election.cc:304] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a
I20260709 12:28:55.696236 15708 catalog_manager.cc:5697] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 1 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:55.697409 16044 raft_consensus.cc:2804] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:55.697958 16044 raft_consensus.cc:697] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:28:55.698931 16044 consensus_queue.cc:237] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.703347 16048 raft_consensus.cc:493] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.704079 16048 raft_consensus.cc:515] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.707885 16048 leader_election.cc:290] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.709460 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.709682 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.711014 16022 raft_consensus.cc:2749] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.711054 15961 raft_consensus.cc:2393] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b36129f73bf485eb51ccf20370db33a in current term 1: Already voted for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in this term.
I20260709 12:28:55.713892 15853 leader_election.cc:304] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:28:55.715270 16048 raft_consensus.cc:3060] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:55.723906 15709 catalog_manager.cc:5697] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:55.729280 16022 consensus_queue.cc:1048] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:55.732643 16048 raft_consensus.cc:2749] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:28:55.823968 16048 consensus_queue.cc:1048] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:55.837422 16009 consensus_queue.cc:1048] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:55.918903 16044 consensus_queue.cc:1048] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:55.939474 16045 consensus_queue.cc:1048] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:55.984967 16009 raft_consensus.cc:493] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:55.985809 16009 raft_consensus.cc:515] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:55.987890 16009 leader_election.cc:290] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:55.989009 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:55.989168 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:55.989903 15812 raft_consensus.cc:2468] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 0.
I20260709 12:28:55.990051 15961 raft_consensus.cc:2468] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 0.
I20260709 12:28:55.991647 15851 leader_election.cc:304] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:28:55.992596 16009 raft_consensus.cc:2804] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:28:55.993193 16009 raft_consensus.cc:493] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:55.993611 16009 raft_consensus.cc:3060] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:28:55.997931 15758 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:28:56.001101 16009 raft_consensus.cc:515] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:56.003314 16009 leader_election.cc:290] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:56.004489 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:56.004554 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:56.005465 15812 raft_consensus.cc:3060] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:56.005545 15961 raft_consensus.cc:3060] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:28:56.013684 15961 raft_consensus.cc:2468] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 1.
I20260709 12:28:56.013684 15812 raft_consensus.cc:2468] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 1.
I20260709 12:28:56.015298 15853 leader_election.cc:304] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:56.016196 16042 raft_consensus.cc:2804] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:28:56.016772 16042 raft_consensus.cc:697] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:56.017577 16042 consensus_queue.cc:237] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:56.019268 16009 consensus_queue.cc:1048] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:56.028301 15708 catalog_manager.cc:5697] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 1 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:56.029781 16055 consensus_queue.cc:1048] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:56.036499 16009 consensus_queue.cc:1048] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:56.053351 16024 consensus_queue.cc:1048] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:56.165148 16024 consensus_queue.cc:1048] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:56.178661 16022 consensus_queue.cc:1048] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:56.278861 16045 consensus_queue.cc:1048] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:56.292106 16044 consensus_queue.cc:1048] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:56.558290 16009 consensus_queue.cc:1048] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:56.570070 16048 consensus_queue.cc:1048] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:28:56.889204 16048 raft_consensus.cc:493] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:28:56.889832 16048 raft_consensus.cc:515] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:56.891875 16048 leader_election.cc:290] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:56.893112 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:28:56.893196 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:28:56.893887 15812 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 1.
I20260709 12:28:56.894168 15960 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 1.
I20260709 12:28:56.895334 15851 leader_election.cc:304] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:28:56.896219 16048 raft_consensus.cc:2804] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:28:56.896620 16048 raft_consensus.cc:493] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:28:56.897012 16048 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:56.902357 16048 raft_consensus.cc:515] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:56.904103 16048 leader_election.cc:290] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:56.905150 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:56.905794 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:56.905872 15812 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:56.906592 15960 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:56.914122 15960 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:28:56.914242 15812 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:28:56.915438 15853 leader_election.cc:304] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:56.916397 16048 raft_consensus.cc:2804] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:56.916917 16048 raft_consensus.cc:697] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:56.917749 16048 consensus_queue.cc:237] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:56.926061 15706 catalog_manager.cc:5697] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 0 to 2, leader changed from <none> to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 2 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:57.306268 15812 raft_consensus.cc:1275] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:28:57.307917 16042 consensus_queue.cc:1048] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:57.323125 15960 raft_consensus.cc:1275] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:28:57.325008 16080 consensus_queue.cc:1048] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:28:57.351486 15960 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
mode: GRACEFUL
new_leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:37734
I20260709 12:28:57.352257 15960 raft_consensus.cc:606] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Received request to transfer leadership to TS cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:28:57.369169 15886 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
mode: GRACEFUL
new_leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:59312
I20260709 12:28:57.369781 15886 raft_consensus.cc:606] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Received request to transfer leadership to TS cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:28:57.377038 15886 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
mode: GRACEFUL
new_leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:59312
I20260709 12:28:57.377660 15886 raft_consensus.cc:606] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Received request to transfer leadership to TS cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:28:57.383916 15960 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
mode: GRACEFUL
new_leader_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:37734
I20260709 12:28:57.384588 15960 raft_consensus.cc:606] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Received request to transfer leadership to TS 9b36129f73bf485eb51ccf20370db33a
I20260709 12:28:57.401876 15812 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea"
dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
mode: GRACEFUL
new_leader_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:43554
I20260709 12:28:57.402626 15812 raft_consensus.cc:606] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Received request to transfer leadership to TS 9b36129f73bf485eb51ccf20370db33a
I20260709 12:28:57.414184 15812 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a"
dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
mode: GRACEFUL
new_leader_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:43554
I20260709 12:28:57.414981 15812 raft_consensus.cc:606] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Received request to transfer leadership to TS 9b36129f73bf485eb51ccf20370db33a
I20260709 12:28:57.421502 15886 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
mode: GRACEFUL
new_leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
 from {username='slave'} at 127.0.0.1:59312
I20260709 12:28:57.422171 15886 raft_consensus.cc:606] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Received request to transfer leadership to TS d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:28:57.556802 16090 raft_consensus.cc:993] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51: : Instructing follower 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:28:57.557417 16044 raft_consensus.cc:1081] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Signalling peer 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:28:57.560114 15886 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:59290
I20260709 12:28:57.560689 15886 raft_consensus.cc:493] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:57.560990 15886 raft_consensus.cc:3060] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.566883 15886 raft_consensus.cc:515] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.568828 15886 leader_election.cc:290] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:57.570143 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:57.570259 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:57.570864 15960 raft_consensus.cc:3055] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:57.571195 15812 raft_consensus.cc:3060] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.571338 15960 raft_consensus.cc:740] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:28:57.572431 15960 consensus_queue.cc:260] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.573721 15960 raft_consensus.cc:3060] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.579536 15812 raft_consensus.cc:2468] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:28:57.579826 15960 raft_consensus.cc:2468] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:28:57.581090 15851 leader_election.cc:304] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:28:57.582038 16080 raft_consensus.cc:2804] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:57.582585 16080 raft_consensus.cc:697] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:57.583549 16080 consensus_queue.cc:237] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.591960 15708 catalog_manager.cc:5697] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 1 to 2, leader changed from d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67) to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 2 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:57.606302 16042 raft_consensus.cc:993] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a: : Instructing follower d5923b92f8db4416a3c43a90bc1cdd51 to start an election
I20260709 12:28:57.606995 16080 raft_consensus.cc:1081] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Signalling peer d5923b92f8db4416a3c43a90bc1cdd51 to start an election
I20260709 12:28:57.608800 15960 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
 from {username='slave'} at 127.0.0.1:37714
I20260709 12:28:57.609426 15960 raft_consensus.cc:493] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:57.609741 15960 raft_consensus.cc:3060] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.615311 15960 raft_consensus.cc:515] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.617123 15960 leader_election.cc:290] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 2 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:28:57.618399 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:57.618458 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "869265d9a2fa4ae6b4c5b58c1c4cc1f5" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:57.619154 15886 raft_consensus.cc:3055] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:57.619563 15812 raft_consensus.cc:3060] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.619578 15886 raft_consensus.cc:740] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:57.620745 15886 consensus_queue.cc:260] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.621980 15886 raft_consensus.cc:3060] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.626761 16042 raft_consensus.cc:993] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a: : Instructing follower cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:28:57.627487 16080 raft_consensus.cc:1081] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Signalling peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:28:57.627768 15812 raft_consensus.cc:2468] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 2.
I20260709 12:28:57.629161 15925 leader_election.cc:304] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:57.629878 15886 raft_consensus.cc:2468] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 2.
I20260709 12:28:57.630081 16044 raft_consensus.cc:2804] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:57.629915 15812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f"
dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:43534
I20260709 12:28:57.631163 16044 raft_consensus.cc:697] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:28:57.631373 15812 raft_consensus.cc:493] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:57.631960 15812 raft_consensus.cc:3060] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.632165 16044 consensus_queue.cc:237] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.638865 15812 raft_consensus.cc:515] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.641465 15812 leader_election.cc:290] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 2 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:57.641232 15708 catalog_manager.cc:5697] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 1 to 2, leader changed from 9b36129f73bf485eb51ccf20370db33a (127.0.140.66) to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 2 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:28:57.642189 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:57.642922 15886 raft_consensus.cc:3055] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:57.642869 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc9602a9b0514fe7abc499687775488f" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:57.643551 15886 raft_consensus.cc:740] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:57.643833 15960 raft_consensus.cc:3060] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.644526 15886 consensus_queue.cc:260] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.646296 15886 raft_consensus.cc:3060] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.651506 15960 raft_consensus.cc:2468] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 2.
I20260709 12:28:57.653396 15778 leader_election.cc:304] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:57.654448 16024 raft_consensus.cc:2804] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:57.654846 15886 raft_consensus.cc:2468] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 2.
I20260709 12:28:57.655961 16024 raft_consensus.cc:697] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:28:57.656999 16024 consensus_queue.cc:237] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.665259 15708 catalog_manager.cc:5697] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 1 to 2, leader changed from 9b36129f73bf485eb51ccf20370db33a (127.0.140.66) to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 2 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:57.701175 16024 raft_consensus.cc:993] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: : Instructing follower 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:28:57.701642 16096 raft_consensus.cc:1081] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Signalling peer 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:28:57.703117 15886 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:59304
I20260709 12:28:57.703574 15886 raft_consensus.cc:493] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:57.703826 15886 raft_consensus.cc:3060] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.709064 15886 raft_consensus.cc:515] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.710748 15886 leader_election.cc:290] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:57.712015 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:57.711894 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7bbd23a4b0424576ba4cc5df1d5a7aea" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:57.712738 15960 raft_consensus.cc:3060] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.712787 15812 raft_consensus.cc:3055] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:57.713294 15812 raft_consensus.cc:740] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:28:57.714186 15812 consensus_queue.cc:260] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.715516 15812 raft_consensus.cc:3060] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.720305 15960 raft_consensus.cc:2468] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:28:57.721938 15853 leader_election.cc:304] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:57.722571 15812 raft_consensus.cc:2468] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:28:57.722909 16080 raft_consensus.cc:2804] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:57.723727 16080 raft_consensus.cc:697] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:57.724757 16080 consensus_queue.cc:237] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.733620 15708 catalog_manager.cc:5697] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 1 to 2, leader changed from cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65) to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 2 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:57.792306 16090 raft_consensus.cc:993] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51: : Instructing follower cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:28:57.792840 16090 raft_consensus.cc:1081] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Signalling peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:28:57.794870 15812 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96"
dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:43538
I20260709 12:28:57.795588 15812 raft_consensus.cc:493] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:57.795965 15812 raft_consensus.cc:3060] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.797225 16042 raft_consensus.cc:993] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a: : Instructing follower cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:28:57.797889 16080 raft_consensus.cc:1081] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Signalling peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:28:57.799723 15811 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa"
dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:43534
I20260709 12:28:57.800263 15811 raft_consensus.cc:493] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:57.800581 15811 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:57.803408 15812 raft_consensus.cc:515] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.805823 15812 leader_election.cc:290] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 2 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:57.806887 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:57.807305 15811 raft_consensus.cc:515] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.807897 15886 raft_consensus.cc:3060] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.807343 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f67bf24fa1b54ae2bf3ef7afe02cbd96" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:57.808590 15960 raft_consensus.cc:3055] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:57.809007 15960 raft_consensus.cc:740] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:28:57.810766 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:28:57.810853 15811 leader_election.cc:290] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 3 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:57.811583 15887 raft_consensus.cc:3055] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:28:57.811630 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:57.812106 15887 raft_consensus.cc:740] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:57.812570 15961 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:57.810029 15960 consensus_queue.cc:260] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.813258 15887 consensus_queue.cc:260] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.814304 15960 raft_consensus.cc:3060] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.814946 15887 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:28:57.816224 15886 raft_consensus.cc:2468] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 2.
I20260709 12:28:57.817698 15778 leader_election.cc:304] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:28:57.818579 16096 raft_consensus.cc:2804] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:57.819178 16096 raft_consensus.cc:697] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:28:57.820286 16096 consensus_queue.cc:237] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.823079 15961 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 3.
I20260709 12:28:57.824743 15778 leader_election.cc:304] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:57.825902 16024 raft_consensus.cc:2804] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:28:57.827207 15960 raft_consensus.cc:2468] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 2.
I20260709 12:28:57.829815 16024 raft_consensus.cc:697] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:28:57.830173 15887 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 3.
I20260709 12:28:57.831277 16024 consensus_queue.cc:237] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.835893 15708 catalog_manager.cc:5697] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 1 to 2, leader changed from d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67) to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 2 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:57.838269 16100 raft_consensus.cc:993] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: : Instructing follower 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:28:57.838948 16024 raft_consensus.cc:1081] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Signalling peer 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:28:57.840878 15887 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:59304
I20260709 12:28:57.841634 15887 raft_consensus.cc:493] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:28:57.842028 15887 raft_consensus.cc:3060] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.851642 15887 raft_consensus.cc:515] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.854036 15887 leader_election.cc:290] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:28:57.853356 15708 catalog_manager.cc:5697] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 2 to 3, leader changed from 9b36129f73bf485eb51ccf20370db33a (127.0.140.66) to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 3 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:57.854987 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:28:57.855350 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "22fa1d49fd24457395605b0c9943dc4a" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:28:57.855860 15811 raft_consensus.cc:3055] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:28:57.856325 15811 raft_consensus.cc:740] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:28:57.855939 15961 raft_consensus.cc:3060] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.857241 15811 consensus_queue.cc:260] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.858594 15811 raft_consensus.cc:3060] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:28:57.864951 15961 raft_consensus.cc:2468] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:28:57.865948 15811 raft_consensus.cc:2468] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:28:57.866954 15853 leader_election.cc:304] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:28:57.867882 16080 raft_consensus.cc:2804] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:28:57.869236 16080 raft_consensus.cc:697] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:28:57.870115 16080 consensus_queue.cc:237] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:28:57.878226 15708 catalog_manager.cc:5697] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 1 to 2, leader changed from cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65) to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 2 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:28:58.024586 15811 raft_consensus.cc:1275] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Refusing update from remote peer d5923b92f8db4416a3c43a90bc1cdd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.026299 16090 consensus_queue.cc:1048] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:58.038486 15887 raft_consensus.cc:1275] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Refusing update from remote peer d5923b92f8db4416a3c43a90bc1cdd51: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.040804 16090 consensus_queue.cc:1048] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:58.091248 15961 raft_consensus.cc:1275] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.093112 16080 consensus_queue.cc:1048] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:58.097641 15960 raft_consensus.cc:1275] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.099741 16100 consensus_queue.cc:1048] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:58.108384 15811 raft_consensus.cc:1275] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.110230 16080 consensus_queue.cc:1048] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:58.117714 15887 raft_consensus.cc:1275] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.120034 16100 consensus_queue.cc:1048] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:58.171190 15811 raft_consensus.cc:1275] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.172955 16080 consensus_queue.cc:1048] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:58.185111 15961 raft_consensus.cc:1275] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.186851 16080 consensus_queue.cc:1048] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:58.215633 15887 raft_consensus.cc:1275] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 3 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:58.217078 16110 consensus_queue.cc:1048] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:58.225972 15961 raft_consensus.cc:1275] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 3 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:28:58.228044 16100 consensus_queue.cc:1048] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:58.297614 15811 raft_consensus.cc:1275] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.299222 16080 consensus_queue.cc:1048] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:28:58.311789 15961 raft_consensus.cc:1275] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.314402 16116 consensus_queue.cc:1048] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:58.358359 15887 raft_consensus.cc:1275] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.359957 16100 consensus_queue.cc:1048] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:28:58.363788 15961 raft_consensus.cc:1275] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:28:58.365677 16100 consensus_queue.cc:1048] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:00.453516 15708 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53900:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:29:00.456305 15708 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:29:00.514325 15801 tablet_service.cc:1511] Processing CreateTablet for tablet a1cbc7ee5bdf434192dea928f8ca1a96 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:29:00.515157 15874 tablet_service.cc:1511] Processing CreateTablet for tablet a1cbc7ee5bdf434192dea928f8ca1a96 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:29:00.515766 15801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cbc7ee5bdf434192dea928f8ca1a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.516503 15874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cbc7ee5bdf434192dea928f8ca1a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.518760 15876 tablet_service.cc:1511] Processing CreateTablet for tablet 16d196f6fcd6474b9567e2fbac53ae77 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:29:00.519958 15950 tablet_service.cc:1511] Processing CreateTablet for tablet a1cbc7ee5bdf434192dea928f8ca1a96 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:29:00.520363 15876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d196f6fcd6474b9567e2fbac53ae77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.521384 15950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1cbc7ee5bdf434192dea928f8ca1a96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.524240 15877 tablet_service.cc:1511] Processing CreateTablet for tablet d81824aa48c14c71b39f71ffa84ff0dc (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:29:00.525709 15877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d81824aa48c14c71b39f71ffa84ff0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.528661 15948 tablet_service.cc:1511] Processing CreateTablet for tablet d81824aa48c14c71b39f71ffa84ff0dc (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:29:00.530208 15948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d81824aa48c14c71b39f71ffa84ff0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.531821 15798 tablet_service.cc:1511] Processing CreateTablet for tablet 7616a7a446f945a5b651eddd03b25e45 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:29:00.532192 15947 tablet_service.cc:1511] Processing CreateTablet for tablet 7616a7a446f945a5b651eddd03b25e45 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:29:00.533325 15798 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7616a7a446f945a5b651eddd03b25e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.533829 15947 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7616a7a446f945a5b651eddd03b25e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.535473 15797 tablet_service.cc:1511] Processing CreateTablet for tablet d81824aa48c14c71b39f71ffa84ff0dc (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:29:00.535815 15871 tablet_service.cc:1511] Processing CreateTablet for tablet 7616a7a446f945a5b651eddd03b25e45 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:29:00.536963 15797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d81824aa48c14c71b39f71ffa84ff0dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.537024 15871 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7616a7a446f945a5b651eddd03b25e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.528728 15949 tablet_service.cc:1511] Processing CreateTablet for tablet 16d196f6fcd6474b9567e2fbac53ae77 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:29:00.540117 15949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d196f6fcd6474b9567e2fbac53ae77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.530234 15799 tablet_service.cc:1511] Processing CreateTablet for tablet 16d196f6fcd6474b9567e2fbac53ae77 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:29:00.542032 15799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16d196f6fcd6474b9567e2fbac53ae77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.554944 15795 tablet_service.cc:1511] Processing CreateTablet for tablet 96fefd41ad4643c28334c4353ffe8ff1 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:29:00.556530 15875 tablet_service.cc:1511] Processing CreateTablet for tablet 96fefd41ad4643c28334c4353ffe8ff1 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:29:00.558153 15875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fefd41ad4643c28334c4353ffe8ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.559825 15946 tablet_service.cc:1511] Processing CreateTablet for tablet 96fefd41ad4643c28334c4353ffe8ff1 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:29:00.560717 15800 tablet_service.cc:1511] Processing CreateTablet for tablet 3a4eb0279d894156b7094538b6a5713b (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:29:00.561411 15946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fefd41ad4643c28334c4353ffe8ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.563529 15795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96fefd41ad4643c28334c4353ffe8ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.564487 15800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4eb0279d894156b7094538b6a5713b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.588231 15872 tablet_service.cc:1511] Processing CreateTablet for tablet 3a4eb0279d894156b7094538b6a5713b (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:29:00.589995 15872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4eb0279d894156b7094538b6a5713b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.593968 15951 tablet_service.cc:1511] Processing CreateTablet for tablet 3a4eb0279d894156b7094538b6a5713b (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:29:00.595643 15951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a4eb0279d894156b7094538b6a5713b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.596828 15794 tablet_service.cc:1511] Processing CreateTablet for tablet c47c39a171ab4d13990ce8a3a20ca869 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:29:00.598361 15794 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c47c39a171ab4d13990ce8a3a20ca869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.601104 15873 tablet_service.cc:1511] Processing CreateTablet for tablet c47c39a171ab4d13990ce8a3a20ca869 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:29:00.603576 15873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c47c39a171ab4d13990ce8a3a20ca869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.614565 15945 tablet_service.cc:1511] Processing CreateTablet for tablet c47c39a171ab4d13990ce8a3a20ca869 (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:29:00.616350 15945 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c47c39a171ab4d13990ce8a3a20ca869. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.621068 15944 tablet_service.cc:1511] Processing CreateTablet for tablet fdd3f394cc164d108a7f276e6cfcf8eb (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:29:00.623883 15802 tablet_service.cc:1511] Processing CreateTablet for tablet fdd3f394cc164d108a7f276e6cfcf8eb (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:29:00.622601 15944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd3f394cc164d108a7f276e6cfcf8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.632157 15802 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd3f394cc164d108a7f276e6cfcf8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.632404 15796 tablet_service.cc:1511] Processing CreateTablet for tablet e897991c3158463f8147cc5ea24425ec (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:29:00.633746 15796 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e897991c3158463f8147cc5ea24425ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.639991 16133 tablet_bootstrap.cc:492] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:29:00.641508 15949 tablet_service.cc:1511] Processing CreateTablet for tablet e897991c3158463f8147cc5ea24425ec (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:29:00.642921 15949 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e897991c3158463f8147cc5ea24425ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.648690 15874 tablet_service.cc:1511] Processing CreateTablet for tablet fdd3f394cc164d108a7f276e6cfcf8eb (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:29:00.651978 15874 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fdd3f394cc164d108a7f276e6cfcf8eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.695263 16134 tablet_bootstrap.cc:492] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:29:00.699769 16135 tablet_bootstrap.cc:492] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:29:00.705232 16133 tablet_bootstrap.cc:654] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.711786 15875 tablet_service.cc:1511] Processing CreateTablet for tablet e897991c3158463f8147cc5ea24425ec (DEFAULT_TABLE table=soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:29:00.741395 16134 tablet_bootstrap.cc:654] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.755534 15875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e897991c3158463f8147cc5ea24425ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:29:00.773643 16133 tablet_bootstrap.cc:492] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:29:00.774662 16133 ts_tablet_manager.cc:1403] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.135s	user 0.013s	sys 0.003s
I20260709 12:29:00.775205 16135 tablet_bootstrap.cc:654] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.778702 16133 raft_consensus.cc:359] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.782285 16133 raft_consensus.cc:385] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.782651 16133 raft_consensus.cc:740] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.783859 16133 consensus_queue.cc:260] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.788570 16133 ts_tablet_manager.cc:1434] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.001s
I20260709 12:29:00.791390 16133 tablet_bootstrap.cc:492] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:29:00.793848 16134 tablet_bootstrap.cc:492] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:29:00.794656 16134 ts_tablet_manager.cc:1403] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.100s	user 0.011s	sys 0.010s
I20260709 12:29:00.804303 16133 tablet_bootstrap.cc:654] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.806551 16134 raft_consensus.cc:359] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.807750 16134 raft_consensus.cc:385] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.808156 16134 raft_consensus.cc:740] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.809182 16134 consensus_queue.cc:260] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.809533 16135 tablet_bootstrap.cc:492] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:29:00.810328 16135 ts_tablet_manager.cc:1403] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.111s	user 0.009s	sys 0.011s
I20260709 12:29:00.810701 16133 tablet_bootstrap.cc:492] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:29:00.811309 16133 ts_tablet_manager.cc:1403] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.020s	user 0.009s	sys 0.004s
I20260709 12:29:00.814143 16135 raft_consensus.cc:359] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.814218 16134 ts_tablet_manager.cc:1434] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.001s
I20260709 12:29:00.815096 16135 raft_consensus.cc:385] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.814385 16133 raft_consensus.cc:359] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.815678 16135 raft_consensus.cc:740] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.815775 16133 raft_consensus.cc:385] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.816146 16133 raft_consensus.cc:740] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.816459 16134 tablet_bootstrap.cc:492] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:29:00.816735 16135 consensus_queue.cc:260] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.817008 16133 consensus_queue.cc:260] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.819473 16133 ts_tablet_manager.cc:1434] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:29:00.819751 16135 ts_tablet_manager.cc:1434] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.004s
I20260709 12:29:00.820667 16133 tablet_bootstrap.cc:492] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:29:00.824316 16135 tablet_bootstrap.cc:492] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:29:00.827499 16133 tablet_bootstrap.cc:654] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.835284 16135 tablet_bootstrap.cc:654] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.845428 16134 tablet_bootstrap.cc:654] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.850210 16135 tablet_bootstrap.cc:492] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:29:00.850770 16135 ts_tablet_manager.cc:1403] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.027s	user 0.005s	sys 0.016s
I20260709 12:29:00.853547 16133 tablet_bootstrap.cc:492] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:29:00.854239 16133 ts_tablet_manager.cc:1403] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.034s	user 0.008s	sys 0.022s
I20260709 12:29:00.855082 16135 raft_consensus.cc:359] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.855930 16135 raft_consensus.cc:385] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.856243 16135 raft_consensus.cc:740] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.857174 16135 consensus_queue.cc:260] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.857690 16133 raft_consensus.cc:359] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.858597 16133 raft_consensus.cc:385] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.859575 16133 raft_consensus.cc:740] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.860538 16133 consensus_queue.cc:260] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.863322 16133 ts_tablet_manager.cc:1434] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:29:00.865103 16133 tablet_bootstrap.cc:492] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:29:00.868120 16135 ts_tablet_manager.cc:1434] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:29:00.869392 16135 tablet_bootstrap.cc:492] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:29:00.873405 16133 tablet_bootstrap.cc:654] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.880925 16100 raft_consensus.cc:493] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.881714 16100 raft_consensus.cc:515] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.884357 16100 leader_election.cc:290] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:00.886332 16133 tablet_bootstrap.cc:492] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:29:00.887193 16133 ts_tablet_manager.cc:1403] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.011s
I20260709 12:29:00.887491 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
W20260709 12:29:00.889146 15778 leader_election.cc:343] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:00.886325 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:29:00.890640 16133 raft_consensus.cc:359] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
W20260709 12:29:00.891278 15778 leader_election.cc:343] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:00.891538 16133 raft_consensus.cc:385] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.891824 15778 leader_election.cc:304] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:29:00.892004 16133 raft_consensus.cc:740] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.893029 16100 raft_consensus.cc:2749] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:00.892869 16133 consensus_queue.cc:260] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.895947 16133 ts_tablet_manager.cc:1434] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:29:00.897178 16133 tablet_bootstrap.cc:492] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:29:00.897246 16135 tablet_bootstrap.cc:654] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.898079 16134 tablet_bootstrap.cc:492] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:29:00.898758 16134 ts_tablet_manager.cc:1403] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.083s	user 0.028s	sys 0.023s
I20260709 12:29:00.904739 16134 raft_consensus.cc:359] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.906481 16134 raft_consensus.cc:385] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.906867 16134 raft_consensus.cc:740] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.905033 16133 tablet_bootstrap.cc:654] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.907963 16134 consensus_queue.cc:260] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.910445 16134 ts_tablet_manager.cc:1434] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 12:29:00.911762 16134 tablet_bootstrap.cc:492] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:29:00.913098 16121 raft_consensus.cc:493] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.917186 16135 tablet_bootstrap.cc:492] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:29:00.917892 16135 ts_tablet_manager.cc:1403] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.049s	user 0.013s	sys 0.004s
I20260709 12:29:00.917363 16121 raft_consensus.cc:515] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.920935 16121 leader_election.cc:290] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:00.921944 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81824aa48c14c71b39f71ffa84ff0dc" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:00.922042 16135 raft_consensus.cc:359] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.923391 16135 raft_consensus.cc:385] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.923686 16135 raft_consensus.cc:740] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.924351 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81824aa48c14c71b39f71ffa84ff0dc" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:00.925307 16134 tablet_bootstrap.cc:654] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
W20260709 12:29:00.926375 15927 leader_election.cc:343] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:00.927434 16133 tablet_bootstrap.cc:492] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:29:00.928148 16133 ts_tablet_manager.cc:1403] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.031s	user 0.005s	sys 0.009s
I20260709 12:29:00.924552 16135 consensus_queue.cc:260] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.931356 16135 ts_tablet_manager.cc:1434] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.005s
I20260709 12:29:00.932443 16135 tablet_bootstrap.cc:492] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:29:00.932320 16133 raft_consensus.cc:359] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.933564 16133 raft_consensus.cc:385] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.933894 16133 raft_consensus.cc:740] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.934684 16133 consensus_queue.cc:260] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.935168 16134 tablet_bootstrap.cc:492] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
W20260709 12:29:00.928239 15925 leader_election.cc:343] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:00.937821 16133 ts_tablet_manager.cc:1434] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:29:00.939029 15925 leader_election.cc:304] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:29:00.935838 16134 ts_tablet_manager.cc:1403] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.024s	user 0.006s	sys 0.010s
I20260709 12:29:00.940027 16121 raft_consensus.cc:2749] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:00.941888 16133 tablet_bootstrap.cc:492] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:29:00.943224 16134 raft_consensus.cc:359] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.944370 16134 raft_consensus.cc:385] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.944705 16134 raft_consensus.cc:740] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.946337 16135 tablet_bootstrap.cc:654] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.945902 16134 consensus_queue.cc:260] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.950939 16134 ts_tablet_manager.cc:1434] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.001s
I20260709 12:29:00.952198 16134 tablet_bootstrap.cc:492] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:29:00.958093 16133 tablet_bootstrap.cc:654] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.959738 16134 tablet_bootstrap.cc:654] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:00.965086 16121 raft_consensus.cc:493] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:00.965955 16121 raft_consensus.cc:515] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.975670 16121 leader_election.cc:290] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:00.977182 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16d196f6fcd6474b9567e2fbac53ae77" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:00.979060 15811 raft_consensus.cc:2468] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:29:00.977182 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16d196f6fcd6474b9567e2fbac53ae77" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:00.980636 15887 raft_consensus.cc:2468] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:29:00.980875 15925 leader_election.cc:304] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:29:00.982914 16121 raft_consensus.cc:2804] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:00.984218 16121 raft_consensus.cc:493] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:00.985394 16121 raft_consensus.cc:3060] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:00.986167 16135 tablet_bootstrap.cc:492] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:29:00.986963 16135 ts_tablet_manager.cc:1403] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.055s	user 0.015s	sys 0.003s
I20260709 12:29:00.987176 16134 tablet_bootstrap.cc:492] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:29:00.987850 16133 tablet_bootstrap.cc:492] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:29:00.987892 16134 ts_tablet_manager.cc:1403] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.036s	user 0.017s	sys 0.017s
I20260709 12:29:00.988796 16133 ts_tablet_manager.cc:1403] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.049s	user 0.006s	sys 0.025s
I20260709 12:29:00.990876 16135 raft_consensus.cc:359] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.991878 16135 raft_consensus.cc:385] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.992353 16135 raft_consensus.cc:740] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.992018 16134 raft_consensus.cc:359] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.993180 16134 raft_consensus.cc:385] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.992697 16133 raft_consensus.cc:359] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.993652 16134 raft_consensus.cc:740] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.993995 16133 raft_consensus.cc:385] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:00.993490 16135 consensus_queue.cc:260] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.994576 16133 raft_consensus.cc:740] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:29:00.995554 16134 consensus_queue.cc:260] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.995692 16133 consensus_queue.cc:260] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:00.996991 16135 ts_tablet_manager.cc:1434] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:29:00.998714 16133 ts_tablet_manager.cc:1434] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:29:00.999151 16134 ts_tablet_manager.cc:1434] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.003s
I20260709 12:29:01.000097 16133 tablet_bootstrap.cc:492] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:29:01.000751 16134 tablet_bootstrap.cc:492] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:29:00.998373 16135 tablet_bootstrap.cc:492] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:29:01.007428 16136 raft_consensus.cc:493] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.009245 16133 tablet_bootstrap.cc:654] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.009099 16136 raft_consensus.cc:515] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.010152 16134 tablet_bootstrap.cc:654] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.021970 16133 tablet_bootstrap.cc:492] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:29:01.022583 16133 ts_tablet_manager.cc:1403] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.023s	user 0.018s	sys 0.002s
I20260709 12:29:01.025805 16133 raft_consensus.cc:359] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.026855 16133 raft_consensus.cc:385] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.027189 16133 raft_consensus.cc:740] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.028033 16133 consensus_queue.cc:260] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.031626 16133 ts_tablet_manager.cc:1434] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:29:01.032876 16133 tablet_bootstrap.cc:492] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:29:01.032943 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cbc7ee5bdf434192dea928f8ca1a96" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:01.033634 16136 leader_election.cc:290] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
W20260709 12:29:01.034943 15851 leader_election.cc:343] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.034997 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cbc7ee5bdf434192dea928f8ca1a96" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
W20260709 12:29:01.037031 15853 leader_election.cc:343] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.037565 15853 leader_election.cc:304] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:29:01.039137 16136 raft_consensus.cc:2749] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:01.040565 16145 raft_consensus.cc:493] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.041199 16133 tablet_bootstrap.cc:654] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.041232 16135 tablet_bootstrap.cc:654] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.041222 16145 raft_consensus.cc:515] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.045629 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.046002 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:01.046763 15887 raft_consensus.cc:2468] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:29:01.047535 16145 leader_election.cc:290] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
W20260709 12:29:01.047673 15925 leader_election.cc:343] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:00.995157 16121 raft_consensus.cc:515] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.049235 15927 leader_election.cc:304] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:29:01.050879 16145 raft_consensus.cc:2804] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:01.051290 16145 raft_consensus.cc:493] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.051676 16145 raft_consensus.cc:3060] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.052198 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16d196f6fcd6474b9567e2fbac53ae77" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:01.052915 15811 raft_consensus.cc:3060] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.058303 16135 tablet_bootstrap.cc:492] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:29:01.059162 16135 ts_tablet_manager.cc:1403] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.061s	user 0.015s	sys 0.006s
W20260709 12:29:01.060595 15758 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:29:01.060927 16145 raft_consensus.cc:515] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.062915 16133 tablet_bootstrap.cc:492] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:29:01.064565 16133 ts_tablet_manager.cc:1403] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.032s	user 0.010s	sys 0.011s
I20260709 12:29:01.063750 16157 raft_consensus.cc:493] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.065943 16157 raft_consensus.cc:515] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.062718 16135 raft_consensus.cc:359] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.067278 16135 raft_consensus.cc:385] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.066987 16133 raft_consensus.cc:359] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.067778 16135 raft_consensus.cc:740] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.067831 16133 raft_consensus.cc:385] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.068418 16133 raft_consensus.cc:740] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.068964 16135 consensus_queue.cc:260] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.070026 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.070223 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:01.070818 15887 raft_consensus.cc:2468] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:29:01.069340 16133 consensus_queue.cc:260] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.072224 16135 ts_tablet_manager.cc:1434] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.001s
I20260709 12:29:01.072178 15927 leader_election.cc:304] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
W20260709 12:29:01.073081 15925 leader_election.cc:343] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.075209 16158 raft_consensus.cc:2804] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:01.076141 16133 ts_tablet_manager.cc:1434] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.005s
I20260709 12:29:01.076751 16135 tablet_bootstrap.cc:492] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:29:01.077236 16133 tablet_bootstrap.cc:492] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51: Bootstrap starting.
I20260709 12:29:01.084015 16121 leader_election.cc:290] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.084374 16133 tablet_bootstrap.cc:654] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.084374 16135 tablet_bootstrap.cc:654] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.086661 16157 leader_election.cc:290] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.084432 16158 raft_consensus.cc:493] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.087352 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16d196f6fcd6474b9567e2fbac53ae77" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:01.087747 16158 raft_consensus.cc:3060] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.088151 15887 raft_consensus.cc:3060] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.093227 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:01.094012 16145 leader_election.cc:290] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.095744 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:01.096750 15886 raft_consensus.cc:3060] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:29:01.099802 15925 leader_election.cc:343] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.108129 16135 tablet_bootstrap.cc:492] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:29:01.109156 16135 ts_tablet_manager.cc:1403] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.004s
I20260709 12:29:01.088147 15811 raft_consensus.cc:2468] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:29:01.114634 15925 leader_election.cc:304] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:29:01.115993 16145 raft_consensus.cc:2804] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:01.116643 16145 raft_consensus.cc:697] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:01.117805 16145 consensus_queue.cc:237] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.121517 16133 tablet_bootstrap.cc:492] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51: No bootstrap required, opened a new log
I20260709 12:29:01.122218 16133 ts_tablet_manager.cc:1403] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent bootstrapping tablet: real 0.045s	user 0.014s	sys 0.022s
I20260709 12:29:01.121686 16135 raft_consensus.cc:359] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.123669 16135 raft_consensus.cc:385] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.124037 16135 raft_consensus.cc:740] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.125435 16133 raft_consensus.cc:359] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.125624 16158 raft_consensus.cc:515] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.126289 16133 raft_consensus.cc:385] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.126680 16133 raft_consensus.cc:740] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.128454 16158 leader_election.cc:290] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.129120 16133 consensus_queue.cc:260] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.129436 16135 consensus_queue.cc:260] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.130613 15885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:01.131563 15885 raft_consensus.cc:3060] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.131654 16133 ts_tablet_manager.cc:1434] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:29:01.132867 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:01.133141 16135 ts_tablet_manager.cc:1434] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.000s
I20260709 12:29:01.134428 16135 tablet_bootstrap.cc:492] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
W20260709 12:29:01.134509 15925 leader_election.cc:343] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.142374 16135 tablet_bootstrap.cc:654] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.146842 15885 raft_consensus.cc:2468] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:29:01.148576 15927 leader_election.cc:304] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:29:01.121655 15887 raft_consensus.cc:2468] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:29:01.149657 16158 raft_consensus.cc:2804] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:01.150000 16157 raft_consensus.cc:493] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.150287 16158 raft_consensus.cc:697] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:01.150643 16157 raft_consensus.cc:515] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.151535 16158 consensus_queue.cc:237] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.155193 15886 raft_consensus.cc:2468] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:29:01.160266 16135 tablet_bootstrap.cc:492] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:29:01.155375 16157 leader_election.cc:290] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.161032 15927 leader_election.cc:304] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:29:01.161051 16135 ts_tablet_manager.cc:1403] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.027s	user 0.009s	sys 0.008s
I20260709 12:29:01.163007 16157 raft_consensus.cc:2804] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:01.163527 16157 raft_consensus.cc:697] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:01.164327 16157 consensus_queue.cc:237] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.155611 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:01.167501 15811 raft_consensus.cc:2468] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:29:01.167768 16134 tablet_bootstrap.cc:492] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:29:01.168465 16134 ts_tablet_manager.cc:1403] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.168s	user 0.040s	sys 0.020s
I20260709 12:29:01.169123 15925 leader_election.cc:304] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:29:01.170331 16145 raft_consensus.cc:2804] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:01.169878 16135 raft_consensus.cc:359] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.170873 16145 raft_consensus.cc:493] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.171110 16135 raft_consensus.cc:385] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.171561 16135 raft_consensus.cc:740] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.171685 16145 raft_consensus.cc:3060] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.172586 16134 raft_consensus.cc:359] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.172725 16135 consensus_queue.cc:260] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.155952 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.175526 16135 ts_tablet_manager.cc:1434] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
W20260709 12:29:01.176816 15927 leader_election.cc:343] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.178496 16135 tablet_bootstrap.cc:492] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:29:01.173629 16134 raft_consensus.cc:385] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.182479 16145 raft_consensus.cc:515] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.185945 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:01.186523 15811 raft_consensus.cc:3060] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.188728 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
W20260709 12:29:01.190395 15927 leader_election.cc:343] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.191344 16145 leader_election.cc:290] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.194521 16135 tablet_bootstrap.cc:654] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.182538 16134 raft_consensus.cc:740] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.198036 16134 consensus_queue.cc:260] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.198017 15709 catalog_manager.cc:5697] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:01.201411 16134 ts_tablet_manager.cc:1434] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.033s	user 0.010s	sys 0.000s
I20260709 12:29:01.201622 15811 raft_consensus.cc:2468] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:29:01.201131 15709 catalog_manager.cc:5697] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:01.202617 16134 tablet_bootstrap.cc:492] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:29:01.204182 15925 leader_election.cc:304] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a
I20260709 12:29:01.206077 16135 tablet_bootstrap.cc:492] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:29:01.206964 16135 ts_tablet_manager.cc:1403] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.000s
I20260709 12:29:01.207098 16157 raft_consensus.cc:2804] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:01.207767 16157 raft_consensus.cc:697] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:01.208837 16157 consensus_queue.cc:237] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.210633 16134 tablet_bootstrap.cc:654] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.211838 16135 raft_consensus.cc:359] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.212735 16135 raft_consensus.cc:385] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.213364 16135 raft_consensus.cc:740] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.214579 16100 raft_consensus.cc:493] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.215217 16100 raft_consensus.cc:515] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.217658 16100 leader_election.cc:290] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.218259 16145 raft_consensus.cc:493] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.218896 16145 raft_consensus.cc:515] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.220333 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81824aa48c14c71b39f71ffa84ff0dc" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.221259 15887 raft_consensus.cc:2468] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 0.
I20260709 12:29:01.214232 16135 consensus_queue.cc:260] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.223639 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e897991c3158463f8147cc5ea24425ec" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.224511 16135 ts_tablet_manager.cc:1434] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.017s	user 0.007s	sys 0.000s
I20260709 12:29:01.225060 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81824aa48c14c71b39f71ffa84ff0dc" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:29:01.225476 15778 leader_election.cc:304] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:29:01.225847 15961 raft_consensus.cc:2468] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 0.
I20260709 12:29:01.226612 16134 tablet_bootstrap.cc:492] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:29:01.227262 16134 ts_tablet_manager.cc:1403] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.025s	user 0.009s	sys 0.010s
I20260709 12:29:01.227639 16145 leader_election.cc:290] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.227190 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e897991c3158463f8147cc5ea24425ec" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:01.228964 15811 raft_consensus.cc:2468] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:29:01.227702 16138 raft_consensus.cc:2804] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:01.231371 16138 raft_consensus.cc:493] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.231699 16138 raft_consensus.cc:3060] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.232707 16134 raft_consensus.cc:359] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.233850 16134 raft_consensus.cc:385] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:29:01.233273 15927 leader_election.cc:343] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.234164 16134 raft_consensus.cc:740] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.235062 16134 consensus_queue.cc:260] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.237762 16134 ts_tablet_manager.cc:1434] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:29:01.238430 16100 raft_consensus.cc:493] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.238883 16134 tablet_bootstrap.cc:492] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:29:01.238963 16100 raft_consensus.cc:515] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.241318 16100 leader_election.cc:290] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.248379 16134 tablet_bootstrap.cc:654] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.250874 15925 leader_election.cc:304] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a
I20260709 12:29:01.251766 16135 tablet_bootstrap.cc:492] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Bootstrap starting.
I20260709 12:29:01.255195 16145 raft_consensus.cc:2804] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:01.255893 16145 raft_consensus.cc:493] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.256335 16145 raft_consensus.cc:3060] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.258419 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.258666 16100 raft_consensus.cc:493] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.259353 15887 raft_consensus.cc:2393] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in current term 1: Already voted for candidate d5923b92f8db4416a3c43a90bc1cdd51 in this term.
I20260709 12:29:01.259202 16100 raft_consensus.cc:515] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.261514 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:29:01.248832 16138 raft_consensus.cc:515] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.311619 16138 leader_election.cc:290] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.312556 16138 raft_consensus.cc:493] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:29:01.314798 15925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.65:43921, user_credentials={real_user=slave}} blocked reactor thread for 64067.9us
W20260709 12:29:01.314996 15778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.140.67:35041, user_credentials={real_user=slave}} blocked reactor thread for 56220.2us
I20260709 12:29:01.320878 16157 raft_consensus.cc:493] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.321420 16157 raft_consensus.cc:515] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.323607 16157 leader_election.cc:290] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.325838 16135 tablet_bootstrap.cc:654] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.327378 16157 raft_consensus.cc:493] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.339238 16157 raft_consensus.cc:515] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.343441 16157 leader_election.cc:290] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.328328 16138 raft_consensus.cc:515] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.328723 15778 leader_election.cc:304] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:29:01.329921 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81824aa48c14c71b39f71ffa84ff0dc" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:29:01.345001 15884 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd3f394cc164d108a7f276e6cfcf8eb" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.345679 16170 raft_consensus.cc:3060] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.345978 15884 raft_consensus.cc:2468] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:29:01.331928 16100 leader_election.cc:290] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.333807 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d81824aa48c14c71b39f71ffa84ff0dc" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:01.334388 15886 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.347602 16138 leader_election.cc:290] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.337064 16145 raft_consensus.cc:515] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.347169 15887 raft_consensus.cc:3060] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.338397 15885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.347997 15927 leader_election.cc:304] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:29:01.345286 15961 raft_consensus.cc:3060] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.355011 16170 raft_consensus.cc:2749] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:01.363125 16157 raft_consensus.cc:2804] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:01.363929 16157 raft_consensus.cc:493] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.364511 16157 raft_consensus.cc:3060] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:29:01.379595 15927 leader_election.cc:343] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.385289 16145 leader_election.cc:290] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.387566 15885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e897991c3158463f8147cc5ea24425ec" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
W20260709 12:29:01.389910 15927 leader_election.cc:343] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.393321 15961 raft_consensus.cc:2468] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 1.
I20260709 12:29:01.395133 16134 tablet_bootstrap.cc:492] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:29:01.396088 16134 ts_tablet_manager.cc:1403] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.157s	user 0.019s	sys 0.019s
I20260709 12:29:01.397432 15887 raft_consensus.cc:2468] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 1.
I20260709 12:29:01.401719 16134 raft_consensus.cc:359] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.403157 16134 raft_consensus.cc:385] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.403633 16134 raft_consensus.cc:740] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.405022 16134 consensus_queue.cc:260] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.413436 16134 ts_tablet_manager.cc:1434] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.017s	user 0.010s	sys 0.000s
I20260709 12:29:01.414963 16135 tablet_bootstrap.cc:492] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: No bootstrap required, opened a new log
I20260709 12:29:01.415582 16135 ts_tablet_manager.cc:1403] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent bootstrapping tablet: real 0.164s	user 0.035s	sys 0.050s
I20260709 12:29:01.414229 16157 raft_consensus.cc:515] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
W20260709 12:29:01.416661 15778 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.67:35041, user_credentials={real_user=slave}} blocked reactor thread for 70748us
I20260709 12:29:01.418620 15778 leader_election.cc:304] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:29:01.419524 16100 raft_consensus.cc:2804] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:01.419992 16100 raft_consensus.cc:697] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:29:01.420825 16100 consensus_queue.cc:237] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.420922 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd3f394cc164d108a7f276e6cfcf8eb" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:01.421985 15887 raft_consensus.cc:3060] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.423861 16135 raft_consensus.cc:359] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.424718 16135 raft_consensus.cc:385] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.424978 16135 raft_consensus.cc:740] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.425642 16135 consensus_queue.cc:260] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.428270 16135 ts_tablet_manager.cc:1434] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:29:01.429504 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cbc7ee5bdf434192dea928f8ca1a96" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:29:01.429876 15885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cbc7ee5bdf434192dea928f8ca1a96" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.430317 15960 raft_consensus.cc:2468] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 0.
I20260709 12:29:01.430733 15885 raft_consensus.cc:2468] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 0.
I20260709 12:29:01.431082 16134 tablet_bootstrap.cc:492] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:29:01.428277 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:29:01.436739 15961 raft_consensus.cc:2468] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 0.
I20260709 12:29:01.437078 15706 catalog_manager.cc:5697] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:01.438939 16157 leader_election.cc:290] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.440690 15812 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:01.441458 15887 raft_consensus.cc:2468] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:29:01.442603 15810 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd3f394cc164d108a7f276e6cfcf8eb" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
W20260709 12:29:01.442932 15778 leader_election.cc:343] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:29:01.443624 15810 raft_consensus.cc:2468] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:29:01.444455 15778 leader_election.cc:304] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:29:01.445083 15927 leader_election.cc:304] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:29:01.443519 15809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e897991c3158463f8147cc5ea24425ec" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:01.446310 15809 raft_consensus.cc:3060] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.446026 15706 catalog_manager.cc:5697] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:01.447299 16134 tablet_bootstrap.cc:654] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.449864 16175 raft_consensus.cc:2804] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:01.450332 16175 raft_consensus.cc:493] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.441592 15812 raft_consensus.cc:2468] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 0.
I20260709 12:29:01.450748 16175 raft_consensus.cc:3060] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.450418 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd3f394cc164d108a7f276e6cfcf8eb" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:01.451713 15811 raft_consensus.cc:3060] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.452193 16157 raft_consensus.cc:2804] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:01.451882 15709 catalog_manager.cc:5697] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 1 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:01.452965 16157 raft_consensus.cc:697] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:01.454064 16157 consensus_queue.cc:237] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.461665 15811 raft_consensus.cc:2468] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:29:01.464322 15809 raft_consensus.cc:2468] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:29:01.467696 16175 raft_consensus.cc:515] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.471246 16134 tablet_bootstrap.cc:492] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:29:01.470928 15925 leader_election.cc:304] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a
I20260709 12:29:01.471988 16134 ts_tablet_manager.cc:1403] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.057s	user 0.010s	sys 0.008s
I20260709 12:29:01.472548 16175 leader_election.cc:290] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.472688 16157 raft_consensus.cc:2804] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:01.473300 16157 raft_consensus.cc:493] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.473702 16157 raft_consensus.cc:3060] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.475870 16134 raft_consensus.cc:359] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.476902 16134 raft_consensus.cc:385] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.477244 16134 raft_consensus.cc:740] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.478377 15706 catalog_manager.cc:5697] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:01.478725 16134 consensus_queue.cc:260] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.480583 15925 leader_election.cc:304] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a
I20260709 12:29:01.481580 16145 raft_consensus.cc:2804] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:01.481776 16134 ts_tablet_manager.cc:1434] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:29:01.482092 16145 raft_consensus.cc:697] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:01.483096 16145 consensus_queue.cc:237] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.483982 16134 tablet_bootstrap.cc:492] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a: Bootstrap starting.
I20260709 12:29:01.483595 16157 raft_consensus.cc:515] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.488245 16157 leader_election.cc:290] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:01.489866 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:01.490629 15887 raft_consensus.cc:3060] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.492041 15778 leader_election.cc:304] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 9b36129f73bf485eb51ccf20370db33a
I20260709 12:29:01.493516 16100 raft_consensus.cc:2804] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:29:01.493880 16100 raft_consensus.cc:493] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:29:01.494213 16100 raft_consensus.cc:3060] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.498195 16134 tablet_bootstrap.cc:654] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a: Neither blocks nor log segments found. Creating new log.
I20260709 12:29:01.504045 15887 raft_consensus.cc:2468] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 1.
I20260709 12:29:01.508235 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cbc7ee5bdf434192dea928f8ca1a96" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:29:01.508975 15961 raft_consensus.cc:3060] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.505173 15885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a1cbc7ee5bdf434192dea928f8ca1a96" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:01.510921 15885 raft_consensus.cc:3060] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.511518 15927 leader_election.cc:304] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:29:01.513587 16175 raft_consensus.cc:493] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.514138 16175 raft_consensus.cc:515] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.513875 15706 catalog_manager.cc:5697] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:01.516237 16175 leader_election.cc:290] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.516888 16145 raft_consensus.cc:2804] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:01.517678 16145 raft_consensus.cc:697] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:01.518571 16145 consensus_queue.cc:237] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.525194 15885 raft_consensus.cc:2468] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 1.
I20260709 12:29:01.527395 15809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:01.528455 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:29:01.529047 15887 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a" is_pre_election: true
I20260709 12:29:01.529820 15887 raft_consensus.cc:2393] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in current term 1: Already voted for candidate d5923b92f8db4416a3c43a90bc1cdd51 in this term.
I20260709 12:29:01.530506 16134 tablet_bootstrap.cc:492] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a: No bootstrap required, opened a new log
I20260709 12:29:01.529254 16100 raft_consensus.cc:515] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.530817 15778 leader_election.cc:304] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:29:01.531306 16134 ts_tablet_manager.cc:1403] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a: Time spent bootstrapping tablet: real 0.048s	user 0.018s	sys 0.023s
I20260709 12:29:01.525629 15961 raft_consensus.cc:2468] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 1.
I20260709 12:29:01.531754 15809 raft_consensus.cc:2393] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d5923b92f8db4416a3c43a90bc1cdd51 in current term 1: Already voted for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in this term.
I20260709 12:29:01.532267 15778 leader_election.cc:304] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:29:01.534183 16138 raft_consensus.cc:2804] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:29:01.534885 16138 raft_consensus.cc:697] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:29:01.535045 16170 raft_consensus.cc:3060] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.535007 16134 raft_consensus.cc:359] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.536127 16134 raft_consensus.cc:385] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:29:01.536450 16134 raft_consensus.cc:740] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Initialized, Role: FOLLOWER
I20260709 12:29:01.537379 16134 consensus_queue.cc:260] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.537206 16138 consensus_queue.cc:237] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.543838 16134 ts_tablet_manager.cc:1434] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 12:29:01.551911 16170 raft_consensus.cc:2749] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:01.557161 16100 leader_election.cc:290] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.557192 15709 catalog_manager.cc:5697] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 0 to 1, leader changed from <none> to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 1 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:01.559538 15885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:01.560633 15885 raft_consensus.cc:2393] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in current term 1: Already voted for candidate d5923b92f8db4416a3c43a90bc1cdd51 in this term.
I20260709 12:29:01.560817 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:29:01.560348 15706 catalog_manager.cc:5697] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 1 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:01.562865 15778 leader_election.cc:304] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:29:01.563951 16100 raft_consensus.cc:2749] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:29:01.565763 16145 consensus_queue.cc:1048] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:01.583644 16145 consensus_queue.cc:1048] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:01.605018 16145 consensus_queue.cc:1048] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:01.617455 16159 consensus_queue.cc:1048] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.623494 16136 raft_consensus.cc:493] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.624120 16136 raft_consensus.cc:515] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.626327 16136 leader_election.cc:290] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.627187 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:01.628021 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:29:01.628291 15811 raft_consensus.cc:2393] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b36129f73bf485eb51ccf20370db33a in current term 1: Already voted for candidate d5923b92f8db4416a3c43a90bc1cdd51 in this term.
I20260709 12:29:01.632030 15851 leader_election.cc:304] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:29:01.633407 16136 raft_consensus.cc:3060] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.641108 16136 raft_consensus.cc:2749] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:01.666676 16159 consensus_queue.cc:1048] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.679804 16159 consensus_queue.cc:1048] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.680570 16145 consensus_queue.cc:1048] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.694897 16145 consensus_queue.cc:1048] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.717649 15961 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "16d196f6fcd6474b9567e2fbac53ae77"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
mode: GRACEFUL
new_leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:37734
I20260709 12:29:01.718454 15961 raft_consensus.cc:606] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Received request to transfer leadership to TS cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:29:01.735183 15961 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
mode: GRACEFUL
new_leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:37734
I20260709 12:29:01.735870 15961 raft_consensus.cc:606] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Received request to transfer leadership to TS cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:29:01.742977 15961 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
mode: GRACEFUL
new_leader_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:37734
I20260709 12:29:01.743595 15961 raft_consensus.cc:606] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Received request to transfer leadership to TS 9b36129f73bf485eb51ccf20370db33a
I20260709 12:29:01.749650 15961 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
mode: GRACEFUL
new_leader_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:37734
I20260709 12:29:01.750321 15961 raft_consensus.cc:606] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Received request to transfer leadership to TS 9b36129f73bf485eb51ccf20370db33a
I20260709 12:29:01.756304 15961 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
mode: GRACEFUL
new_leader_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:37734
I20260709 12:29:01.756922 15961 raft_consensus.cc:606] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Received request to transfer leadership to TS 9b36129f73bf485eb51ccf20370db33a
I20260709 12:29:01.763311 15961 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fdd3f394cc164d108a7f276e6cfcf8eb"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
mode: GRACEFUL
new_leader_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:37734
I20260709 12:29:01.764037 15961 raft_consensus.cc:606] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Received request to transfer leadership to TS 9b36129f73bf485eb51ccf20370db33a
I20260709 12:29:01.840094 16145 consensus_queue.cc:1048] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.848168 16145 raft_consensus.cc:993] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51: : Instructing follower 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:29:01.848810 16145 raft_consensus.cc:1081] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Signalling peer 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:29:01.851249 15885 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fdd3f394cc164d108a7f276e6cfcf8eb"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:59290
I20260709 12:29:01.854362 15885 raft_consensus.cc:493] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:01.854883 15885 raft_consensus.cc:3060] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:01.858289 16157 consensus_queue.cc:1048] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:01.861690 16100 consensus_queue.cc:1048] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.862707 15885 raft_consensus.cc:515] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.867015 15885 leader_election.cc:290] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.867973 15809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd3f394cc164d108a7f276e6cfcf8eb" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:01.869423 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fdd3f394cc164d108a7f276e6cfcf8eb" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:29:01.870059 15960 raft_consensus.cc:3055] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:01.870340 15960 raft_consensus.cc:740] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:01.871207 15960 consensus_queue.cc:260] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.872368 15960 raft_consensus.cc:3060] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:01.879504 16138 consensus_queue.cc:1048] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.881259 15960 raft_consensus.cc:2468] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:29:01.882539 15853 leader_election.cc:304] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a
I20260709 12:29:01.883867 16136 raft_consensus.cc:2804] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:01.884413 16136 raft_consensus.cc:697] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:29:01.885211 16136 consensus_queue.cc:237] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.885330 16122 raft_consensus.cc:493] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:29:01.886296 16122 raft_consensus.cc:515] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:01.889266 16122 leader_election.cc:290] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:01.892009 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e897991c3158463f8147cc5ea24425ec" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" is_pre_election: true
I20260709 12:29:01.895936 15811 raft_consensus.cc:2393] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9b36129f73bf485eb51ccf20370db33a in current term 1: Already voted for candidate d5923b92f8db4416a3c43a90bc1cdd51 in this term.
I20260709 12:29:01.897465 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e897991c3158463f8147cc5ea24425ec" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" is_pre_election: true
I20260709 12:29:01.899258 15853 leader_election.cc:304] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a; no voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:29:01.900156 16122 raft_consensus.cc:3060] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:29:01.908329 16122 raft_consensus.cc:2749] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:29:01.908213 15710 catalog_manager.cc:5697] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 1 to 2, leader changed from d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67) to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 2 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:01.929306 16138 consensus_queue.cc:1048] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.942128 16170 consensus_queue.cc:1048] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.957988 16157 consensus_queue.cc:1048] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.972291 16157 consensus_queue.cc:1048] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:29:01.991559 16159 raft_consensus.cc:993] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51: : Instructing follower cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:29:01.992071 16157 raft_consensus.cc:1081] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Signalling peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:29:01.993862 15811 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45"
dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:43538
I20260709 12:29:01.994643 15811 raft_consensus.cc:493] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:01.995065 15811 raft_consensus.cc:3060] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.000628 15811 raft_consensus.cc:515] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.002245 15811 leader_election.cc:290] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 2 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:02.003434 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:29:02.003528 15885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7616a7a446f945a5b651eddd03b25e45" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:02.004101 15960 raft_consensus.cc:3055] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:02.004406 15885 raft_consensus.cc:3060] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.004526 15960 raft_consensus.cc:740] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:02.005501 15960 consensus_queue.cc:260] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.006481 15960 raft_consensus.cc:3060] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.011137 15885 raft_consensus.cc:2468] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 2.
I20260709 12:29:02.011765 15960 raft_consensus.cc:2468] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 2.
I20260709 12:29:02.012740 15778 leader_election.cc:304] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:29:02.013584 16170 raft_consensus.cc:2804] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:02.014178 16170 raft_consensus.cc:697] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:29:02.015120 16170 consensus_queue.cc:237] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.024783 15707 catalog_manager.cc:5697] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 1 to 2, leader changed from d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67) to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 2 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:02.029795 16157 consensus_queue.cc:1048] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:02.045804 16159 consensus_queue.cc:1048] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:29:02.060967 16145 raft_consensus.cc:993] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51: : Instructing follower 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:29:02.061594 16204 raft_consensus.cc:1081] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Signalling peer 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:29:02.064275 15885 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:59290
I20260709 12:29:02.064936 15885 raft_consensus.cc:493] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:02.065441 15885 raft_consensus.cc:3060] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.073452 15885 raft_consensus.cc:515] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.075943 15885 leader_election.cc:290] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:02.077047 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:02.077947 15811 raft_consensus.cc:3060] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.077673 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a4eb0279d894156b7094538b6a5713b" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:29:02.078706 15960 raft_consensus.cc:3055] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:02.079195 15960 raft_consensus.cc:740] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:02.080063 15960 consensus_queue.cc:260] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.081504 15960 raft_consensus.cc:3060] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.085345 15811 raft_consensus.cc:2468] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:29:02.087028 15851 leader_election.cc:304] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:29:02.088014 16122 raft_consensus.cc:2804] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:02.088445 15960 raft_consensus.cc:2468] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:29:02.088616 16122 raft_consensus.cc:697] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:29:02.089769 16122 consensus_queue.cc:237] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.098613 15709 catalog_manager.cc:5697] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 1 to 2, leader changed from d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67) to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 2 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:02.126005 16159 raft_consensus.cc:993] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51: : Instructing follower cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:29:02.126562 16204 raft_consensus.cc:1081] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Signalling peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a to start an election
I20260709 12:29:02.128443 15811 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "16d196f6fcd6474b9567e2fbac53ae77"
dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
 from {username='slave'} at 127.0.0.1:43538
I20260709 12:29:02.129001 15811 raft_consensus.cc:493] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:02.129371 15811 raft_consensus.cc:3060] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.134637 15811 raft_consensus.cc:515] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.136201 15811 leader_election.cc:290] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 2 election: Requested vote from peers 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:02.137405 15885 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16d196f6fcd6474b9567e2fbac53ae77" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:02.137776 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16d196f6fcd6474b9567e2fbac53ae77" candidate_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:29:02.138026 15885 raft_consensus.cc:3060] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.138631 15960 raft_consensus.cc:3055] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:02.139024 15960 raft_consensus.cc:740] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:02.139851 15960 consensus_queue.cc:260] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.141172 15960 raft_consensus.cc:3060] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.144141 15885 raft_consensus.cc:2468] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 2.
I20260709 12:29:02.145522 15778 leader_election.cc:304] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:29:02.146315 16170 raft_consensus.cc:2804] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:02.146771 16170 raft_consensus.cc:697] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 LEADER]: Becoming Leader. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:29:02.147655 16170 consensus_queue.cc:237] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.149144 15960 raft_consensus.cc:2468] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdaf8c0ff80d4f3b874c1b0e5fc6cc4a in term 2.
I20260709 12:29:02.157152 15709 catalog_manager.cc:5697] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a reported cstate change: term changed from 1 to 2, leader changed from d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67) to cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65). New cstate: current_term: 2 leader_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:02.213722 16159 raft_consensus.cc:993] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51: : Instructing follower 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:29:02.214206 16204 raft_consensus.cc:1081] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Signalling peer 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:29:02.216470 15885 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:59290
I20260709 12:29:02.217118 15885 raft_consensus.cc:493] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:02.217478 15885 raft_consensus.cc:3060] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.219449 16204 raft_consensus.cc:993] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51: : Instructing follower 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:29:02.220260 16159 raft_consensus.cc:1081] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Signalling peer 9b36129f73bf485eb51ccf20370db33a to start an election
I20260709 12:29:02.222203 15887 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
 from {username='slave'} at 127.0.0.1:59290
I20260709 12:29:02.222761 15887 raft_consensus.cc:493] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:02.223137 15887 raft_consensus.cc:3060] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.224395 15885 raft_consensus.cc:515] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.226754 15885 leader_election.cc:290] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:02.227835 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:02.228566 15811 raft_consensus.cc:3060] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.228700 15960 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c47c39a171ab4d13990ce8a3a20ca869" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:29:02.229436 15960 raft_consensus.cc:3055] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:02.229741 15960 raft_consensus.cc:740] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:02.230538 15960 consensus_queue.cc:260] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.231133 15887 raft_consensus.cc:515] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.231822 15960 raft_consensus.cc:3060] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.233357 15887 leader_election.cc:290] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041)
I20260709 12:29:02.234750 15961 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
I20260709 12:29:02.234750 15811 raft_consensus.cc:2468] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:29:02.235663 15961 raft_consensus.cc:3055] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:29:02.236008 15961 raft_consensus.cc:740] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:02.237479 15851 leader_election.cc:304] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:29:02.237514 15811 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96fefd41ad4643c28334c4353ffe8ff1" candidate_uuid: "9b36129f73bf485eb51ccf20370db33a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:02.238353 15811 raft_consensus.cc:3060] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.238498 16122 raft_consensus.cc:2804] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:02.239138 16122 raft_consensus.cc:697] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:29:02.240042 16122 consensus_queue.cc:237] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.240366 15960 raft_consensus.cc:2468] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:29:02.236801 15961 consensus_queue.cc:260] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.243650 15961 raft_consensus.cc:3060] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.245900 15811 raft_consensus.cc:2468] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:29:02.248379 15709 catalog_manager.cc:5697] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 1 to 2, leader changed from d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67) to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 2 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:02.250409 15961 raft_consensus.cc:2468] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b36129f73bf485eb51ccf20370db33a in term 2.
I20260709 12:29:02.251262 15851 leader_election.cc:304] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, cdaf8c0ff80d4f3b874c1b0e5fc6cc4a; no voters: 
I20260709 12:29:02.252256 16122 raft_consensus.cc:2804] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:29:02.252842 16122 raft_consensus.cc:697] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Leader. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:29:02.253621 16122 consensus_queue.cc:237] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:02.264524 15709 catalog_manager.cc:5697] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a reported cstate change: term changed from 1 to 2, leader changed from d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67) to 9b36129f73bf485eb51ccf20370db33a (127.0.140.66). New cstate: current_term: 2 leader_uuid: "9b36129f73bf485eb51ccf20370db33a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: UNKNOWN } } }
I20260709 12:29:02.328032 15961 raft_consensus.cc:1275] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.329329 16122 consensus_queue.cc:1048] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.340023 15811 raft_consensus.cc:3060] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:29:02.347237 15811 raft_consensus.cc:1275] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.348686 16136 consensus_queue.cc:1048] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.525516 15811 raft_consensus.cc:1275] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.527235 16136 consensus_queue.cc:1048] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.533845 15887 raft_consensus.cc:1275] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.538113 15961 raft_consensus.cc:1275] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.539932 16175 consensus_queue.cc:1048] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:29:02.542419 16136 consensus_queue.cc:1048] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.555140 15961 raft_consensus.cc:1275] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.556732 16175 consensus_queue.cc:1048] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.602591 15961 raft_consensus.cc:1275] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.604259 16175 consensus_queue.cc:1048] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.614715 15887 raft_consensus.cc:1275] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Refusing update from remote peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.616662 16170 consensus_queue.cc:1048] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.778570 15960 raft_consensus.cc:1275] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.779942 16136 consensus_queue.cc:1048] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.789826 15811 raft_consensus.cc:1275] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.791299 16122 consensus_queue.cc:1048] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.827001 15811 raft_consensus.cc:1275] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.828363 16206 consensus_queue.cc:1048] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:02.838054 15960 raft_consensus.cc:1275] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Refusing update from remote peer 9b36129f73bf485eb51ccf20370db33a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:29:02.839799 16210 consensus_queue.cc:1048] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:29:04.773144 15707 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53900:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:29:04.803086 15707 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:53900:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:29:04.838362   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:29:05.013254 16234 log.cc:927] Time spent T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a: Append to log took a long time: real 0.087s	user 0.002s	sys 0.000s
W20260709 12:29:05.035909 16242 log.cc:927] Time spent T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Append to log took a long time: real 0.108s	user 0.004s	sys 0.000s
I20260709 12:29:05.106851 16252 tablet.cc:1724] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.136070 16260 tablet.cc:1724] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:29:05.154058 16255 log.cc:927] Time spent T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51: Append to log took a long time: real 0.058s	user 0.001s	sys 0.000s
I20260709 12:29:05.175676 16256 tablet.cc:1724] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.187979 16269 tablet.cc:1724] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.196269 16266 tablet.cc:1724] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:29:05.223357 16254 log.cc:927] Time spent T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:29:05.227092 16259 tablet.cc:1724] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:29:05.253433 16258 log.cc:927] Time spent T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
W20260709 12:29:05.268224 16261 log.cc:927] Time spent T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:29:05.273624 16263 log.cc:927] Time spent T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51: Append to log took a long time: real 0.097s	user 0.002s	sys 0.000s
I20260709 12:29:05.291518 15707 catalog_manager.cc:5789] TS d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67:35041) does not have the latest schema for tablet e897991c3158463f8147cc5ea24425ec (table soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]). Expected version 1 got 0
I20260709 12:29:05.357000 16260 tablet.cc:1724] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.300896 16252 tablet.cc:1724] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:29:05.365880 15925 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.140.65:43921, user_credentials={real_user=slave}} blocked reactor thread for 63265.5us
I20260709 12:29:05.350847 16280 tablet.cc:1724] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.389920 15883 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5"
dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
mode: GRACEFUL
new_leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
 from {username='slave'} at 127.0.0.1:44796
I20260709 12:29:05.413679 15883 raft_consensus.cc:606] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Received request to transfer leadership to TS d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:29:05.421357 16278 tablet.cc:1724] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.441023 16256 tablet.cc:1724] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.442993 16136 raft_consensus.cc:993] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a: : Instructing follower d5923b92f8db4416a3c43a90bc1cdd51 to start an election
I20260709 12:29:05.448964 16217 raft_consensus.cc:1081] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Signalling peer d5923b92f8db4416a3c43a90bc1cdd51 to start an election
I20260709 12:29:05.455135 16282 tablet.cc:1724] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.463553 16276 tablet.cc:1724] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.456619 16259 tablet.cc:1724] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.456763 16266 tablet.cc:1724] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.458756 15956 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
 from {username='slave'} at 127.0.0.1:37714
I20260709 12:29:05.459380 16275 tablet.cc:1724] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.468691 15956 raft_consensus.cc:493] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:05.469352 15956 raft_consensus.cc:3060] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:29:05.467450 16277 tablet.cc:1724] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.495540 16269 tablet.cc:1724] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.494381 16260 tablet.cc:1724] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.506762 15956 raft_consensus.cc:515] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:05.525699 15710 catalog_manager.cc:5789] TS 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631) does not have the latest schema for tablet 3a4eb0279d894156b7094538b6a5713b (table soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]). Expected version 1 got 0
I20260709 12:29:05.526273 16276 tablet.cc:1724] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.539990 15809 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa"
dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
mode: GRACEFUL
new_leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
 from {username='slave'} at 127.0.0.1:35904
I20260709 12:29:05.516234 16256 tablet.cc:1724] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.529165 16269 tablet.cc:1724] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.540750 15809 raft_consensus.cc:606] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 LEADER]: Received request to transfer leadership to TS d5923b92f8db4416a3c43a90bc1cdd51
I20260709 12:29:05.517122 16277 tablet.cc:1724] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.542827   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:29:05.521579 16278 tablet.cc:1724] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.543845   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:05.547997 16252 tablet.cc:1724] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.538437 15710 catalog_manager.cc:5789] TS 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631) does not have the latest schema for tablet fdd3f394cc164d108a7f276e6cfcf8eb (table soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]). Expected version 1 got 0
I20260709 12:29:05.512133 16280 tablet.cc:1724] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.554893 15809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:05.568782 16259 tablet.cc:1724] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:29:05.557050 15883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "41db3c9b613f4023b493ebaa3343faf5" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:05.569599 15710 catalog_manager.cc:5789] TS 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631) does not have the latest schema for tablet c47c39a171ab4d13990ce8a3a20ca869 (table soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d]). Expected version 1 got 0
I20260709 12:29:05.570892 15883 raft_consensus.cc:3055] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:29:05.555301 15956 leader_election.cc:290] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 3 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:05.571368 15883 raft_consensus.cc:740] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9b36129f73bf485eb51ccf20370db33a, State: Running, Role: LEADER
I20260709 12:29:05.574162 15883 consensus_queue.cc:260] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:05.576371 15883 raft_consensus.cc:3060] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:29:05.572391 15809 raft_consensus.cc:3060] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:29:05.617072 15809 raft_consensus.cc:2468] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 3.
I20260709 12:29:05.619725 15925 leader_election.cc:304] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:29:05.621536 16145 raft_consensus.cc:2804] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:29:05.622072 16145 raft_consensus.cc:697] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 3 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:05.623093 16145 consensus_queue.cc:237] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:05.635501 15883 raft_consensus.cc:2468] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 3.
I20260709 12:29:05.642341 15709 catalog_manager.cc:6242] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe: soft_deleted_table [id=3316e4a547cb4f94b1b2046a41daa56d] alter complete (version 1)
I20260709 12:29:05.651352 15707 catalog_manager.cc:5697] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 2 to 3, leader changed from 9b36129f73bf485eb51ccf20370db33a (127.0.140.66) to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 3 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:06.047403 15883 raft_consensus.cc:1275] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 3 FOLLOWER]: Refusing update from remote peer d5923b92f8db4416a3c43a90bc1cdd51: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:29:06.049089 16145 consensus_queue.cc:1048] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:29:06.067230 15809 raft_consensus.cc:1275] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 FOLLOWER]: Refusing update from remote peer d5923b92f8db4416a3c43a90bc1cdd51: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:29:06.069157 16274 consensus_queue.cc:1048] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:29:06.103365 16250 raft_consensus.cc:993] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: : Instructing follower d5923b92f8db4416a3c43a90bc1cdd51 to start an election
I20260709 12:29:06.103895 16284 raft_consensus.cc:1081] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 LEADER]: Signalling peer d5923b92f8db4416a3c43a90bc1cdd51 to start an election
I20260709 12:29:06.105644 15956 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa"
dest_uuid: "d5923b92f8db4416a3c43a90bc1cdd51"
 from {username='slave'} at 127.0.0.1:37724
I20260709 12:29:06.106353 15956 raft_consensus.cc:493] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:29:06.106848 15956 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:29:06.114352 15956 raft_consensus.cc:515] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:06.116336 15956 leader_election.cc:290] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 4 election: Requested vote from peers cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921), 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631)
I20260709 12:29:06.117326 15809 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a"
I20260709 12:29:06.117614 15883 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a95ea2d71cdf4e7c989ceea499c747aa" candidate_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "9b36129f73bf485eb51ccf20370db33a"
I20260709 12:29:06.118129 15809 raft_consensus.cc:3055] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:29:06.118360 15883 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:29:06.118556 15809 raft_consensus.cc:740] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cdaf8c0ff80d4f3b874c1b0e5fc6cc4a, State: Running, Role: LEADER
I20260709 12:29:06.119822 15809 consensus_queue.cc:260] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:06.120929 15809 raft_consensus.cc:3060] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:29:06.125151 15883 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 4.
I20260709 12:29:06.126895 15927 leader_election.cc:304] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b36129f73bf485eb51ccf20370db33a, d5923b92f8db4416a3c43a90bc1cdd51; no voters: 
I20260709 12:29:06.128031 16274 raft_consensus.cc:2804] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:29:06.128563 16274 raft_consensus.cc:697] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 4 LEADER]: Becoming Leader. State: Replica: d5923b92f8db4416a3c43a90bc1cdd51, State: Running, Role: LEADER
I20260709 12:29:06.129500 16274 consensus_queue.cc:237] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } }
I20260709 12:29:06.129870 15809 raft_consensus.cc:2468] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5923b92f8db4416a3c43a90bc1cdd51 in term 4.
I20260709 12:29:06.141031 15707 catalog_manager.cc:5697] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 reported cstate change: term changed from 3 to 4, leader changed from cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65) to d5923b92f8db4416a3c43a90bc1cdd51 (127.0.140.67). New cstate: current_term: 4 leader_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5923b92f8db4416a3c43a90bc1cdd51" member_type: VOTER last_known_addr { host: "127.0.140.67" port: 35041 } health_report { overall_health: HEALTHY } } }
I20260709 12:29:06.545167   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:06.550971   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:06.551462   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:06.598515 15809 raft_consensus.cc:1275] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 4 FOLLOWER]: Refusing update from remote peer d5923b92f8db4416a3c43a90bc1cdd51: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:29:06.600350 16274 consensus_queue.cc:1048] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cdaf8c0ff80d4f3b874c1b0e5fc6cc4a" member_type: VOTER last_known_addr { host: "127.0.140.65" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:29:06.613688 15883 raft_consensus.cc:1275] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 4 FOLLOWER]: Refusing update from remote peer d5923b92f8db4416a3c43a90bc1cdd51: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:29:06.617614 16274 consensus_queue.cc:1048] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b36129f73bf485eb51ccf20370db33a" member_type: VOTER last_known_addr { host: "127.0.140.66" port: 35631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:29:08.197486   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:08.203496   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:08.204027   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:29:08.498894 15982 debug-util.cc:398] Leaking SignalData structure 0x7b08002ded60 after lost signal to thread 564
W20260709 12:29:08.500367 15982 debug-util.cc:398] Leaking SignalData structure 0x7b08002df440 after lost signal to thread 15744
W20260709 12:29:08.501993 15982 debug-util.cc:398] Leaking SignalData structure 0x7b0800085d60 after lost signal to thread 15836
W20260709 12:29:08.503623 15982 debug-util.cc:398] Leaking SignalData structure 0x7b08003629e0 after lost signal to thread 15911
W20260709 12:29:08.504993 15982 debug-util.cc:398] Leaking SignalData structure 0x7b08001bbf00 after lost signal to thread 15985
I20260709 12:29:09.205175   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:09.211731   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:09.212363   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:10.213560   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:10.220440   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:10.221025   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:11.222299   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:11.231145   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:11.231964   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:12.233215   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:12.238534   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:12.239101   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:13.240470   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:13.246157   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:13.246748   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:14.248147   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:14.254679   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:14.255388   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:15.256914   561 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:29:15.262831   561 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:29:15.263509   561 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:29:16.339576   561 tablet_server.cc:179] TabletServer@127.0.140.65:0 shutting down...
W20260709 12:29:16.348277 15851 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 11 similar messages]
W20260709 12:29:16.353981 15851 consensus_peers.cc:597] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:16.366066 15851 consensus_peers.cc:597] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:16.406167   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:29:16.407295   561 tablet_replica.cc:333] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.408753   561 raft_consensus.cc:2243] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.409641   561 raft_consensus.cc:2272] T e897991c3158463f8147cc5ea24425ec P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.412724   561 tablet_replica.cc:333] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.413486   561 raft_consensus.cc:2243] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.414124   561 raft_consensus.cc:2272] T 96fefd41ad4643c28334c4353ffe8ff1 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.416932   561 tablet_replica.cc:333] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.417650   561 raft_consensus.cc:2243] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.419009   561 raft_consensus.cc:2272] T 7616a7a446f945a5b651eddd03b25e45 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.423365   561 tablet_replica.cc:333] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.424047   561 raft_consensus.cc:2243] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.425108   561 raft_consensus.cc:2272] T d81824aa48c14c71b39f71ffa84ff0dc P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.428020   561 tablet_replica.cc:333] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.428663   561 raft_consensus.cc:2243] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.429618   561 raft_consensus.cc:2272] T 16d196f6fcd6474b9567e2fbac53ae77 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.432421   561 tablet_replica.cc:333] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.433023   561 raft_consensus.cc:2243] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.433512   561 raft_consensus.cc:2272] T 5e42ffbebd604354925aebaa0fe7ef82 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.437556   561 tablet_replica.cc:333] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.438412   561 raft_consensus.cc:2243] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.439680   561 raft_consensus.cc:2272] T 2e18dfee472b4c60b724b43004e356e9 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.442421   561 tablet_replica.cc:333] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.443177   561 raft_consensus.cc:2243] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.443727   561 raft_consensus.cc:2272] T c47c39a171ab4d13990ce8a3a20ca869 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.446238   561 tablet_replica.cc:333] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.446935   561 raft_consensus.cc:2243] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.447475   561 raft_consensus.cc:2272] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.449963   561 tablet_replica.cc:333] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.450511   561 raft_consensus.cc:2243] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.451079   561 raft_consensus.cc:2272] T 3a4eb0279d894156b7094538b6a5713b P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.453727   561 tablet_replica.cc:333] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.454281   561 raft_consensus.cc:2243] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.454869   561 raft_consensus.cc:2272] T fdd3f394cc164d108a7f276e6cfcf8eb P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.457254   561 tablet_replica.cc:333] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.457747   561 raft_consensus.cc:2243] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.458604   561 raft_consensus.cc:2272] T a1cbc7ee5bdf434192dea928f8ca1a96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.461179   561 tablet_replica.cc:333] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.461704   561 raft_consensus.cc:2243] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.462203   561 raft_consensus.cc:2272] T 7bbd23a4b0424576ba4cc5df1d5a7aea P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.464748   561 tablet_replica.cc:333] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.465296   561 raft_consensus.cc:2243] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.465816   561 raft_consensus.cc:2272] T 41db3c9b613f4023b493ebaa3343faf5 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.468379   561 tablet_replica.cc:333] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.468904   561 raft_consensus.cc:2243] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.469383   561 raft_consensus.cc:2272] T 22fa1d49fd24457395605b0c9943dc4a P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.471972   561 tablet_replica.cc:333] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.472452   561 raft_consensus.cc:2243] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.473369   561 raft_consensus.cc:2272] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.476015   561 tablet_replica.cc:333] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.476512   561 raft_consensus.cc:2243] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.476975   561 raft_consensus.cc:2272] T a95ea2d71cdf4e7c989ceea499c747aa P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.479715   561 tablet_replica.cc:333] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a: stopping tablet replica
I20260709 12:29:16.480237   561 raft_consensus.cc:2243] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.481405   561 raft_consensus.cc:2272] T dc9602a9b0514fe7abc499687775488f P cdaf8c0ff80d4f3b874c1b0e5fc6cc4a [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:29:16.485855 15851 consensus_peers.cc:597] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:16.540141   561 tablet_server.cc:196] TabletServer@127.0.140.65:0 shutdown complete.
W20260709 12:29:16.541731 15851 consensus_peers.cc:597] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:16.542461 15851 consensus_peers.cc:597] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:16.593262   561 tablet_server.cc:179] TabletServer@127.0.140.66:0 shutting down...
W20260709 12:29:16.607595 15925 consensus_peers.cc:597] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:16.632201 15925 consensus_peers.cc:597] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:16.651506 15925 consensus_peers.cc:597] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:16.654132   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:29:16.655475   561 tablet_replica.cc:333] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.656857   561 raft_consensus.cc:2243] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.657560   561 raft_consensus.cc:2272] T e897991c3158463f8147cc5ea24425ec P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.660693   561 tablet_replica.cc:333] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.661351   561 raft_consensus.cc:2243] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.661893   561 raft_consensus.cc:2272] T d81824aa48c14c71b39f71ffa84ff0dc P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.664791   561 tablet_replica.cc:333] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.665478   561 raft_consensus.cc:2243] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.666217   561 raft_consensus.cc:2272] T 7616a7a446f945a5b651eddd03b25e45 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.669070   561 tablet_replica.cc:333] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.669752   561 raft_consensus.cc:2243] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.671093   561 raft_consensus.cc:2272] T 3a4eb0279d894156b7094538b6a5713b P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.673900   561 tablet_replica.cc:333] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.674582   561 raft_consensus.cc:2243] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.675171   561 raft_consensus.cc:2272] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.678051   561 tablet_replica.cc:333] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.678865   561 raft_consensus.cc:2243] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.679987   561 raft_consensus.cc:2272] T 22fa1d49fd24457395605b0c9943dc4a P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.682767   561 tablet_replica.cc:333] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.683588   561 raft_consensus.cc:2243] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.685221   561 raft_consensus.cc:2272] T 96fefd41ad4643c28334c4353ffe8ff1 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.689687   561 tablet_replica.cc:333] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.690907   561 raft_consensus.cc:2243] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:29:16.691272 15851 consensus_peers.cc:597] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:16.691666   561 raft_consensus.cc:2272] T dc9602a9b0514fe7abc499687775488f P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.695015   561 tablet_replica.cc:333] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.695829   561 raft_consensus.cc:2243] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.696437   561 raft_consensus.cc:2272] T 41db3c9b613f4023b493ebaa3343faf5 P 9b36129f73bf485eb51ccf20370db33a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.699340   561 tablet_replica.cc:333] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.699992   561 raft_consensus.cc:2243] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.701045   561 raft_consensus.cc:2272] T fdd3f394cc164d108a7f276e6cfcf8eb P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.703980   561 tablet_replica.cc:333] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.704650   561 raft_consensus.cc:2243] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.705308   561 raft_consensus.cc:2272] T a1cbc7ee5bdf434192dea928f8ca1a96 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.708619   561 tablet_replica.cc:333] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.709362   561 raft_consensus.cc:2243] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.710619   561 raft_consensus.cc:2272] T 7bbd23a4b0424576ba4cc5df1d5a7aea P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.713508   561 tablet_replica.cc:333] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.714391   561 raft_consensus.cc:2243] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.715814   561 raft_consensus.cc:2272] T c47c39a171ab4d13990ce8a3a20ca869 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.719074   561 tablet_replica.cc:333] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.720024   561 raft_consensus.cc:2243] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.721300   561 raft_consensus.cc:2272] T 5e42ffbebd604354925aebaa0fe7ef82 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.725415   561 tablet_replica.cc:333] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.726276   561 raft_consensus.cc:2243] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.726958   561 raft_consensus.cc:2272] T 2e18dfee472b4c60b724b43004e356e9 P 9b36129f73bf485eb51ccf20370db33a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.730389   561 tablet_replica.cc:333] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.731133   561 raft_consensus.cc:2243] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.731846   561 raft_consensus.cc:2272] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.736274   561 tablet_replica.cc:333] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.737219   561 raft_consensus.cc:2243] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.738559   561 raft_consensus.cc:2272] T 16d196f6fcd6474b9567e2fbac53ae77 P 9b36129f73bf485eb51ccf20370db33a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.743445   561 tablet_replica.cc:333] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a: stopping tablet replica
I20260709 12:29:16.744287   561 raft_consensus.cc:2243] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.745072   561 raft_consensus.cc:2272] T a95ea2d71cdf4e7c989ceea499c747aa P 9b36129f73bf485eb51ccf20370db33a [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:29:16.763095 15927 consensus_peers.cc:597] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 -> Peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Couldn't send request to peer 9b36129f73bf485eb51ccf20370db33a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:29:16.794487 15927 consensus_peers.cc:597] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 -> Peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Couldn't send request to peer 9b36129f73bf485eb51ccf20370db33a. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:35631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:16.805256   561 tablet_server.cc:196] TabletServer@127.0.140.66:0 shutdown complete.
W20260709 12:29:16.836054 15927 consensus_peers.cc:597] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 -> Peer 9b36129f73bf485eb51ccf20370db33a (127.0.140.66:35631): Couldn't send request to peer 9b36129f73bf485eb51ccf20370db33a. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.66:35631: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:16.852653   561 tablet_server.cc:179] TabletServer@127.0.140.67:0 shutting down...
W20260709 12:29:16.886704 15925 consensus_peers.cc:597] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 -> Peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a (127.0.140.65:43921): Couldn't send request to peer cdaf8c0ff80d4f3b874c1b0e5fc6cc4a. Status: Network error: Client connection negotiation failed: client connection to 127.0.140.65:43921: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:29:16.900447   561 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:29:16.901646   561 tablet_replica.cc:333] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.902685   561 raft_consensus.cc:2243] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.903576   561 raft_consensus.cc:2272] T 96fefd41ad4643c28334c4353ffe8ff1 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.906577   561 tablet_replica.cc:333] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.907456   561 raft_consensus.cc:2243] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.908670   561 raft_consensus.cc:2272] T e897991c3158463f8147cc5ea24425ec P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.911787   561 tablet_replica.cc:333] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.912629   561 raft_consensus.cc:2243] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.913393   561 raft_consensus.cc:2272] T d81824aa48c14c71b39f71ffa84ff0dc P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.916175   561 tablet_replica.cc:333] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.916887   561 raft_consensus.cc:2243] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.917518   561 raft_consensus.cc:2272] T 7616a7a446f945a5b651eddd03b25e45 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.920467   561 tablet_replica.cc:333] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.921169   561 raft_consensus.cc:2243] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.921679   561 raft_consensus.cc:2272] T dc9602a9b0514fe7abc499687775488f P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.924515   561 tablet_replica.cc:333] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.925251   561 raft_consensus.cc:2243] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.925853   561 raft_consensus.cc:2272] T 3a4eb0279d894156b7094538b6a5713b P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.928787   561 tablet_replica.cc:333] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.929486   561 raft_consensus.cc:2243] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.930550   561 raft_consensus.cc:2272] T 869265d9a2fa4ae6b4c5b58c1c4cc1f5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.933326   561 tablet_replica.cc:333] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.934002   561 raft_consensus.cc:2243] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.934597   561 raft_consensus.cc:2272] T 22fa1d49fd24457395605b0c9943dc4a P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.937215   561 tablet_replica.cc:333] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.937867   561 raft_consensus.cc:2243] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.938454   561 raft_consensus.cc:2272] T c47c39a171ab4d13990ce8a3a20ca869 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.940955   561 tablet_replica.cc:333] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.941553   561 raft_consensus.cc:2243] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.942076   561 raft_consensus.cc:2272] T 5e42ffbebd604354925aebaa0fe7ef82 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.944747   561 tablet_replica.cc:333] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.945405   561 raft_consensus.cc:2243] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.945966   561 raft_consensus.cc:2272] T 2e18dfee472b4c60b724b43004e356e9 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.948453   561 tablet_replica.cc:333] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.949057   561 raft_consensus.cc:2243] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.949615   561 raft_consensus.cc:2272] T f67bf24fa1b54ae2bf3ef7afe02cbd96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.952155   561 tablet_replica.cc:333] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.952800   561 raft_consensus.cc:2243] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.953776   561 raft_consensus.cc:2272] T 41db3c9b613f4023b493ebaa3343faf5 P d5923b92f8db4416a3c43a90bc1cdd51 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.956331   561 tablet_replica.cc:333] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.957208   561 raft_consensus.cc:2243] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.957885   561 raft_consensus.cc:2272] T a1cbc7ee5bdf434192dea928f8ca1a96 P d5923b92f8db4416a3c43a90bc1cdd51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.960772   561 tablet_replica.cc:333] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.961457   561 raft_consensus.cc:2243] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.962105   561 raft_consensus.cc:2272] T fdd3f394cc164d108a7f276e6cfcf8eb P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.964845   561 tablet_replica.cc:333] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.965502   561 raft_consensus.cc:2243] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.966074   561 raft_consensus.cc:2272] T 7bbd23a4b0424576ba4cc5df1d5a7aea P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.968812   561 tablet_replica.cc:333] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.969465   561 raft_consensus.cc:2243] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:29:16.970103   561 raft_consensus.cc:2272] T 16d196f6fcd6474b9567e2fbac53ae77 P d5923b92f8db4416a3c43a90bc1cdd51 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:16.972803   561 tablet_replica.cc:333] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51: stopping tablet replica
I20260709 12:29:16.973672   561 raft_consensus.cc:2243] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:29:16.974817   561 raft_consensus.cc:2272] T a95ea2d71cdf4e7c989ceea499c747aa P d5923b92f8db4416a3c43a90bc1cdd51 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:17.045017   561 tablet_server.cc:196] TabletServer@127.0.140.67:0 shutdown complete.
I20260709 12:29:17.094630   561 master.cc:562] Master@127.0.140.126:37015 shutting down...
I20260709 12:29:17.128165   561 raft_consensus.cc:2243] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:29:17.128876   561 raft_consensus.cc:2272] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:29:17.129227   561 tablet_replica.cc:333] T 00000000000000000000000000000000 P f84b7681764f482aadad4c622874defe: stopping tablet replica
I20260709 12:29:17.152945   561 master.cc:584] Master@127.0.140.126:37015 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (23405 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (45009 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (532898 ms total)
[  PASSED  ] 8 tests.
I20260709 12:29:17.215281   561 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 2241 messages since previous log ~34 seconds ago
I20260709 12:29:17.215559   561 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 2241 messages since previous log ~34 seconds ago
I20260709 12:29:17.215764   561 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 2237 messages since previous log ~34 seconds ago
I20260709 12:29:17.216008   561 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~53 seconds ago
I20260709 12:29:17.216225   561 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~53 seconds ago
I20260709 12:29:17.216519   561 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 5 messages since previous log ~193 seconds ago
I20260709 12:29:17.216727   561 logging.cc:436] LogThrottler Service Queue Overflow: suppressed but not reported on 1 messages since previous log ~310 seconds ago
I20260709 12:29:17.216965   561 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/service_pool.cc:136: suppressed but not reported on 1 messages since previous log ~310 seconds ago
I20260709 12:29:17.217268   561 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 12 messages since previous log ~0 seconds ago
I20260709 12:29:17.217625   561 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 52 messages since previous log ~22 seconds ago
